devices/loopback: clear the buffer cache

Manually clear the buffer cache of the loop device, which seems to
be required in order to make sure that data written via the loop
device is actually landing in the file:
Since commit c1379f6 the file descriptor of the loop device is
explicitly cleared. This broke manifests that involved creating a
FAT filesystem. Said file system could later not be mounted. The
breaking change was identified to indeed be commit c1379f6. Using
`biosnoop` we saw that some write operations were missing when
clearing the file descriptor that were present when using the
auto-clearing feature of the loop device (see below). Reading the
corresponding kernel source (v5.13.8), the current theory is that
when using the auto clear feature, once the last handle on the
loop device is closed, the code path in the kernel is:
    blkdev_close (fs/block_dev.c)
    blkdev_put (fs/block_dev.c)
    __blkdev_put (fs/block_dev.c)
    sync_blockdev (fs/block_dev.c)

On the other hand when manually clearing the file descriptor, the
code path seems to be:
    loop_clr_fd (fs/loop.c)
    __loop_clr_fd (fs/loop.c)

The latter first removes the backing file and then calls `bdput`,
and thus no call to sync_blockdev is made.

Luckily, sync_blockdev can be called via an ioctl, `BLKFLSBUF`,
which we no do, via the new helper function `lo.flush_buf`. This
fixes the observed issue and leads to the same biosnoop trace as
observed when using the auto clear feature without explicitly
clearing the fd.

NB: we considered reverting the commit c1379f6, but we want to make
sure that we control to point when the backing file is cleared from
the fd, since sub-sequent osbuild stages will re-use the file and
we want to ensure no loop device still has the file open and that
all the data in is in the file.

-- biosnoop trace --
4.115946    mkfs.fat       731297 loop1   R 0          4096      0.08
4.116096    mkfs.fat       731297 loop1   R 8          4096      0.02
4.116176    mkfs.fat       731297 loop1   R 16         4096      0.02
 [...]
4.120632    mkfs.fat       731297 loop1   R 400        4096      0.02
4.200354    org.osbuild.lo 731281 vda     W 4182432    32768     0.64
4.200429    org.osbuild.lo 731281 vda     W 6279584    32768     0.70
4.200657    ?              0              R 0          0         0.19
4.200946    org.osbuild.lo 731281 vda     W 3328128    4096      0.20
4.201109    ?              0              R 0          0         0.13
 [the following entires were missing with manual flushing:]
4.201601    org.osbuild.lo 731281 loop1   W 0          4096      0.24
4.201634    org.osbuild.lo 731281 loop1   W 8          4096      0.26
4.201645    org.osbuild.lo 731281 loop1   W 16         4096      0.27
 [...]
4.203118    org.osbuild.lo 731281 loop1   W 432        4096      0.25

Reported-by: Achilleas Koutsou <achilleas@koutsou.net>
Reported-by: Tomas Hozza <thozza@redhat.com>

devices/org.osbuild.loopback

@@ -119,6 +119,12 @@ class LoopbackService(devices.DeviceService):
         self.ctl.close()
 
         if self.lo:
+            # Flush the buffer cache of the loop device. This
+            # seems to be required when clearing the fd of the
+            # loop device (as of kernel 5.13.8) or otherwise
+            # it leads to data loss.
+            self.lo.flushbuf()
+
             # clear the fd. Since it might not immediately be
             # cleared (due to a race with udev or some other
             # process still having a reference to the loop dev)