It's sometimes useful to set up a loop device for an already formatted
disk/filesystem image to derive new artifacts from it. In that case, we
want to make sure it's impossible to modify its contents in any way in
that process, both for our own purposes and for other stages operating
on it.
Notably, mounting some filesystems read-only still seem to touch the
disk (like XFS).
This reverts commit 158acaac78.
With https://github.com/osbuild/bootc-image-builder/pull/238 the
original reason to call mknod goes away so we can just revert
it. osbuild now requires not only the loop device but also uses
`losetup --partscan` quite a lot now so the mknod approach becomes
impractical and the consumers of osbuild in a container should
just setup devtmpfs.
For the org.osbuild.loopback the user can set the sector size, but
it had no effect on the underlying loopback device. Let's make it
meaningful by passing along the given value to the underlying code.
When loop.Loop() is called and a new loop device must be allocated
there is no gurantee that the correct device node is available on
the system. In containers /dev is often just a tmpfs with static
device nodes. So when /dev/loopN is not available when the
container is created the device node will be missing even if
`get_unbound()` create a new loop device for us.
This commit ensures that the device node is available. It creates
it unconditionally and ignores any EEXIST errors to ensure there
is no TOCTOU issue.
Note that the test could have passed a `Loop(dir_fd=open(tmpdir))`
instead of creating/patching loop.DEV_PATH but it seems slightly
nicer to test the flow without a custom dir_path as this is what
the real code that creates a loop device is also using.
When osbuild.loop.Loop calls `__init__()` it assigns the `self.fd`
on open. However if that open call fails for whatever reason
(not found, permissions) the cleanup in `__del__` will fail in
confusing ways because `self.fd` is not initialized yet. It
also prevents the correct error from getting reported. A tiny
test is added to ensure this does not regress.
`tox` is a standard testing tool for Python projects, this allows you to
test locally with all your installed Python version with the following
command:
`tox -m test -p all`
To run the tests in parallel for all supported Python versions.
To run linters or type analysis:
```
tox -m lint -p all
tox -m type -p all
```
This commit *also* disables the `import-error` warning from `pylint`,
not all Python versions have the system-installed Python libraries
available and they can't be fetched from PyPI.
Some linters have been added and the general order linters run in has
been changed. This allows for quicker test failure when running
`tox -m lint`. As a consequence the `test_pylint` test has been removed
as it's role can now be fulfilled by `tox`.
Other assorted linter fixes due to newer versions:
- use a str.join method (`consider-using-join`)
- fix various (newer) mypy and pylint issues
- comments starting with `#` and no space due to `autopep8`
This also changes our CI to use the new `tox` setup and on top of that
pins the versions of linters used. This might move into separate
requirements.txt files later on to allow for easier updating of those
dependencies.
LOOP_CONFIGURE allows to atomically configure the decive when opening
it. This avoid the possibility of a race condition where between set_fd
and set_status some operations are already accepted by the loopback
device. See https://lwn.net/Articles/820408/
This feature was included in the linux kernel 5.8 however it is safe to
not include any kind of fallback to the previous method as @obudai
points out that:
LOOP_CONFIGURE was backported into RHEL 8 kernel in RHEL 8.4 as a part
of https://bugzilla.redhat.com/show_bug.cgi?id=1881760 (block layer:
update to upstream v5.8).
Since RHEL 8.4 is currently the oldest supported release that we support
running osbuild on, it might be just fine implementing this without the
fallback.
From a centos stream 8 container:
kernel-4.18.0-448.el8.x86_64
- loop: Fix missing discard support when using LOOP_CONFIGURE (Ming Lei) [1997338]
- [block] loop: Set correct device size when using LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: unset GENHD_FL_NO_PART_SCAN on LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: Add LOOP_CONFIGURE ioctl (Ming Lei) [1881760]
Silence pylint warning W0201 (attribute-defined-outside-init) in
`set_status`; it sets dynamic attributes on the LoopInfo class
which pylint does not recognize.
Add a new callback parameter to `LoopControl` that, if specified,
will be invoked after the loop device is opened but before any
other operation is done, like setting the backing file. Can be
used to perform custom setup tasks.
Add a new signal like callback to the `Loop` class which will be
invoked before the actual loop device is closed, i.e. the loop
device has an open file descriptor to the device node and it is
being closed. Can be used to perform custom cleanup tasks.
Add support for locking the loopback block device via `flock(2)`.
The main use case for this is to prevent systemd-udevd from
proben the device while any modification is done to it. See the
systemd page, https://www.freedesktop.org/software/systemd, for
more details.
Add the corresponding tests to it.
Add a helper method that clears the fd for a given loop device but
also ensures that the loop device is not bound to the supplied fd
anymore. Check the function documentation for more information.
Add a corresponding test.
Add a `Loop.is_bound_to` helper that checks if the looback device is
bound if is so if the backing file refers to the same file as `fd`.
The latter is done by comparing the device and inode information.
Add a helper that will check if the loop devices is backed by
the file identified via the stat(2) result, i.e. the inode on
the correspoding device.
Add a correspoding test for the new helper.
Implement a `Loop.get_status` method, to get the properties of the
loop device, corresponding to LOOP_GET_STATUS64, and counterpart
to the existing `Loop.set_status` method. Use the new `get_status`
call in the `set_status` call, replacing the existing code that
does the same thing.
Add a basic test for the `get_status` method. Also fix an actual
leak, where the loop device was closed but the fd was not cleared
inside the test.
Add a new helper, `loop_for_fd` that will get (or create) an
unbounded loop device, bind it to an fd and then set its
status. Since this is racy and can fail the method does these
steps in a retry-loop.
Add a `close` method to the loop controller class `LoopControl` since
it actually opens a file descriptor, which should be closed once the
loop controller is no longer needed.
Assert that the controlling file descriptor is open for all methods
that require this.
The mknod() method currently allows passing no dir_fd, in which case an
internal one is opened. This FD is then never closed, though.
Fix this by simply making the dir_fd mandatory. All callers pass it
(there is actually only a single caller), so no need for the fallback.
Don't wait until python's garbage collector closes the file descriptors
to loop devices. Close them when the `LoopServer` context manager exits,
after an assembler has finished running.
Support the LOOP_SET_DIRECT_IO ioctl, which alows us to control
whether or not a loopback device should perform its own buffering
or rely on the one done by the underlying backing file.
Enabling this should improve both throughput and memory consumption,
it is not currently hooked up as more testing would be required.
loop.py is a simple wrapper around the kernel loop API. remoteloop.py
uses this to create a server/clinet pair that communicates over an
AF_UNIX/SOCK_DGRAM socket to allow the server to create loop devices
for the client.
The client passes a fd that should be bound to the resulting loop
device, and a dir-fd where the loop device node should be created.
The server returns the name of the device node to the client.
The idea is that the client is run from whithin a container without
access to devtmpfs (and hence /dev/loop-control), and the server
runs on the host. The client would typically pass its (fake) /dev
as the output directory.
For the client this will be similar to `losetup -f foo.img --show`.
[@larskarlitski: pylint: ignore the new LoopInfo class, because it
only has dynamic attributes. Also disable attribute-defined-outside-init,
which (among other problems) is not ignored for that class.]
Signed-off-by: Tom Gundersen <teg@jklm.no>
