Add an new module with utility functions to inspect PE32+ files,
mainly listing the sections and their addresses and sizes.
Include a simple test to check that we can successfully parse the
EFI stub contained in systemd (systemd-udev package).
Use the new `Index.detect_runner` method that will give us the best
available runner for a requested one. To do so a new `pipeline.Runner`
class is introduced that stores the `meta.RunnerInfo` class for the
specific runner and the original name that was requested.
In the manifest loading and describing functions of the formats, use
`Index.detect_runner` to get the `RunnerInfo` for a requested runner
and then wrap it in a `pipeline.Runner` object, which is then passed
to the `Manifest.add_pipeline` method.
See also commit "meta: ability to auto-detect runner".
Adjust all test.
Instead of using a non-existing runner `org.osbuild.test` use an
existing one `org.osbuild.linux`. This prepares the switch to
using runner auto-detection, which will rely on existing runners.
For some reasons I forgot to fix those in the previous runs. Fix a
linter and pep8 warning.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Newer warning from pylint, also consistent with how we do things
elsewhere. Note that this only applies to one file in the tests but
disabling it would be very weird for such a small fix.
The idea of this test case was to check that two identical trees are
only stored once, via their treesum in the object store; but this
functionality was removed in commit e97f6ef34 and instead of treesums
random uuids are now used. As a result there is no de-duplication
anymore -- the subject of the test. So remove the test.
Add a new class `SubIdsDB` as a database of subordinate Ids, like the
ones in `/etc/subuid` and `/etc/subgid`. Methods to read and write
data from these two files are provided.
Add corresponding unit tests.
Add a new member variable `caps` that if not `None` indicates the
capabilities to retain, i.e. all other capabilities not specified
will be dropped via `bubblewrap` (`--cap-drop`).
Add corresponding tests.
This extends the possible ways of passing references to inputs. The
current ways possible are:
1) "plain references", an array of strings:
["ref1", "ref2", ...]
2) "object references", a mapping of keys to objects:
{"ref1": { <options> }, "ref2": { <options> }, ...}
This patch adds a new way:
3) "array of object references":
[{"id": "ref1", "options": { ... }}, {"id": ... }, ]
While osbuild promises to preserves the order for "object references"
not all JSON serialization libraries preserve the order since the
JSON specification does leave this up to the implementation.
The new "array of object references" thus allows for specifying the
references together with reference specific options and this in a
specific order.
Additionally this paves the way for specifying the same input twice,
e.g. in the case of the `org.osbuild.files` input where a pipeline
could then be specified twice with different files. This needs core
rework though, since internally we use dictionaries right now.
Specifically this test checks that the order given in the manifest is
preserved when loaded, i.e. the internal dict has the keys ordered in
the same way, independently in which way they were specified -- list
or object.
This is basically a re-implementation of `setfilecon(3)` minus the
translation of human readable context to raw context. Add test for
the new function.
In all the invocation of `subprocess.run` stderr and stdout were both
combined in a shared pipe, but lvm sometimes spits out notices and
informational messages on stderr and thus potentially interfering
with the data we are interested in on stdout. Separate the two.
Add support for emitting signals to host.Service which can be used to
transmit data back to the client during an ongoing method call. This
provides the possibility for the services to send information to their
client counterpart while running. The signal can take file descriptors
as extra parameters to send data on separate files.
The treesum of a filesystem tree is the content hash of all its
files, its directory structure and file metadata.
By storing trees by their treesum we avoid storing duplicates of
identical trees, at the cost of computing the hashes for every
commit to the store.
This has limited benefit as the likelihood of two trees being
identical is slim, in particular when we already have the ability
to cache based on pipeline/stage ID (i.e., we can avoid rebuilding
trees if the pipelines that built them were the same).
Drop the concept of a treesum entirely, even though I very much
liked the idea in theory...
Signed-off-by: Tom Gundersen <teg@jklm.no>
Set the container environment variable to indicate to programs
inside the build root that they are indeed running inside a
container (see also https://systemd.io/CONTAINER_INTERFACE/).
Create a well-defined environment with and use that for the build
root. It is not desirable to have the host's environment leak
into the container. Add a test to ensure that this works.
NB: This was probably an oversight when we switched from systemd-
nspawn to bubblewrap.
Add a new signal like callback to the `Loop` class which will be
invoked before the actual loop device is closed, i.e. the loop
device has an open file descriptor to the device node and it is
being closed. Can be used to perform custom cleanup tasks.
Certain udev rules for block devices are problematic for osbuild.
One prominent example is LVM2 related rules that would trigger
a scan and auto-activation of logical volumes. This rules are
triggered for new block devices or when the backing file of an
loop devices changes. The rules will lead to a `lvm pvscan
--cache --activate ay` via the `lvm2-pvscan@.service` systemd
service. This will auto-activate all LVM2 logical volumes and
thus interfering with our own device handling in `devices/
org.osbuild.lvm2.lv`, where we only want to activate a single
logical volume.
Also, if the lvm2 devices get activated after the manual metadata
change done in `org.osbuild.lvm2.metadata` the volume group names
might conflict which results in all lvm2 based tooling to be very,
ver sad and also said stage to hang since the loopback device can
not be detached since the activate logical volumes keep it open.
To work-around this we therefore implement a udev rule inhibition
mechanism: on the osbuild side a lock file is created via the new
class called `UdevInhibitor` in `utils/udev.py`. A custom set of
udev rules in `10-osbuild-inhibitor.rules` is then acting on the
existence of that lock file and if present will opt-out of certain
further processing. See the udev rules file for more details.
In fact, we want this custom inhibition mechanism, for all block
devices that are under osbuild's control, since these rules are
there to provide automatisms and integrations with the host,
something we never want.
NB: this should not affect the detection of devices, since lvm2
does do a scan of devices when we call `lvdisplay` in `lvm2.lv`.
The call chain as of lvm2 git rev f773040:
_lvdisplay_single [tools/lvdisplay.c
process_each_lv [tools/toollib.c
lvmcache_label_scan [lib/cache/lvmcache.c
label_scan [ibidem, here is the device detection!
lvdisplay_full [lib/display/display.c
Use the new Manifest.depsolve function to only build the pipelines that
were explicitly requested and their dependencies, taking into account
what is already present in the store.
Since now not all pipeline will be built, there wont be a result entry
for all the pipelines, thus the format version 2 result formatting was
changed to not require the pipeline to be present in result set.
New function that take a list of pipelines and return the list of
pipelines that need to be build, i.e. the pipelines and all their
dependencies that are not already present in the store.
Add corresponding test.
When building a version 1 manifest, the assembler would always be
exported, even when not requested via the `--export` command line
option. This was done for backwards compatibility so to not break
tools relying on that behavior. The problem is that support for
this uses a completely different code path and might also now be
confusing behavior. Thus remove the implicit and really only ever
export what was explicitly requested by the caller.
Since we bind `/proc` inside the container, we leak certain information
that comes with it. One of this is the kernel command line. None of the
decisions done by software running inside the container should depend
on the kernel command line on the host, so overwrite the kernel command
line by creating a temporary directory and mapping it inside the build-
root. For now we default to a simple `root=/dev/osbuild` fake kernel
command line.
Add a simple check for it as well.
Instead of having custom code that basically duplicates the
functionality of `LoopControl.loop_for_fd` use that instead.
Additionally, the version used in the test had a bug where
it did not re-create the Loop device in the main loop when
it was close due to an error, leading errors in subsequent
usages of the device that would often manifest in CI runs:
fcntl.ioctl(self.fd, self.LOOP_SET_FD, fd)
ValueError: file descriptor cannot be a negative integer (-1)
This stage takes /usr/lib/passwd and /usr/etc/passwd from an OSTree
checkout, merges them into one file, and store it as /etc/passwd in the
buildroot.
It does the same for /etc/group.
The reason for doing this is that there is an issue with unstable UIDs
and GIDs when creating OSTree commits from scratch. When there is a
package that creates a system user or a system group, it can change the
UID and GID of users and groups that are created later.
This is not a problem in traditional deployments because already created
users and groups never change their UIDs and GIDs, but with OSTree we
recreate the files from scratch and then replace the previous one so it
can actually change.
By copying the files to the build root before doing any other
operations, we can make sure that the UIDs and GIDs of already existing
users and groups won't change.
Co-author: Christian Kellner <christian@kellner.me>
Add a simple check that data written through the loop device is
actually ending up in the file. NB: this this will _fail_ if the
fd is cleared via `clear_fd` without the use of `flush_buf`. It
seems that the kernel (as of 5.13.8) will indeed not clear the
buffer cache of the loop device if the backing file is detached
via `LOOP_CLR_FD`. On the other hand, if the autoclear flag is,
i.e. the backing file cleared when the last file descriptor of
the loop device is closed, the buffer cached will be cleared as
part of the `release` operation of the block device.
Add support for locking the loopback block device via `flock(2)`.
The main use case for this is to prevent systemd-udevd from
proben the device while any modification is done to it. See the
systemd page, https://www.freedesktop.org/software/systemd, for
more details.
Add the corresponding tests to it.
Add a helper method that clears the fd for a given loop device but
also ensures that the loop device is not bound to the supplied fd
anymore. Check the function documentation for more information.
Add a corresponding test.
