Set the container environment variable to indicate to programs
inside the build root that they are indeed running inside a
container (see also https://systemd.io/CONTAINER_INTERFACE/).
Create a well-defined environment with and use that for the build
root. It is not desirable to have the host's environment leak
into the container. Add a test to ensure that this works.
NB: This was probably an oversight when we switched from systemd-
nspawn to bubblewrap.
Wrap the LVM volume group in a LUKS container with the passphrase
`osbuild` (yes, really, super secure). NB: the kernel command line
is changed to include `luks.uuid` which is needed so that dracut
will attempt to open the luks container. This corresponds to an
crypttab entry `luks-uuid UUID`. We cannot use the /etc/crypttab
for ostree based images because the initrd is created at commit
time but they luks volume is created at deployment time, we have
to use the kernel command line instead. See the man page for the
systemd-cryptsetup-generator(8) for more information.
The `cryptsetup` package is included in the build root since it is
needed by the `org.osbuild.luks2.format` stage. All manifests that
are using the `f34-build-v2` build root change as a result.
OSTree tests, especially the fedora-ostree-image one, will soon
need the tight integration with the host for LVM2/LUKS support.
This we cannot run them in github action containers. Move them
to Schutzbot.
Explicitly install the new sub-package until composer gains the
needed requirement.
Add a new signal like callback to the `Loop` class which will be
invoked before the actual loop device is closed, i.e. the loop
device has an open file descriptor to the device node and it is
being closed. Can be used to perform custom cleanup tasks.
Certain udev rules for block devices are problematic for osbuild.
One prominent example is LVM2 related rules that would trigger
a scan and auto-activation of logical volumes. This rules are
triggered for new block devices or when the backing file of an
loop devices changes. The rules will lead to a `lvm pvscan
--cache --activate ay` via the `lvm2-pvscan@.service` systemd
service. This will auto-activate all LVM2 logical volumes and
thus interfering with our own device handling in `devices/
org.osbuild.lvm2.lv`, where we only want to activate a single
logical volume.
Also, if the lvm2 devices get activated after the manual metadata
change done in `org.osbuild.lvm2.metadata` the volume group names
might conflict which results in all lvm2 based tooling to be very,
ver sad and also said stage to hang since the loopback device can
not be detached since the activate logical volumes keep it open.
To work-around this we therefore implement a udev rule inhibition
mechanism: on the osbuild side a lock file is created via the new
class called `UdevInhibitor` in `utils/udev.py`. A custom set of
udev rules in `10-osbuild-inhibitor.rules` is then acting on the
existence of that lock file and if present will opt-out of certain
further processing. See the udev rules file for more details.
In fact, we want this custom inhibition mechanism, for all block
devices that are under osbuild's control, since these rules are
there to provide automatisms and integrations with the host,
something we never want.
NB: this should not affect the detection of devices, since lvm2
does do a scan of devices when we call `lvdisplay` in `lvm2.lv`.
The call chain as of lvm2 git rev f773040:
_lvdisplay_single [tools/lvdisplay.c
process_each_lv [tools/toollib.c
lvmcache_label_scan [lib/cache/lvmcache.c
label_scan [ibidem, here is the device detection!
lvdisplay_full [lib/display/display.c
Add support for `PermitRootLogin` option in the
`org.osbuild.sshd.config` stage.
I kept the "yes" and "no" values for consistency with other stage
options. While it will make the implementation in osbuild-composer
harder, it won't be impossible as we already have a precedence for doing
it this way (e.g. in the `org.osbuild.pam.limits.conf`).
Modify the stage unit tests to check the new option.
Remove the empty `org.osbuild.sshd.config` stage from `a.mpp.json`
since it does not add any value and it actually made the `tree-diff`
tool provide a weird tree diff results.
Fix#910
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use the new Manifest.depsolve function to only build the pipelines that
were explicitly requested and their dependencies, taking into account
what is already present in the store.
Since now not all pipeline will be built, there wont be a result entry
for all the pipelines, thus the format version 2 result formatting was
changed to not require the pipeline to be present in result set.
New function that take a list of pipelines and return the list of
pipelines that need to be build, i.e. the pipelines and all their
dependencies that are not already present in the store.
Add corresponding test.
When building a version 1 manifest, the assembler would always be
exported, even when not requested via the `--export` command line
option. This was done for backwards compatibility so to not break
tools relying on that behavior. The problem is that support for
this uses a completely different code path and might also now be
confusing behavior. Thus remove the implicit and really only ever
export what was explicitly requested by the caller.
Instead of checkpointing the tree and then accessing the generated
tree inside the store via the `map_object` function we not just
export the tree and use that. This better hides the internals of
the store and also allows us to activate on-demand building that
does not rely on checkpoints being implicitly built like exports.
Require all the tests that compile a manifest to either specify
checkpoints or exports. Convert all the tests that were relying
on implicit exports with v1 manifests to use explicit exports.
Since we bind `/proc` inside the container, we leak certain information
that comes with it. One of this is the kernel command line. None of the
decisions done by software running inside the container should depend
on the kernel command line on the host, so overwrite the kernel command
line by creating a temporary directory and mapping it inside the build-
root. For now we default to a simple `root=/dev/osbuild` fake kernel
command line.
Add a simple check for it as well.
This is to showcase it as much as to test its functionality. For this
the tar and xz stage tests have been converted. NB: only the mpp file
for each test is changed but the corresponding manifest is not.
The `sources/org.osbuild.inline` section has been kept otherwise the
ordering in the result manifest would change.
Instead of having custom code that basically duplicates the
functionality of `LoopControl.loop_for_fd` use that instead.
Additionally, the version used in the test had a bug where
it did not re-create the Loop device in the main loop when
it was close due to an error, leading errors in subsequent
usages of the device that would often manifest in CI runs:
fcntl.ioctl(self.fd, self.LOOP_SET_FD, fd)
ValueError: file descriptor cannot be a negative integer (-1)
This applies the default authconfig settings to the tree.
Note that the `/backups` directory is removed. The tool creaset
this, and by default it should not exist, so this should be a
noop. However, if you run this on a tree with existing backups,
they would be lost.
Add a new stage for modifying YUM global configuration.
Add a unit test case for the newly added stage.
Because we test stages on Fedora, where there is no YUM, and this stage
is mostly intended for being used with RHEL-7 images, the stage does not
produce error in case the `/etc/yum.conf` file does not exist. It rather
produces a warning and creates the file. Ideally the stage would produce
an error in case the configuration file does not exist, but that would
be impossible to test on recent Fedora.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
New stage to schedule a script via a /etc/cron.{hourly, daily ...}.
Currently only a simple command that will be put into an generated
script with the given name. Later more options might be added.
This commit adds options to the org.osbuild.grub2 stage to configure
terminal input and output, serial console and timeout.
The functionality and configuration schema is the same as in the legacy
grub2 stage.
This is requried to comply with Azure marketplace best
practices. The WALinuxAgent should not handle formating or
swap, as that is done by cloud-init.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is required for images to be importable to the AWS
marketplace. Both PasswordAuthentication and
ChallengeResponseAuthentication must be disabled.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add support for new 'install' command in the org.osbuild.modprobe stage.
Extend the unit test coverage to test the new command.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The `size` option was using `-l`, which in fact was `--extents`.
Fix that to use `--size` and add a new option `extents` that
will in fact call translate to `--extents` and this replace the
current use of the `size` option.
Adapt the `fedora-ostree-image` test manifest to use `extents`.
Introduce a new class to manage devices, `DeviceManger` and move the
code to open devices from the `Device` here. The main insight of why
the logic should be place here is that certain information is needed
to open the devices, independently of specific type: the path to the
device node directory, `devpath`, the actual `tree` and the service
manager instance to start the actual service. Instead of passing all
this information again and again to the `Device` class, we now have
a specialized (service) manager class for devices that has all the
needed information all the time. Additionally, the special handling
of parent devices is moved from the pipeline to the service manager,
which is where it belongs.
This will make even more sense for mounts, where the `DeviceManger`
can then be passed to access the individual devices.
Port the test to use the `DeviceManager`.
Add RHEL 7.9 example manifests. Add them to a `rhel` sub-directory in
the test/data/manifests directory since we cannot re-generate them
in the normal github actions, because they require access to RHEL
content.
Include a new test that writes a partition table to a disk and
then reads it back via `sfdisk` compares it against an layout
that was generated.
NB: This test needs `sfdisk` with `--json` support on the use host.
Port sources to also use the host services infrastructure that is
used by inputs, devices and mounts. Sources are a bit different
from the other services that they don't run for the duration of
the stage but are run before anything is built. By using the same
infrastructure we re-use the process management and inter process
communcation. Additionally, this will forward all messages from
sources to the existing monitoring framework.
Adapt all existing sources and tests.
Add a new stage `org.osbuild.pam_limits.conf`, which created
configuration files for `pam_limits` module in /etc/security/limits.d.
Add unit test for the new stage.
Fix#788
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Ability to set the bootloader backend that OSTree should use. NB:
normally this should be set to `none` since in modern distros and
bootloaders the BLS is used and the BLS snippets are generated on
`none` but none of the of the specific bootloader tools are run,
like `grub2-mkconfig` for grub.
Update the fedora image manifest to use that config setting.
Add a new stage `org.osbuild.sysctld` for setting kernel parameters at
boot by creating a configuration file in /usr/lib/sysctl.d. At least
one parameter must be specified for the stage.
Add unit test for the new stage.
Fix#790
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new stage `org.osbuild.tmpfilesd` for creating tmpfiles.d
configuration files in /usr/lib/tmpfiles.d. Provided list of
configuration directives is written as separate lines into
the configuration file. At least one configuration directive
must be specified.
Fix#786
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new `org.osbuild.selinux.config` stage to configure the
SELinux state on the system.
The stage configures the SELinux state on the system in /etc/selinux/config.
The policy enforcement state and active policy type can be configured.
Fix#785
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new `org.osbuild.dnf.config` stage for changing persistent DNF
configuration. Currently only DNF variables can be defined.
Fix#791
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new stage for setting active TuneD profile. The stage checks the
value of chosen TuneD profile(s) with the list of available TuneD
profiles installed in the filesystem root. If any of the chosen profiles
does not exist, the stage raises an exception.
Add unit tests for the new stage.
Fix#792
Signed-off-by: Tomas Hozza <thozza@redhat.com>
