When loop.Loop() is called and a new loop device must be allocated
there is no gurantee that the correct device node is available on
the system. In containers /dev is often just a tmpfs with static
device nodes. So when /dev/loopN is not available when the
container is created the device node will be missing even if
`get_unbound()` create a new loop device for us.
This commit ensures that the device node is available. It creates
it unconditionally and ignores any EEXIST errors to ensure there
is no TOCTOU issue.
Note that the test could have passed a `Loop(dir_fd=open(tmpdir))`
instead of creating/patching loop.DEV_PATH but it seems slightly
nicer to test the flow without a custom dir_path as this is what
the real code that creates a loop device is also using.
Quick rename to have our wording be in-line with the new differences
between stage unit tests and stage integration tests; also being applied
to the guides.
When osbuild.loop.Loop calls `__init__()` it assigns the `self.fd`
on open. However if that open call fails for whatever reason
(not found, permissions) the cleanup in `__del__` will fail in
confusing ways because `self.fd` is not initialized yet. It
also prevents the correct error from getting reported. A tiny
test is added to ensure this does not regress.
This commit removes some unnecessary custom tmpdir() fixtures
and uses the pytest buildin tmp_path instead.
Some custom tmpdir fixtures are left in place as they configure
the tmp location to be under `/var/tmp` which is not trivial to
do with pytests `tmp_path`. Not sure or not if the is a deep
reason there for using /var/tmp. I assume it's to ensure that
the tests run on a real FS not on a potential tmpfs but I don't
have the full background so didn't want to change anything.
This stage calls `update-crypto-policies` to set the
policy applicable for the various cryptographic back-ends,
such as SSL/TLS libraries.
Signed-off-by: Miguel Martín <mmartinv@redhat.com>
Also switch the qcow2 output to be a "qemu" platform image. This
change takes advantage of the recent org.osbuild.kernel-cmdline.bls-append
stage addition to modify kernel arguments in later pipelines.
To get a qemu image output:
- osbuild --output-directory out/ --export qemu test/data/manifests/fedora-coreos-container.json
To get a metal image output:
- osbuild --output-directory out/ --export metal test/data/manifests/fedora-coreos-container.json
This was done via:
```
$ sudo osbuild --export tree --output-directory /tmp/devnull test/data/stages/rpm/b.json --json | jq .metadata >test/data/stages/rpm/metadata.json
```
a outlined in `test/data/stages/rpm`.
Moving to the newer fedora 38 snapshot pulled in some changes
to the `/etc/dnf/automatic.conf` (e.g. [0]) when moving from
dnf 4.16.1 to 4.17.0. This commit updates the diff.
[0] a0acc88efc
The authselect upstream code dropped adding a timestamp to the
auto-geneated content in /etc/authselect [0]. With that the
content diff can be compared again.
[0] 44b9d87f90
Changes in `/etc/shadow` depend on the date, i.e. the last time
the password changed field. So for now exclude them from the
tree diff (until we use `faketime` to fix this :)
With lvm2 the generated fedora fc38 boot image boots in degraded
mode with the following error:
```
[root@localhost ~]# journalctl -u lvm2-monitor.service|more
Nov 13 12:52:04 localhost.localdomain lvm[431]: Failed to create /etc/lvm/devi
ces 2
Nov 13 12:52:04 localhost.localdomain lvm[431]: Failed to set up devices.
Nov 13 12:52:04 localhost.localdomain systemd[1]: lvm2-monitor.service: Main pro
cess exited, code=exited, status=5/NOTINSTALLED
Nov 13 12:52:04 localhost.localdomain systemd[1]: lvm2-monitor.service: Failed w
ith result 'exit-code'.
Nov 13 12:52:04 localhost.localdomain systemd[1]: Failed to start lvm2-monitor.s
ervice - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress p
olling.
```
This breaks the `test_boot.py` which expects the system after booting
in `running` state (from `systemd is-system-running`).
It looks like this is some sort of race with our generated image,
potentially related to selinux, see
https://github.com/lvmteam/lvm2/blob/v2_03_18/lib/device/dev-cache.c#L1842
and note the lines around dm_prepare_selinux_context(). Note
also that `lvm2-monitor.service` runs with `DefaultDependencies=no`
(c.f.
https://github.com/lvmteam/lvm2/blob/v2_03_18/scripts/lvm2_monitoring_systemd_red_hat.service.in#L7)
Given that the official fc38 cloud image does not use lvm2 and that
it's not needed for the boot test this commit simply removes it
from the fedora-boot manifest. This fixes the test.
The `test_assemblers.py` has an `assertGRUB2` helper that ensures
that the data written in the mbr and first megabyte is unchanged
by the `org.osbuild.qemu` assembler.
With the move to f38 the digests change. The mbr digest is
computed via the following python code and it matches that value
used in the test (440 byte only because the rest is the partition
table).
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f34/usr/lib/grub/i386-pc/boot.img
26e3327c6b5ac9b5e21d8b86f19ff7cb4d12fb2d0406713f936997d9d89de3ee
```
So with that code we can update the f38 mbr value now.
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f38/usr/lib/grub/i386-pc/boot.img
b8cea7475422d35cd6f85ad099fb4f921557fd1b25db62cd2a92709ace21cf0f
However computing the second sha256 for the `512:1024*1024` is much
harder to do from first principles because the value depends on the
image generated via `grub2-mkimage` and the hash changes with each
different module or config option. This means one needs to replicate
the exact inputs of:
```
subprocess.run(["grub2-mkimage",
"--verbose",
"--directory", f"/usr/lib/grub/{platform}",
"--prefix", f"(,{partid})/{grub_path}",
"--format", platform,
"--compression", "auto",
"--output", core_path] +
modules,
check=True)
```
in the test. At this point I cheated and just update to the computed
value inside the test.
The authselect stage will generate timestamps as part of the
update to `/etc/authselect/*`, e.g.:
```
Generate by authselect on Fri Nov 10 16:06:29 2023
...
```
this makes the content diff for those files unusable (until we
start to use faketime which maybe we should ;)
Instead of using the F34-based manifests, let's switch to F38. I tried my
best to import the vars into the new manifest, but I don't think that's
currently supported.
Let's not depend on f34-base.json anymore, but instead of a separate,
standardized manifest for it. The test was modified so it supports
v2 manifests. Also, the new manifest installs just a very minimal system.
There's no need to install the whole @core.
Authconfig was completely retired from Fedora. In order to keep this stage
covered, this commit changes the test to use CentOS Stream 9, which still
ships this package.
This gets rid of the old F34 manifest and migrates the to test to the
standardized V2 one based on F38.
Since the metadata format is much saner in V2, I was able to simplify the
assertion quite a lot.
This commit migrates the test to a brand new V2, F38-based manifest. It's
actually based on osbuild-composer interpretation of the Fedora Cloud Base
image.
This commit adds `osbuild.testutil.imports.import_module_from_path`
that can be used to import arbitrary python source files. This
allows importing files from the stages directory that have a
non python friendly filename like `org.osbuild.kickstart`.
The items of an array are defined under the `items` key, not under `paths`.
Let's fix this.
Btw, this is possible because JSON Schema itself doesn't use
additionalProperties = false. This allows extending the schemas easily, but
is sadly a bit error-prone.
Sadly, since this issue effectively disabled validation of the stage options,
we also need to relax the schema a bit:
We found out that there are manifests in the wild, that use relative paths,
instead of absolute ones. Thus, this commit changes the validation regex to
allow relative paths. However, this now emits a warning and it's strongly
discouraged. The associated stage test was modified to accommodate for this.
Co-authored-by: Tomáš Hozza <thozza@redhat.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This example shows how to build a qcow2 very similar to the one
delivered by Fedora CoreOS. It uses an input container that has
the Fedora CoreOS OSTree already baked into it, which means that
the OSTree isn't built using this manifest, but taken as an input
from a remote registry.
This drains some of the logic out of `main()` into a
`ostree_commit_deploy()` function. Doing this will make it easier
to diff this stage with the recently added `ostree.deploy.container`
stage.
This commit also changes the `ref` in the schema to be optional,
which is a fixup for 3cc733d. We need to make the ref optional because
the ref could come from the user in the toplevel schema or it could
come from input commit in the schema.
The test for this stage is failing because etc/shadow changes content
depending on the date that it runs on (due to the "date of last password
change" field). This causes the checksums to not be constant for our
tests and depends on the date.
This commit removes the checksums for etc/shadow from the test so that
they are not checked as part of the test. This worksaround the test
failure issue for now until a solution to the dynamic contents is
determined.
Signed-off-by: Michael Ho <michael.ho@ieee.org>
Added three users to the test to test all combinations of ssh key
definitions:
- a user called `onekey` that has a single key defined in the `key`
option.
- a user called `multikey` that has a multiple keys defined in the
`keys` array.
- a user called `bothy` that defines both a single key in the `key`
option and two other keys in the `keys` array.
This will allow a user to specify an input directly to the deploy
stage rather than requiring a ostree.pull stage to be called first.
Adding inputs will also be useful when we soon add support for
deploying from a container rather than just deploying from an existing
ostree commit in a repo.
Some platforms like the TI AM62 require a particular FAT geometry for
their CPU to read the file system (and thus the bootloader). Failing
that the CPU will simply not boot and keep looking for a bootloader.
Let's add some options to enforce a particular filesystem geometry
through the -g option of mkfs.fat.
Signed-off-by: Maxime Ripard <mripard@kernel.org>
This should just defer to rpm-ostree.
xref https://github.com/coreos/rpm-ostree/issues/4530
(If someone cares about reproducibility here, they can specify it;
alternatively and more reliably, they can create builder container
images and pin to those)
A simple test to make sure that our btrfs stages work.
Updating the testing container is needed, because the old version
didn't have btrfs-progs in it.
Osbuild doesn't support creating btrfs over multiple devices, thus it should
be fine to have only single and dup in the schema (other options are for raid)
So we have a quick reference how to work with btrfs-based images.
Co-authored-by: Christian Kellner <christian@kellner.me>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
systemd unit stage will now be able to add user unit dropin,
this is done by adding an additional field:unit-type
which is set to system by default. It also adds ability
to update the unit config with ConditionPathExists.
Relevant testing updated for the new workflow.
Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
