If you do math in mpp-format-int it could end up getting converted
to a float. Of course if you end up with a decimal value that isn't
`.0` that's a problem for an int, but if it is `.0` let's handle it
gracefully.
For example, math like this could end up with a value with `.0`:
mpp-format-int: "{bios_boot_size_mb * 1024 * 1024 / sector_size_bytes}"
For the org.osbuild.loopback the user can set the sector size, but
it had no effect on the underlying loopback device. Let's make it
meaningful by passing along the given value to the underlying code.
I'm trying to debug some failures, and having no feedback as to
what file we're parsing or what code is evaluated when something
fails makes it hard to debug manifests.
This adds some nice error messages that will help.
Check for valid ipv4 addresses via a regex in the schema and
add matching tests. This will ensure that only valid ipv4
addresses can be entereed in "ip", "gateway" or "nameservers".
Note that libc/kernel accept invalid ipv4 addresses and do
"interesting" things with them. So they accept `127.1` and
turn that into `127.0.0.1` or even `127.256` and turn that
into `127.0.1.0` because 256 overflows into the next segment
(thanks to Simon for poiting this out). If this becomes a
problem and customers rely on invalid ipv4 addresses we will
need to relax the rules but let's start strict and help our
users with more guardrails.
Note that no ipv6 validation via regex is done. The regex
on stackoverflow for validating ipv6 is 660 chars long
and that seems a bit too long for our schemas and putting
and error with that in front of our users.
This commit adds a small stage unit test and most importantly
a comemnt why `devices` is part of the schema (but appears unused).
The reason "devices" is explained by Alex Larsson:
"""
The mounts don't work without devices that have the filesystems.
In sample-images for example, this is typically used like so:
```
type: org.osbuild.ostree.post-copy
devices:
root:
type: org.osbuild.loopback
options:
filename: disk.img
mounts:
- name: root
type: org.osbuild.ext4
source: root
target: /
```
"""
When loop.Loop() is called and a new loop device must be allocated
there is no gurantee that the correct device node is available on
the system. In containers /dev is often just a tmpfs with static
device nodes. So when /dev/loopN is not available when the
container is created the device node will be missing even if
`get_unbound()` create a new loop device for us.
This commit ensures that the device node is available. It creates
it unconditionally and ignores any EEXIST errors to ensure there
is no TOCTOU issue.
Note that the test could have passed a `Loop(dir_fd=open(tmpdir))`
instead of creating/patching loop.DEV_PATH but it seems slightly
nicer to test the flow without a custom dir_path as this is what
the real code that creates a loop device is also using.
Quick rename to have our wording be in-line with the new differences
between stage unit tests and stage integration tests; also being applied
to the guides.
When osbuild.loop.Loop calls `__init__()` it assigns the `self.fd`
on open. However if that open call fails for whatever reason
(not found, permissions) the cleanup in `__del__` will fail in
confusing ways because `self.fd` is not initialized yet. It
also prevents the correct error from getting reported. A tiny
test is added to ensure this does not regress.
This commit removes some unnecessary custom tmpdir() fixtures
and uses the pytest buildin tmp_path instead.
Some custom tmpdir fixtures are left in place as they configure
the tmp location to be under `/var/tmp` which is not trivial to
do with pytests `tmp_path`. Not sure or not if the is a deep
reason there for using /var/tmp. I assume it's to ensure that
the tests run on a real FS not on a potential tmpfs but I don't
have the full background so didn't want to change anything.
If fs-verity is configured in ostree then ostree will (try to) enable
fs-verity on various repo files. However, in osbuild this will happen
in a separate pipeline, and these files will later be copied to the
final location on the physical filesystem, and any fs-verity status
then is lost.
To support fs-verity we need to run this stage after copying the image
to the filesystem. It uses the ostree "admin post-copy" operation.
which it will re-enable fs-verity as needed.
This adds a new stage that allows you to set the experimental new
`ex-integrity.composefs` option. If set to true, it means that when
deploying from this repository a composefs image will be created.
A value of `maybe` is also supported, which means composefs will only
be created if support is built into ostree.
Support for this was added in ostree 2023.4, earlier versions ignore
this key.
This stage uses the new prefix org.osbuild.experimental. This way
users will not accidentally enable an experimental option, and allows
us (and ostree) some leeway in making changes over time to this
feature.
In OSBuild we'll often be operating on sparse files. Let's make the
tabulation of the size of files on disk used when determining cache
size for pruning consider the actual size of the file usage on disk
rather than the size the file reports to be.
This means using os.lstat().st_blocks * 512 versus os.lstat().st_size.
See https://stackoverflow.com/a/55203604
Add comment why the `ModuleInfo.load()` code uses open()/ast.parse()
instead of just using `importlib`.
The reason is that while `importlib` is more convenient and much
shorter it would require that all python modules of the osbuild
modules are actually installed on the system just to inspect the
schema/documentation of the stage.
The existing man-page is a bit misleading as it claims that the
command in the example will generate an image. Because osbuild
generates what is requested this is not actually true. This
commit tweaks the examples a bit to make them more useful. It
also fixes a numbering bug (we had two "Example 1").
It also tweaks the description of `fedora-boot.json` as it is no
longer fedora 34 but 38.
The `shutil.rmtree(onerror=...)` kwarg got deprecated with py3.12.
We still need to support older version of python all the way
back to 3.6 so just ignore this pylint error for a while.
This stage calls `update-crypto-policies` to set the
policy applicable for the various cryptographic back-ends,
such as SSL/TLS libraries.
Signed-off-by: Miguel Martín <mmartinv@redhat.com>
Link "/proc/self/fd" to "/dev/fd" within the tree
to avoid "'/dev/fd/63': No such file or directory" errors
Signed-off-by: Miguel Martín <mmartinv@redhat.com>
This implements the display mode options `text`, `graphical`,
`cmdline` as an enum with the name `display_mode`.
See PR#1442 for the rational/discussion of this over using
three boolean options.
Thanks to Achilleas and Tom!
Also switch the qcow2 output to be a "qemu" platform image. This
change takes advantage of the recent org.osbuild.kernel-cmdline.bls-append
stage addition to modify kernel arguments in later pipelines.
To get a qemu image output:
- osbuild --output-directory out/ --export qemu test/data/manifests/fedora-coreos-container.json
To get a metal image output:
- osbuild --output-directory out/ --export metal test/data/manifests/fedora-coreos-container.json
This adds a stage to be able to add kernel arguments on a system by
appending to the BLS [1] config directly either in the tree or in
a mount. This is useful on say systems that don't use `grubby` and
thus can't use the org.osbuild.kernel-cmdline stage.
[1] https://freedesktop.org/wiki/Specifications/BootLoaderSpec/
Instead of running osbuild as a binary use `python3 -m osbuild`
(just like in `test/test.py:compile()`) so that it will use
osbuild fromgit and can be run from a checkout without the need
for an installed osbuild.
This was done via:
```
$ sudo osbuild --export tree --output-directory /tmp/devnull test/data/stages/rpm/b.json --json | jq .metadata >test/data/stages/rpm/metadata.json
```
a outlined in `test/data/stages/rpm`.
Moving to the newer fedora 38 snapshot pulled in some changes
to the `/etc/dnf/automatic.conf` (e.g. [0]) when moving from
dnf 4.16.1 to 4.17.0. This commit updates the diff.
[0] a0acc88efc
The authselect upstream code dropped adding a timestamp to the
auto-geneated content in /etc/authselect [0]. With that the
content diff can be compared again.
[0] 44b9d87f90
Changes in `/etc/shadow` depend on the date, i.e. the last time
the password changed field. So for now exclude them from the
tree diff (until we use `faketime` to fix this :)
With lvm2 the generated fedora fc38 boot image boots in degraded
mode with the following error:
```
[root@localhost ~]# journalctl -u lvm2-monitor.service|more
Nov 13 12:52:04 localhost.localdomain lvm[431]: Failed to create /etc/lvm/devi
ces 2
Nov 13 12:52:04 localhost.localdomain lvm[431]: Failed to set up devices.
Nov 13 12:52:04 localhost.localdomain systemd[1]: lvm2-monitor.service: Main pro
cess exited, code=exited, status=5/NOTINSTALLED
Nov 13 12:52:04 localhost.localdomain systemd[1]: lvm2-monitor.service: Failed w
ith result 'exit-code'.
Nov 13 12:52:04 localhost.localdomain systemd[1]: Failed to start lvm2-monitor.s
ervice - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress p
olling.
```
This breaks the `test_boot.py` which expects the system after booting
in `running` state (from `systemd is-system-running`).
It looks like this is some sort of race with our generated image,
potentially related to selinux, see
https://github.com/lvmteam/lvm2/blob/v2_03_18/lib/device/dev-cache.c#L1842
and note the lines around dm_prepare_selinux_context(). Note
also that `lvm2-monitor.service` runs with `DefaultDependencies=no`
(c.f.
https://github.com/lvmteam/lvm2/blob/v2_03_18/scripts/lvm2_monitoring_systemd_red_hat.service.in#L7)
Given that the official fc38 cloud image does not use lvm2 and that
it's not needed for the boot test this commit simply removes it
from the fedora-boot manifest. This fixes the test.
The `test_assemblers.py` has an `assertGRUB2` helper that ensures
that the data written in the mbr and first megabyte is unchanged
by the `org.osbuild.qemu` assembler.
With the move to f38 the digests change. The mbr digest is
computed via the following python code and it matches that value
used in the test (440 byte only because the rest is the partition
table).
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f34/usr/lib/grub/i386-pc/boot.img
26e3327c6b5ac9b5e21d8b86f19ff7cb4d12fb2d0406713f936997d9d89de3ee
```
So with that code we can update the f38 mbr value now.
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f38/usr/lib/grub/i386-pc/boot.img
b8cea7475422d35cd6f85ad099fb4f921557fd1b25db62cd2a92709ace21cf0f
However computing the second sha256 for the `512:1024*1024` is much
harder to do from first principles because the value depends on the
image generated via `grub2-mkimage` and the hash changes with each
different module or config option. This means one needs to replicate
the exact inputs of:
```
subprocess.run(["grub2-mkimage",
"--verbose",
"--directory", f"/usr/lib/grub/{platform}",
"--prefix", f"(,{partid})/{grub_path}",
"--format", platform,
"--compression", "auto",
"--output", core_path] +
modules,
check=True)
```
in the test. At this point I cheated and just update to the computed
value inside the test.
