The skopeo stage is updated to accommodate two types of
destinations: 'containers-storage' and 'oci'. Now, it can
copy a container image to either a container store or
a directory as per 'Open Container Image Layout
Specification'.
Pytoml is no longer being maintained: https://github.com/avakar/pytoml
The author suggest switching to toml.
We already use the
```
try:
import toml
except ModuleNotFoundError:
import pytoml as toml
```
pattern in stages/org.osbuild.containers.storage.conf so use it in the tests too to prefer "toml" instead of pytoml.
This helper can be used to implement a strategy to find the oldest
cache entries and evict them when the cache is full.
The implementation uses the `atime` of the per object `cache.lock`
file and ensures in `load()` that it's actually updated.
This commit adds mount output to the error raised by
FileSystemMountService.mount(). This is useful when running into
mount failures during osbuild runs.
The issue was discovered while debugging a mount failure for
osbuild-composer PR#3820. Initially osbuild PR#1490 was meant
to fix it but it turned out there is a third mount helper in
the code that was originally overlooked (sorry for that!).
This is a convenient way for tests to assert that some nested dicts
(like a parsed json) has a particular key/value somewhere in it.
For example:
assert_dict_has(config, "toplevel.subitem.key", True)
While debugging a failure of osbuild-composer [0] on fc39 it was
noticed that a mount failure does not include the output of
the mount command:
```
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 78, in mount
path = client.call("mount", args)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 348, in call
ret, _ = self.call_with_fds(method, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 384, in call_with_fds
raise error
osbuild.host.RemoteError: CalledProcessError: Command '['mount', '-t', 'xfs', '-o', 'ro,norecovery', '--source', '/dev/rootvg/applv', '--target', '/tmp/tmpjtfmth56/app']' returned non-zero exit status 32.
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 268, in serve
reply, reply_fds = self._handle_message(msg, fds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 301, in _handle_message
ret, fds = self.dispatch(name, args, fds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 111, in dispatch
r = self.mount(args)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 160, in mount
subprocess.run(
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
```
which makes diagnostic errors harder of course. This commit adds
a test that ensures that mount output is visbile and also changes
the code to include it.
[0] https://github.com/osbuild/osbuild-composer/pull/3820
osbuild-composer project has build related section in the main README.
For the osbuild it was missing. Even if this is script-based project
external developers may not be aware of this just by looking at README.
Missing build related section has been added for consistency.
This commit extends the current support for OpenSCAP
tailoring by accepting an array of key/value overrides.
Users will be able to specify override values for specific
rules that will update the value when remediating the
image.
The next commit will add a stage test that requires erofs-utils. Let's add it
into the buildroot in a separate commit, so the history is more readable.
Add a simple stage test for the erofs stage. It uses dump.erofs
instead of mounting the file because the kernel in the GH runners
do not support mounting erofs just yet.
This adds tests for the erofs stage. The tests are slightly different
from the existing tests that run the filesystem utils inside the
stages. Depending on what exactly we want to test we may still need
a run inside the stages. However running this inside a container
should be good enough if we just want to validate that the options
are passed correctly and the file is created.
Erofs is "a lightweight read-only file system"[1]. Imagine squashfs, but with
faster reads.
This commit adds support for creating it. The new stage is heavily inspired by
the squashfs one. I've decided to add all features of mkfs.erofs that looked
useful: All compression types and most of extended options (excluding the
compatibility ones, we can always add them later).
[1]: https://en.wikipedia.org/wiki/EROFS
Add a new Packit "copr_build" job, which will build new upstream
releases in a dedicated COPR project "osbuild-stable". This will allow
people to consume stable builds of osbuild ASAP (including some
sub-packages on CS which are not part of the official distro builds).
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
`mpp-define-images` can create an image file, using `losetup` to deal
with non-standard sector sizes requires root. Not all users run
`osbuild-mpp` as root.
While I am not a fan of "suddenly sudo" based on the input manifest this
does alleviate builds breaking for manifests with default sector sizes
when non-root.
This is a variation of PR https://github.com/osbuild/osbuild/pull/960
that put the machine-id handling into it's own stage and adds
explicit handling what should happen with it.
For machine-id(5) we essentially want the following three states
implemented:
1. `first-boot: yes` will ensure that /etc/machine-id is
in the "uninitialized" state. This means on boot the systemd
`ConditionFirstBoot` is triggered and a new id in `/etc/machine-id`
is created. This will work for systemd v247+.
2. `first-boot: no` will ensure that /etc/machine-id exists but
is empty. This will trigger the creation of a new machine-id but
will *not* trigger `ConditionFirstBoot`.
3. `first-boot: preserve` will just keep the existing machine-id.
Note that it will error if there is no /etc/machine-id
Note that the `org.osbuild.rpm` will also create a
`{tree}/etc/machine-id` while it runs to ensure that postinst
scripts will not fail that rely on this file. This is an
implementation detail but unfortunately the rpm stage will
leave an empty machine-id file if it was missing. So we cannot
just remove /etc/machine-id because any following rpm stage
would re-create it again (and we cannot change that without
breaking backward compatiblity). Thanks to the special semantic
that a missing /etc/machine-id and an /etc/machine-id with
the `uninitialized` string are equivalent we don't care.
To support systemd versions below v247 we could offer an option
to remove /etc/machine-id. But the downside of this is that
it would only work if the org.osbuild.machine-id stage is after
the rpm stage.
See also the discussion in PR#960.
Thanks to Tom, Christian for the PR and the background.
For our Fedora CoreOS disk images we set the partition labels (name)
for the partitions. This is also supported using the primitives here
in OSBuild, but it wasn't obvious that I needed to set the name in
the mpp-define-images definition. Let's set the name there, but let's
also allow osbuild-mpp to set the `id`, which is what is used later
to access that partition from the `name` too if `id` isn't set.
This means we allow something like:
- name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
rather than requiring something like:
- id: BIOS-BOOT
name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
Now you can specify a sector_size in `mpp-define-images` to support
creating a 4k native disk image (sector_size=4096).
This does use a loopback device, which means osbuild-mpp also needs
to run as root, when previously that wasn't necessary.
If you do math in mpp-format-int it could end up getting converted
to a float. Of course if you end up with a decimal value that isn't
`.0` that's a problem for an int, but if it is `.0` let's handle it
gracefully.
For example, math like this could end up with a value with `.0`:
mpp-format-int: "{bios_boot_size_mb * 1024 * 1024 / sector_size_bytes}"
For the org.osbuild.loopback the user can set the sector size, but
it had no effect on the underlying loopback device. Let's make it
meaningful by passing along the given value to the underlying code.
I'm trying to debug some failures, and having no feedback as to
what file we're parsing or what code is evaluated when something
fails makes it hard to debug manifests.
This adds some nice error messages that will help.
Check for valid ipv4 addresses via a regex in the schema and
add matching tests. This will ensure that only valid ipv4
addresses can be entereed in "ip", "gateway" or "nameservers".
Note that libc/kernel accept invalid ipv4 addresses and do
"interesting" things with them. So they accept `127.1` and
turn that into `127.0.0.1` or even `127.256` and turn that
into `127.0.1.0` because 256 overflows into the next segment
(thanks to Simon for poiting this out). If this becomes a
problem and customers rely on invalid ipv4 addresses we will
need to relax the rules but let's start strict and help our
users with more guardrails.
Note that no ipv6 validation via regex is done. The regex
on stackoverflow for validating ipv6 is 660 chars long
and that seems a bit too long for our schemas and putting
and error with that in front of our users.
This commit adds a small stage unit test and most importantly
a comemnt why `devices` is part of the schema (but appears unused).
The reason "devices" is explained by Alex Larsson:
"""
The mounts don't work without devices that have the filesystems.
In sample-images for example, this is typically used like so:
```
type: org.osbuild.ostree.post-copy
devices:
root:
type: org.osbuild.loopback
options:
filename: disk.img
mounts:
- name: root
type: org.osbuild.ext4
source: root
target: /
```
"""
When loop.Loop() is called and a new loop device must be allocated
there is no gurantee that the correct device node is available on
the system. In containers /dev is often just a tmpfs with static
device nodes. So when /dev/loopN is not available when the
container is created the device node will be missing even if
`get_unbound()` create a new loop device for us.
This commit ensures that the device node is available. It creates
it unconditionally and ignores any EEXIST errors to ensure there
is no TOCTOU issue.
Note that the test could have passed a `Loop(dir_fd=open(tmpdir))`
instead of creating/patching loop.DEV_PATH but it seems slightly
nicer to test the flow without a custom dir_path as this is what
the real code that creates a loop device is also using.
