The `tree-diff` tool is used by the `gen-stage-diff` tool to generate
the stage `diff.json` files. A few of these stages have timestamped
files that need to be edited after the tool is run to replace the
`sha256` content hash with `null` so the stage diff tests ignore these
files.
This commit updates the `tree-diff` tool to check through a list of the
files that contain timestamps and to use the `null` value rather than
the `sha256` content hash so the stage tests don't fail.
Since the `/etc/shadow` file contains a timestamp we need to add a
`null` value rather than a `sha256` hash to tell the diff tool to ignore
these fields. The issue is that the timestamp will always be different
meaning the tests will pass for a day, but then fail after that.
I'm not sure what happened, but the test case started failing on the
diff on 'main'. I didn't change anything related to this test case in my
PR. The previous changes adjusted the vars, specifically the Fedora
snapshot date used to generate the manifests, but the test passed on
it.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
With the `oscap-utils-1.3.10` release, the `autotailor` command now
supports importing a JSON tailoring file[1] that is then converted to the
XML tailoring file which is consumed by the `oscap` command in the
remediation stage.
[1] https://github.com/ComplianceAsCode/schemas/blob/main/tailoring/schema.json
Since updating the snapshots the diffs for some stage tests have
changed. This commit updates the diffs accordingly.
I followed the same steps used in 1148a6e.
The optional_metadata_types option isn't available in older versions of
dnf. We could version-guard the option, but let's be more explicit and
check if it exists instead.
If it doesn't we can safely ignore it and rely on dnf's default
behaviour.
The original CentOS Stream GPG key uses SHA-1 in its signature. However,
SHA-1 is by default not allowed by the c10s / el10 crypto policy. As a
result, running the stage tests which use c9s on c10s / el10 are failing
when rpmkeys tries to import the key.
As part of CS-1616 [1], the CS GPG key has been resigned using SHA256,
however only in c10s for now. Let's use the SHA256 signed GPG key from
c10s for c9s manifests, to make tests pass also on c10s / el10.
[1] https://issues.redhat.com/browse/CS-1616
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When the filelists are enabled in the optional metadata, the local cache
for the depsolve will include a filelist file for each repository.
Count the files matching *filelists* using glob() and compare them with
the number of repositories when the option is enabled.
When the option is not enabled, there should be no filelists.
We originally enabled filelists unconditionally in dnf5. Let's make it
optional as part of the 'optional-metadata' option in the arguments so
that the caller can decide if it's needed, for example when using dnf5
with older repositories that have packages with file dependencies, but
don't download them when they're not needed, since the filelists are
quite large.
Filelist repo metadata is required when using newer versions of libdnf
with old repositories or packages that specify dependencies on files.
For example, depsolving with RHEL 9.3 repos on Fedora 40 fails to
resolve platform-python with the message
nothing provides /usr/libexec/platform-python needed by platform-python-...
Not loading filelists is beneficial because it greatly reduces the size
of the metadata to download. Filelists were downloaded by default for
repositories in the past, but with newer versions of dnf/libdnf (in
Fedora 40), they are downloaded on-demand (e.g. when running a query
that requires file information). Newer package guidelines prohibit
depending on file paths, so filelists are not needed for package
depsolving with newer distro repositories.
Add an 'optional-metadata' property to the 'arguments' object of the
depsolve request, so that callers can enable any optional metadata they
need.
Further reading:
- https://libdnf.readthedocs.io/en/stable/tutorial-py/#case-for-loading-the-filelists
- https://github.com/rpm-software-management/dnf/releases/tag/4.19.0
- https://dnf.readthedocs.io/en/stable/user_faq.html#starting-with-fedora-40-i-noticed-repository-metadata-is-synchronized-much-faster-what-happened
The `test_osbuild_mount_failure_msg` currently fails on fc40 when
run in tmt, see:
https://artifacts.dev.testing-farm.io/c6588a82-a2cb-46df-8ca8-85dd809465f2/
This is because the failure output is slightly different between
a container and a VM/real-machine. The test ensures that we capture
the output of mount and present to the user (for easier debugging).
So this commit updates this test once more for the error string
(that part of the error comes directly from the kernels fsconfig).
If we need another update of the string we should reconsider this
test and e.g. just use `testutil.mock_command()` for this. But
for now it's easier to just add this one more failure string.
Add the force_password_reset field to the users option in the
org.osbuild.users stage. This option forces the user to change their
password on first login.
This reverts commit bc04bfc366.
The `remove_tmpfiles()` helper is nice but it is also problematic
because it creates extra output after the command was run and
created output. E.g. a test failure on centos stream9 [0]
```
r = root.run(["stat", "--format=%a", "/var/tmp"], monitor)
assert r.returncode == 0
> assert r.stdout.strip().split("\n")[-1] == "1777"
E AssertionError: assert '/usr/lib/tmp... such process' == '1777'
E
E - 1777
E + /usr/lib/tmpfiles.d/rpcbind.conf:2: Failed to resolve user 'rpc': No such process
```
Here the output from "stat" is not the last output because the
rempve_tmpfiles runs `systemd-tmpfiles --clean --remove` which
produces some noisy output after stat was run.
This was found by @thozza (thanks!) and discussed in osbuild PR#1785.
There are various ways to fix this, the one is to use the
`--graceful` option of systemd-tmpfiles. However that only got added in
systemd v256 and centos-stream9 has v252 so that is sadly not an option.
Plus even when avaialble it will produce some informational output like
```
All rules containing unresolvable specifiers will be skipped.
```
Another way would be to sent the output from systemd-tmpfiles cleanup
to /dev/null. Not really great as we will not know about real problems
or warnings that we should care about.
None of the option above is good. So I started looking at the tmpfiles.d
rules and the cleanup and why we are doing it. It was added relatively
recently in https://github.com/osbuild/osbuild/pull/1458 and after
some medidiation not having it seems to do no harm (details below). The
tl;dr is that the buildroot is created inside bubblewrap and the
dirs that `--clean` and `--remove` touch are already tmpdirs created
just for the buildroot so the cleanup in the runner is redundant
(and because the cleanup is now run for each buidlroot.run() command
there *might* be unintended conequences but the current rules seem
to not have any).
In detail, the tmpfiles_cleanup() does two things:
1. `--clean`
It will remove files that are older then the given age
in tmpfiles.d. The tmpfiles in centos9 give me the following ages:
```
$ systemd-tmpfiles --cat-config|grep -E '[0-9]+d$'
d /var/lib/systemd/pstore 0755 root root 14d
d /var/lib/systemd/coredump 0755 root root 3d
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
```
Given that we run our commands inside a bubblewrap environment and
give it a fresh /run, /tmp, /var [1] there really should be no long
lived things and even if there are they are cleaned up from the
buildroot itself
2. `--remove`
It will remove files marked for removal in tmpdfiles.d. Running
it on a centos9 env it yields for me:
```
$ systemd-tmpfiles --cat-config|grep -E '^[rRD]'
R /var/tmp/dnf*/locks/*
r /var/cache/dnf/download_lock.pid
r /var/cache/dnf/metadata_lock.pid
r /var/lib/dnf/rpmdb_lock.pid
r /var/log/log_lock.pid
r! /forcefsck
r! /fastboot
r! /forcequotacheck
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
R! /var/tmp/container_images*
D /run/rpcbind 0700 rpc rpc - -
D /run/sudo/ts 0700 root root
R! /tmp/systemd-private-*
R! /var/tmp/systemd-private-*
r! /var/lib/systemd/coredump/.#*
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
r! /tmp/.X[0-9]*-lock
```
which is also covered by the bwrap cleanup.
[0] https://artifacts.dev.testing-farm.io/2d07b8f3-5f52-4e61-b1fa-5328a0ff1058/#artifacts-/plans/unit-tests
[1] https://github.com/osbuild/osbuild/blob/main/osbuild/buildroot.py#L218
Based on the filename, different sections of the unit configuration are
valid. Service (.service) units require a [Service] section and mount
(.mount) units requires a [Mount] section.
It might be possible to validate this in the schema but it would
probably require a major rewrite and could get complicated. For now,
validate it in the stage code itself.
This test ensures that the inputs of devices/mounts we generate for
bootc are actually considered valid by the schema. This is a more
blackbox style test compared to `test_get_schema_automatically_added`
which just checks that we get the expected schema but not that the
expected schema actually parses our inputs.
With PR#1727 merged there is no need to test for individual stages
if they support the bootc style mounting in their schema. All
stages now support devices/mounts now. So let's remove this
redundant code.
This will include the latest osbuild-ci container changes from
https://github.com/osbuild/containers/pull/73
This includes a bunch of binaries that were missing and lead
to skpped tests.
On Fedora 40 we switched to dnf5 too early. There is going to be a new
version in Fedora 41 soon, with breaking API changes, so in order to
reduce the number of things we need to support, switch back to using
libdnf on Fedora 40 (and 41 until dnf5 version 5.2.0 is available,
see #1769)
