Disable `curl --parallel` by default until the failure in
https://github.com/osbuild/osbuild-composer/pull/4247
is fully understood. It can be enabled via the environment:
```
OSBUILD_SOURCES_CURL_USE_PARALLEL=1
```
in the osbuild-composer test.
When using a modern curl we can download download multiple urls
in parallel which avoids connection setup overhead and is generally
more efficient. Use when it's detected.
TODO: ensure both old and new curl are tested automatically via
the testsuite.
Modern curl (7.68+) has a --parallel option that will download
multiple sources in parallel. This commit adds detection for this
feature as it is only available after RHEL 8.
In addition we need some more feature to properly support --parallel,
i.e. `--write-out` with json and exitcode options. This bumps the
requirements to 7.75+ which is still fine, centos9/RHEL9 have
7.76.
Instead of passing the url and options on the commandline this
commit moves it into a config file. This is not useful just yet
but it will be once we download multiple urls per curl instance.
Setting the user-agent using `--header` is broken in combination with
`--location`, `--proxy`, and an https endpoint which redirects. The
user-agent sent to the proxy changes after the client is redirected,
tripping up proxies.
For more information see https://issues.redhat.com/browse/RHEL-45364
Don't instruct mock to not bootstrap the chroot. This is making builds
to fail on F40 due to DNF5. The error message suggests to bootstrap
chroot.
Example:
https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/jobs/7239614060#L1363
Note that e.g. osbuild-composer does not specify the
--no-bootstrap-chroot option either.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update RHEL runners to the same versions that correspond to the actual
state of which release is currently GA and which is in development
(nightly). Specifically, run jobs only on:
- RHEL-8.10 GA
- RHEL-9.4 GA
- RHEL-9.5 nightly
This is the same set of RHEL workers, as used by 'manifest-db'
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update manifest-db ref to the latest one. This should solve the problem
with GCP el8 repos being gone in the old manifests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Similar to what was explained in 2e6d49fbe this commit updates
the l2hash in test_assemblers to the new values from fc40 images.
Sadly it is hard to derive them from first principles (see the
other commit) and given that this is legacy code it is probably
fine this way.
In the release loop upstream changes are merged to Centos every two
weeks. This creates a delay in error detection when new tests
being added upstream.
Running tests in Centos based containers on top of the upstream code
more frequently should speed up error detection.
Newly add building of RPMs for c10s on all supported arches.
Extend the c9s config to build RPMs for all supported arches.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `tree-diff` tool is used by the `gen-stage-diff` tool to generate
the stage `diff.json` files. A few of these stages have timestamped
files that need to be edited after the tool is run to replace the
`sha256` content hash with `null` so the stage diff tests ignore these
files.
This commit updates the `tree-diff` tool to check through a list of the
files that contain timestamps and to use the `null` value rather than
the `sha256` content hash so the stage tests don't fail.
Since the `/etc/shadow` file contains a timestamp we need to add a
`null` value rather than a `sha256` hash to tell the diff tool to ignore
these fields. The issue is that the timestamp will always be different
meaning the tests will pass for a day, but then fail after that.
I'm not sure what happened, but the test case started failing on the
diff on 'main'. I didn't change anything related to this test case in my
PR. The previous changes adjusted the vars, specifically the Fedora
snapshot date used to generate the manifests, but the test passed on
it.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
With the `oscap-utils-1.3.10` release, the `autotailor` command now
supports importing a JSON tailoring file[1] that is then converted to the
XML tailoring file which is consumed by the `oscap` command in the
remediation stage.
[1] https://github.com/ComplianceAsCode/schemas/blob/main/tailoring/schema.json
Since updating the snapshots the diffs for some stage tests have
changed. This commit updates the diffs accordingly.
I followed the same steps used in 1148a6e.
The optional_metadata_types option isn't available in older versions of
dnf. We could version-guard the option, but let's be more explicit and
check if it exists instead.
If it doesn't we can safely ignore it and rely on dnf's default
behaviour.
The original CentOS Stream GPG key uses SHA-1 in its signature. However,
SHA-1 is by default not allowed by the c10s / el10 crypto policy. As a
result, running the stage tests which use c9s on c10s / el10 are failing
when rpmkeys tries to import the key.
As part of CS-1616 [1], the CS GPG key has been resigned using SHA256,
however only in c10s for now. Let's use the SHA256 signed GPG key from
c10s for c9s manifests, to make tests pass also on c10s / el10.
[1] https://issues.redhat.com/browse/CS-1616
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When the filelists are enabled in the optional metadata, the local cache
for the depsolve will include a filelist file for each repository.
Count the files matching *filelists* using glob() and compare them with
the number of repositories when the option is enabled.
When the option is not enabled, there should be no filelists.
We originally enabled filelists unconditionally in dnf5. Let's make it
optional as part of the 'optional-metadata' option in the arguments so
that the caller can decide if it's needed, for example when using dnf5
with older repositories that have packages with file dependencies, but
don't download them when they're not needed, since the filelists are
quite large.
Filelist repo metadata is required when using newer versions of libdnf
with old repositories or packages that specify dependencies on files.
For example, depsolving with RHEL 9.3 repos on Fedora 40 fails to
resolve platform-python with the message
nothing provides /usr/libexec/platform-python needed by platform-python-...
Not loading filelists is beneficial because it greatly reduces the size
of the metadata to download. Filelists were downloaded by default for
repositories in the past, but with newer versions of dnf/libdnf (in
Fedora 40), they are downloaded on-demand (e.g. when running a query
that requires file information). Newer package guidelines prohibit
depending on file paths, so filelists are not needed for package
depsolving with newer distro repositories.
Add an 'optional-metadata' property to the 'arguments' object of the
depsolve request, so that callers can enable any optional metadata they
need.
Further reading:
- https://libdnf.readthedocs.io/en/stable/tutorial-py/#case-for-loading-the-filelists
- https://github.com/rpm-software-management/dnf/releases/tag/4.19.0
- https://dnf.readthedocs.io/en/stable/user_faq.html#starting-with-fedora-40-i-noticed-repository-metadata-is-synchronized-much-faster-what-happened
The `test_osbuild_mount_failure_msg` currently fails on fc40 when
run in tmt, see:
https://artifacts.dev.testing-farm.io/c6588a82-a2cb-46df-8ca8-85dd809465f2/
This is because the failure output is slightly different between
a container and a VM/real-machine. The test ensures that we capture
the output of mount and present to the user (for easier debugging).
So this commit updates this test once more for the error string
(that part of the error comes directly from the kernels fsconfig).
If we need another update of the string we should reconsider this
test and e.g. just use `testutil.mock_command()` for this. But
for now it's easier to just add this one more failure string.
Add the force_password_reset field to the users option in the
org.osbuild.users stage. This option forces the user to change their
password on first login.
This reverts commit bc04bfc366.
The `remove_tmpfiles()` helper is nice but it is also problematic
because it creates extra output after the command was run and
created output. E.g. a test failure on centos stream9 [0]
```
r = root.run(["stat", "--format=%a", "/var/tmp"], monitor)
assert r.returncode == 0
> assert r.stdout.strip().split("\n")[-1] == "1777"
E AssertionError: assert '/usr/lib/tmp... such process' == '1777'
E
E - 1777
E + /usr/lib/tmpfiles.d/rpcbind.conf:2: Failed to resolve user 'rpc': No such process
```
Here the output from "stat" is not the last output because the
rempve_tmpfiles runs `systemd-tmpfiles --clean --remove` which
produces some noisy output after stat was run.
This was found by @thozza (thanks!) and discussed in osbuild PR#1785.
There are various ways to fix this, the one is to use the
`--graceful` option of systemd-tmpfiles. However that only got added in
systemd v256 and centos-stream9 has v252 so that is sadly not an option.
Plus even when avaialble it will produce some informational output like
```
All rules containing unresolvable specifiers will be skipped.
```
Another way would be to sent the output from systemd-tmpfiles cleanup
to /dev/null. Not really great as we will not know about real problems
or warnings that we should care about.
None of the option above is good. So I started looking at the tmpfiles.d
rules and the cleanup and why we are doing it. It was added relatively
recently in https://github.com/osbuild/osbuild/pull/1458 and after
some medidiation not having it seems to do no harm (details below). The
tl;dr is that the buildroot is created inside bubblewrap and the
dirs that `--clean` and `--remove` touch are already tmpdirs created
just for the buildroot so the cleanup in the runner is redundant
(and because the cleanup is now run for each buidlroot.run() command
there *might* be unintended conequences but the current rules seem
to not have any).
In detail, the tmpfiles_cleanup() does two things:
1. `--clean`
It will remove files that are older then the given age
in tmpfiles.d. The tmpfiles in centos9 give me the following ages:
```
$ systemd-tmpfiles --cat-config|grep -E '[0-9]+d$'
d /var/lib/systemd/pstore 0755 root root 14d
d /var/lib/systemd/coredump 0755 root root 3d
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
```
Given that we run our commands inside a bubblewrap environment and
give it a fresh /run, /tmp, /var [1] there really should be no long
lived things and even if there are they are cleaned up from the
buildroot itself
2. `--remove`
It will remove files marked for removal in tmpdfiles.d. Running
it on a centos9 env it yields for me:
```
$ systemd-tmpfiles --cat-config|grep -E '^[rRD]'
R /var/tmp/dnf*/locks/*
r /var/cache/dnf/download_lock.pid
r /var/cache/dnf/metadata_lock.pid
r /var/lib/dnf/rpmdb_lock.pid
r /var/log/log_lock.pid
r! /forcefsck
r! /fastboot
r! /forcequotacheck
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
R! /var/tmp/container_images*
D /run/rpcbind 0700 rpc rpc - -
D /run/sudo/ts 0700 root root
R! /tmp/systemd-private-*
R! /var/tmp/systemd-private-*
r! /var/lib/systemd/coredump/.#*
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
r! /tmp/.X[0-9]*-lock
```
which is also covered by the bwrap cleanup.
[0] https://artifacts.dev.testing-farm.io/2d07b8f3-5f52-4e61-b1fa-5328a0ff1058/#artifacts-/plans/unit-tests
[1] https://github.com/osbuild/osbuild/blob/main/osbuild/buildroot.py#L218
