This moves the dnf and dnf5 code into a new osbuild module called
solver. The dnf specific code is in dnf.py and dnf5 is in dnf5.py
At runtime the osbuild-depsolve-dnf script reads a config file from
/usr/lib/osbuild/solver.json and imports the selected solver. This
currently just contains a 'use_dnf5' bool but can be extended to support
other configuration options or depsolvers.
At build time a config file is selected from tools/solver-dnf.json or
tools/solver-dnf5.json and installed. Currently dnf5 is not installed,
it will be added when dnf5 5.2.1.0 becomes available in rawhide (Fedora
41).
The error messages have been normalized since the top level functions in
osbuild-depsolve-dnf do not know which version of dnf is being used.
The existing code to record progress was a bit too naive. Instead
of just counting the number os pipelines in a manifest to get the
total steps we need to look at the resolved pipelines.
with this fix `bib` will report the correct number of steps left
when doing e.g. a qcow2 image build. Right now the number of
steps is incorrect because the osbuild manifest contains pipelines
for qcow2,vdmk,raw,ami and all are currently considered steps
that need to be completed. With this commit this is fixed.
When using `--write-out` we are not using %{json} because older curl
(7.76) will write {"http_connect":000} which python cannot parse.
So we had a custom `--write-out` with `\1xc` as "record" separators
between the fields. This is a bit old-school and not very extensible
so Achilleas had the idea to still use json but "define" our own
subset via the variables that curl provides. This commit does that.
This commit adds a new `https_serve_directory()` test helper
and some custom self-signed and worthless certs that are used
during testing. They are not dynamically generated to avoid the
extra compuation time during tests (but they could be).
Generated via:
```
$ openssl req -new -newkey rsa:2048 -nodes -x509 \
-subj "/C=DE/ST=Berlin/L=Berlin/O=Org/CN=localhost" \
-keyout "key1.pem" -out "cert1.pem"
```
This will allow us to test `https` download URLs as well in e.g.
the curl source.
The test case still fails on RHEL-10.0 Beta, even when not using dnf5,
with:
```
for line in r.stdout.splitlines():
> package, mark = line.strip().split(",")
E ValueError: not enough values to unpack (expected 2, got 1)
```
Make debugging of failures like this easier by printing the line when
the issue happens.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Let's revert to using plain 'dnf', add an explicit newline in the query
format and skip empty lines when processing the output. This makes the
test case compatible with all DNF versions, even with dnf5 once this
issue gets fixed.
The previous approach didn't work on c9s / el9, because there is no
'/usr/bin/dnf4 -> dnf-3' symlink.
Also see:
https://github.com/osbuild/osbuild/actions/runs/10136827918/job/28026181824
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
There was a small mistake in the schema since either one of
`new_profile` or `tailored_profile_id` is required. This commit fixes
this and updates the tests to check for this case.
This commit allows to the user to define the "container-transport"
variable when they are defining a container image to be resolved by the
osbuild-mpp tool. When the user defines the "container-storage" as
"container-transport" osbuild-mpp will look for the image in the local
container-storage.
Signed-off-by: Roberto Majadas <rmajadas@redhat.com>
SELinux policy forbids systemd-machined from creating a varlink socket,
which it does since v256. This makes it fail to start. Subsequently,
mock fails to build the RPM, because it uses systemd-nspawn as an
isolation and it fails to register the machine with systemd-machined.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
According to `autotailor(8)` arguments passed in via the cli take
precedence over the JSON tailoring file contents.
Make the `new_profile` a required field for the json tailoring too and
pass it as an option to the `autotailor` command. This approach has some
trade-offs. It allows us to maintain the explicitness of the manifest
that is consumed by `osbuild`. The downside is that it will override the
profile id that is set by the user in the JSON tailoring file.
Rename the `new_profile` option to `tailoring_profile_id` for clarity.
This also ensures that the change is backwards compatible by falling
back to the `new_profile` option if that was set instead of the
`tailoring_profile` id option.
As part of the investigation of the CI failure in
https://github.com/osbuild/osbuild-composer/pull/4247
we noticed that curl can return a return_code of `0` even
when it did not downloaded all the urls in a `--config` provided
file. This seems to be curl version dependent, I had a hard
time writing a test-case with the real curl (8.6.0) that
reproduces this so I went with mocking it. We definietly saw
this failure with the centos 9 version (7.76).
Our current code is buggy and assumes that the exit status
of curl is always non-zero if any download fails but that is
only the case when `--fail-early` is used.
The extra paranoia will not hurt even when relying on the
exit code of curl is fixed.
Manifest tests on Fedora 39 sometimes fail, because checking out the
specific manifest-db commit fails with:
"error: The following untracked working tree files would be
overwritten by checkout"
Use --force when checking out the ref, which will hopefully solve any
error like this.
[1] https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/jobs/7317097983
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This is a small adjustment to how the `osbuild.util.mnt.mount` fuction
is imported. While adding unit tests, the tests failed when trying to
patch the function, this commit fixes the issue.
Disable `curl --parallel` by default until the failure in
https://github.com/osbuild/osbuild-composer/pull/4247
is fully understood. It can be enabled via the environment:
```
OSBUILD_SOURCES_CURL_USE_PARALLEL=1
```
in the osbuild-composer test.
When using a modern curl we can download download multiple urls
in parallel which avoids connection setup overhead and is generally
more efficient. Use when it's detected.
TODO: ensure both old and new curl are tested automatically via
the testsuite.
Modern curl (7.68+) has a --parallel option that will download
multiple sources in parallel. This commit adds detection for this
feature as it is only available after RHEL 8.
In addition we need some more feature to properly support --parallel,
i.e. `--write-out` with json and exitcode options. This bumps the
requirements to 7.75+ which is still fine, centos9/RHEL9 have
7.76.
Instead of passing the url and options on the commandline this
commit moves it into a config file. This is not useful just yet
but it will be once we download multiple urls per curl instance.
Setting the user-agent using `--header` is broken in combination with
`--location`, `--proxy`, and an https endpoint which redirects. The
user-agent sent to the proxy changes after the client is redirected,
tripping up proxies.
For more information see https://issues.redhat.com/browse/RHEL-45364
Don't instruct mock to not bootstrap the chroot. This is making builds
to fail on F40 due to DNF5. The error message suggests to bootstrap
chroot.
Example:
https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/jobs/7239614060#L1363
Note that e.g. osbuild-composer does not specify the
--no-bootstrap-chroot option either.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update RHEL runners to the same versions that correspond to the actual
state of which release is currently GA and which is in development
(nightly). Specifically, run jobs only on:
- RHEL-8.10 GA
- RHEL-9.4 GA
- RHEL-9.5 nightly
This is the same set of RHEL workers, as used by 'manifest-db'
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update manifest-db ref to the latest one. This should solve the problem
with GCP el8 repos being gone in the old manifests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Similar to what was explained in 2e6d49fbe this commit updates
the l2hash in test_assemblers to the new values from fc40 images.
Sadly it is hard to derive them from first principles (see the
other commit) and given that this is legacy code it is probably
fine this way.
