Recently [1], the update-crypto-policies script added a check to verify
that the FIPS policy was automounted by reading the
/proc/self/mountinfo. The script will fail if the proc filesystem isn't
available.
Use the new Chroot context to set up the environment for the command.
[1] 04ceadccfc
New chroot utility module that sets up a tree with the necessary virtual
filesystems needed for running commands in the root tree in a similar
environment as they would run in the build root.
This is needed for some stages, but may also be used for all chroot
calls to unify the setup and teardown of the root environment.
The Chroot context class was previously part of the org.osbuild.dracut
stage, which was the first stage to need this setup.
osbuild 126 added a new "solver" field in the osbuild-dnf-json
output and osbuild-composer only supports this since 116.
This broke production (sorry!). This commit helps mitigate it.
dracut expects the environment, in which it is run, to have properly
mounted /proc, /dev and /sys. Otherwise, some of its modules don't work
properly. E.g. dracut fails to embed the CA cert bundle into the initram
disk, which means that HTTPS won't work in it. dracut also prints a lot
of errors and warnings about this, but we used to ignore them until now.
The buildroot environment in which the stage runs is OK, but we actually
run dracut using 'chroot', which is the core of the problem. The runtime
environment in such case lacks the necessary mounts.
Add a context manager for setting up and cleaning up all the necessary
mounts in the image FS tree when running dracut.
This change is related to:
https://bugzilla.redhat.com/show_bug.cgi?id=1962975
And the implementation has been inspired by the fix in lorax:
https://github.com/weldr/lorax/pull/1151
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the dracut stage test case with checks for error / warning
messages complaining about unsupported / incorrect runtime environment.
Messages such as:
```
/dev/fd/63: No such file or directory
```
or
```
/proc/ is not mounted. This is not a supported mode of operation.
Please fix your invocation environment to mount /proc/ and /sys/
properly. Proceeding anyway. Your mileage may vary.
```
The stage will be fixed in the next commit.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
We recently hit the issue that `osbuild` crashed with:
```
Unable to decode response body "Traceback (most recent call last):
File \"/usr/bin/osbuild\", line 33, in <module>
sys.exit(load_entry_point('osbuild==124', 'console_scripts', 'osbuild')())
File \"/usr/lib/python3.9/site-packages/osbuild/main_cli.py\", line 181, in osbuild_cli
r = manifest.build(
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 477, in build
res = pl.run(store, monitor, libdir, debug_break, stage_timeout)
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 376, in run
results = self.build_stages(store,
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 348, in build_stages
r = stage.run(tree,
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 213, in run
data = ipmgr.map(ip, store)
File \"/usr/lib/python3.9/site-packages/osbuild/inputs.py\", line 94, in map
reply, _ = client.call_with_fds(\"map\", {}, fds)
File \"/usr/lib/python3.9/site-packages/osbuild/host.py\", line 373, in call_with_fds
kind, data = self.protocol.decode_message(ret)
File \"/usr/lib/python3.9/site-packages/osbuild/host.py\", line 83, in decode_message
raise ProtocolError(\"message empty\")
osbuild.host.ProtocolError: message empty
cannot run osbuild: exit status 1" into osbuild result: invalid character 'T' looking for beginning of value
...
input/packages (org.osbuild.files): Traceback (most recent call last):
input/packages (org.osbuild.files): File "/usr/lib/osbuild/inputs/org.osbuild.files", line 226, in <module>
input/packages (org.osbuild.files): main()
input/packages (org.osbuild.files): File "/usr/lib/osbuild/inputs/org.osbuild.files", line 222, in main
input/packages (org.osbuild.files): service.main()
input/packages (org.osbuild.files): File "/usr/lib/python3.11/site-packages/osbuild/host.py", line 250, in main
input/packages (org.osbuild.files): self.serve()
input/packages (org.osbuild.files): File "/usr/lib/python3.11/site-packages/osbuild/host.py", line 284, in serve
input/packages (org.osbuild.files): self.sock.send(reply, fds=reply_fds)
input/packages (org.osbuild.files): File "/usr/lib/python3.11/site-packages/osbuild/util/jsoncomm.py", line 407, in send
input/packages (org.osbuild.files): n = self._socket.sendmsg([serialized], cmsg, 0)
input/packages (org.osbuild.files): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
input/packages (org.osbuild.files): OSError: [Errno 90] Message too long
```
The underlying issue is that the reply of the `map()` call is too
big for the buffer that `jsoncomm` uses. This problem existed before
for the args of map and was fixed by introducing a temporary file
in https://github.com/osbuild/osbuild/pull/1331 (and similarly
before in https://github.com/osbuild/osbuild/pull/824).
This commit writes the return values also into a file. This should
fix the crash above and make the function more symetrical as well.
Alternative/complementary version of
https://github.com/osbuild/osbuild/pull/1833
Closes: HMS-4537
When `jsoncomm` fails because the message is too big it currently
does not indicate just how big the message was. This commit adds
this information so that it's easier for us to determine what to
do about it.
We could also include a pointer to `/proc/sys/net/core/wmem_defaults`
but it seems we want to not require fiddling with that so let's
not do it for now.
See also https://github.com/osbuild/osbuild/pull/1838
The DNF5 implementation has known issues, which are not yet fixed and in
some cases, they can't be fixed due to the limitations and state of the
DNF5 implementation itself. Skip them for now.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case for installation of a package, which is excluded in a
different transaction. This is a common scenario in osbuild-composer,
where the image base package set is depsolved in the first transaction
and can contain package excludes. The user must be able to install these
excluded packages when specified explicitly in the Blueprint.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case for depsolving a package group with specific optional
packages of the group being excluded. This is a common scenario in
many image definitions.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case, that the repositories not enabled for a transaction are
not really used to depsolve its package specs. This is a common use case
for osbuild-composer, where the custom repositories specified by the
user are enabled only for the second transaction and not for the first
one (when depsolving the image base package set). This is important, so
that the user can't possibly replace important packages shipped by
the "system" repositories (e.g. kernel, systemd, etc.) with versions
from their custom repositories.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Fix a bug in the `osbuild-depsolve-dnf`, which would cause the
`RepoError` to not be handled and producing a traceback.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Implement the generator for repo config combinations, instead of
hard-coding it. The motivation is to be able to optionally add e.g.
q broken repo to the list to test `RepoError`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
A wrong exception type was returned for the same kind of issues,
compared to the DNF4 version. Specifically, the DNF4 version returned
`MarkingErrors`, while the DNF5 version returned `DepsolveError`, when
a non-existent package was specified in the depsplve request. Make the
behavior consistent and return `MarkingErrors` also from the DNF5
version.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The returned error reason didn't contain any details after the merge
with DNF4 version. The reason is that previously, the actual exception
returned by the DNF library was appended to the error reason. However,
now it is wrapped by a custom `MarkingErrors` exception, which didn't
have any details set. The wrapped exception in the `__cause__`
property was not taken into account. Revert to the original behavior
by reusing the wrapped exception message as the message for the
wrapper exception.
Extend the unit test to allow testing of depsolving failures.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Porting the latests osbuild version in `osbuild/images` revealed a
regression in error messages returned by the tool in case a non-existent
package is requested in the depsolve request.
Test that requesting a non-existent package results in `MarkingErrors`,
which was the original behavior of the DNF4-based
`osbuild-depsolve-dnf`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Remove one test case, which does not really add value or extend the test
coverage, since the same thing is already tested by previous test case.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Until now, the unit test tested only very simple case with a single
transaction, which happens only for vanilla images. Any user
customization would result in multiple transactions in the depsolve
request. This case is not yet tested at all.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Modify the function that executes the depsolve command, so that does not
produce exception in case the command fails. Instead, return the message
returned by `osbuild-depsolve-dnf` and the exit code. This will allow to
test also other scenarios (such as depsolve errors) in the test case.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Use the latest c9s BaseOS repodata snapshot, specifically so that it
contains multiple versions of the same packages. This will allow to test
the `osbuild-depsolve-dnf` 'search' command. The previous metadata
contained only single version of each package.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When subscription-manager DNF plugins are enabled (e.g. on RHEL), they
produce messages to the stdout on any DNF command execution. E.g.
"Updating Subscription Management repositories.".
Disable all plugins when inspecting package markings so prevent them
from modifying the output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The spinner is mostly distracting IMHO and it makes the CI logs
harder to read as it generates several dozens of lines like:
```
...
⠋ [3/5] pylint | autopep8 | mypy-strict
...
```
I was considering to make it conditional on GH but decided to
just disable globally. If someone feels attached to the spinner
I only disable it in GH.
This commit includes the used sovler in the dnfjson reply. This
is mostly information (e.g. in service logs) but also useful in
tests to ensure that the expected solver was really run.
Note that this needs https://github.com/osbuild/images/pull/723
first.
When I rolled back from using 'dnf4', to check package markings, to
using 'dnf', I didn't verify the test case on Fedora Rawhide with DNF5.
It turns out that the strings reported by DNF5 differ and make the test
case fail. This time I tested the change on Fedora Rawhide with DNF5 and
it works.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This moves the dnf and dnf5 code into a new osbuild module called
solver. The dnf specific code is in dnf.py and dnf5 is in dnf5.py
At runtime the osbuild-depsolve-dnf script reads a config file from
/usr/lib/osbuild/solver.json and imports the selected solver. This
currently just contains a 'use_dnf5' bool but can be extended to support
other configuration options or depsolvers.
At build time a config file is selected from tools/solver-dnf.json or
tools/solver-dnf5.json and installed. Currently dnf5 is not installed,
it will be added when dnf5 5.2.1.0 becomes available in rawhide (Fedora
41).
The error messages have been normalized since the top level functions in
osbuild-depsolve-dnf do not know which version of dnf is being used.
The existing code to record progress was a bit too naive. Instead
of just counting the number os pipelines in a manifest to get the
total steps we need to look at the resolved pipelines.
with this fix `bib` will report the correct number of steps left
when doing e.g. a qcow2 image build. Right now the number of
steps is incorrect because the osbuild manifest contains pipelines
for qcow2,vdmk,raw,ami and all are currently considered steps
that need to be completed. With this commit this is fixed.
When using `--write-out` we are not using %{json} because older curl
(7.76) will write {"http_connect":000} which python cannot parse.
So we had a custom `--write-out` with `\1xc` as "record" separators
between the fields. This is a bit old-school and not very extensible
so Achilleas had the idea to still use json but "define" our own
subset via the variables that curl provides. This commit does that.
This commit adds a new `https_serve_directory()` test helper
and some custom self-signed and worthless certs that are used
during testing. They are not dynamically generated to avoid the
extra compuation time during tests (but they could be).
Generated via:
```
$ openssl req -new -newkey rsa:2048 -nodes -x509 \
-subj "/C=DE/ST=Berlin/L=Berlin/O=Org/CN=localhost" \
-keyout "key1.pem" -out "cert1.pem"
```
This will allow us to test `https` download URLs as well in e.g.
the curl source.
The test case still fails on RHEL-10.0 Beta, even when not using dnf5,
with:
```
for line in r.stdout.splitlines():
> package, mark = line.strip().split(",")
E ValueError: not enough values to unpack (expected 2, got 1)
```
Make debugging of failures like this easier by printing the line when
the issue happens.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Let's revert to using plain 'dnf', add an explicit newline in the query
format and skip empty lines when processing the output. This makes the
test case compatible with all DNF versions, even with dnf5 once this
issue gets fixed.
The previous approach didn't work on c9s / el9, because there is no
'/usr/bin/dnf4 -> dnf-3' symlink.
Also see:
https://github.com/osbuild/osbuild/actions/runs/10136827918/job/28026181824
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
