If one of the chroot mounts fails to unmount, keep iterating so that we
don't stop and continue to unmount the rest.
Print an error message with the failed mounts, but don't fail the build.
Since failing to unmount doesn't fail the exiting of the context, and
the context itself doesn't know what will be running in the chroot,
do a lazy unmount.
This commit keeps track of individual errors as curl will only report
the last download operation success/failure via it's exit code.
There is "--fail-early" but the downside of that is that abort all in
progress downloads too.
curl keeps a global parser state. This means that if there are
multiple "cacert =" values they are just overriden and the last
one wins. This is why the `test_curl_download_many_mixed_certs`
test did not work - the second `cacert = ` overwrites the previous
one.
To fix this we need to use `--next` when we need to change options
on a per url (like `cacert`) basis. With `--next` curl starts a
new parser state for the next url (but keeps the options for the
previous ones set). This commit does that in a slightly naive
way by just repeating our options for each url. Technically
we could sort the sources so that we have less repetition but
other then slightly smaller auto-generated files it has no
advantage.
With this commit the `test_curl_download_many_mixed_certs` test
works.
When investigating https://github.com/osbuild/osbuild-composer/pull/4247
we found that it would fail when a download required two sets of
`--cacert` keys. This commits adds a test for this that fails on
the centos9 7.76.1 version.
When we create the device node inside the buildroot so far it's
very minimal - just `/dev/{vg}-{lv}` with the appopriate major/minor.
However when mount runs it will create a mapper device with the
same major/minor under `/dev/mapper/{escaped(vg)}-{escaped(lv)}`
and use that to mount the actual filesystem. Without this additional
device findmnt will not be able to detect the udev attributes of
the source (as the source is just missing from /dev).
This commit create the right mapper in the same way that we
create the non-mapper device node.
When generating the original test certs no `-days` paramter was
passed which resulted in a too low `notAfter` value.
This commit fixes this and uses 100y also updates the README:
```
$ openssl x509 -enddate -noout -in test/data/certs/cert1.pem
notAfter=Aug 2 10:42:40 2124 GMT
$ openssl x509 -enddate -noout -in test/data/certs/cert2.pem
notAfter=Aug 2 10:42:45 2124 GMT
```
This fixes a test failure in https://github.com/osbuild/osbuild/pull/1819
for the `test_curl_download_many_mixed_certs` test.
The libdir is passed down for sources but it is never used in
any of our sources. As this is confusing and we want to eventually
support multiple libdirs remove this code.
It looks like the libdir for soruces was added a long time ago in 8423da3
but there is no indication if/how it is/was supposed to get used and
AFACT from going over the git history it was very used.
SourceService:dispatch() never sends "libdir" to the actual sources,
so it is not an even technically an API break.
This commit extract a helper `get_parent_path()` that is unit
tested and also uses this generated parent_path for the call
to manage_devices_file to be consistent with the exiting behavior
of only including the device that actually contains the VG.
This is needed for bootc where all mounts need to be from the same
physical disk/loop so that bootupd works. The idea is that in the
manifest the new option `vg_partnum` is added and the parent VG
is found via the partition number of the full image similar to
the `partnum` from https://github.com/osbuild/osbuild/pull/1501
A manifest using this feature looks like this:
```json
"devices": {
"disk": {
"type": "org.osbuild.loopback",
"options": {
"filename": "disk.raw",
"partscan": true
}
},
"rootlv": {
"type": "org.osbuild.lvm2.lv",
"parent": "disk",
"options": {
"volume": "rootlv",
"vg_partnum": 4
}
}
}
```
Co-authored-by: Michael Vogt <mvogt@redhat.com>
storage_conf was never not None, so the loading was called every time.
This never crashed because conf was always being set, but it wasn't
working properly regardless.
Add two unit tests for our toml util module.
- Write an object with util.toml, read it with util.toml, and compare
written and read objects.
- Write an object directly as a string, read it with util.toml,
comparing with an expected object.
A test that writes with util.toml, reads as string, and verifies the
read string is difficult to do in a general way, because each toml
module we support writes files in a slightly different way.
Different toml libraries write arrays differently, so we can't know
exactly what the file contents will look like. Some will write an array
in a single line (toml) while others will break it into one element per
line (tomli_w). Parse the file that's written by the stage so we can
compare the objects instead of the text contents directly.
Add a new util module called host which is used for functions that are
meant for interactions with the host. These functions should not be
used in stages.
The containers.get_host_storage() function is renamed to
host.get_container_storage() for clarity, since it is no longer
namespaced under containers.
The containers.storage.conf stage writes a header explaining what the
configuration is doing and its origin. It also supports adding extra
comments via stage options, which we need to support. Add support for
writing comments at the top of the file in the toml.dump_to_file()
function.
The toml module situation in Python is a bit of a mess. Different
distro versions have different modules packaged or built-in, sometimes
with different capabilities (no writing). Since we need to support
reading and writing toml files both on the host (osbuild internals,
sources, inputs) and in the build root (stages), let's centralise the
import decision making in an internal utility module that covers all
cases.
Two of the modules we might import (tomli and tomllib) don't support
writing, so we need to either import a separate module (tomli_w) or
raise an exception when dump() is called without a write-capable module.
The tomli and tomllib modules require files be opened in binary mode
(not text) while the others require text mode. So we can't wrap the
toml.load() and toml.dump() functions directly; the caller doesn't know
which module it will be using. Let's keep track of the mode based on
which import succeeded and have our functions open the files as needed.
The wrapper functions are named load_from_file() and dump_to_file() to
avoid confusion with the load() and dump() functions that take a file
object.
See also #1847
With the mounting of /dev (among others) into the chroot for the
update-crypto-policies, the leftover /dev/null is now removed.
This was created by the update-crypto-policies script, running in the
chroot, by multiple output redirects into /dev/null. Without a /dev fs,
the file was being created in the tree and would remain on the image.
Recently [1], the update-crypto-policies script added a check to verify
that the FIPS policy was automounted by reading the
/proc/self/mountinfo. The script will fail if the proc filesystem isn't
available.
Use the new Chroot context to set up the environment for the command.
[1] 04ceadccfc
New chroot utility module that sets up a tree with the necessary virtual
filesystems needed for running commands in the root tree in a similar
environment as they would run in the build root.
This is needed for some stages, but may also be used for all chroot
calls to unify the setup and teardown of the root environment.
The Chroot context class was previously part of the org.osbuild.dracut
stage, which was the first stage to need this setup.
osbuild 126 added a new "solver" field in the osbuild-dnf-json
output and osbuild-composer only supports this since 116.
This broke production (sorry!). This commit helps mitigate it.
dracut expects the environment, in which it is run, to have properly
mounted /proc, /dev and /sys. Otherwise, some of its modules don't work
properly. E.g. dracut fails to embed the CA cert bundle into the initram
disk, which means that HTTPS won't work in it. dracut also prints a lot
of errors and warnings about this, but we used to ignore them until now.
The buildroot environment in which the stage runs is OK, but we actually
run dracut using 'chroot', which is the core of the problem. The runtime
environment in such case lacks the necessary mounts.
Add a context manager for setting up and cleaning up all the necessary
mounts in the image FS tree when running dracut.
This change is related to:
https://bugzilla.redhat.com/show_bug.cgi?id=1962975
And the implementation has been inspired by the fix in lorax:
https://github.com/weldr/lorax/pull/1151
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the dracut stage test case with checks for error / warning
messages complaining about unsupported / incorrect runtime environment.
Messages such as:
```
/dev/fd/63: No such file or directory
```
or
```
/proc/ is not mounted. This is not a supported mode of operation.
Please fix your invocation environment to mount /proc/ and /sys/
properly. Proceeding anyway. Your mileage may vary.
```
The stage will be fixed in the next commit.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
We recently hit the issue that `osbuild` crashed with:
```
Unable to decode response body "Traceback (most recent call last):
File \"/usr/bin/osbuild\", line 33, in <module>
sys.exit(load_entry_point('osbuild==124', 'console_scripts', 'osbuild')())
File \"/usr/lib/python3.9/site-packages/osbuild/main_cli.py\", line 181, in osbuild_cli
r = manifest.build(
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 477, in build
res = pl.run(store, monitor, libdir, debug_break, stage_timeout)
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 376, in run
results = self.build_stages(store,
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 348, in build_stages
r = stage.run(tree,
File \"/usr/lib/python3.9/site-packages/osbuild/pipeline.py\", line 213, in run
data = ipmgr.map(ip, store)
File \"/usr/lib/python3.9/site-packages/osbuild/inputs.py\", line 94, in map
reply, _ = client.call_with_fds(\"map\", {}, fds)
File \"/usr/lib/python3.9/site-packages/osbuild/host.py\", line 373, in call_with_fds
kind, data = self.protocol.decode_message(ret)
File \"/usr/lib/python3.9/site-packages/osbuild/host.py\", line 83, in decode_message
raise ProtocolError(\"message empty\")
osbuild.host.ProtocolError: message empty
cannot run osbuild: exit status 1" into osbuild result: invalid character 'T' looking for beginning of value
...
input/packages (org.osbuild.files): Traceback (most recent call last):
input/packages (org.osbuild.files): File "/usr/lib/osbuild/inputs/org.osbuild.files", line 226, in <module>
input/packages (org.osbuild.files): main()
input/packages (org.osbuild.files): File "/usr/lib/osbuild/inputs/org.osbuild.files", line 222, in main
input/packages (org.osbuild.files): service.main()
input/packages (org.osbuild.files): File "/usr/lib/python3.11/site-packages/osbuild/host.py", line 250, in main
input/packages (org.osbuild.files): self.serve()
input/packages (org.osbuild.files): File "/usr/lib/python3.11/site-packages/osbuild/host.py", line 284, in serve
input/packages (org.osbuild.files): self.sock.send(reply, fds=reply_fds)
input/packages (org.osbuild.files): File "/usr/lib/python3.11/site-packages/osbuild/util/jsoncomm.py", line 407, in send
input/packages (org.osbuild.files): n = self._socket.sendmsg([serialized], cmsg, 0)
input/packages (org.osbuild.files): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
input/packages (org.osbuild.files): OSError: [Errno 90] Message too long
```
The underlying issue is that the reply of the `map()` call is too
big for the buffer that `jsoncomm` uses. This problem existed before
for the args of map and was fixed by introducing a temporary file
in https://github.com/osbuild/osbuild/pull/1331 (and similarly
before in https://github.com/osbuild/osbuild/pull/824).
This commit writes the return values also into a file. This should
fix the crash above and make the function more symetrical as well.
Alternative/complementary version of
https://github.com/osbuild/osbuild/pull/1833
Closes: HMS-4537
When `jsoncomm` fails because the message is too big it currently
does not indicate just how big the message was. This commit adds
this information so that it's easier for us to determine what to
do about it.
We could also include a pointer to `/proc/sys/net/core/wmem_defaults`
but it seems we want to not require fiddling with that so let's
not do it for now.
See also https://github.com/osbuild/osbuild/pull/1838
The DNF5 implementation has known issues, which are not yet fixed and in
some cases, they can't be fixed due to the limitations and state of the
DNF5 implementation itself. Skip them for now.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case for installation of a package, which is excluded in a
different transaction. This is a common scenario in osbuild-composer,
where the image base package set is depsolved in the first transaction
and can contain package excludes. The user must be able to install these
excluded packages when specified explicitly in the Blueprint.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case for depsolving a package group with specific optional
packages of the group being excluded. This is a common scenario in
many image definitions.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case, that the repositories not enabled for a transaction are
not really used to depsolve its package specs. This is a common use case
for osbuild-composer, where the custom repositories specified by the
user are enabled only for the second transaction and not for the first
one (when depsolving the image base package set). This is important, so
that the user can't possibly replace important packages shipped by
the "system" repositories (e.g. kernel, systemd, etc.) with versions
from their custom repositories.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
