This commit adds a new `transform` option to the tar stages that
maps directly to the `--transform=` comamndline argument of tar(1).
This allows to transform the names while files/dirs are added to
a tarfile. This is useful for the `gcp` pipeline for
bootc-image-builder where we want to create a gcp tar file that
expects the disk image filename in the tar to be exactly `disk.raw`.
Note that tar allows only a single `--transform` and we leave it
to the user to construct `sed` expressions if multiple renames
are required.
With the labels option the user is specifying the exact context
they want to set on the path so it's not necessary to supply a
context here. This can be also useful in the case where you want
to set some labels and you haven't yet populated the tree yet.
For the DNF4 version, we actually use 'dnf' package and not 'libdnf'.
Fix the SPEC file dependencies and also the check in unit test.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This is essentially org.osbuild.systemd.unit.create but creates
file where podman looks for quadlets instead.
For now only container, volume and network is supported.
Not all quadlet options are supported, but at least the most usef ones, and enough for the automotive sample-images.
Its not uncommon for systemd unit key values to contain things like
"%t", as these are magic values expanded by systemd. We need to
disable the ConfigParser default interpolation that treats '%' as
meaning interpolation.
Otherwise you will get errors like:
```
File "/run/osbuild/bin/org.osbuild.systemd.unit.create", line 66, in <module>
r = main(args["tree"], args["options"])
File "/run/osbuild/bin/org.osbuild.systemd.unit.create", line 46, in main
config.set(section, option, str(value))
File "/usr/lib64/python3.9/configparser.py", line 1204, in set
super().set(section, option, value)
File "/usr/lib64/python3.9/configparser.py", line 894, in set
value = self._interpolation.before_set(self, section, option,
File "/usr/lib64/python3.9/configparser.py", line 402, in before_set
raise ValueError("invalid interpolation syntax in %r at "
ValueError: invalid interpolation syntax in '%t/asil-ipc-demo/asil_ipc.socket' at position 0
```
Currently if one uses `mpp-embed` with URLs, osbuild-mpp still wants to
download the full file just so it can hash it. Make this more efficient
by hashing from the stream instead, which `hashlib` natively supports.
This also makes osbuild-mpp work with large artifacts in environments
that may not have enough space to temporarily save the data.
'_hawkey.Reldep' object has no attribute 'name' in the version shipped
on RHEL-8. Add code to handle this situation in case it happens.
Default to using named attributes if these are available.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Since the `with_sbom` variable was used only in a single place, we can
simplify the code (and remove one extra line of it) to just directly use
the if condition.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Instead of hard-coding the Python version that the installed
python3-dnf has been built against on the latest Fedora, read the
value from the osbuild-ci container. The container now has the version
written in /osb/libdnf-python-version.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new stage, which allows analyzing the installed packages in a
given filesystem tree using DNF4 API and generating an SPDX v2.3 SBOM
document for it.
One can provide the filesystem tree to be analyzed as a stage input. If
no input is provided, the stage will analyze the filesystem tree of the
current pipeline.
Add tests cases for both usage variants of the stage, as well as the
unit test for stage schema validation.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend osbuild-depsolve-dnf, to return JSON with SPDX SBOM that
corresponds to the depsolved package set, if it has been requested.
For now, only DNF4 is supported.
Cover the new functionality with unit test.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This will allow validating request arguments in the solver method in a
different way for dnf4 and dnf5 and raising an exception if needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add implementation of standard-agnostic model for SBOM, and simple SPDX
v2.3 model. Also add convenience functions for converting DNF4 package
set to the standard-agnostic model and for converting it to SPDX model.
Cover the functionality with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
'dnf' Python package can't be installed using pip in the tox
environment. In order to test the code which uses it, we need to use the
system version. Our testing environment uses Fedora as the system,
therefore we can reasonably use the system version of 'dnf' only with
Python version which is on Fedora.
Enable site packages in tox for Python 3.12 when testing osbuild
internals.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
It's sometimes useful to set up a loop device for an already formatted
disk/filesystem image to derive new artifacts from it. In that case, we
want to make sure it's impossible to modify its contents in any way in
that process, both for our own purposes and for other stages operating
on it.
Notably, mounting some filesystems read-only still seem to touch the
disk (like XFS).
The existing jsoncomm is a work of beautiy. For very big arguments
however the used `SOCK_SEQPACKET` hits the limitations of the
kernel network buffer size (see also [0]). This lead to various
workarounds in #824,#1331,#1836 where parts of the request are
encoded as part of the json method call and parts are done via
a side-channel via fd-passing.
This commit changes the code so that the fd channel is automatically
and transparently created and the workarounds are removed. A test
is added that ensures that very big messages can be passed.
[0] https://github.com/osbuild/osbuild/pull/1833
The code currently does not support btrfs subvolumes that are not
directly under the root directory. This commit fixes this by adding
`-p` to `btrfs subvolume create` and adding an integration test.
Closes: https://github.com/osbuild/osbuild/issues/1882
We need to pass loopback devices for these properties, but the schema
says that there will be a `path` property, so osbuild complains.
osbuild is right of course, but this definitely *did* work in an earlier
version, so something changed. Ideally, we'd narrow down here what
happened exactly, but at the same time this approach of just making the
property more generic matches what's done in e.g. the `zipl.inst` stage
where we also use a loopback device.
For reference, this is where we use this stage:
ba45b296ec/src/osbuild-manifests/platform.qemu.ipp.yaml (L100-L119)
The current error message when an export is not found could be
improved by printing what exports are actually availalble to make
it easier for the user to e.g. spot typos.
Adds a new stage that calls update-ca-trust tool with extract argument
to extract CA certificates. It is expected that one or more CAs are
placed in the /etc/pki/ca-trust/source/anchors directory in PEM format.
Filenames do not matter but must be unique enough. See the
update-ca-trust man page for more details on what it does.
Current osbuild will always print some non output even
when run with `--monitor=JSONSeqMonitor` because of the
unconditional `print/sys.stdout.write()` in `main_cli.py`.
This commit adds a new `-q` option to silence this so that something
like osbuild-composer can run `osbuild -q --monitor=JSONSeqMonitor`
to get pure json-seq output during the build.
The use-case is to run `osbuild --monitor-fd` from e.g. bib and
osbuild-composer so that we get pure json from the monitor-fd
and anything that goes on std{out,err} can be logged as it is
most likely error output.
Quick check to see if checkpointing "build" helps with the
runtime. Note that the cache size is already 20GB, I doubled
it for good measure but we probably can go back to 20, just
want to make sure this is not the bottleneck.
Closes: https://github.com/osbuild/osbuild/issues/1874
- Add an extra call to `/bin/false` and explicitly set the `check`
argument for both `run()` calls.
- Compare full call_args_list. This checks that all the options are as
expected, that the `check` argument is set properly, and that the full
order of all the calls is as expected, including the chroot path.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
For consistency, use subprocess.run() with check=True for the calls that
were previously using subprocess.check_call().
Update the affected tests to match.
Add a test for the chroot context that mocks subprocess.run() and
subprocess.check_call(). The test verifies that the functions are
called the expected number of times with the expected command (first
arg).
If one of the chroot mounts fails to unmount, keep iterating so that we
don't stop and continue to unmount the rest.
Print an error message with the failed mounts, but don't fail the build.
Since failing to unmount doesn't fail the exiting of the context, and
the context itself doesn't know what will be running in the chroot,
do a lazy unmount.
This commit keeps track of individual errors as curl will only report
the last download operation success/failure via it's exit code.
There is "--fail-early" but the downside of that is that abort all in
progress downloads too.
