Add new implementation of the manifest tests, which goal is to ensure
that the osbuild behavior didn't change. This is ensured by comparing
image-info report produced for image artifact built using older
(known-to-be-good) osbuild version and the latest osbuild version
(potentially from a PR).
Previously, we used the osbuild/manifest-db repository, which contained
pre-generated manifests with their corresponding image-info report.
Unfortunately, this setup prooved to be cumbersome to maintain and keep
updated.
We are already building images for known manifests in the osbuild/images
repository. These are then uploaded to AWS S3 cache. The images are
built with a pinned osbuild version, which will be always older than the
one that we would be using for image build in osbuild PR.
So the intention of this new script is to take advantage of the
osbuild/images S3 cache. As part of the test case (for a specific distro
/ arch / image_type / config):
- download the manifest from S3
- download the image artifact, built from the manifest, from S3
- generate image-info report for the downloaded image
- rebuild the downloaded manifest using current version of osbuild
- generate image-info report for the rebuilt image
- compare the two image-info reports. If there is no difference, the
test case PASS, otherwise it will FAIL.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit fixes a race/threading issue with the way the monitor
works. The osbuild monitor can be called from multiple threads,
e.g. in buildroot.py:run() monitor.log() is called but also
in host.py:_stdout_ready(). This can lead to out-of-order writes
when many messages need to be processed.
We did not notice this so far because we were lucky and also
log was just used for information. But now it is used to transmit
the jsonseq data which means out-of-order communication results
in broken json.
Closes: https://github.com/osbuild/image-builder-cli/issues/110
The CentOS Stream 9 repository metadata contains modules; these are
necessary for testing modularity depsolving.
Note that the filelists metadata is kept empty to keep repository size
down.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Add mpp files to create manifests that will embed the container (that
contains /dev/null) in an OS tree and compose an ostree commit from it.
This used to cause the rpm-ostree compose command to fail with:
error: Writing commit: While writing rootfs to mtree: Not a regular file or symlink: null
This commit adds error reporting from source download errors
to the monitor. It reuses the `BuildResult` for symmetry but
we probably want to refactor this a bit to make source handling
a bit more similar to stages.
In order to avoid having to rely on the output of `osbuild --json`
when using `--progress=JSONSeqMonitor` the monitor needs to include
the `osbuild.pipeline.BuildResult` for each individual stage.
This commit adds those to the montior.
This commit just ran:
```
find -iname "*.json" -print -exec sh -c 'jq --indent 2 . {} | sponge {}' \;
```
to ensure that the new test json data from librepo is in the format
the linter expects.
Most modules do not support both schema versions. This is masked by
module type code in get_schema() in most cases, but really should not be
tested. This skips running check_moduleinfo if the module doesn't
support the version. eg. org.osbuild.librepo only supports v2.
This helps prevent testing against invalid manifest data. It runs on the
source's manifest data, using the highest schema version parsed for the
source.
These tests test downloading packages using mirrorlist:
* Using a single mirrorlist repo
* Using two mirrorlist repos
* A bad checksum (or file) from the download which is expected to fail
* Two mirrorlist paths with the first one returning a 404, expected to
succeed with the 2nd mirrorlist path.
NOTE: The metalink xml file points to a repomd.xml file which does not
exist. In practice librepo doesn't download this, it only uses the url
to construct the mirror urls, so it is not present.
These tests test downloading packages using metalink:
* Using a single metalink repo
* Using two metalink repos
* A bad checksum (or file) from the download which is expected to fail
* Two metalink paths with the first one returning a 404, expected to
succeed with the 2nd metalink path.
It turned out that in many cases, stages need to join two absolute
paths, the pipeline tree path and the path on a booted system. However,
the standard `os.path.join()` function can't handle such situation as
just prepending the root to the subsequent paths.
Add a new helper function, which is able to join any paths together,
regardless if any of them is absolute or not. If the root is not
absolute, the result will be made absolute to the filesystem root `/`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This adds a new `org.osbuild.coreos.live-artifacts.mono` stage to build
CoreOS Live ISO/PXE artifacts. The code is heavily based on the
`cmd-buildextend-live` script from coreos-assembler [1], but a lot of
things had to be adapted:
- the stage is provided the deployed oscontainer tree, metal, and
metal4k images as inputs
- we use chroot instead of supermin to execute some commands in the
context of the target oscontainer
- a bunch of calls that were wrapped by libguestfs for us (e.g.
mkfs.vfat, mksquashfs), we now have to call ourselves; to retain
maximum compatibility, we ensured that we still effectively use the
same args that libguestfs passed
And various other minor adjustments.
Of course, this is not really in line with the OSBuild philosophy
of having smaller-scoped stages. We have labeled this with a .mono
suffix to denote it is monolithic, similar to the existing
`org.osbuild.bootiso.mono` stage today.
Eventually we may be able to break this stage down if we find it worth
the effort. Alternatively the need for it may go away as we align more
with Image Mode.
[1] 43a9c80e1f/src/cmd-buildextend-live
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
Co-authored-by: Renata Ravanelli <renata.ravanelli@gmail.com>
There have been a lot of changes to the CoreOS definitions in [1].
Let's update the test manifest here to more closely match what is
running in the field there.
[1] dcd60cfe01/src/osbuild-manifests
The org.osbuild.container-deploy stage uses podman. Including it
in the build here will allow that stage to be used with this
pipeline as the buildroot.
Include a workaround here for what I consider to be a bug [1] in that
`podman` will create `/etc/containers/networks` on first run if it
doesn't exist. That dir should just be created by an RPM. If we
don't include this workaround then the stage will fail when `podman`
attempts the `mkdir` because `/etc/containers` is mounted in from
the buildroot readonly.
[1] https://github.com/containers/common/pull/2265
Add DNF5 version of the `sbom.spdx` stage, which generates an SPDX SBOM
document from a given FS tree using libdnf5 API.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit adds a tiny unit test for the new `shlex` based
os-release parsing and tweaks the error message in a small
and non-functional way (just because it's slightly nicer
for a user). The test checks for three keys NAME which is
quoted with `"`, ID which is not quoted and OSTREE_VERSION
which is quoted with `'`.
- Replaced string stripping with `shlex.split()` to properly
handle values in the os-release file;
- This ensures cleaner and more accurate key-value assignments,
follwing a broader set of shell-like parsing rules;
- Add os-release file for Fedora CoreOS 40 for testing.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
Add functions for transforming package sets depsolved using libdnf5 to
the SBOM standard-agnostic model. Cover the function with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update the fedora manifest template to F41. This is a preparation for
testing a new `org.osbuild.dnf5.sbom.spdx` stage. In addition to that,
F39 is already EOL and unsupported, so moving to a newer version is
desirable.
Regenerate all testing manifests.
Notes about specific changes:
- Remove `pcmciautils` package from ostree manifest, because it has
been deprecated and it is not available on F41 any more.
- Add `python3-dnf` package to the Fedora build pipeline. The reason
is that DNF5 is installed by default since F41 and the module is
needed to test e.g. `org.osbuild.dnf4.sbom.spdx` or
`org.osbuild.dnf4.mark` stages.
- Add `python3-libdnf5` package to the Fedora build pipeline. This is a
preparation for testing a new `org.osbuild.dnf5.sbom.spdx` stage.
- Install `pam` in `authselect` stage test manifest, because otherwise
authselect fails on the nonexistence of /etc/pam.d.
- Adjust the `dnf4.mark` test case to also accept `dnf5` as the marked
package, because on F41, `dnf` is a virtual provide of it.
- Adjust stage tests and their diffs as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Since 9071cd0abb the commands in the
org.osbuild.users stage are run using the osbuild.util.chroot.Chroot
context manager. This includes mounting /proc into the chroot.
The `usermod` command now fails with
usermod: user root is currently used by process N
when modifying any property of the root user that requires the user to
not be logged in (like the home directory).
Change the home directory of the root user in the stage test to check
for this issue and catch regressions.
Fix:
assemblers/org.osbuild.qemu:310:36: E0606: Possibly using variable 'prep_type' before assignment (possibly-used-before-assignment)
inputs/org.osbuild.tree:85:15: E0606: Possibly using variable 'path' before assignment (possibly-used-before-assignment)
stages/org.osbuild.sfdisk:58:36: E0606: Possibly using variable 'prep_type' before assignment (possibly-used-before-assignment)
stages/org.osbuild.systemd.unit:23:16: E0606: Possibly using variable 'unit_dropins_dir' before assignment (possibly-used-before-assignment)
test/mod/test_meta.py:219:29: E0606: Possibly using variable 'schema_part' before assignment (possibly-used-before-assignment)
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
New fucntion returns tuple of 'root' and relative 'file path', which could be
useful in contexts, where knowing 'root' is required, for example setting
selinux labels.
Michael Vogt pointed out that testcases start failing when we describe
the new format. Let's add a test case and fix the describe to include
the metadata.
Metadata is freeform in the `Manifest` instance but it is stored on it
during loading (at which time its properties are validated) and returned
as-is on describe.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Allows a new object under the top-level key `metadata` which contains a
`generators` property. This property is a list of all generators involved
in the creation of a manifest.
Each generator can add its name and version to this list.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
We currently use the absolute path of these binaries in the
helper. This has some advantages but given that we control the
inputs for PATH in general it seems unnecessary.
We are also slightly inconsistent about this in the codebase but
favor the non absolute path version. A quick count:
```
$ git grep '"chroot"'|wc -l
13
$ git grep '"/usr/sbin/chroot"'|grep -v test_|wc -l
8
```
for `mount` and `umount` it seems this is the only place that uses
the absolute path.
It's not an important change but it has the nice property that it
allows us to use e.g. `testutil.mock_command()` in our tests and
it would be nice to be consistent.
This commit moves the joining of path fragements from f-strings
to pathlib and simplifies some of the map/filter/lambda expressions
into more standard list comprehensions.
Add a new stage, which allows analyzing the installed packages in a
given filesystem tree using DNF4 API and generating an SPDX v2.3 SBOM
document for it.
One can provide the filesystem tree to be analyzed as a stage input. If
no input is provided, the stage will analyze the filesystem tree of the
current pipeline.
Add tests cases for both usage variants of the stage, as well as the
unit test for stage schema validation.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add implementation of standard-agnostic model for SBOM, and simple SPDX
v2.3 model. Also add convenience functions for converting DNF4 package
set to the standard-agnostic model and for converting it to SPDX model.
Cover the functionality with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The existing jsoncomm is a work of beautiy. For very big arguments
however the used `SOCK_SEQPACKET` hits the limitations of the
kernel network buffer size (see also [0]). This lead to various
workarounds in #824,#1331,#1836 where parts of the request are
encoded as part of the json method call and parts are done via
a side-channel via fd-passing.
This commit changes the code so that the fd channel is automatically
and transparently created and the workarounds are removed. A test
is added that ensures that very big messages can be passed.
[0] https://github.com/osbuild/osbuild/pull/1833
