Introduce a new class `SpdxLicenseExpressionCreator`, responsible for
converting license texts extracted from packages, into an SPDX-compliant
license expressions. If the `license_expression` Python package is
available on the system, it is used to determine the license text
extracted from a package is a valid SPDX license expression. If it is,
it's returned as is back to the caller. If it is not, or of the package
is not available on the system, the license text is wrapped in a
`ExtractedLicensingInfo` instance.
The `SpdxLicenseExpressionCreator` object keeps track of all generated
`ExtractedLicensingInfo` instances and de-duplicates them based on the
license text. This means that if two packages use the same
SPDX-non-compliant license text, they will be wrapped by an
`ExtractedLicensingInfo` instance with the same `LicenseRef-` ID.
The reason for fallback when `license_expression` package is not
available is that it is not available on RHEL and CentOS Stream. This
implementation allows us to ship the functionality in RHEL and
optionally enabling it by installing `license_expression` from a 3rd
party repository. In any case, the generated SBOM document will always
contain valid SPDX license expressions.
Extend unit tests to cover the newly added functionality.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
FIXUP: sbom/spdx: use compliant license expressions
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the SPDX v2 model to support referencing extracted licensing
information, which is either not in the SPDX license list or can't be
expressed by the SPDX-compliant license expression.
Cover the new functionality by unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit limits the output in the json pipeline to a "reasonable"
length. We ran into issues (e.g. [0]) from a combination of a stage
that produce tons of output (dracut, ~256 kb, see issue#1976) and
the consumer ("images" osbuild/monitor.go) that used a golang scanner
with a max default buffer of 64kb before erroring. So limit it
here.
The stage result from via json is mostly for information and any error
will most likely at the end. Plus consumers can collect the individual
log lines on their own if desired via the "log()" messages that are
stream in "real-time" with the added benefit that e.g. timestamps
can be added to the logs etc.
[0] https://issues.redhat.com/browse/RHEL-77988
In case the workdir is not provided to the script explicitly as an
argument, the script will use a temporary directory under /var/tmp as
its workdir. In such case, the workdir will be deleted on exit. This
should mitigate potentially confusing behavior when executing the script
multiple times with different arguments, while never specifying the
workdir.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add new implementation of the manifest tests, which goal is to ensure
that the osbuild behavior didn't change. This is ensured by comparing
image-info report produced for image artifact built using older
(known-to-be-good) osbuild version and the latest osbuild version
(potentially from a PR).
Previously, we used the osbuild/manifest-db repository, which contained
pre-generated manifests with their corresponding image-info report.
Unfortunately, this setup prooved to be cumbersome to maintain and keep
updated.
We are already building images for known manifests in the osbuild/images
repository. These are then uploaded to AWS S3 cache. The images are
built with a pinned osbuild version, which will be always older than the
one that we would be using for image build in osbuild PR.
So the intention of this new script is to take advantage of the
osbuild/images S3 cache. As part of the test case (for a specific distro
/ arch / image_type / config):
- download the manifest from S3
- download the image artifact, built from the manifest, from S3
- generate image-info report for the downloaded image
- rebuild the downloaded manifest using current version of osbuild
- generate image-info report for the rebuilt image
- compare the two image-info reports. If there is no difference, the
test case PASS, otherwise it will FAIL.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit fixes a race/threading issue with the way the monitor
works. The osbuild monitor can be called from multiple threads,
e.g. in buildroot.py:run() monitor.log() is called but also
in host.py:_stdout_ready(). This can lead to out-of-order writes
when many messages need to be processed.
We did not notice this so far because we were lucky and also
log was just used for information. But now it is used to transmit
the jsonseq data which means out-of-order communication results
in broken json.
Closes: https://github.com/osbuild/image-builder-cli/issues/110
The CentOS Stream 9 repository metadata contains modules; these are
necessary for testing modularity depsolving.
Note that the filelists metadata is kept empty to keep repository size
down.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Add mpp files to create manifests that will embed the container (that
contains /dev/null) in an OS tree and compose an ostree commit from it.
This used to cause the rpm-ostree compose command to fail with:
error: Writing commit: While writing rootfs to mtree: Not a regular file or symlink: null
This commit adds error reporting from source download errors
to the monitor. It reuses the `BuildResult` for symmetry but
we probably want to refactor this a bit to make source handling
a bit more similar to stages.
In order to avoid having to rely on the output of `osbuild --json`
when using `--progress=JSONSeqMonitor` the monitor needs to include
the `osbuild.pipeline.BuildResult` for each individual stage.
This commit adds those to the montior.
This commit just ran:
```
find -iname "*.json" -print -exec sh -c 'jq --indent 2 . {} | sponge {}' \;
```
to ensure that the new test json data from librepo is in the format
the linter expects.
Most modules do not support both schema versions. This is masked by
module type code in get_schema() in most cases, but really should not be
tested. This skips running check_moduleinfo if the module doesn't
support the version. eg. org.osbuild.librepo only supports v2.
This helps prevent testing against invalid manifest data. It runs on the
source's manifest data, using the highest schema version parsed for the
source.
These tests test downloading packages using mirrorlist:
* Using a single mirrorlist repo
* Using two mirrorlist repos
* A bad checksum (or file) from the download which is expected to fail
* Two mirrorlist paths with the first one returning a 404, expected to
succeed with the 2nd mirrorlist path.
NOTE: The metalink xml file points to a repomd.xml file which does not
exist. In practice librepo doesn't download this, it only uses the url
to construct the mirror urls, so it is not present.
These tests test downloading packages using metalink:
* Using a single metalink repo
* Using two metalink repos
* A bad checksum (or file) from the download which is expected to fail
* Two metalink paths with the first one returning a 404, expected to
succeed with the 2nd metalink path.
It turned out that in many cases, stages need to join two absolute
paths, the pipeline tree path and the path on a booted system. However,
the standard `os.path.join()` function can't handle such situation as
just prepending the root to the subsequent paths.
Add a new helper function, which is able to join any paths together,
regardless if any of them is absolute or not. If the root is not
absolute, the result will be made absolute to the filesystem root `/`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This adds a new `org.osbuild.coreos.live-artifacts.mono` stage to build
CoreOS Live ISO/PXE artifacts. The code is heavily based on the
`cmd-buildextend-live` script from coreos-assembler [1], but a lot of
things had to be adapted:
- the stage is provided the deployed oscontainer tree, metal, and
metal4k images as inputs
- we use chroot instead of supermin to execute some commands in the
context of the target oscontainer
- a bunch of calls that were wrapped by libguestfs for us (e.g.
mkfs.vfat, mksquashfs), we now have to call ourselves; to retain
maximum compatibility, we ensured that we still effectively use the
same args that libguestfs passed
And various other minor adjustments.
Of course, this is not really in line with the OSBuild philosophy
of having smaller-scoped stages. We have labeled this with a .mono
suffix to denote it is monolithic, similar to the existing
`org.osbuild.bootiso.mono` stage today.
Eventually we may be able to break this stage down if we find it worth
the effort. Alternatively the need for it may go away as we align more
with Image Mode.
[1] 43a9c80e1f/src/cmd-buildextend-live
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
Co-authored-by: Renata Ravanelli <renata.ravanelli@gmail.com>
There have been a lot of changes to the CoreOS definitions in [1].
Let's update the test manifest here to more closely match what is
running in the field there.
[1] dcd60cfe01/src/osbuild-manifests
The org.osbuild.container-deploy stage uses podman. Including it
in the build here will allow that stage to be used with this
pipeline as the buildroot.
Include a workaround here for what I consider to be a bug [1] in that
`podman` will create `/etc/containers/networks` on first run if it
doesn't exist. That dir should just be created by an RPM. If we
don't include this workaround then the stage will fail when `podman`
attempts the `mkdir` because `/etc/containers` is mounted in from
the buildroot readonly.
[1] https://github.com/containers/common/pull/2265
Add DNF5 version of the `sbom.spdx` stage, which generates an SPDX SBOM
document from a given FS tree using libdnf5 API.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit adds a tiny unit test for the new `shlex` based
os-release parsing and tweaks the error message in a small
and non-functional way (just because it's slightly nicer
for a user). The test checks for three keys NAME which is
quoted with `"`, ID which is not quoted and OSTREE_VERSION
which is quoted with `'`.
- Replaced string stripping with `shlex.split()` to properly
handle values in the os-release file;
- This ensures cleaner and more accurate key-value assignments,
follwing a broader set of shell-like parsing rules;
- Add os-release file for Fedora CoreOS 40 for testing.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
Add functions for transforming package sets depsolved using libdnf5 to
the SBOM standard-agnostic model. Cover the function with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update the fedora manifest template to F41. This is a preparation for
testing a new `org.osbuild.dnf5.sbom.spdx` stage. In addition to that,
F39 is already EOL and unsupported, so moving to a newer version is
desirable.
Regenerate all testing manifests.
Notes about specific changes:
- Remove `pcmciautils` package from ostree manifest, because it has
been deprecated and it is not available on F41 any more.
- Add `python3-dnf` package to the Fedora build pipeline. The reason
is that DNF5 is installed by default since F41 and the module is
needed to test e.g. `org.osbuild.dnf4.sbom.spdx` or
`org.osbuild.dnf4.mark` stages.
- Add `python3-libdnf5` package to the Fedora build pipeline. This is a
preparation for testing a new `org.osbuild.dnf5.sbom.spdx` stage.
- Install `pam` in `authselect` stage test manifest, because otherwise
authselect fails on the nonexistence of /etc/pam.d.
- Adjust the `dnf4.mark` test case to also accept `dnf5` as the marked
package, because on F41, `dnf` is a virtual provide of it.
- Adjust stage tests and their diffs as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Since 9071cd0abb the commands in the
org.osbuild.users stage are run using the osbuild.util.chroot.Chroot
context manager. This includes mounting /proc into the chroot.
The `usermod` command now fails with
usermod: user root is currently used by process N
when modifying any property of the root user that requires the user to
not be logged in (like the home directory).
Change the home directory of the root user in the stage test to check
for this issue and catch regressions.
Fix:
assemblers/org.osbuild.qemu:310:36: E0606: Possibly using variable 'prep_type' before assignment (possibly-used-before-assignment)
inputs/org.osbuild.tree:85:15: E0606: Possibly using variable 'path' before assignment (possibly-used-before-assignment)
stages/org.osbuild.sfdisk:58:36: E0606: Possibly using variable 'prep_type' before assignment (possibly-used-before-assignment)
stages/org.osbuild.systemd.unit:23:16: E0606: Possibly using variable 'unit_dropins_dir' before assignment (possibly-used-before-assignment)
test/mod/test_meta.py:219:29: E0606: Possibly using variable 'schema_part' before assignment (possibly-used-before-assignment)
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
New fucntion returns tuple of 'root' and relative 'file path', which could be
useful in contexts, where knowing 'root' is required, for example setting
selinux labels.
Michael Vogt pointed out that testcases start failing when we describe
the new format. Let's add a test case and fix the describe to include
the metadata.
Metadata is freeform in the `Manifest` instance but it is stored on it
during loading (at which time its properties are validated) and returned
as-is on describe.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Allows a new object under the top-level key `metadata` which contains a
`generators` property. This property is a list of all generators involved
in the creation of a manifest.
Each generator can add its name and version to this list.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
