In some (ununderstood) cases the combination of `--no-single-branch` and
`--depth=1` leads to the revision we want to check out not being
available.
Achilleas suggested to change the command to this instead.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
RPM 6.0 (landed in F43, current rawhide), no longer uses short key ID
for the 'gpg-pubkey' pseudo-package, but to prevent any collision, it
uses full key ID.
This means that we can't consistently just compare the pipeline rpm
stage metadata from the test data with what we get from building the
test manifest. The reason is that we need to keep running the unit test
in upstream and downstream CI pipelines for OSes, which may ship RPM
6.0, but also older RPM version.
Extend the test case to do a special case check for any 'gpg-pubkey'
package in the metadata if the package version length differs between
the test data and data got from building the manifest.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Rename the 'success' case to 'base64_success' as a preparation for
adding test case for other encodings.
The test case was originally embedding the tools/inline-source.py
script. Since the script has been updated, let's update the embedded
date in the test case as well.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the stage to support setting new options:
- Provisioning.UseCloudInit
- Provisioning.Enabled
Extend the stage test to use them and add a simple stage unit test for
the schema.
Related to https://github.com/osbuild/images/issues/1416
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Support setting the `auto_enable_yum_plugins` option in the `rhsm`
section of `rhsm.conf`.
Add a unit test for the stage schema and also adjust the stage test
manifest.
Related to: https://github.com/osbuild/images/issues/1408
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit adds two new helpers:
- util.experimentalflags.get_bool()
- util.experimentalflags.get_string()
similar to what we added in the images library in PR:
https://github.com/osbuild/images/pull/1248
The idea is that we provide experimentalflags for osbuild via
an environment like `OSBUILD_EXPERIMENTAL` and for those we
make no API promises. This will be initially used for better
debug of qemu-user.
Add a stage build test for the boot_root option of the org.osbuild.rpm
stage. The test defined two manifests that both install the
kernel-uki-virt kernel but differ in the value of the boot_root option.
The diff shows the effect of the value on the install location of the
kernel.
Always return License ref IDs as is, if used as package license,
regardless if license_expression package is available. This will prevent
wrapping them again as extracted license info and generating yet another
license ref ID.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Move the call to `pytest.importorskip()` function into a specific test
case that relies on imported modules. This will make test cases in the
same file to be run, even if importing the modules fail.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Introduce a new class `SpdxLicenseExpressionCreator`, responsible for
converting license texts extracted from packages, into an SPDX-compliant
license expressions. If the `license_expression` Python package is
available on the system, it is used to determine the license text
extracted from a package is a valid SPDX license expression. If it is,
it's returned as is back to the caller. If it is not, or of the package
is not available on the system, the license text is wrapped in a
`ExtractedLicensingInfo` instance.
The `SpdxLicenseExpressionCreator` object keeps track of all generated
`ExtractedLicensingInfo` instances and de-duplicates them based on the
license text. This means that if two packages use the same
SPDX-non-compliant license text, they will be wrapped by an
`ExtractedLicensingInfo` instance with the same `LicenseRef-` ID.
The reason for fallback when `license_expression` package is not
available is that it is not available on RHEL and CentOS Stream. This
implementation allows us to ship the functionality in RHEL and
optionally enabling it by installing `license_expression` from a 3rd
party repository. In any case, the generated SBOM document will always
contain valid SPDX license expressions.
Extend unit tests to cover the newly added functionality.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
FIXUP: sbom/spdx: use compliant license expressions
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the SPDX v2 model to support referencing extracted licensing
information, which is either not in the SPDX license list or can't be
expressed by the SPDX-compliant license expression.
Cover the new functionality by unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit limits the output in the json pipeline to a "reasonable"
length. We ran into issues (e.g. [0]) from a combination of a stage
that produce tons of output (dracut, ~256 kb, see issue#1976) and
the consumer ("images" osbuild/monitor.go) that used a golang scanner
with a max default buffer of 64kb before erroring. So limit it
here.
The stage result from via json is mostly for information and any error
will most likely at the end. Plus consumers can collect the individual
log lines on their own if desired via the "log()" messages that are
stream in "real-time" with the added benefit that e.g. timestamps
can be added to the logs etc.
[0] https://issues.redhat.com/browse/RHEL-77988
In case the workdir is not provided to the script explicitly as an
argument, the script will use a temporary directory under /var/tmp as
its workdir. In such case, the workdir will be deleted on exit. This
should mitigate potentially confusing behavior when executing the script
multiple times with different arguments, while never specifying the
workdir.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add new implementation of the manifest tests, which goal is to ensure
that the osbuild behavior didn't change. This is ensured by comparing
image-info report produced for image artifact built using older
(known-to-be-good) osbuild version and the latest osbuild version
(potentially from a PR).
Previously, we used the osbuild/manifest-db repository, which contained
pre-generated manifests with their corresponding image-info report.
Unfortunately, this setup prooved to be cumbersome to maintain and keep
updated.
We are already building images for known manifests in the osbuild/images
repository. These are then uploaded to AWS S3 cache. The images are
built with a pinned osbuild version, which will be always older than the
one that we would be using for image build in osbuild PR.
So the intention of this new script is to take advantage of the
osbuild/images S3 cache. As part of the test case (for a specific distro
/ arch / image_type / config):
- download the manifest from S3
- download the image artifact, built from the manifest, from S3
- generate image-info report for the downloaded image
- rebuild the downloaded manifest using current version of osbuild
- generate image-info report for the rebuilt image
- compare the two image-info reports. If there is no difference, the
test case PASS, otherwise it will FAIL.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit fixes a race/threading issue with the way the monitor
works. The osbuild monitor can be called from multiple threads,
e.g. in buildroot.py:run() monitor.log() is called but also
in host.py:_stdout_ready(). This can lead to out-of-order writes
when many messages need to be processed.
We did not notice this so far because we were lucky and also
log was just used for information. But now it is used to transmit
the jsonseq data which means out-of-order communication results
in broken json.
Closes: https://github.com/osbuild/image-builder-cli/issues/110
The CentOS Stream 9 repository metadata contains modules; these are
necessary for testing modularity depsolving.
Note that the filelists metadata is kept empty to keep repository size
down.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Add mpp files to create manifests that will embed the container (that
contains /dev/null) in an OS tree and compose an ostree commit from it.
This used to cause the rpm-ostree compose command to fail with:
error: Writing commit: While writing rootfs to mtree: Not a regular file or symlink: null
This commit adds error reporting from source download errors
to the monitor. It reuses the `BuildResult` for symmetry but
we probably want to refactor this a bit to make source handling
a bit more similar to stages.
In order to avoid having to rely on the output of `osbuild --json`
when using `--progress=JSONSeqMonitor` the monitor needs to include
the `osbuild.pipeline.BuildResult` for each individual stage.
This commit adds those to the montior.
This commit just ran:
```
find -iname "*.json" -print -exec sh -c 'jq --indent 2 . {} | sponge {}' \;
```
to ensure that the new test json data from librepo is in the format
the linter expects.
Most modules do not support both schema versions. This is masked by
module type code in get_schema() in most cases, but really should not be
tested. This skips running check_moduleinfo if the module doesn't
support the version. eg. org.osbuild.librepo only supports v2.
This helps prevent testing against invalid manifest data. It runs on the
source's manifest data, using the highest schema version parsed for the
source.
These tests test downloading packages using mirrorlist:
* Using a single mirrorlist repo
* Using two mirrorlist repos
* A bad checksum (or file) from the download which is expected to fail
* Two mirrorlist paths with the first one returning a 404, expected to
succeed with the 2nd mirrorlist path.
NOTE: The metalink xml file points to a repomd.xml file which does not
exist. In practice librepo doesn't download this, it only uses the url
to construct the mirror urls, so it is not present.
These tests test downloading packages using metalink:
* Using a single metalink repo
* Using two metalink repos
* A bad checksum (or file) from the download which is expected to fail
* Two metalink paths with the first one returning a 404, expected to
succeed with the 2nd metalink path.
It turned out that in many cases, stages need to join two absolute
paths, the pipeline tree path and the path on a booted system. However,
the standard `os.path.join()` function can't handle such situation as
just prepending the root to the subsequent paths.
Add a new helper function, which is able to join any paths together,
regardless if any of them is absolute or not. If the root is not
absolute, the result will be made absolute to the filesystem root `/`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This adds a new `org.osbuild.coreos.live-artifacts.mono` stage to build
CoreOS Live ISO/PXE artifacts. The code is heavily based on the
`cmd-buildextend-live` script from coreos-assembler [1], but a lot of
things had to be adapted:
- the stage is provided the deployed oscontainer tree, metal, and
metal4k images as inputs
- we use chroot instead of supermin to execute some commands in the
context of the target oscontainer
- a bunch of calls that were wrapped by libguestfs for us (e.g.
mkfs.vfat, mksquashfs), we now have to call ourselves; to retain
maximum compatibility, we ensured that we still effectively use the
same args that libguestfs passed
And various other minor adjustments.
Of course, this is not really in line with the OSBuild philosophy
of having smaller-scoped stages. We have labeled this with a .mono
suffix to denote it is monolithic, similar to the existing
`org.osbuild.bootiso.mono` stage today.
Eventually we may be able to break this stage down if we find it worth
the effort. Alternatively the need for it may go away as we align more
with Image Mode.
[1] 43a9c80e1f/src/cmd-buildextend-live
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
Co-authored-by: Renata Ravanelli <renata.ravanelli@gmail.com>
There have been a lot of changes to the CoreOS definitions in [1].
Let's update the test manifest here to more closely match what is
running in the field there.
[1] dcd60cfe01/src/osbuild-manifests
The org.osbuild.container-deploy stage uses podman. Including it
in the build here will allow that stage to be used with this
pipeline as the buildroot.
Include a workaround here for what I consider to be a bug [1] in that
`podman` will create `/etc/containers/networks` on first run if it
doesn't exist. That dir should just be created by an RPM. If we
don't include this workaround then the stage will fail when `podman`
attempts the `mkdir` because `/etc/containers` is mounted in from
the buildroot readonly.
[1] https://github.com/containers/common/pull/2265
Add DNF5 version of the `sbom.spdx` stage, which generates an SPDX SBOM
document from a given FS tree using libdnf5 API.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit adds a tiny unit test for the new `shlex` based
os-release parsing and tweaks the error message in a small
and non-functional way (just because it's slightly nicer
for a user). The test checks for three keys NAME which is
quoted with `"`, ID which is not quoted and OSTREE_VERSION
which is quoted with `'`.
- Replaced string stripping with `shlex.split()` to properly
handle values in the os-release file;
- This ensures cleaner and more accurate key-value assignments,
follwing a broader set of shell-like parsing rules;
- Add os-release file for Fedora CoreOS 40 for testing.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
