- Add debian-forge-apparmor package with AppArmor stage support
- Create example AppArmor stage (org.osbuild.apparmor)
- Update workflow to build 9 packages total
- Add AppArmor manifest example for Debian Atomic
- Update todo with complete package structure
Although duration of each stage could be calculated from the start and
end timestamps, it is more convenient to have it directly in the log
entry. This way we can avoid calculating it in the client code.
When fully cached manifest is used, the duration used to print an
incorrect value of 55 years:
python3 -m osbuild --libdir . ./test/data/manifests/fedora-boot.json
⏱ Duration: 1753191578s
This patch fixes the duration calculation to use the correct timestamp
from the manifest by using monotonic timer instead. Additionally, it
prints nothing when there was no module executed. Finally, it improves
the formatting of the duration output.
Update test manifests by running tools/update-test-manifests.
This bumps the snapshot date for all CentOS Stream 9 test manifests,
which currently is only the authconfig and dnf4.versionlock.
A couple of CentOS Stream 9 manifests were setting 'el9' as their
module-platform-id, making the depsolver complain.
Fixed to the correct 'platform:el9'.
Add a test that builds the stage test manifest with the versionlock
plugin (test/stages/dnf4.versionlock/b.json) to export the tree and read
the file created by the tree.
The test then chroots into the tree and dnf versionlock add to create a
file using the actual plugin and compares the created file with the one
created by the stage.
This adds support generating a virtualbox vagrant image. It differs from
libvirt by requiring an xml file and a vmdk image.
When the provider is set to libvirt it is required to pass a
`virtualbox` configuration section to this stage which must include the
mac address.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
In some (ununderstood) cases the combination of `--no-single-branch` and
`--depth=1` leads to the revision we want to check out not being
available.
Achilleas suggested to change the command to this instead.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
RPM 6.0 (landed in F43, current rawhide), no longer uses short key ID
for the 'gpg-pubkey' pseudo-package, but to prevent any collision, it
uses full key ID.
This means that we can't consistently just compare the pipeline rpm
stage metadata from the test data with what we get from building the
test manifest. The reason is that we need to keep running the unit test
in upstream and downstream CI pipelines for OSes, which may ship RPM
6.0, but also older RPM version.
Extend the test case to do a special case check for any 'gpg-pubkey'
package in the metadata if the package version length differs between
the test data and data got from building the manifest.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Rename the 'success' case to 'base64_success' as a preparation for
adding test case for other encodings.
The test case was originally embedding the tools/inline-source.py
script. Since the script has been updated, let's update the embedded
date in the test case as well.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the stage to support setting new options:
- Provisioning.UseCloudInit
- Provisioning.Enabled
Extend the stage test to use them and add a simple stage unit test for
the schema.
Related to https://github.com/osbuild/images/issues/1416
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Support setting the `auto_enable_yum_plugins` option in the `rhsm`
section of `rhsm.conf`.
Add a unit test for the stage schema and also adjust the stage test
manifest.
Related to: https://github.com/osbuild/images/issues/1408
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit adds two new helpers:
- util.experimentalflags.get_bool()
- util.experimentalflags.get_string()
similar to what we added in the images library in PR:
https://github.com/osbuild/images/pull/1248
The idea is that we provide experimentalflags for osbuild via
an environment like `OSBUILD_EXPERIMENTAL` and for those we
make no API promises. This will be initially used for better
debug of qemu-user.
Add a stage build test for the boot_root option of the org.osbuild.rpm
stage. The test defined two manifests that both install the
kernel-uki-virt kernel but differ in the value of the boot_root option.
The diff shows the effect of the value on the install location of the
kernel.
Always return License ref IDs as is, if used as package license,
regardless if license_expression package is available. This will prevent
wrapping them again as extracted license info and generating yet another
license ref ID.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Move the call to `pytest.importorskip()` function into a specific test
case that relies on imported modules. This will make test cases in the
same file to be run, even if importing the modules fail.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Introduce a new class `SpdxLicenseExpressionCreator`, responsible for
converting license texts extracted from packages, into an SPDX-compliant
license expressions. If the `license_expression` Python package is
available on the system, it is used to determine the license text
extracted from a package is a valid SPDX license expression. If it is,
it's returned as is back to the caller. If it is not, or of the package
is not available on the system, the license text is wrapped in a
`ExtractedLicensingInfo` instance.
The `SpdxLicenseExpressionCreator` object keeps track of all generated
`ExtractedLicensingInfo` instances and de-duplicates them based on the
license text. This means that if two packages use the same
SPDX-non-compliant license text, they will be wrapped by an
`ExtractedLicensingInfo` instance with the same `LicenseRef-` ID.
The reason for fallback when `license_expression` package is not
available is that it is not available on RHEL and CentOS Stream. This
implementation allows us to ship the functionality in RHEL and
optionally enabling it by installing `license_expression` from a 3rd
party repository. In any case, the generated SBOM document will always
contain valid SPDX license expressions.
Extend unit tests to cover the newly added functionality.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
FIXUP: sbom/spdx: use compliant license expressions
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the SPDX v2 model to support referencing extracted licensing
information, which is either not in the SPDX license list or can't be
expressed by the SPDX-compliant license expression.
Cover the new functionality by unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit limits the output in the json pipeline to a "reasonable"
length. We ran into issues (e.g. [0]) from a combination of a stage
that produce tons of output (dracut, ~256 kb, see issue#1976) and
the consumer ("images" osbuild/monitor.go) that used a golang scanner
with a max default buffer of 64kb before erroring. So limit it
here.
The stage result from via json is mostly for information and any error
will most likely at the end. Plus consumers can collect the individual
log lines on their own if desired via the "log()" messages that are
stream in "real-time" with the added benefit that e.g. timestamps
can be added to the logs etc.
[0] https://issues.redhat.com/browse/RHEL-77988
In case the workdir is not provided to the script explicitly as an
argument, the script will use a temporary directory under /var/tmp as
its workdir. In such case, the workdir will be deleted on exit. This
should mitigate potentially confusing behavior when executing the script
multiple times with different arguments, while never specifying the
workdir.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add new implementation of the manifest tests, which goal is to ensure
that the osbuild behavior didn't change. This is ensured by comparing
image-info report produced for image artifact built using older
(known-to-be-good) osbuild version and the latest osbuild version
(potentially from a PR).
Previously, we used the osbuild/manifest-db repository, which contained
pre-generated manifests with their corresponding image-info report.
Unfortunately, this setup prooved to be cumbersome to maintain and keep
updated.
We are already building images for known manifests in the osbuild/images
repository. These are then uploaded to AWS S3 cache. The images are
built with a pinned osbuild version, which will be always older than the
one that we would be using for image build in osbuild PR.
So the intention of this new script is to take advantage of the
osbuild/images S3 cache. As part of the test case (for a specific distro
/ arch / image_type / config):
- download the manifest from S3
- download the image artifact, built from the manifest, from S3
- generate image-info report for the downloaded image
- rebuild the downloaded manifest using current version of osbuild
- generate image-info report for the rebuilt image
- compare the two image-info reports. If there is no difference, the
test case PASS, otherwise it will FAIL.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit fixes a race/threading issue with the way the monitor
works. The osbuild monitor can be called from multiple threads,
e.g. in buildroot.py:run() monitor.log() is called but also
in host.py:_stdout_ready(). This can lead to out-of-order writes
when many messages need to be processed.
We did not notice this so far because we were lucky and also
log was just used for information. But now it is used to transmit
the jsonseq data which means out-of-order communication results
in broken json.
Closes: https://github.com/osbuild/image-builder-cli/issues/110
The CentOS Stream 9 repository metadata contains modules; these are
necessary for testing modularity depsolving.
Note that the filelists metadata is kept empty to keep repository size
down.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Add mpp files to create manifests that will embed the container (that
contains /dev/null) in an OS tree and compose an ostree commit from it.
This used to cause the rpm-ostree compose command to fail with:
error: Writing commit: While writing rootfs to mtree: Not a regular file or symlink: null
This commit adds error reporting from source download errors
to the monitor. It reuses the `BuildResult` for symmetry but
we probably want to refactor this a bit to make source handling
a bit more similar to stages.
In order to avoid having to rely on the output of `osbuild --json`
when using `--progress=JSONSeqMonitor` the monitor needs to include
the `osbuild.pipeline.BuildResult` for each individual stage.
This commit adds those to the montior.
This commit just ran:
```
find -iname "*.json" -print -exec sh -c 'jq --indent 2 . {} | sponge {}' \;
```
to ensure that the new test json data from librepo is in the format
the linter expects.
Most modules do not support both schema versions. This is masked by
module type code in get_schema() in most cases, but really should not be
tested. This skips running check_moduleinfo if the module doesn't
support the version. eg. org.osbuild.librepo only supports v2.
This helps prevent testing against invalid manifest data. It runs on the
source's manifest data, using the highest schema version parsed for the
source.
These tests test downloading packages using mirrorlist:
* Using a single mirrorlist repo
* Using two mirrorlist repos
* A bad checksum (or file) from the download which is expected to fail
* Two mirrorlist paths with the first one returning a 404, expected to
succeed with the 2nd mirrorlist path.
