RPM 6.0 (landed in F43, current rawhide), no longer uses short key ID
for the 'gpg-pubkey' pseudo-package, but to prevent any collision, it
uses full key ID.
This means that we can't consistently just compare the pipeline rpm
stage metadata from the test data with what we get from building the
test manifest. The reason is that we need to keep running the unit test
in upstream and downstream CI pipelines for OSes, which may ship RPM
6.0, but also older RPM version.
Extend the test case to do a special case check for any 'gpg-pubkey'
package in the metadata if the package version length differs between
the test data and data got from building the manifest.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This helps prevent testing against invalid manifest data. It runs on the
source's manifest data, using the highest schema version parsed for the
source.
Update the fedora manifest template to F41. This is a preparation for
testing a new `org.osbuild.dnf5.sbom.spdx` stage. In addition to that,
F39 is already EOL and unsupported, so moving to a newer version is
desirable.
Regenerate all testing manifests.
Notes about specific changes:
- Remove `pcmciautils` package from ostree manifest, because it has
been deprecated and it is not available on F41 any more.
- Add `python3-dnf` package to the Fedora build pipeline. The reason
is that DNF5 is installed by default since F41 and the module is
needed to test e.g. `org.osbuild.dnf4.sbom.spdx` or
`org.osbuild.dnf4.mark` stages.
- Add `python3-libdnf5` package to the Fedora build pipeline. This is a
preparation for testing a new `org.osbuild.dnf5.sbom.spdx` stage.
- Install `pam` in `authselect` stage test manifest, because otherwise
authselect fails on the nonexistence of /etc/pam.d.
- Adjust the `dnf4.mark` test case to also accept `dnf5` as the marked
package, because on F41, `dnf` is a virtual provide of it.
- Adjust stage tests and their diffs as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The current error message when an export is not found could be
improved by printing what exports are actually availalble to make
it easier for the user to e.g. spot typos.
Current osbuild will always print some non output even
when run with `--monitor=JSONSeqMonitor` because of the
unconditional `print/sys.stdout.write()` in `main_cli.py`.
This commit adds a new `-q` option to silence this so that something
like osbuild-composer can run `osbuild -q --monitor=JSONSeqMonitor`
to get pure json-seq output during the build.
The use-case is to run `osbuild --monitor-fd` from e.g. bib and
osbuild-composer so that we get pure json from the monitor-fd
and anything that goes on std{out,err} can be logged as it is
most likely error output.
Quick check to see if checkpointing "build" helps with the
runtime. Note that the cache size is already 20GB, I doubled
it for good measure but we probably can go back to 20, just
want to make sure this is not the bottleneck.
Closes: https://github.com/osbuild/osbuild/issues/1874
The libdir is passed down for sources but it is never used in
any of our sources. As this is confusing and we want to eventually
support multiple libdirs remove this code.
It looks like the libdir for soruces was added a long time ago in 8423da3
but there is no indication if/how it is/was supposed to get used and
AFACT from going over the git history it was very used.
SourceService:dispatch() never sends "libdir" to the actual sources,
so it is not an even technically an API break.
Extend the dracut stage test case with checks for error / warning
messages complaining about unsupported / incorrect runtime environment.
Messages such as:
```
/dev/fd/63: No such file or directory
```
or
```
/proc/ is not mounted. This is not a supported mode of operation.
Please fix your invocation environment to mount /proc/ and /sys/
properly. Proceeding anyway. Your mileage may vary.
```
The stage will be fixed in the next commit.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When subscription-manager DNF plugins are enabled (e.g. on RHEL), they
produce messages to the stdout on any DNF command execution. E.g.
"Updating Subscription Management repositories.".
Disable all plugins when inspecting package markings so prevent them
from modifying the output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When I rolled back from using 'dnf4', to check package markings, to
using 'dnf', I didn't verify the test case on Fedora Rawhide with DNF5.
It turns out that the strings reported by DNF5 differ and make the test
case fail. This time I tested the change on Fedora Rawhide with DNF5 and
it works.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The test case still fails on RHEL-10.0 Beta, even when not using dnf5,
with:
```
for line in r.stdout.splitlines():
> package, mark = line.strip().split(",")
E ValueError: not enough values to unpack (expected 2, got 1)
```
Make debugging of failures like this easier by printing the line when
the issue happens.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Let's revert to using plain 'dnf', add an explicit newline in the query
format and skip empty lines when processing the output. This makes the
test case compatible with all DNF versions, even with dnf5 once this
issue gets fixed.
The previous approach didn't work on c9s / el9, because there is no
'/usr/bin/dnf4 -> dnf-3' symlink.
Also see:
https://github.com/osbuild/osbuild/actions/runs/10136827918/job/28026181824
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Similar to what was explained in 2e6d49fbe this commit updates
the l2hash in test_assemblers to the new values from fc40 images.
Sadly it is hard to derive them from first principles (see the
other commit) and given that this is legacy code it is probably
fine this way.
The `test_assembler.py` hardcods some filesystem and partition
UUIDs. This leads to hard to diagnose test failures when the
test is run in parallel. The btrfs and xfs filesystem drivers
will see the same uuid for multi created images and error sometimes with
someting like:
```
Mar 06 10:22:54 top kernel: BTRFS error: device /dev/loop104 belongs to fsid aff010e9-df95-4f81-be6b-e22317251033, and the fs is already mounted, scanned by mount (123856)
```
Its a race that only happens when two images are checked at the
same time.
This commit fixes the issue by just using a randomized UUID in
the test_assemblers.py. It also re-enables running the test in
parallel (which make it run a lot faster, from 34min to 14min).
Older versions of RHEL/Centos do not have `mkfs.fat -g` yet so
this test will fail. Detect this and skip the test if mkfs.fat
is too old (see 7af2f1a for the original commit).
When constructing a minimal environment for osbuild the exported
dirs lacked "schemas" so the test_exports.py test failed on RHEL8.
This commit adds it (and also "assemblers" for good measure). With
that the test will pass.
Older version of `lsblk --json` will not have the plural `mountpoints`
but only a singular `mountpoint`. But newer version lost the
singular `mountpoint` in the json. Adjust the test accordingly.
The new `create_image_with_partitions()` helper fails under rhel8
currently. The reason is that `mkfs.ext4 -E offset=` will warn
in older versions about a partition table and require user input.
This got fixed `e2fsprogs` 1.46.3 in Jul 2021 but RHEL8 still
has 1.45.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989612
Add a simple stage test for the erofs stage. It uses dump.erofs
instead of mounting the file because the kernel in the GH runners
do not support mounting erofs just yet.
This commit removes some unnecessary custom tmpdir() fixtures
and uses the pytest buildin tmp_path instead.
Some custom tmpdir fixtures are left in place as they configure
the tmp location to be under `/var/tmp` which is not trivial to
do with pytests `tmp_path`. Not sure or not if the is a deep
reason there for using /var/tmp. I assume it's to ensure that
the tests run on a real FS not on a potential tmpfs but I don't
have the full background so didn't want to change anything.
The `test_assemblers.py` has an `assertGRUB2` helper that ensures
that the data written in the mbr and first megabyte is unchanged
by the `org.osbuild.qemu` assembler.
With the move to f38 the digests change. The mbr digest is
computed via the following python code and it matches that value
used in the test (440 byte only because the rest is the partition
table).
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f34/usr/lib/grub/i386-pc/boot.img
26e3327c6b5ac9b5e21d8b86f19ff7cb4d12fb2d0406713f936997d9d89de3ee
```
So with that code we can update the f38 mbr value now.
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f38/usr/lib/grub/i386-pc/boot.img
b8cea7475422d35cd6f85ad099fb4f921557fd1b25db62cd2a92709ace21cf0f
However computing the second sha256 for the `512:1024*1024` is much
harder to do from first principles because the value depends on the
image generated via `grub2-mkimage` and the hash changes with each
different module or config option. This means one needs to replicate
the exact inputs of:
```
subprocess.run(["grub2-mkimage",
"--verbose",
"--directory", f"/usr/lib/grub/{platform}",
"--prefix", f"(,{partid})/{grub_path}",
"--format", platform,
"--compression", "auto",
"--output", core_path] +
modules,
check=True)
```
in the test. At this point I cheated and just update to the computed
value inside the test.
Instead of using the F34-based manifests, let's switch to F38. I tried my
best to import the vars into the new manifest, but I don't think that's
currently supported.
Let's not depend on f34-base.json anymore, but instead of a separate,
standardized manifest for it. The test was modified so it supports
v2 manifests. Also, the new manifest installs just a very minimal system.
There's no need to install the whole @core.
This gets rid of the old F34 manifest and migrates the to test to the
standardized V2 one based on F38.
Since the metadata format is much saner in V2, I was able to simplify the
assertion quite a lot.
This commit migrates the test to a brand new V2, F38-based manifest. It's
actually based on osbuild-composer interpretation of the Fedora Cloud Base
image.
Some platforms like the TI AM62 require a particular FAT geometry for
their CPU to read the file system (and thus the bootloader). Failing
that the CPU will simply not boot and keep looking for a bootloader.
Let's add some options to enforce a particular filesystem geometry
through the -g option of mkfs.fat.
Signed-off-by: Maxime Ripard <mripard@kernel.org>
A simple test to make sure that our btrfs stages work.
Updating the testing container is needed, because the old version
didn't have btrfs-progs in it.
This adjustment allows the definition of the mark with the RPMs and runs
DNF after installing the RPMs to put the proper markings in the DNF
state database. See #455.
This ensures that packages don't get removed during `autoremove` leading
to broken systems.
This commit adds support for a new field in tree diffs fed to the stage tests.
This is useful when you care that a directory is added, but you don't care
about its content.
Since this is useful only for the expected tree diff, it's supported only
there. The actual tree diff doesn't support the new field.
Refactor unit test implementation for `parted`, `sfdisk` and `sgdisk`
stages by extracting the common parts into a helper function. Each stage
now implements only its own function for filtering `sfdisk --json`
output and calls the common helper function.
In addition, flip the oder when comparing the expected and actual output
from `sfdisk --json`, to make it more easier to comprehend.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `sfdisk` tool on RHEL-8 does not include the `sectorsize` in its
output when we are testing the `sgdisk` stage. As a result, the test
case fails, because the expected and actual output differs. Modify the
test to delete the `sectorsize` key from the expected output if it is
not present in the actual output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
