When subscription-manager DNF plugins are enabled (e.g. on RHEL), they
produce messages to the stdout on any DNF command execution. E.g.
"Updating Subscription Management repositories.".
Disable all plugins when inspecting package markings so prevent them
from modifying the output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When I rolled back from using 'dnf4', to check package markings, to
using 'dnf', I didn't verify the test case on Fedora Rawhide with DNF5.
It turns out that the strings reported by DNF5 differ and make the test
case fail. This time I tested the change on Fedora Rawhide with DNF5 and
it works.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The test case still fails on RHEL-10.0 Beta, even when not using dnf5,
with:
```
for line in r.stdout.splitlines():
> package, mark = line.strip().split(",")
E ValueError: not enough values to unpack (expected 2, got 1)
```
Make debugging of failures like this easier by printing the line when
the issue happens.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Let's revert to using plain 'dnf', add an explicit newline in the query
format and skip empty lines when processing the output. This makes the
test case compatible with all DNF versions, even with dnf5 once this
issue gets fixed.
The previous approach didn't work on c9s / el9, because there is no
'/usr/bin/dnf4 -> dnf-3' symlink.
Also see:
https://github.com/osbuild/osbuild/actions/runs/10136827918/job/28026181824
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Similar to what was explained in 2e6d49fbe this commit updates
the l2hash in test_assemblers to the new values from fc40 images.
Sadly it is hard to derive them from first principles (see the
other commit) and given that this is legacy code it is probably
fine this way.
The `test_assembler.py` hardcods some filesystem and partition
UUIDs. This leads to hard to diagnose test failures when the
test is run in parallel. The btrfs and xfs filesystem drivers
will see the same uuid for multi created images and error sometimes with
someting like:
```
Mar 06 10:22:54 top kernel: BTRFS error: device /dev/loop104 belongs to fsid aff010e9-df95-4f81-be6b-e22317251033, and the fs is already mounted, scanned by mount (123856)
```
Its a race that only happens when two images are checked at the
same time.
This commit fixes the issue by just using a randomized UUID in
the test_assemblers.py. It also re-enables running the test in
parallel (which make it run a lot faster, from 34min to 14min).
Older versions of RHEL/Centos do not have `mkfs.fat -g` yet so
this test will fail. Detect this and skip the test if mkfs.fat
is too old (see 7af2f1a for the original commit).
When constructing a minimal environment for osbuild the exported
dirs lacked "schemas" so the test_exports.py test failed on RHEL8.
This commit adds it (and also "assemblers" for good measure). With
that the test will pass.
Older version of `lsblk --json` will not have the plural `mountpoints`
but only a singular `mountpoint`. But newer version lost the
singular `mountpoint` in the json. Adjust the test accordingly.
The new `create_image_with_partitions()` helper fails under rhel8
currently. The reason is that `mkfs.ext4 -E offset=` will warn
in older versions about a partition table and require user input.
This got fixed `e2fsprogs` 1.46.3 in Jul 2021 but RHEL8 still
has 1.45.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989612
Add a simple stage test for the erofs stage. It uses dump.erofs
instead of mounting the file because the kernel in the GH runners
do not support mounting erofs just yet.
This commit removes some unnecessary custom tmpdir() fixtures
and uses the pytest buildin tmp_path instead.
Some custom tmpdir fixtures are left in place as they configure
the tmp location to be under `/var/tmp` which is not trivial to
do with pytests `tmp_path`. Not sure or not if the is a deep
reason there for using /var/tmp. I assume it's to ensure that
the tests run on a real FS not on a potential tmpfs but I don't
have the full background so didn't want to change anything.
The `test_assemblers.py` has an `assertGRUB2` helper that ensures
that the data written in the mbr and first megabyte is unchanged
by the `org.osbuild.qemu` assembler.
With the move to f38 the digests change. The mbr digest is
computed via the following python code and it matches that value
used in the test (440 byte only because the rest is the partition
table).
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f34/usr/lib/grub/i386-pc/boot.img
26e3327c6b5ac9b5e21d8b86f19ff7cb4d12fb2d0406713f936997d9d89de3ee
```
So with that code we can update the f38 mbr value now.
```
$ python3 -c 'import hashlib,sys; f=open(sys.argv[1], "rb"); m1=hashlib.sha256();m1.update(f.read()[:440]);print(m1.hexdigest())' ./f38/usr/lib/grub/i386-pc/boot.img
b8cea7475422d35cd6f85ad099fb4f921557fd1b25db62cd2a92709ace21cf0f
However computing the second sha256 for the `512:1024*1024` is much
harder to do from first principles because the value depends on the
image generated via `grub2-mkimage` and the hash changes with each
different module or config option. This means one needs to replicate
the exact inputs of:
```
subprocess.run(["grub2-mkimage",
"--verbose",
"--directory", f"/usr/lib/grub/{platform}",
"--prefix", f"(,{partid})/{grub_path}",
"--format", platform,
"--compression", "auto",
"--output", core_path] +
modules,
check=True)
```
in the test. At this point I cheated and just update to the computed
value inside the test.
Instead of using the F34-based manifests, let's switch to F38. I tried my
best to import the vars into the new manifest, but I don't think that's
currently supported.
Let's not depend on f34-base.json anymore, but instead of a separate,
standardized manifest for it. The test was modified so it supports
v2 manifests. Also, the new manifest installs just a very minimal system.
There's no need to install the whole @core.
This gets rid of the old F34 manifest and migrates the to test to the
standardized V2 one based on F38.
Since the metadata format is much saner in V2, I was able to simplify the
assertion quite a lot.
This commit migrates the test to a brand new V2, F38-based manifest. It's
actually based on osbuild-composer interpretation of the Fedora Cloud Base
image.
Some platforms like the TI AM62 require a particular FAT geometry for
their CPU to read the file system (and thus the bootloader). Failing
that the CPU will simply not boot and keep looking for a bootloader.
Let's add some options to enforce a particular filesystem geometry
through the -g option of mkfs.fat.
Signed-off-by: Maxime Ripard <mripard@kernel.org>
A simple test to make sure that our btrfs stages work.
Updating the testing container is needed, because the old version
didn't have btrfs-progs in it.
This adjustment allows the definition of the mark with the RPMs and runs
DNF after installing the RPMs to put the proper markings in the DNF
state database. See #455.
This ensures that packages don't get removed during `autoremove` leading
to broken systems.
This commit adds support for a new field in tree diffs fed to the stage tests.
This is useful when you care that a directory is added, but you don't care
about its content.
Since this is useful only for the expected tree diff, it's supported only
there. The actual tree diff doesn't support the new field.
Refactor unit test implementation for `parted`, `sfdisk` and `sgdisk`
stages by extracting the common parts into a helper function. Each stage
now implements only its own function for filtering `sfdisk --json`
output and calls the common helper function.
In addition, flip the oder when comparing the expected and actual output
from `sfdisk --json`, to make it more easier to comprehend.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `sfdisk` tool on RHEL-8 does not include the `sectorsize` in its
output when we are testing the `sgdisk` stage. As a result, the test
case fails, because the expected and actual output differs. Modify the
test to delete the `sectorsize` key from the expected output if it is
not present in the actual output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `sfdisk` tool on RHEL-8 does not include the `sectorsize` in its
output when we are testing the `parted` stage. As a result, the test
case fails, because the expected and actual output differs. Modify the
test to delete the `sectorsize` key from the expected output if it is
not present in the actual output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
LOOP_CONFIGURE allows to atomically configure the decive when opening
it. This avoid the possibility of a race condition where between set_fd
and set_status some operations are already accepted by the loopback
device. See https://lwn.net/Articles/820408/
This feature was included in the linux kernel 5.8 however it is safe to
not include any kind of fallback to the previous method as @obudai
points out that:
LOOP_CONFIGURE was backported into RHEL 8 kernel in RHEL 8.4 as a part
of https://bugzilla.redhat.com/show_bug.cgi?id=1881760 (block layer:
update to upstream v5.8).
Since RHEL 8.4 is currently the oldest supported release that we support
running osbuild on, it might be just fine implementing this without the
fallback.
From a centos stream 8 container:
kernel-4.18.0-448.el8.x86_64
- loop: Fix missing discard support when using LOOP_CONFIGURE (Ming Lei) [1997338]
- [block] loop: Set correct device size when using LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: unset GENHD_FL_NO_PART_SCAN on LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: Add LOOP_CONFIGURE ioctl (Ming Lei) [1881760]
Add conditional skip to some tests that depend on rpm-ostree
availability, but were not checking for its presence. These tests would
previously fail if rpm-ostree is not available. They will be skipped
now.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `directory` argument has been added only since Python 3.7, which
breaks the unit test on Python 3.6.
Reimplement the intended behavior by overriding the `translate_path()`
method, which takes the `directory` value into account on newer Python
versions.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Do not specify the default value for 'expected_size' argument in
assertImageFile() function declaration. Previously, it was set to
`None`, which was never taken into account. Moreover, all callers of the
function always provide an explicit value, so the default was never
really used.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new optional pytest CLI argument `--unsupported-fs` allowing to
specify file-systems which should be treated as unsupported in the
platform where running tests. Any test cases dependent on such
file-system support will be sipped.
This will allow to run unit tests and selectively skipping test cases
for unsupported file-systems.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Port assembler tests from unittest to pytest. In addition, use
parametrized tests when testing various filesystems and various
combinations.
This is important to be able to selectively skip the test for if a
specific filesystem is not supported by the kernel (e.g. btrfs is not
supported on RHEL). Skipping a unittest subtest is not possible, which
is the motivation to move away from it and use only pytest.
Test output is now also much nicer for parametrized test cases.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the unit test depended on osbuild modules being installed on
the system. As a result, this made the test not work in CI where we do
not install osbuild when running unit tests. In addition, the stage
executed by the unit test would use different version of osbuild
internals than the version that is being tests, which could result in
issues or not testing the intended code.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
