We have been observing a race condition in our CI when installing
packages at the beginning of the SUT setup. This happens only on RHEL
and it is caused by the `rhc` tool, which executes some Ansible
playbooks on system startup, which install packages using dnf. This
interferes with dnf commands ran by the `mockbuild.sh` script, which
results in a job failure.
Since there seems to be no way to determine if `rhc` finished "its
thing", let's retry dnf install of packages up to 5 times with
exponential backoff in between retries.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Use %forgeautosetup macro to prepare sources in the %prep phase to
auto-apply any potential downstream patches just by listing them in the
spec file using `PatchX`. Otherwise, the macro needs to be modified each
time a downstream patch needs to be applied in downstream.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The partition name in the sgdisk stage was previously quoted when
passed to sgdisk as an argument. I think that this was done because
the sgdisk man page states that:
```
If you want to set a name that includes a space, enclose it in
quotation marks, as in sgdisk -c 1:"Sample Name" /dev/sdb.
```
However, this should apply only when sgdisk is run in a shell, so that
the argument is not split by shell into multiple arguments and passes
as a single string.
The stage is executing sgdisk using Python `subprocess` module, which
does not need strings with spaces to be quoted, because they are passed
to the command as separate items which are not split in any way.
The previous behavior of the stage was that these quotes became part of
the actual partition name in the partition table.
After a discussion within the team, we determined that this is a bug.
However, fixing it would result in osbuild producing a different
artifact for the same manifest, compared to osbuild version without such
fix. This is undesired.
For backward compatibility, a new `quote_partition_name` property is
added to the stage options, which can be used to make the stage not
quote the partition name when passed to `sgdisk`. As a result, the
partition name won't be quoted in the partition table.
The default stage behavior us kept.
Modify unit tests to use this option by default.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
util-linux 2.38.1, at least, does not accept raw bit indexes for the
reserved bits (Bit0-2). The undefined ones are out of reach as well and
will have sfdisk throw an error. Only the GUID specific ones can be
passed as raw indexes. This can be verified with the --part-attrs
option. It replicates the format of the --dump output:
$ dd if=/dev/zero of=disk.img bs=$((4<<10)) count=$((32<<10)
$ sgdisk disk.img -n 0:0:+64M -t 0:0FC63DAF-8483-4772-8E79-3D69D8477DE4 -c 0:root
$ sgdisk disk.img -A1:set:{0,1,2,3,48}
$ sfdisk --dump disk.img
label: gpt
label-id: 7484F730-3429-47BF-8A72-3A7AE1F2D86C
device: disk.img
unit: sectors
first-lba: 34
last-lba: 262110
sector-size: 512
disk.img1 : start= 2048, size= 131072, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=404694AC-247D-43B3-9907-A468E5C038A3, name="root", attrs="RequiredPartition NoBlockIOProtocol LegacyBIOSBootable GUID:48"
$ sfdisk --part-attrs disk.img 1 0
unsupported GPT attribute bit '0'
sfdisk: disk.img: partition 1: failed to set partition attributes
While the --dump output prefixes the GUID specific bits with "GUID:",
that is not necessary for setting them, which is consistent with the
man-page.
Signed-off-by: Eric Chanudet <echanude@redhat.com>
Add the translation logic to handle the attrs field of sfdisk as
supported by org.osbuild.sfdisk and documented in its schema. With the
schema taking an int array, some translation is required to populate the
sfdisk command appropriately.
Amend the example schema to reflect the change.
Signed-off-by: Eric Chanudet <echanude@redhat.com>
This fixes an issue where Fedora-38 hosts can not build CentOS-Stream-9
images due to an incompatible gpg key with the new default settings for
rpm.
On Fedora-38, rpm has changed to use a new backend for key verification
and by default does not support SHA1 anymore, although the support for
SHA1 can be re-enabled via a config file. The (current) CentOS-Stream-9
keys however still require SHA1 support in order to be importable. So
they are now unusable on Fedora-38 unless SHA1 support is re-enabled.
In OSBuild, the initial chroot does not contain the config files and so
SHA1 support is disabled when rpmkeys from the host is called. It does
not matter if the crypto-policies on the host machine is configured with
the exception to support SHA1 because the chroot filters that out. This
means it may not be possible to assemble CentOS-Stream-9 based images
without disabling the key check.
This patch adds an explicit conditional case for Fedora-38 to inject the
needed configuration file into /etc/crypto-policies/back-ends to enable
SHA1 support for rpm by default. It does this by copying the default
policies from /usr/share/crypto-policies. The result is OSBuild behaving
similar to the previous behaviour seen on Fedora-37 and earlier.
This allows enabling the ext4 "verity" feature (which is currently
default to off). This will be needed in the automotive work we're
doing.
Signed-off-by: Alexander Larsson <alexl@redhat.com>
LOOP_CONFIGURE allows to atomically configure the decive when opening
it. This avoid the possibility of a race condition where between set_fd
and set_status some operations are already accepted by the loopback
device. See https://lwn.net/Articles/820408/
This feature was included in the linux kernel 5.8 however it is safe to
not include any kind of fallback to the previous method as @obudai
points out that:
LOOP_CONFIGURE was backported into RHEL 8 kernel in RHEL 8.4 as a part
of https://bugzilla.redhat.com/show_bug.cgi?id=1881760 (block layer:
update to upstream v5.8).
Since RHEL 8.4 is currently the oldest supported release that we support
running osbuild on, it might be just fine implementing this without the
fallback.
From a centos stream 8 container:
kernel-4.18.0-448.el8.x86_64
- loop: Fix missing discard support when using LOOP_CONFIGURE (Ming Lei) [1997338]
- [block] loop: Set correct device size when using LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: unset GENHD_FL_NO_PART_SCAN on LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: Add LOOP_CONFIGURE ioctl (Ming Lei) [1881760]
`os.scandir()` can accept file descriptors only since Python 3.7. The
tool would previously fail with exception when run using Python 3.6.
The solution is to provide a path, which is done using a symlink in
procfs (this is already used within the tool).
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add conditional skip to some tests that depend on rpm-ostree
availability, but were not checking for its presence. These tests would
previously fail if rpm-ostree is not available. They will be skipped
now.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `directory` argument has been added only since Python 3.7, which
breaks the unit test on Python 3.6.
Reimplement the intended behavior by overriding the `translate_path()`
method, which takes the `directory` value into account on newer Python
versions.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
DNF has more elaborate locking system and can wait for other instances of
itself when installing packages. Using rpm directly to install local
package is causing failures in CI due to it not being able to acquire
lock on `/var/lib/rpm/.rpm.lock`.
Using DNF should improve the situation, although there is no good
documentation to link and support this claim for sure.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This is how it is done also in osbuild-composer. The downside of doing
this in `deploy.sh` is that team SSH keys are not set as authorized for
mock-build jobs, which make it impossible to log into the machine and
debug anything.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
`bandit` is a fast Python vulnerability scanner, this provides a default
configuration file for it for those that want to run it.
`bandit --ini .bandit -r osbuild/` will get you results.
Do not specify the default value for 'expected_size' argument in
assertImageFile() function declaration. Previously, it was set to
`None`, which was never taken into account. Moreover, all callers of the
function always provide an explicit value, so the default was never
really used.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new optional pytest CLI argument `--unsupported-fs` allowing to
specify file-systems which should be treated as unsupported in the
platform where running tests. Any test cases dependent on such
file-system support will be sipped.
This will allow to run unit tests and selectively skipping test cases
for unsupported file-systems.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Port assembler tests from unittest to pytest. In addition, use
parametrized tests when testing various filesystems and various
combinations.
This is important to be able to selectively skip the test for if a
specific filesystem is not supported by the kernel (e.g. btrfs is not
supported on RHEL). Skipping a unittest subtest is not possible, which
is the motivation to move away from it and use only pytest.
Test output is now also much nicer for parametrized test cases.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The code uses x86_64 as a default, see:
basearch = options.get("basearch", "x86_64")
Let's declare that explicitly in the schema to prevent any confusion.
The test case was testing a subset of the functionality, which is
already tested by test case in `tests.yml`. Delete it, since it does not
add any value.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Fix the following errors:
```
osbuild/util/lvm2.py:117: error: Only instance methods can be decorated with @property
osbuild/api.py:50: error: Only instance methods can be decorated with @property
osbuild/sources.py:85: error: Only instance methods can be decorated with @property
```
Chaining of `@classmethod` and `@property` has been deprecated since
Python 3.11 with a note that chaining didn't work correctly in some
cases.
Relevant links:
https://github.com/python/mypy/issues/13746https://docs.python.org/3.11/whatsnew/3.11.html#language-builtins
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the unit test depended on osbuild modules being installed on
the system. As a result, this made the test not work in CI where we do
not install osbuild when running unit tests. In addition, the stage
executed by the unit test would use different version of osbuild
internals than the version that is being tests, which could result in
issues or not testing the intended code.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The directory does not exist when the unit test is run in CI. Handle
this case by ensuring that parent directories are created as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
In CI, we list specific tests from `test/run` to run them in parallel.
This is different than what we do with tests in `test/mod` and
`test/src`.
It seems that as a result, we did not run tests from the following
files in CI:
- `test_devices.py`
- `test_mount.py`
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The right way to enable services is to use a preset file instead of
writing directly into /etc. This adds a new stage called
`org.osbuild.systemd.preset` to do so.
