The base url for the osbuild-worker container was
missing the `https` protocol in the docker-compose file.
This commit adds this to prevent the worker from exiting
The `nss-altfiles` package is actually needed for OSTree based
systems, since in those the user/groups database is located in
the /usr/lib directory (in addition to /etc). It was removed
because it was not available in RHEL 9 (rhbz#1916260). This has
been fixed, so we need to re-include the package.
Generating image test case with all possible blueprint customizations is
supported only for `qcow2` image type. Generating `qcow2` images with
customizations was not a lot of fun. One had to use a special CLI option
for this case.
To streamline the generation of image test cases, drop the
`--with-customizations` option from the `generate-test-cases` script and
move the functionality to `format-request-map.json` by defining a new
`qcow2-customize` test case. This is very similar to what `*edge-rt`
test case already uses. This mean that when the test case for `qcow2`
image type is being generated, actually two test cases will be
generated. The filesystem customizations since RHEL-8.5 are handled
through distro-specific `overrides`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Some of the `overrides` specified in `format-request-map.json` were
redundant, because the same blueprint customizations were already
specified in the image compose-request. Thus these values had no
additional values.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Unfortunately, after the addition of `supported_arches` item to some
of the image type definitions in `format-request-map.json`, it started
to leak to the `compose-request` object in the generated image test
case.
Ensure that the item does not leak to the generated image test case,
since it is relevant only for the generator script.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a CLI option to keep the workdir created on the runner, after it
finishes its work. The workdir is deleted by default.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Report per-distro list of image test cases which were successfully
generated or failed, after the Runner finishes its work.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Remove the '--keep-image-info' script option. The output directory on
the Runner is always newly created at the beginning and empty, therefore
this option effectively can not be used with the `generate-test-cases`
script.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The script currently allows generating image test cases only using QEMU
VMs. As part of the process to allow using different approaches for
generating image test cases, introduce CLI commands using subparsers and
move the current functionality under 'qemu' command.
As part of extracting the generic parts from the TestCaseMatrixGenerator
class, rename it to QEMUTestCaseMatrixGenerator and move all generic
parts to a new BaseTestCaseMatrixGenerator class.
Change filtering options such as '--arch', '--distro', '--image-type'
to take only one argument and append the value to a list if used
multiple times, instead of accepting multiple values. This resolves an
issue of 'nargs=*' argument being too greedy and consuming command as
an additional argument.
Modify the script doc text to match the changes.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When one run the script with debug logging (`-d`), the set log level
in the logger was not preserved in the newly created multiprocessing
processes.
Explicitly set the log level in newly created processes.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Previously the QEMU virtfs devices were used to export osbuild-composer
sources to the QEMU VMs and also to get generated image test cases from
them. This had multiple downsides. The virtfs QEMU option is not
supported on MacOS. In addition, the 9p protocol, which was needed to
mount these devices in the VM, is not supported on RHEL.
Remove all code related to mounting. The data is now copied to and from
the VM using rsync instead.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Don't use paramiko library for SSH connections to the Runner, but
instead execute the `ssh` command using Subprocess.
When one uses SSH ID files protected by password, the paramiko library
can not access them without it, even if the password is stored in the
ssh-agent running in the user session. On the other hand, running the
`ssh` command using Subprocess works just fine in this scenario.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Previously passwords were used to log into provisioned QEMU VMs. This is
not practical if one would like to use e.g. rsync to transfer files from
and to the VM. The script now does not use passwords at all, but instead
configures the most recent SSH key from the system matching
'~/.ssh/id*.pub' as an authorized key on the VM. Alternatively the SSH
key to be used can be provided as an argument to the script.
In addition, the script no longer relies on external files for
cloud-init user-data. If no cloud-init user-data are provided as an
argument to the script creates default user-data file in a temporary
work directory and uses it. The reason for this change is mostly that the
default user-data became very short and need to be always extended with
the authorized SSH key anyway. In addition, this makes the script more
standalone by relying on fewer external file paths.
Delete the `tools/deploy/gen-test-data` which held the cloud-init
user-data previously used by default by the script.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Previously cloud-init was used to install necessary packages on the
runner. This would not be practical in the future with other types of
runners, which would not use cloud-init. Install all necessary RPMs by
directly running DNF command on the runner.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The `BaseRunner` class represented a base QEMU runner. Rename it to
`BaseQEMURunner` and extract parts which are not QEMU specific to a new
`BaseRunner` class. This new base class will be later used as a baseline
for other types of runners which don't rely on QEMU.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Some image type test cases require additional repositories, which are
not available for all architectures. However, when an image type test
case is specified in the `format-request-map.json`, it is generated on any
architecture.
This behavior is creating issues when generating `*edge-commit` image
type test cases. This is because the `format-request-map.json` contains one
additional definition for `*edge-commit-rt`, which includes `kernel-rt`
package. However repositories with this package are available only for
x86_64. Therefore, when generating image test cases for `*edge-commit`,
the `generate-test-cases` script always generates two test cases,
but the generation of `*edge-commit-rt` always fails on non-x86_64
architectures.
Add a new optional member to the image type test case object in
`format-request-map.json`, called `supported_arches`. Its value is a
list of strings, specifying the supported architectures of the image
type test case. In case the member is not specified, the image test
case is supported on any architecture.
Extend the `generate-test-cases` script to skip image type test case
generation in case the case has the `supported_arches` specified and the
requested architecture is not in the list.
Fix#1478
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Instead of keeping a howto on releasing in every repository we move the
documentation to a central place in our "developer guide".
See https://github.com/osbuild/guides/pull/45
Add support for a new osbuild stage `org.osbuild.tuned`, for
setting TuneD profile.
Add unit tests for the new stage.
Related to https://github.com/osbuild/osbuild/pull/797.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for a new osbuild stage `org.osbuild.dnf.config`, for
configuring DNF.
Add unit tests for the new stage.
Related to https://github.com/osbuild/osbuild/pull/798.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for a new osbuild stage `org.osbuild.sysctld`, for
creating tmpfiles.d configuration files.
Add unit tests for the new stage.
Related to https://github.com/osbuild/osbuild/pull/804.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for a new osbuild stage `org.osbuild.tmpfilesd`, for
creating tmpfiles.d configuration files.
Add unit tests for the new stage.
Related to https://github.com/osbuild/osbuild/pull/801.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for a new osbuid stage `org.osbuild.selinux.config`,
for setting the desired SELinux policy state and type on the system.
Add unit tests for the new stage.
Related to https://github.com/osbuild/osbuild/pull/799.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This commit also fixes the rpmrepo for 8.5 rt. The previously used one
was wrongly generated and had to be regenerated.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When we firstly introduced the v2 manifests, we excluded docs from all rpm
stages because it doesn't make sense to include docs in edge images.
However, when we ported the other image types to v2, we left the flag on.
The side effect of --excludedocs is that we no longer install man pages into
the image. This the default behaviour of rpm and can be seen here:
b88f43b9a3/macros.in (L555)
This is actually quite a major regression in non-edge images, thus this commit
reverts this setting. It would be great to have the option not to install
docs in some rpm stages but as this is a difficult topic and we're already
past the deadline, we will consider this in the future.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When we firstly introduced the v2 manifests, we excluded docs from all rpm
stages because it doesn't make sense to include docs in edge images.
However, when we ported the other image types to v2, we left the flag on.
The side effect of --excludedocs is that we no longer install man pages into
the image. This the default behaviour of rpm and can be seen here:
b88f43b9a3/macros.in (L555)
This is actually quite a major regression in non-edge images, thus this commit
reverts this setting. It would be great to have the option not to install
docs in some rpm stages but as this is a difficult topic and we're already
past the deadline, we will consider this in the future.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add test case to check the ability to create partitions for
the /var, /var/log & /var/log/audit mountpoints. This test is
checking that mountpoints with a depth of 3 or greater can be configured.
Previously it was only possible to configure separate partitions
for mountpoints in the allow list and their immediate subdirectories
only i.e. /var & /var/log
This fix allows for an arbitrary level of mountpoints, i.e. /var/log/audit,
/var/a/b/c/d/e and so on
Previously it was only possible to configure separate partitions
for mountpoints in the allow list and their immediate subdirectories
only i.e. /var & /var/log
This fix allows for an arbitrary level of mountpoints, i.e. /var/log/audit,
/var/a/b/c/d/e and so on
V2 is compliant with api.openshift.com design guidelines.
Errors are predefined, have codes, and are queryable.
All requests have an operationId set: a unique identifier which is
sortable by time. This is added to the response in case of an error.
All returned objects have the href, id, and kind field set.
The RHSM DNF plugins `product-id` and `subscription-manager` are now
by default enabled on the RHEL-8.5 and RHEL-9.0 `ec2`, `ec2-ha` and
`ami` images.
The desired default state of the RHSM DNF plugins has been decided by
the RHSM team.
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1996670
Signed-off-by: Tomas Hozza <thozza@redhat.com>
We don't want to give the impression we support building Fedora on RHEL or
CentOS Stream, or CentOS Stream on RHEL, so drop these repositories from
the package.
Also adjust dnf-json to use test repositories, rather than the upstream ones.
These are shipped in the test package, so makes sure we are unaffected by
what is shipped in the RPM shipped in the distros.
