Add support for building images for the Azure marketplace: add a
new image type "azure-rhui" that can be used to build images
tailored to the Azure marketplace.
Add two sample manifests for 8.5 and 8.6, but note that even the
8.5 is using the 8.6 distro definitions. Also no image-info is
included since `image-info` cannot (yet) handle LVM setups and
the azure marketplace images use the LVM setup.
When we are running inside a container we generally wont be booted
with systemd and thus systemctl will fail. Fall back to check for
the dnf json socket by checking the path exists and bail otherwise.
Adds ostree params to the request to generate a test manifest for
edge-installers and necessary customisations in the blueprint for the
edge-simplified installer. The manifest is not buildable but works for
checking for changes in the pipeline and packages for the installers.
Add rpmrepo repositories for testing.
Symlink centos-9 to centos-stream-9.
Add rpmrepo snapshot repositories to test and
test-case-generator repos.
Remove unused repositories from cs9 test repositories.
test/data/repositories:
new snapshots for RHEL 8.6 (20220201) and 9.0 (20220208)
Switching RHEL 8.6 and RHEL 9.0 to the latest rpmrepo snapshots:
We need updated packages for greenboot, used by edge image types, so
we technically only need to change the architectures that are
supported by edge, but let's update all of them for consistency.
test/data/repositories:
Remove rhel-90 symlink to beta
Keep the rhel-90-beta.json repository definitions, but make rhel-90.json
a file that contains the previously named rhel-90-ga.json definitions.
This is needed to satisfy RPM dependencies of GCP guest tools, which
will be installed on the `gce` image type.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add F34 and F35 image test cases for all remaining image types, which
were previously not tested. With this PR, image test cases are now
generated for all image types on all architectures as supported by the
Fedora distro definition.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The existing test cases for `image-installer` image type were not
regenerated as part of PR #2135 which introduced 'image_type_tags'
because regenerating them using `generate-all-test-cases` failed on them
due to trying to analyze them using 'image-info'. These test cases were
most probably originally generated manually with --keep-image-info
option passed to `generate-test-cases`. And I missed that their
regeneration failed at that time.
Add the `image-installer` image type to all distros in
`distro-arch-imagetype-map.json` and (re)generate the test cases for it.
The image test case for CentOS Stream 9 is not generated, because there
are no repos defined yet for the distro. This is part of the CentOS 9
PR #2142.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new option `no-image-info`, which can be specified for an image
type in `format-request-map.json` in case the image can not be analyzed
using `image-info` and thus it should not be run as part of generating
the image test case.
This is the case for e.g. the `image-installer` image type in RHEL-8.6,
which effectively breaks generating all image test cases for RHEL-8.6
due to `image-info` failing to analyze the resulting image.
This change will allow us to consistently generate image test cases
(manifests only) for all image types, which were so far not covered by
the image testing.
Modify the `generate-test-cases` script to recognize the `no-image-info`
option in the `format-request-map.json`
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use `make scratch` for building RPMs without tests. This fixes the case,
when the RPM build with test fails to build due to changes in image
manifests. The whole reason of running the script is to regenerate image
test cases when the manifest changed, so this was a chicken and egg
problem.
No CI testing is needed, as this is a development tool.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use image_type_tags in repo definitions used for generating image test
cases. Enhance the `generate-test-cases` script to take the
image_type_tags into account when creating a compose request for a
specific image test case.
The motivation for the change is to keep the list of repositories in the
compose request of a image test case as small as possible. Previously,
all of the defined repositories were part of each image test cases for a
specific architecture, even those that were not needed.
Regenerate affected image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add `--build-rpms` option which will make the script build osbuild-composer
RPMs on the remote runner from the sources which were copied over. These
RPMs are then installed on the system, before any image test cases are
generated.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use newly added RHUI-4 repo snapshots for all RHEL-9 EC2* image test
cases. This includes RHEL-9.0 and RHEL-9.0-Beta images. The removal
of installed repos on EC2-HA and EC2-SAP images is expected, as RHUI
client RPMs for these variants are empty for 9.0 Beta. This should
change for GA once there are updated RHUI clients RPMs available.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
- Updated repos to use latest nightly (20211105)
- Added rt repository
- Both -ga and -beta are identical
- rhel-90.json repo file is a symlink to rhel-90-beta.json
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Add support for building the official RHEL EC2 SAP image for RHEL-8.6.
The new image type is available under the `ec2-sap` name. Since the
image type's package set includes RHUI client RPMs, which are not
available publicly, it is by default not exposed through the Weldr API.
Extend the relevant files and definitions used for generating image test
cases. This includes addition of repository definitions needed by the
image type, specifically: `sap`, `saphana` and `ansible`.
Generate image test case for the new image type.
Add News entry for the new image type.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Because the RHUI client RPMs for RHEL-9 don't exist yet, this test
case uses the RHUI client RPMs for RHEL-8 to satisfy the required
package set. These RPMs contain the entitlement certificate and repo
definitions for RHEL-8 AWS RHUI repos, but RHUI will not work without
the image being imported with the proper Billing Code to AWS. So these
installed RHUI RPMs are technically useless on the RHEL-9 image, but
this approach is good enough to allow us to at least build and test
the image from configuration point of view.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Temporarily use the RHEL-8 RHUI client repo, because RHUI for RHEL-9
does not exist yet. Using at least RHEL-8 RHUI repo for EC2 images
allows us to at least successfully build them and inspect them.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The image is not available via Weldr API, because it requires RHUI
client RPMs.
The content and configuration is based on RHEL-8.6 EC2 SAP image, since
there is no definition for the RHEL-9 SAP image yet.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Because the RHUI client RPMs for RHEL-9 don't exist yet, these test
cases use the RHUI client RPMs for RHEL-8 to satisfy the required
package set. These RPMs contain the entitlement certificate and repo
definitions for RHEL-8 AWS RHUI repos, but RHUI will not work without
the image being imported with the proper Billing Code to AWS. So these
installed RHUI RPMs are technically useless on the RHEL-9 image, but
this approach is good enough to allow us to at least build and test
these images from configuration point of view.
Regenerate all image test cases because of the updated repositories.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Temporarily use the RHEL-8 RHUI client repo, because RHUI for RHEL-9
does not exist yet. Using at least RHEL-8 RHUI repo for EC2 images
allows us to at least successfully build them and inspect them.
Update all RHEL-9 repos to the latest snapshot to ensure the consistency
of package split among repositories.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new option `--repofrompath` allowing to specify additional DNF
repositories, which will be used on the Runner when installing any
packages (such as osbuild).
Extend the `test/README.md` to mention the new option. In addition,
specify some aspects of the script in more detail, because some of
them were not easy to figure out by users.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Generating image test case with all possible blueprint customizations is
supported only for `qcow2` image type. Generating `qcow2` images with
customizations was not a lot of fun. One had to use a special CLI option
for this case.
To streamline the generation of image test cases, drop the
`--with-customizations` option from the `generate-test-cases` script and
move the functionality to `format-request-map.json` by defining a new
`qcow2-customize` test case. This is very similar to what `*edge-rt`
test case already uses. This mean that when the test case for `qcow2`
image type is being generated, actually two test cases will be
generated. The filesystem customizations since RHEL-8.5 are handled
through distro-specific `overrides`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Some of the `overrides` specified in `format-request-map.json` were
redundant, because the same blueprint customizations were already
specified in the image compose-request. Thus these values had no
additional values.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Unfortunately, after the addition of `supported_arches` item to some
of the image type definitions in `format-request-map.json`, it started
to leak to the `compose-request` object in the generated image test
case.
Ensure that the item does not leak to the generated image test case,
since it is relevant only for the generator script.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a CLI option to keep the workdir created on the runner, after it
finishes its work. The workdir is deleted by default.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Report per-distro list of image test cases which were successfully
generated or failed, after the Runner finishes its work.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Remove the '--keep-image-info' script option. The output directory on
the Runner is always newly created at the beginning and empty, therefore
this option effectively can not be used with the `generate-test-cases`
script.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The script currently allows generating image test cases only using QEMU
VMs. As part of the process to allow using different approaches for
generating image test cases, introduce CLI commands using subparsers and
move the current functionality under 'qemu' command.
As part of extracting the generic parts from the TestCaseMatrixGenerator
class, rename it to QEMUTestCaseMatrixGenerator and move all generic
parts to a new BaseTestCaseMatrixGenerator class.
Change filtering options such as '--arch', '--distro', '--image-type'
to take only one argument and append the value to a list if used
multiple times, instead of accepting multiple values. This resolves an
issue of 'nargs=*' argument being too greedy and consuming command as
an additional argument.
Modify the script doc text to match the changes.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When one run the script with debug logging (`-d`), the set log level
in the logger was not preserved in the newly created multiprocessing
processes.
Explicitly set the log level in newly created processes.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Previously the QEMU virtfs devices were used to export osbuild-composer
sources to the QEMU VMs and also to get generated image test cases from
them. This had multiple downsides. The virtfs QEMU option is not
supported on MacOS. In addition, the 9p protocol, which was needed to
mount these devices in the VM, is not supported on RHEL.
Remove all code related to mounting. The data is now copied to and from
the VM using rsync instead.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Don't use paramiko library for SSH connections to the Runner, but
instead execute the `ssh` command using Subprocess.
When one uses SSH ID files protected by password, the paramiko library
can not access them without it, even if the password is stored in the
ssh-agent running in the user session. On the other hand, running the
`ssh` command using Subprocess works just fine in this scenario.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
