Our current testing environment cannot run VMs for aarch64, therefore we
cannot test the qcow2 image. Also, we don't have an openstack account with
aarch64 so also we cannot test openstack image type. Additionaly, we cannot
selectively disable boot testing for now.
Let's do the smallest change possible and just add ami test case - we can boot
test these images in AWS.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
x86_64 and aarch64 rhel 8.4 images support hybrid boot. qcow2 images are
now tested with both BIOS and UEFI boot. The Jenkinsfile now specifies
which boot type is supported when calling run_tests. For the qemu tests,
this boot type can be passed as an option so the test can boot using
UEFI.
In order to boot using uefi from a rhel machine, we must pass the --boot
uefi flag as well as tell libvirt where the variable store template is.
Since we are not testing secureboot, we must pass
nvram_template=/usr/share/edk2/ovmf/OVMF_VARS.fd to the --boot flag.
This is the location of our non-secureboot varstore template.
rhel 8.4 tests are added. The configs are based off of those used for
rhel 8.3. The Schutzbot Mockbuild, Base, Image, Integration, and OSTree
tests are added for 8.4. Repo overrides are added for the rhel 8.4 tests
so that the tests use rpmrepo snapshots.
The mockbuild uses the jenkins rhel84-nightly-repo credential to
override the rhel mock template's repos with rhel 8.4 nightly repos.
These repos are stored in a credential because they are internal links.
The image tests and koji tests need a special distro selector since the
rhel-8 test cases are only for rhel 8 versions less than 8.4. The rhel
8.4 tests are named with the rhel-84 pattern whereas the other rhel 8
versions have the rhel-8 pattern.
Also, instead of having only rhel-8 and rhel-8-beta repo configs for the
tests, we now have a specific repo config for each rhel release we test.
The repo is also now pulled from an rpmrepo snapshot. For whichever
distro is being tested, the approriate repo config will be copied to
/etc/osbuild-composer/repositories as rhel-8 and rhel-8-beta since this
is the naming osbuild-composer looks for. For testing purposes, the
rhel-8 and rhel-8-beta repo should be the same since eventually all rhel
releases will go from beta to not beta. The fedora repo overrides are
already done in tools/provision.sh so the rhel override is set there as
well. Currently, only rhel 8.4 requires an override.
cockpit-composer can now build rhel 8.4 images. Our distro name for
rhel 8.4 is rhel-84 unlike prior rhel releases which fall
under the umbrella name rhel-8. rhel 8.4 still uses the same
repos as the rest of the rhel 8 releases but points to a different
nightly repo for testing purposes. Test cases are added. The changes
between rhel 8.3 and 8.4 are as follows:
There is now a hybrid boot partition scheme for x86_64. x86_64 images
now use uefi boot and have 3 gpt partitions: a small unformated
partition for mbr compatibility, an efi boot partition of type vfat, and
a root partition of type xfs. The packages grub2-efi-x64 and shim-x64
are added as bootloader packages for all x86_64 images.
For qcow2 images ro is added as a kernel option and the following
packages are added (+) or removed (-):
+ dosfstools
+ efi-filesystem
+ efivar
+ efivar-libs
+ grub2-efi-x64
+ shim-x64
- rhn-client-tools
- rhnlib
- rhnsd
- rhn-setup
Now that all interaciton with the koji API happens in the workers
we can drop koji configuration from composer itself. This means
that composer no longer needs to be provisioned with kerberos
credentials, and does not need to know about which koji servers
the workers support.
This commit does several things:
1) Changes the Fedora 33 repos in the test case generator from development
to release ones.
2) Fixes format-request-map.json so we can generate fedora-iot-commit
"images".
3) Regenerates all the cases.
If the ostree test was run on an unsupported distro, it failed but with a
very weird error message. This commit makes the test fail fast and with a
nice message.
The downloaded image may not fit inside tmpfs, especially when testing
on a constrained VM. This commit makes the test script use a different
temporary directory while handling the possibly big image.
The integration tests are leaving the composes (which include images) in
osbuild-composer. This can lead to exhausting the disk space we have available
on our tiny testing machines. This commit adds a removal of the composes
after each integration test is finished. This issue is not present in koji.sh
and api.sh as they use different osbuild-composer APIs that doesn't use the
artifact feature.
This issue occurred when I worked on enabling the Fedora 33 tests, see:
https://osbuildci.cloud.paas.psi.redhat.com/blue/organizations/jenkins/osbuild%2Fosbuild-composer/detail/PR-1014/23/pipeline
We claim to have self-contained test cases, but the base_tests.sh script
still requires the WORKSPACE environment variable to be set outside of
the script, which is what Jenkins does.
This patch replaces WORKSPACE with a temporary directory and modifies
Jenkinsfile to use it when collecting logs.
We have several repository definitions across the tests which is quite messy.
This commit switches the Koji test to use the "central" repository configs defined in test/data/repositories/
Don't use common.State anymore, because it has different values from
what's defined in openapi.yml. It makes sense to have these strings
defined in the same package as the spec — ideally, the code generator
would make them for us.
While at it, add a "running" status.
Fix the api.sh test to use these new statuses. Thanks to Ondřej Budai
for an additional fix there.
We don't install any packages in test cases anymore, therefore we don't need
to install EPEL there.
A slightly different explanation:
osbuild-composer-tests depends on packages from EPEL on RHEL. Therefore, you
cannot run the test cases without EPEL installed. Therefore, there's no
point in installing EPEL there.
RHEL 8.3 is GA so we can run the ostree test here. This is a first step
to remove the RHEL 8.3 testing pipeline.
8.3 GA also ships a different version of composer-cli, therefore a slight
adjustment was needed.
The certificate generation is based on work by Lars Karlitski in our osbuild
CA. The server and client certs now contains Subject Alternative Name making
Python's request module and Go 1.15 happy (they deprecated certificates
without SAN).
Several reasons why we want to switch to the certificate generation:
1) The pre-generated certificates are not documented. If someone wants
to inspect them, he must know the right openssl incantation. This way,
you are able to see what's inside the certificates in a plain text.
2) The pre-generated certificates are going to expire at one point and
someone will be surprised.
3) Shipping private keys in RPMs is iffy. I know, it's just for testing but
still...
4) Auth tests are generating their own certificates. To achieve consistency,
we have two options:
a) Ship also all certificates for auth tests. That's extra 8 ones or
something like that.
b) Generate all certificates on fly. This commit does that.
5) The setup introduced by this commit is very similar to the one in our CA
making the test environment very similar to what's running in production.
tl;dr: I think this is a good step forward.
The Koji test in Github actions was always a bit quick and dirty solution.
I think it's much nicer solution to run it on Schutzbot.
Therefore, this commit moves the koji_test.go to a new osbuild-koji-tests
executable. This new test isn't run in the base test suite as one would
anticipate but inside the koji.sh test. This is needed because
osbuild-koji-tests requires a running koji instance. This might change
in the future but I think it works for now.
The test package should be self-contained and contain all the configuration
necessary for a known-good test run (minus secrets). This moves repo
overrides from the test orchestration into the test package.
We want all the external sources (including boot isos and repositories) to
be pinned by their content hash, and never use anything that is not strictly
defined. This moves us in the right direction, but we still have some tests
to update to use these shipped repos rather than official mirrors.
One remaining challenge is that we must make sure all our test runners have
access to the RHEL snapshots, which is not currently the case for the runners
in EC2, but a solution is in the works.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Attempt to clarify the structure of our tests. Each test case is now
encapsulated in a script in `test/cases`. Each of these scripts should
be runnable on a pristine machine and be independent of each other. It
is up to the test-orchestractor to decide if they should be run
consequtively instance, or in parallel on separate instances. Each
script can execute several tests and call whatever helper binaries
is desired. However, each case should be assumed to always run as one.
We no longer release into F31, and the right specfile was anyway not
being tested.
This allows us to remove a workaround that updates the VMs during
deploy, and other fedora-31 specific hacks.
`run_{base,image}_tests.sh` are osbuild-composer-specific, and should
live in the `-test` subpackage, rather than as part of the testing
infratructure.
This is not a functional change.
Add an additional integration test, which builds a fedora-32 image via
the newly moved composer API.
This adds a new certificate for clients to authenticate, with
CN=client.osbuild.org.
Also stop allowing certificates with `CN=worker.osbuild.org` in the
`osbuild-composer.toml` used in tests and amend `osbuild-auth-tests` to
use `CN=client.osbuild.org` for accessing the koji API.
Heavily based on an earlier version of this test in `test/cmd/aws.sh` by
Sanne Raymaekers.
This removes the osbuild-composer-cloud package, binary, systemd units,
the (unused) test binary, and the (only-run-on-RHEL) test in aws.sh.
Instead, move the cloud API into the main package, using the same
socket as the koji API, osbuild-composer-api.socket. Expose it next to
the koji API on route `/api/composer/v1`.
This is a backwards incompatible change, but only of the -cloud parts,
which have been marked as subject to change.
The -test subpackage should depend on all that it needs, and we should
never need to install anything explicitly from the test scripts.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This was made a weak dep, so we need to be explicit about it:
```
Author: Jonathan Lebon <jonathan@jlebon.com>
Commit: Lokesh Mandvekar <lsm5@fedoraproject.org>
spec: make podman-plugins a weak dep
That way someone who wants just podman doesn't also get dnsmasq. This is
a point of contention in FCOS currently:
https://github.com/coreos/fedora-coreos-tracker/issues/519
(cherry picked from commit 217f4344ec796dd1134a5ba54a8ce2fb5a32a5b6)
Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>
```
Signed-off-by: Tom Gundersen <teg@jklm.no>
Also install it is part of he tests subpackage. This a helper-tool, not
golang code, so should not live in `internal`. We need access to this
from the integration tests, so install it onto the tests system.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is obviously not a good long-term solution, but at least the fedora
version of the binary has a build-id so RPM will not complain.
Signed-off-by: Tom Gundersen <teg@jklm.no>
No tests should be run directly from git, but should rather be installed
onto the test system using rpm and run from there. This moves towards
unifying our two types of test cases.
The new structure of is now:
`test/cmd`: the executors, one for each test-case. This is installed
into `/usr/libexec/test/osbuild-composer`.
`test/data`: data and config used by the tests. This is installed into
`/usr/share/tests/osbuild-composer`.
`schutzbot`: configuration of the actual test run. In particular, this
is where the distros and repositories to test against are
configured.
This is very much still work-in-progress, and is only the first step
towards simplifying schutzbot. Apart from moving files around, this
should be a noop.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We already have something called image-tests, so this naming was
confusing. This is a collection of generic integration tests, so name
them accordingly.
Signed-off-by: Tom Gundersen <teg@jklm.no>
These had the generic name osbulid-tests, rename the binary to more
closely describe what is under test: weldr API as composed by
composer-cli.
Signed-off-by: Tom Gundersen <teg@jklm.no>
