This commit enables:
- mockbuild for F33 aarch64
- base tests for F33 aarch64
- ami image test for F33 aarch64
This is mostly a MVP - enabling more tests would require more work than I have
a capacity for right now. Still, it gives us at least some coverage for
alt-arches. Most importantly, base tests build qcow2 and image tests build ami
so we have at least some trust that image building for aarch64 works.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
22c9f6a introduced a new environment variable to api.sh,
AWS_API_TEST_SHARE_ACCOUNT, but only set it in the RHEL 8 tests.
Seems like the corresponding branch was not rebased to a version which
already had those 8.4 tests:
https://github.com/osbuild/osbuild-composer/pull/1098
Generalize deploy.sh to allow setting up reverse dependency tests. Those
need the same setup for osbuild and osbuild-composer (i.e., the ones
under test), and in addition need the -tests package of the dependant
project.
Add test runs for koji-osbuild for all operating systems in Jenkinsfile.
Use a commit in koji-osbuild for which a repository was pushed to s3 in
the right format.
heredocs make control flow hard to read. Pull writing the repo file into
its own function and use it for setting up osbuild and osbuild-composer
repos.
x86_64 and aarch64 rhel 8.4 images support hybrid boot. qcow2 images are
now tested with both BIOS and UEFI boot. The Jenkinsfile now specifies
which boot type is supported when calling run_tests. For the qemu tests,
this boot type can be passed as an option so the test can boot using
UEFI.
In order to boot using uefi from a rhel machine, we must pass the --boot
uefi flag as well as tell libvirt where the variable store template is.
Since we are not testing secureboot, we must pass
nvram_template=/usr/share/edk2/ovmf/OVMF_VARS.fd to the --boot flag.
This is the location of our non-secureboot varstore template.
rhel 8.4 tests are added. The configs are based off of those used for
rhel 8.3. The Schutzbot Mockbuild, Base, Image, Integration, and OSTree
tests are added for 8.4. Repo overrides are added for the rhel 8.4 tests
so that the tests use rpmrepo snapshots.
The mockbuild uses the jenkins rhel84-nightly-repo credential to
override the rhel mock template's repos with rhel 8.4 nightly repos.
These repos are stored in a credential because they are internal links.
The image tests and koji tests need a special distro selector since the
rhel-8 test cases are only for rhel 8 versions less than 8.4. The rhel
8.4 tests are named with the rhel-84 pattern whereas the other rhel 8
versions have the rhel-8 pattern.
Also, instead of having only rhel-8 and rhel-8-beta repo configs for the
tests, we now have a specific repo config for each rhel release we test.
The repo is also now pulled from an rpmrepo snapshot. For whichever
distro is being tested, the approriate repo config will be copied to
/etc/osbuild-composer/repositories as rhel-8 and rhel-8-beta since this
is the naming osbuild-composer looks for. For testing purposes, the
rhel-8 and rhel-8-beta repo should be the same since eventually all rhel
releases will go from beta to not beta. The fedora repo overrides are
already done in tools/provision.sh so the rhel override is set there as
well. Currently, only rhel 8.4 requires an override.
Running `make srpm` on the host can have different results from building
the SRPM in mock.
Stop using the Makefile altogether for this, which is meant for
developer convenience only. The interface is clear: a tarball generated
from `git archive` and defining the `commit` variable. Less stuff to
install on the host, too.
To make sure we test against what we will release with, use osbuild from
the target distro rather than rebuilding it.
The only exception is in RHEL8.3, where we are not rebasing any longer,
but expect users who use new osbulid-composer from upstream to use it
with new osbuild from upstream too. In this case, use the RPM from
osbuild upstream, for now pinned to the same version as was in the
submodule.
This introduces a new configuration file: Schutzfile, which is meant
to contain the full test-matrix. For now it only points to the upstream
osbuild commit to add to the distro we are testing against (only relevant
for 8.3).
The submodule is now unused and is therefore removed. The produced repos
now only contain osbuild-composer, osbuild is never built as part of
osbuild-composer CI.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
The length of these is not predictable. It depends on the shortest
unique prefix in the repository and git configuration.
Just use the full one, which also makes it easier to copy the id from
`git log` or GitHub.
Change the repository path on S3 to a more predictable one. We really
only need the name of the project (static osbuild-composer for this
repository), the name of the distro (use the same as osbuild-composer's
API for consistency) and the commit SHA.
In particular, drop the PR number / branch name. Also don't remove the
dots from version numbers. All places we're using them in (paths and
URLs) support dots.
For example, osbuild composer commit xxxxxxx for fedora-33 on x86_64
will result in this URL:
osbuild-composer/fedora-33/x86_64/xxxxxxx
Jenkins has been configured to use the latest commit on a pull request
(instead of merging to master) for a long time now. Rename the variable
to reflect that.
We claim to have self-contained test cases, but the base_tests.sh script
still requires the WORKSPACE environment variable to be set outside of
the script, which is what Jenkins does.
This patch replaces WORKSPACE with a temporary directory and modifies
Jenkinsfile to use it when collecting logs.
RHEL 8.3 is GA so we can run the ostree test here. This is a first step
to remove the RHEL 8.3 testing pipeline.
8.3 GA also ships a different version of composer-cli, therefore a slight
adjustment was needed.
The test package should be self-contained and contain all the configuration
necessary for a known-good test run (minus secrets). This moves repo
overrides from the test orchestration into the test package.
We want all the external sources (including boot isos and repositories) to
be pinned by their content hash, and never use anything that is not strictly
defined. This moves us in the right direction, but we still have some tests
to update to use these shipped repos rather than official mirrors.
One remaining challenge is that we must make sure all our test runners have
access to the RHEL snapshots, which is not currently the case for the runners
in EC2, but a solution is in the works.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Attempt to clarify the structure of our tests. Each test case is now
encapsulated in a script in `test/cases`. Each of these scripts should
be runnable on a pristine machine and be independent of each other. It
is up to the test-orchestractor to decide if they should be run
consequtively instance, or in parallel on separate instances. Each
script can execute several tests and call whatever helper binaries
is desired. However, each case should be assumed to always run as one.
We no longer release into F31, and the right specfile was anyway not
being tested.
This allows us to remove a workaround that updates the VMs during
deploy, and other fedora-31 specific hacks.
`run_{base,image}_tests.sh` are osbuild-composer-specific, and should
live in the `-test` subpackage, rather than as part of the testing
infratructure.
This is not a functional change.
Jenkins doesn't support two pipelines in one file. We had two and it caused
an exception in Jenkins marking all builds as failed even though all stages
passed successfully.
As this pipeline isn't used for anything right now (see #976), it's safe to
delete it.
Fixes#1032
Add an additional integration test, which builds a fedora-32 image via
the newly moved composer API.
This adds a new certificate for clients to authenticate, with
CN=client.osbuild.org.
Also stop allowing certificates with `CN=worker.osbuild.org` in the
`osbuild-composer.toml` used in tests and amend `osbuild-auth-tests` to
use `CN=client.osbuild.org` for accessing the koji API.
Heavily based on an earlier version of this test in `test/cmd/aws.sh` by
Sanne Raymaekers.
Base these on https://osbuild.org/rpmrepo.
Install those repository files when deploying in schutzbot and include
them in tools/deploy/test for local testing.
Also remove copying overrides from deploy.sh, because it copied the
wrong files. We probably want to add this back properly in the future.
Instead, call it osbuild-composer-api.socket, but provide a symlink for
backwards compatibility. Change `schutzbot/provision.sh` to only enable
osbuild-composer-api.socket.
In the future, this new socket is the only API socket, which provides
both the "cloud" API and the one for koji.
This means that the koji API is always enabled.
s3cmd from pip doesn't work with Python 3.9. Fortunately, s3cmd in Fedora has
a downstream patch to fix it.
See rhbz#1884607 and https://github.com/s3tools/s3cmd/pull/1137
In case the job fails completely, resources might leak. This patch
together with the one introducing predictable names for CI artifacts
will make sure we wipe all resources after each CI run.
The -test subpackage should depend on all that it needs, and we should
never need to install anything explicitly from the test scripts.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Deployment is specific to CI, but the provision script is needed to be
run to execute tests locally too.
This should be called from each test, rather than from deploy. For now
this change is a noop.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Our snapshots are currently broken, so until they are fixed revert back
to using an internal RHEL nightly. We choose a slightly older one,
in the hope that it is not going to be deleted anytime soon.
Signed-off-by: Tom Gundersen <teg@jklm.no>
No tests should be run directly from git, but should rather be installed
onto the test system using rpm and run from there. This moves towards
unifying our two types of test cases.
The new structure of is now:
`test/cmd`: the executors, one for each test-case. This is installed
into `/usr/libexec/test/osbuild-composer`.
`test/data`: data and config used by the tests. This is installed into
`/usr/share/tests/osbuild-composer`.
`schutzbot`: configuration of the actual test run. In particular, this
is where the distros and repositories to test against are
configured.
This is very much still work-in-progress, and is only the first step
towards simplifying schutzbot. Apart from moving files around, this
should be a noop.
Signed-off-by: Tom Gundersen <teg@jklm.no>
