When the deploy-qemu script is run with less than 2 arguments, it ended
with error, instead of printing usage. This was due to using 'set -u' and
trying to expand unset variables "$1" and "$2" as part of checking if
they were provided. The issue has been fixed by checking number of
provided arguments, instead of their content. The same approach is used
in 'deploy-openstack' script.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
koji and ansible are not in RHEL repositories. Depending on them breaks RHEL
gating (see OSCI-1541): It tries to build a custom image with -tests package
in it but in the build environment there's no EPEL.
This commit makes the RPM independent from EPEL. However, we still need koji
and ansible, so the provision script now enables EPEL and installs the packages
from there. This is not nice but we have to live with that until OSCI-1541 is
solved.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
All tests in /usr/libexec/tests/osbuild-composer should be able to run without
any arguments. This was not a case of libvirt.sh - it required two arguments
set by some Jenkinsfile logic.
This commit moves test/cases/libvirt.sh to tools/libvirt_test.sh and extracts
the logic controlling the test case from Jenkinsfile to test/cases/libvirt.sh.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This is a developer tool. Allowing setting QEMU_EXTRA_ARGS so that
developers can add arguments that make sense on their machines and for
their workflows.
Just a few tweaks were necessary to add support for macOS:
1. /usr/bin/bash → /bin/bash, which is a link on Linux as well.
2. Use hdiutil instead of genisoimage to make the cloud-init iso.
3. Ask qemu to fall back to macOS' hypervisor hvf.
Instead, append `write_files: <JSON>` to the end of the file. This
works, because JSON is valid YAML.
For two reasons:
1. The generated user-data was hard to read, because python3-pyyaml
outputs weird syntax. Keeping the file as written makes it easier to
recognize when debugging an issue.
2. The tool now only depends on modules that python3 ships, making it
easier to run on a pristine system.
rhel 8.4 tests are added. The configs are based off of those used for
rhel 8.3. The Schutzbot Mockbuild, Base, Image, Integration, and OSTree
tests are added for 8.4. Repo overrides are added for the rhel 8.4 tests
so that the tests use rpmrepo snapshots.
The mockbuild uses the jenkins rhel84-nightly-repo credential to
override the rhel mock template's repos with rhel 8.4 nightly repos.
These repos are stored in a credential because they are internal links.
The image tests and koji tests need a special distro selector since the
rhel-8 test cases are only for rhel 8 versions less than 8.4. The rhel
8.4 tests are named with the rhel-84 pattern whereas the other rhel 8
versions have the rhel-8 pattern.
Also, instead of having only rhel-8 and rhel-8-beta repo configs for the
tests, we now have a specific repo config for each rhel release we test.
The repo is also now pulled from an rpmrepo snapshot. For whichever
distro is being tested, the approriate repo config will be copied to
/etc/osbuild-composer/repositories as rhel-8 and rhel-8-beta since this
is the naming osbuild-composer looks for. For testing purposes, the
rhel-8 and rhel-8-beta repo should be the same since eventually all rhel
releases will go from beta to not beta. The fedora repo overrides are
already done in tools/provision.sh so the rhel override is set there as
well. Currently, only rhel 8.4 requires an override.
cockpit-composer can now build rhel 8.4 images. Our distro name for
rhel 8.4 is rhel-84 unlike prior rhel releases which fall
under the umbrella name rhel-8. rhel 8.4 still uses the same
repos as the rest of the rhel 8 releases but points to a different
nightly repo for testing purposes. Test cases are added. The changes
between rhel 8.3 and 8.4 are as follows:
There is now a hybrid boot partition scheme for x86_64. x86_64 images
now use uefi boot and have 3 gpt partitions: a small unformated
partition for mbr compatibility, an efi boot partition of type vfat, and
a root partition of type xfs. The packages grub2-efi-x64 and shim-x64
are added as bootloader packages for all x86_64 images.
For qcow2 images ro is added as a kernel option and the following
packages are added (+) or removed (-):
+ dosfstools
+ efi-filesystem
+ efivar
+ efivar-libs
+ grub2-efi-x64
+ shim-x64
- rhn-client-tools
- rhnlib
- rhnsd
- rhn-setup
This removes the restriction of only having a single build per compose
and uses the new job types to schedule the broken-appart build.
A small change in behavior is introduced: the koji build ID is not
known when the call to `compose` returns, so it is always set to
`0`. In the future we should remove this from the API, and instead
rely on the status call to return this information, when it is
known.
The status route will be updated in follow-up commits to reflect the
changes introduced here.
This commit does several things:
1) Changes the Fedora 33 repos in the test case generator from development
to release ones.
2) Fixes format-request-map.json so we can generate fedora-iot-commit
"images".
3) Regenerates all the cases.
Add an API route that returns logs for a specific compose.
For now, this contains the result of the job, in JSON. The idea is to
put more and more of this information into structured APIs. This is a
first step to make logs available at all.
Amend koji-compose.py to check that the route exist and contains as many
"image_logs" as images that were requested (currently always 1).
Based on a patch by Chloe Kaubisch <chloe.kaubisch@gmail.com>.
We have several repository definitions across the tests which is quite messy.
This commit switches the Koji test to use the "central" repository configs defined in test/data/repositories/
docker.io has recently introduced a rate limiting on container pulls causing
the koji test to fail quite often.
To fix this issue, I created our own postgress mirror[1]. This commit switches
the Koji test to use it. Note that this change bumps the postgres version from
12 to 13.
[1]: 7db3c6802e
The certificate generation is based on work by Lars Karlitski in our osbuild
CA. The server and client certs now contains Subject Alternative Name making
Python's request module and Go 1.15 happy (they deprecated certificates
without SAN).
Several reasons why we want to switch to the certificate generation:
1) The pre-generated certificates are not documented. If someone wants
to inspect them, he must know the right openssl incantation. This way,
you are able to see what's inside the certificates in a plain text.
2) The pre-generated certificates are going to expire at one point and
someone will be surprised.
3) Shipping private keys in RPMs is iffy. I know, it's just for testing but
still...
4) Auth tests are generating their own certificates. To achieve consistency,
we have two options:
a) Ship also all certificates for auth tests. That's extra 8 ones or
something like that.
b) Generate all certificates on fly. This commit does that.
5) The setup introduced by this commit is very similar to the one in our CA
making the test environment very similar to what's running in production.
tl;dr: I think this is a good step forward.
The test package should be self-contained and contain all the configuration
necessary for a known-good test run (minus secrets). This moves repo
overrides from the test orchestration into the test package.
We want all the external sources (including boot isos and repositories) to
be pinned by their content hash, and never use anything that is not strictly
defined. This moves us in the right direction, but we still have some tests
to update to use these shipped repos rather than official mirrors.
One remaining challenge is that we must make sure all our test runners have
access to the RHEL snapshots, which is not currently the case for the runners
in EC2, but a solution is in the works.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We no longer release into F31, and the right specfile was anyway not
being tested.
This allows us to remove a workaround that updates the VMs during
deploy, and other fedora-31 specific hacks.
Base these on https://osbuild.org/rpmrepo.
Install those repository files when deploying in schutzbot and include
them in tools/deploy/test for local testing.
Also remove copying overrides from deploy.sh, because it copied the
wrong files. We probably want to add this back properly in the future.
tools/gen-user-data generates a cloud-init user-data file from a
configuration directory. It is mostly useful to embed files in the
user-data.
tools/deploy-qemu uses above tool to make a user-data file and spins up
a virtual machine with it. This is useful to locally run, test, and
debug osbuild-composer.
A simple user-data directory for running tests locally is included in
tests/deploy-user-data. It expects a repository with osbuild-composer
rpms to be served on the host's port 8000.
Also install it is part of he tests subpackage. This a helper-tool, not
golang code, so should not live in `internal`. We need access to this
from the integration tests, so install it onto the tests system.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Fedora 33 images can now be built and test cases are added for the new
images. The fedora 33 qcow2 and vmdk images are based off of the
official images and their kickstarters found here:
https://pagure.io/fedora-kickstarts. The fedora 33 iot image is based
off of the the config found here: https://pagure.io/fedora-iot/ostree.
The openstack, azure, and amazon image types have changes made to them
based off of the changes made to the qcow2. The changes between fedora
32 and fedora 33 are as follows:
Grub now loads its kernel command line options from
etc/kernel/cmdline, /usr/lib/kernel/cmdline, and /proc/cmdline instead
of from grub env. This is addressed by adding kernelCmdlineStageOptions
to use osbuild's kernel-cmdline stage to set these options. Alongside
`ro biosdevname=0 net.ifnames=0`, we also set `no_timer_check
console=tty1 console=ttyS0,115200n8` per what is set in the official
qcow2. For azure and amazon, the kernelOptions are still set as they
were in fedora 32.
The timezone is now set to UTC if a user does not set a timezone in the
blueprint customizations. Also, the hostname is set to
localhost.localdomain if the hostname isn't set in the blueprint.
Finally, the following packages have been removed:
polkit
geolite2-city
geolite2-country
zram-generator-defaults
We need this for greenboot-status, in the RHEL for Edge images. This
updates the generator for x86_64 and aarch64 and updates the test cases
for rhel-edge-commit.
Signed-off-by: Tom Gundersen <teg@jklm.no>
All our downstream platforms now support Go 1.13:
RHEL 8.2: golang-1.13.4
Fedora 31: golang-1.13.14
There's no reason anymore to stay on 1.12, therefore this commit bumps
the minimal required Go version to 1.13
The conversion with `qemu-img convert` often fails on aarch64 systems
with LOTS of CPUs. This is fixed in RHEL 8 for aarch64, but not in
Fedora.
Set the maximum coroutines to 1 to avoid this issue until the bug is
fixed.
Bug: https://bugs.launchpad.net/qemu/+bug/1805256
Signed-off-by: Major Hayden <major@redhat.com>
This patch contains refactoring of the part where we handle
customizations. Previously customizations were considered an "image
type" which is not obvious. Now it is a command line switch.
Add Fedora 32 test cases with the base OS + updates (to work around
aarch64 bugs in the original release). Add updates for Fedora 31 for
consistency.
Fixes#861.
Signed-off-by: Major Hayden <major@redhat.com>
Add support for partitions that can 'blkid' can not handle, like
the bios boot partition. For such partitions, 'blkid' will fail;
in that case all additional information will just be 'None'.
Convert any image that is not a raw image, e.g. a qcow2, to a raw
image and open the partitions via loop devices. This replaces the
usage of nbd, which was racy and flaky.
Instead on relying on the kernel for the partition parsing, this
is now done manually via loop devices and start + offset taken
from sfdisk. As a result the read_partition function has been
adapted to be called at later time, after the partitions were
opened via loop devices.
Not using nbd also means that the partition table is not scanned
by the kernel anymore and udev is not triggered. As a result the
'PARTUUID' property is not present for dos/mbr partition layouts,
since it is auto-generated by udev/blkid. Relevant blkid files
and functions are:
blkid_partition_gen_uuid(par)
called from probe_dos_pt()
in file libblkid/src/partitions/dos.c line 295
defined in libblkid/src/partitions/partitions.c line 1374
which generates the uuid via snprintf using the format:
'"%.33s-%02x", par->tab->id, par->partno'
Based on https://github.com/karelzak/util-linux at ce8985cc7
NB: the loop device code is imported from osbuild, making this
tool depend on osbuild's private library.
NB: As of the image conversion, more disk space is required to
examine non-raw images.
If a compressed file is detected (xz, bzip2, gzip), uncompress it
before analyzing it. NB: this has to happen after the is_tarball
check, because a tarball would also count as compressed.
