All our downstream platforms now support Go 1.13:
RHEL 8.2: golang-1.13.4
Fedora 31: golang-1.13.14
There's no reason anymore to stay on 1.12, therefore this commit bumps
the minimal required Go version to 1.13
The conversion with `qemu-img convert` often fails on aarch64 systems
with LOTS of CPUs. This is fixed in RHEL 8 for aarch64, but not in
Fedora.
Set the maximum coroutines to 1 to avoid this issue until the bug is
fixed.
Bug: https://bugs.launchpad.net/qemu/+bug/1805256
Signed-off-by: Major Hayden <major@redhat.com>
This patch contains refactoring of the part where we handle
customizations. Previously customizations were considered an "image
type" which is not obvious. Now it is a command line switch.
Add Fedora 32 test cases with the base OS + updates (to work around
aarch64 bugs in the original release). Add updates for Fedora 31 for
consistency.
Fixes#861.
Signed-off-by: Major Hayden <major@redhat.com>
Add support for partitions that can 'blkid' can not handle, like
the bios boot partition. For such partitions, 'blkid' will fail;
in that case all additional information will just be 'None'.
Convert any image that is not a raw image, e.g. a qcow2, to a raw
image and open the partitions via loop devices. This replaces the
usage of nbd, which was racy and flaky.
Instead on relying on the kernel for the partition parsing, this
is now done manually via loop devices and start + offset taken
from sfdisk. As a result the read_partition function has been
adapted to be called at later time, after the partitions were
opened via loop devices.
Not using nbd also means that the partition table is not scanned
by the kernel anymore and udev is not triggered. As a result the
'PARTUUID' property is not present for dos/mbr partition layouts,
since it is auto-generated by udev/blkid. Relevant blkid files
and functions are:
blkid_partition_gen_uuid(par)
called from probe_dos_pt()
in file libblkid/src/partitions/dos.c line 295
defined in libblkid/src/partitions/partitions.c line 1374
which generates the uuid via snprintf using the format:
'"%.33s-%02x", par->tab->id, par->partno'
Based on https://github.com/karelzak/util-linux at ce8985cc7
NB: the loop device code is imported from osbuild, making this
tool depend on osbuild's private library.
NB: As of the image conversion, more disk space is required to
examine non-raw images.
If a compressed file is detected (xz, bzip2, gzip), uncompress it
before analyzing it. NB: this has to happen after the is_tarball
check, because a tarball would also count as compressed.
Directly iterate over the partition list entries instead if using
indices. Optimize visual flow for the happy case by immediately
continuing if a partition has no file-system.
Read the image information before opening the image. This prepares
for a change where we will use information about the image to
convert it to raw instead of using nbd.
Although the vhdx format is smaller than raw, AWS often has some issues
booting instances from an imported vhdx snapshot. The raw format seems
to boot consistently at AWS.
Fixes#754.
Signed-off-by: Major Hayden <major@redhat.com>
Add one test-case per distro that tries to cover all the customizations
we support.
For now omit firewall customizations, as there are open questions about
how to support that on all image types.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When extracting the commit tarball as well as doing the ostree
commit checkout, use `/var/tmp` instead of `/tmp`, because the
latter might be backed by a tmpfs and thus by RAM; this should
only be used for small files.
The fact that we have two scripts and one captures output of the other
makes it hard to use in scenarios where the test case generation often
fails. This output capturing results in subtle bugs like described in
issue: https://github.com/osbuild/osbuild-composer/issues/769
This patch merge these two scripts and use class to organize it
instead of files.
This patch introduces support for s390x architecture. It has some
specific requirements like using kernel-cmdline stage before the RPM
stage because RPM scriptlets require these values to be present in the
system already. It also uses zipl stage and bootloader as opposed to all
the other architectures.
Also allow a list of them to be passed. Generating 'all' image types
makes less sense now that different distros/arches support different
ones.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Allow individual test-cases or sub-sets of test-cases to be generated
more easily.
We allow explicit skipping of image-info generation (and hence the
osbuild run), and also individual image types to be specified.
Also drop distros and image types that are no longer supported.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Support analyzing ostree repositories or directory that contain an
ostree commit as created by osbuild. Will return the mode of the
repository along with the references and their commits. For the
first references, the commit is resolved and checked-out to a temp
directory. This directory in turn is then analyzed via the existing
append_filesystem function. The latter has gained some small ostree
specific tweaks.
NB: for the ostree checkout /usr/etc/ is bind mounted to /etc in
order to make append_filesystem happy. The rpm verification step
is NOT run, because that is not really compatible with ostree.
Move the code of the current main into is own method and call that
from main. This prepares support for analyzing other types than
images. Additionally, add argument parsing via argparse to get a
help texts.
Allow a test case to be generated for a specific image type, rather than
for all at once.
This is useful when adding additional image types, rather than
regenerating the existing ones.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is a very minimal image type, which is not expected to do anything
with an empty blueprint. It is documented as a way to install RHEL for
Satellite.
This fixes#720.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Commit 8dd4554491 introduced a bug because
the output from osbuild is not captured. The problem is that osbuild
itself writes JSON document to STDOUT so two different JSON documents
are written to STDOUT: osbuild output and generated test case. The
generate-test-cases script then fails because it cannot json.loads() the
test case.
This patch simply supresses the output which fixes the issue.
As it turns out, the default expectation is not to distinguish between
these. We will now produce whatever is the most recent minor release by
default, and image tests will still be pinned at a given snapshot to be
reproducible.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Previously, vhd images were tested using QEMU. This commit changes that to
boot them in the actual Azure infrastructure.
Azure VMs have quite a lot of dependencies - a network interface, a virtual
network, a network security group, a public ip address and a disk. Azure CLI
and Azure Portal handle the creation of all these resources internally.
However, when using the API, the caller is responsible to create all these
resources before creating an actual VM.
To handle the creation of all the resources in the right order, a deployment
is used. A deployment is a set of resources defined in a JSON document.
It can optionally take parameters to customize each deployment. After the
deployment is finished, the VM is up and ready to be tested using SSH.
Sadly, the deployments are a bit hard to clean-up. One would expect that
deleting a deployment removes all the deployed resources. However, it doesn't
work this way and therefore it's needed to clean up all resources "manually".
For this reason, our deployment sets a unique tag on all the resources created
by the deployment. After this test is finished, the API is queried for all
the resources with the tag and then, they're deleted in the right order.
Requests to download.fedoraproject.org are redirected to a regional
mirror that is fairly close to the system that is making the request.
This could speed up downloads slightly since it avoids using the
Fedora proxies.
Signed-off-by: Major Hayden <major@redhat.com>
