%#v was my bad understanding of Go's error formatting. Let's use the standard
%v that gives saner and human-readable error messages.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Test cases shouldn't be smart - if kvm is not available and the boot test
is still run, the test should indeed fail. It's up the test runner to decide
whether the test should be run. (It's currently not, so this is not
a functional change).
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Our current testing environment cannot run VMs for aarch64, therefore we
cannot test the qcow2 image. Also, we don't have an openstack account with
aarch64 so also we cannot test openstack image type. Additionaly, we cannot
selectively disable boot testing for now.
Let's do the smallest change possible and just add ami test case - we can boot
test these images in AWS.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Hybrid boot, i.e. MBR+UEFI, is only supported on x86_64, but not
on aarch64, which only supports UEFI. Therefore, disable legacy,
i.e. MBR, boot for aarch64 and remove the corresponding BIOS boot
partition.
Since hybrid boot is always used on x86_64, the "uefi" parameter
is removed completely, together with the "legacy only" option on
x86_64. Thus we are left with a simple one to one mapping of
architecture to qemu assembler stage options in `qemuAssembler`.
22c9f6a introduced a new environment variable to api.sh,
AWS_API_TEST_SHARE_ACCOUNT, but only set it in the RHEL 8 tests.
Seems like the corresponding branch was not rebased to a version which
already had those 8.4 tests:
https://github.com/osbuild/osbuild-composer/pull/1098
Previously, baseurl was required in openapi.yaml. In order to add support
for metalink and mirrorlist repos as well, make all optional, since openapi
does not support mutually exclusive parameters. Instead, enforce this logic
in server.go, and if no repo has been specified, return a 400 bad request error.
Previously, the checks that dependencies were successful were all over the
Run() method. This led to a issue #1101 (lovely binary number btw).
This commit rewrites the Run() method to:
1) Extract dynamic args. Return an error if they cannot be unmarshalled.
2) Check if dependencies were successful. If not, call kojiFail, update the
job and return.
3) Create the CGImport metadata and call kojiImport.
Fixes#1101
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
test.API is actually just http.Handler, let's use this interface directly
instead of defining our own one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
These packages are architecture-specific. Including them in the generic
package list broke image builds for everything but x86_64.
The corresponding architecture-dependant packages were already included
in `arch.bootloaderPackages` (some via dependencies).
Fixes#1103
osbuild output should be always JSON-parsable. However, if a user has a weird
installation of osbuild, it can return malformed JSON. In this case, it's
very hard to debug what happened because the worker doesn't provide any
useful information. This commit adds the non-parsable osbuild output to the
error so there's at least some clue in the logs what happened.
Let me reiterate: In 99% these cases, this shouldn't be needed. It will give
us some hints in these 1% cases though.
You may want to ask if using json.Decoder isn't better because it doesn't
buffer the input. However, it does buffer. Firstly, json.Decoder.Decode()
buffers the whole JSON value (it contains kinda cool state machine to do it)
and in the second run, it parses the buffer. Therefore, the only thing that
this commit actually does is that it moves the buffer out of json.Decoder.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This is a developer tool. Allowing setting QEMU_EXTRA_ARGS so that
developers can add arguments that make sense on their machines and for
their workflows.
Just a few tweaks were necessary to add support for macOS:
1. /usr/bin/bash → /bin/bash, which is a link on Linux as well.
2. Use hdiutil instead of genisoimage to make the cloud-init iso.
3. Ask qemu to fall back to macOS' hypervisor hvf.
Instead, append `write_files: <JSON>` to the end of the file. This
works, because JSON is valid YAML.
For two reasons:
1. The generated user-data was hard to read, because python3-pyyaml
outputs weird syntax. Keeping the file as written makes it easier to
recognize when debugging an issue.
2. The tool now only depends on modules that python3 ships, making it
easier to run on a pristine system.
Generalize deploy.sh to allow setting up reverse dependency tests. Those
need the same setup for osbuild and osbuild-composer (i.e., the ones
under test), and in addition need the -tests package of the dependant
project.
Add test runs for koji-osbuild for all operating systems in Jenkinsfile.
Use a commit in koji-osbuild for which a repository was pushed to s3 in
the right format.
heredocs make control flow hard to read. Pull writing the repo file into
its own function and use it for setting up osbuild and osbuild-composer
repos.
The previous code was smelling a bit (e.g. Server.server field) so I decided
to rewrite it in the style of the much nicer koji server.
Not a functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
As the team obsessed with immutable test dependencies, how could we use
the ubuntu-latest VM which can change at any time?
Speaking of changes to ubuntu-latest... It will soon be updated from ubuntu
18.04 to 20.04 [1].
This commit switches our testing pipeline to use ubuntu-20.04 to:
1) make our test dependencies immutable (or at least slightly more immutable)
2) make us prepared for the ubuntu-latest changes.
[1]: https://github.com/actions/virtual-environments/issues/1816
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
x86_64 and aarch64 rhel 8.4 images support hybrid boot. qcow2 images are
now tested with both BIOS and UEFI boot. The Jenkinsfile now specifies
which boot type is supported when calling run_tests. For the qemu tests,
this boot type can be passed as an option so the test can boot using
UEFI.
In order to boot using uefi from a rhel machine, we must pass the --boot
uefi flag as well as tell libvirt where the variable store template is.
Since we are not testing secureboot, we must pass
nvram_template=/usr/share/edk2/ovmf/OVMF_VARS.fd to the --boot flag.
This is the location of our non-secureboot varstore template.
rhel 8.4 tests are added. The configs are based off of those used for
rhel 8.3. The Schutzbot Mockbuild, Base, Image, Integration, and OSTree
tests are added for 8.4. Repo overrides are added for the rhel 8.4 tests
so that the tests use rpmrepo snapshots.
The mockbuild uses the jenkins rhel84-nightly-repo credential to
override the rhel mock template's repos with rhel 8.4 nightly repos.
These repos are stored in a credential because they are internal links.
The image tests and koji tests need a special distro selector since the
rhel-8 test cases are only for rhel 8 versions less than 8.4. The rhel
8.4 tests are named with the rhel-84 pattern whereas the other rhel 8
versions have the rhel-8 pattern.
Also, instead of having only rhel-8 and rhel-8-beta repo configs for the
tests, we now have a specific repo config for each rhel release we test.
The repo is also now pulled from an rpmrepo snapshot. For whichever
distro is being tested, the approriate repo config will be copied to
/etc/osbuild-composer/repositories as rhel-8 and rhel-8-beta since this
is the naming osbuild-composer looks for. For testing purposes, the
rhel-8 and rhel-8-beta repo should be the same since eventually all rhel
releases will go from beta to not beta. The fedora repo overrides are
already done in tools/provision.sh so the rhel override is set there as
well. Currently, only rhel 8.4 requires an override.
cockpit-composer can now build rhel 8.4 images. Our distro name for
rhel 8.4 is rhel-84 unlike prior rhel releases which fall
under the umbrella name rhel-8. rhel 8.4 still uses the same
repos as the rest of the rhel 8 releases but points to a different
nightly repo for testing purposes. Test cases are added. The changes
between rhel 8.3 and 8.4 are as follows:
There is now a hybrid boot partition scheme for x86_64. x86_64 images
now use uefi boot and have 3 gpt partitions: a small unformated
partition for mbr compatibility, an efi boot partition of type vfat, and
a root partition of type xfs. The packages grub2-efi-x64 and shim-x64
are added as bootloader packages for all x86_64 images.
For qcow2 images ro is added as a kernel option and the following
packages are added (+) or removed (-):
+ dosfstools
+ efi-filesystem
+ efivar
+ efivar-libs
+ grub2-efi-x64
+ shim-x64
- rhn-client-tools
- rhnlib
- rhnsd
- rhn-setup
It should be totally possible to run osbuild-composer without the Weldr API
and local worker socket. This commit accomplishes this task!
In cloud environments where we don't want the Weldr API, it's now totally
possible to run osbuild-composer without it. If you also want to get rid of
the local worker, this command still works:
systemctl mask osbuild-worker@1.service
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Everybody hates the local workers. The first step of getting rid of them
is to split their socket out of osbuild-composer.socket - we need to keep
this one to support the Weldr API but the local worker socket can live in
its own file.
The behaviour should be the same for now: osbuild-composer.service always
starts the local worker socket.
However, this split allows the osbuild-composer executable to be run without
the Weldr API activated. The following commit explores this option more
in depth.
Note that the new socket can be used by root only because workers are always
run as root.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Not a functional change. I think the previous state was a result of code rotting
and a suboptimal rebase.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Running `make srpm` on the host can have different results from building
the SRPM in mock.
Stop using the Makefile altogether for this, which is meant for
developer convenience only. The interface is clear: a tarball generated
from `git archive` and defining the `commit` variable. Less stuff to
install on the host, too.
Prepending the definition to the spec file feels brittle. Stop doing
that – pass --define to rpmbuild instead.
Keep copying the spec file from git though, so that it's always using
the one from the last commit, and not the current checkout.
To make sure we test against what we will release with, use osbuild from
the target distro rather than rebuilding it.
The only exception is in RHEL8.3, where we are not rebasing any longer,
but expect users who use new osbulid-composer from upstream to use it
with new osbuild from upstream too. In this case, use the RPM from
osbuild upstream, for now pinned to the same version as was in the
submodule.
This introduces a new configuration file: Schutzfile, which is meant
to contain the full test-matrix. For now it only points to the upstream
osbuild commit to add to the distro we are testing against (only relevant
for 8.3).
The submodule is now unused and is therefore removed. The produced repos
now only contain osbuild-composer, osbuild is never built as part of
osbuild-composer CI.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Serializing an interface does not work, let us simply use the string
representation and treat the empty string as no error. This is
compatible with the current API in the success case, and fixes the
error case, which is currently broken.
Also extend the test matrix for the kojiapi to ensure that all the
different kinds of errors can be serialized correctly and leads to
the correct status being returned.
Fixes#1079 and #1080.
osbuild-composer doesn't actually require osbuild. osbuild-composer-worker
does. Let's remove the dependency from osbuild-composer and depend on the
right version of osbuild in the worker.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Soon, we want to begin tagging the jobs with the name of its submitter.
The simplest way to add a tag to a job is to put it into its type string.
However, as we don't know (and don't want to know) the submitters' names when
osbuild-composer is initialized, we need to be able to push arbitrary job
types into the jobqueue.
This commit therefore lifts the restriction that a jobqueue accepts only
a predefined set of job types. Now, jobqueue clients can push jobs of
arbitrary names.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
rpmlint errors are usually not a big deal but it's always nice when we can
catch them early. This commit introduces a new Github workflow that builds
an SRPM and runs rpmlint against it.
#1070 removed osbuild-composer-koji and introduced a new Obsoletes field.
However, rpmlint doesn't like unversioned obsoletes. This commit fixes that
by adding the last version of osbuild-composer that had the koji subpackage
to the Obsoletes field.
Fixes#1076
rpmlint doesn't like SRPMs built from specfiles with a filename different
than what it's in the Name field inside the spec. This commit removes
the renaming. Now, the specfile is always named osbuild-composer.spec.
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
