1. Test retrieving the job arguments from the worker using an
OSBuildJob.
Very basic test with an empty manifest.
2. Test retrieving job manifests through the Koji API. Uses the existing
TestCompose test. The returned manifests are empty for all cases.
Add and implement an API endpoint that returns the manifests for a given
osbuild-koji job. The route returns an array of manifests, one for each
image in the request.
Retrieves the arguments of each Build Job to extract the Manifest.
Wraps jobqueue.JobArgs() method and unmarshals data into the provided
concrete struct value. The struct should match the requested job type of
the given ID.
JobArgs() returns the arguments submitted with a job in raw form.
Since the structure of the args are opaque to job queue, it's the
responsibility of the caller to deserialize the arguments.
Args retrieval is added to the existing TestArgs() function.
With our limited number of machines in OpenStack, we cannot run unlimited jobs
in parallel. When there's a lot of activity in the repository, there's a high
probability that a job must wait for a free VM for quite a long. In these cases,
tests commonly fail due to the timeout.
I decided to prolong the timeout to 12 hours. This should lower the amount of
these kinds of jobs.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add the latest information about the intended use of this type of
image. Also add note about the most common way to consume Red Hat
content using this type of image.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The majority of pull requests do not fix a downstream issue. In
practice, people usually delete the whole suggested content.
Move it into a comment, so that this step is unnecessary.
RHEl 8.4 guest images need to have the default timezone of EST/EDT
unless the user specifies one in their blueprint. New York is a major
location for this timezone.
The directory with image-tests test cases has been renamed from `cases`
to `manifests`. This has not been previously reflected in the test/README.md
and osbuild-image-tests code. osbuild-image-tests hardcodes the test
cases directory path and uses it in case no test case are passed
to it on the command line. Since the image_tests.sh CI test case looks
for image-tests test cases in the correct directory and passes the
relevant ones to osbuild-image-tests, the CI didn't detect this issue.
Running osbuild-images-tests without any argument and let it run all
test cases from the default test cases directory as part of CI probably
does not make sense. Due to this reason, I'm not adding any new test.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
testjobqueue did not implement the JobQueue interface correctly (noted
in its package comment), making it impossible to write tests for
JobQueue itself.
Replace its use everywhere with fsjobqueue operating on a temporary
directory.
When the deploy-qemu script is run with less than 2 arguments, it ended
with error, instead of printing usage. This was due to using 'set -u' and
trying to expand unset variables "$1" and "$2" as part of checking if
they were provided. The issue has been fixed by checking number of
provided arguments, instead of their content. The same approach is used
in 'deploy-openstack' script.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When rpmmd's Depsolve function is called we need to pass in the image
type's excluded packages. These excluded packages are retrieved when we
get the packages we include from each image type.
Fedora 33 and rawhide got an updated version of the azblob library. Sadly, it
introduced a non-compatible API change. This commit does the same thing as
a67baf5a did for kolo/xmlrpc:
We now have two wrappers around the affected part of the API. Fedora 32 uses
the wrapper around the old API, whereas Fedora 33 and 34 (and RHEL with its
vendored deps) use the wrapper around the new API. The switch is implemented
using go build flags and spec file magic.
See a67baf5a for more thoughts.
Also, there's v0.11.1-0.20201209121048-6df5d9af221d in go.mod, why?
The maintainers of azblob probably tagged a wrong commit with v0.12.0 which
breaks go. The long v0.11.1-.* version is basically the proper v0.12.0 commit.
See https://github.com/Azure/azure-storage-blob-go/issues/236 for more
information.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This changes the following:
- Only produce one container. There is no particular benefit to
supporting several different base containers, so unify on ubi
as that is what we need ourselves.
- Build directly from git. Now that the RPM we include in our
container does not have any dependencies and only contains a
couple of executables, the indirection via RPM has less value.
Eventually the value will be reduced even further as we merge
the entrypoint into the main binary and move dnf-json into the
worker, leaving us with only a go binary. The only potential
benefit might be that the build environment of RPMs is more
clearly defined, but there is no real reason to believe that
our mockbuild is any better than using the UBI golang build
container.
This simplifies the container builds, and brings us more in line
with what is done in image-builder, and what is needed to deploy
to openshift.
We do not yet build all our packages for aarch64, so this would
fall back on the RPMs in the repository. In `main` that seems to
work, as osbuild-composer exists in the base repositoires. Though
we obviously want to test the most recent commit, not the released
RPM, so this is maksing the problem.
As of this PR, the build would fail though: We are now build
UBI containers, where osbuild-composer does not exist, and we have
split the RPM into the new osbuild-composer-core, which does not
yet exist in any base repository.
This no longer pulls in systemd/worker, saving space and makes it
suitable for use in a UBI container, where qemu-img is not available.
This drops support for --inbuilt-worker from entrypoint.py. The script
could be simplified further in a future commit, or folded into the
main binary.
koji and ansible are not in RHEL repositories. Depending on them breaks RHEL
gating (see OSCI-1541): It tries to build a custom image with -tests package
in it but in the build environment there's no EPEL.
This commit makes the RPM independent from EPEL. However, we still need koji
and ansible, so the provision script now enables EPEL and installs the packages
from there. This is not nice but we have to live with that until OSCI-1541 is
solved.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
koji-osbuild-tests depends on koji which is not available in RHEL. As we need
to get rid of EPEL from deploy.sh (see the following commit), we need a
mechanism to preinstall EPEL before koji-osbuild-tests is installed. This
commit introduces pre_install_packages to Schutzfile - a simple way to
install packages before ${PROJECT}-tests is installed.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add a Dockerfile that creates a container based on Fedora with
osbuild-composer deployed. Create a suitable entrypoint that runs
osbuild-composer in the container and creates the required sockets
without systemd.
To test this, build the container via:
docker build ./containers/osbuild-composer
Then create your certificates in /etc/osbuild-composer/. Then run
composer with something like:
docker run --rm -v /etc/osbuild-composer:/etc/osbuild-composer <id>
(Where <id> is the container ID returned by `docker build`.)
I thought rand in Go is auto-seeded but I was wrong, see [1].
This commit adds seed initialization.
[1]: https://golang.org/pkg/math/rand/#Seed
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Imagine this situation: You have a RHEL system booted from an image produced
by osbuild-composer. On this system, you want to use osbuild-composer to
create another image of RHEL.
However, there's currently something funny with partitions:
All RHEL images built by osbuild-composer contain a root xfs partition. The
interesting bit is that they all share the same xfs partition UUID. This might
sound like a good thing for reproducibility but it has a quirk.
The issue appears when osbuild runs the qemu assembler: it needs to mount all
partitions of the future image to copy the OS tree into it.
Imagine that osbuild-composer is running on a system booted from an imaged
produced by osbuild-composer. This means that its root xfs partition has this
uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
When osbuild-composer builds an image on this system, it runs osbuild that
runs the qemu assembler at some point. As I said previously, it will mount
all partitions of the future image. That means that it will also try to
mount the root xfs partition with this uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
Do you remember this one? Yeah, it's the same one as before. However, the xfs
kernel driver doesn't like that. It contains a global table[1] of all xfs
partitions that forbids to mount 2 xfs partitions with the same uuid.
I mean... uuids are meant to be unique, right?
This commit changes the way we build RHEL 8.4 images: Each one now has a
unique uuid. It's now literally a unique universally unique identifier. haha
[1]: a349e4c659/fs/xfs/xfs_mount.c (L51)
Now that we have an abstract partition table definition, we can use it to
generate org.osbuild.fstab stage options.
This is extremely nice because it removes magic contains.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Using osbuild.QEMUAssemblerOptions to encode a partition table was weird.
This commit introduces a disk package that contains data types for defining
partition tables. Also, there's a handy function to convert the abstact
partition table to osbuild.QEMUAssemblerOptions.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the partition table definition was defined inside an assembler.
This has an issue though: The partitions and filesystems are needed at several
other places, e.g. grub2 stage and fstab stage. As the partition table was
basically hardcoded, this didn't matter - we could just use constants
in these stages. Not ideal but it worked.
This commit changes the behaviour: A partition table is firstly created and
then it's passed to the assembler function where complete assembler options
are created out of it.
To make this change as small as possible, osbuild.QEMUAssemblerOptions type
is used to encode the partition table for now.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
require.JSONEqf cannot handle diffs of such a big entity as a manifest is.
It just prints an empty string.
This commit unmarshalls the manifests instead and then uses the cmp library
to make a very nice and readable diff.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The kernel options are updated to remove the read only option "ro" from
the qcow2 and amazon image types. Also, the qcow2's kernel options are
updated to only set console=ttyS0 once. It was declared twice which is
redundant so now it is set for both tty0 and ttyS0.
Build the osbuild-composer(7) man-page as part of the RPM build and
include it in the osbuild-composer RPM. Previously the man-page was not
shipped at all in any of the produced RPMs.
make is currently included in the Fedora buildroot, however there is a
plan to remove it since F34. Since make is now used in the %build
section, it should be listed explicitly as a BuildRequires.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
rng-tools is no longer included as a package in the RHEL 8.4 image. This
package is both removed from being an included package and also
specifically declared as excluded. The test manifests are updated.
The RHEL 8.4 specific runner was introduced in osbuild 22, released
on the 8th of October 2020. It should by now be in relevant Fedora
releases and RHEL 8.4.
Adapt the corresponding test cases.
64432c70 promised to remove it but this didn't actually happen. We don't
use the submodule anymore so let's indeed drop it this time.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the compose status returned failure as soon as possible.
koji-osbuild considers the job as done when its status == failure and proceeds
with uploading the logs to koji and marking the job as failed. However, not
all osbuild-composer jobs might be done at this point so the logs might be
incomplete making the debugging hard.
This commit changes the behaviour: Now, the compose status is pending until
ALL jobs belonging to it are finished.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, we had no clue what errors were catched by the default echo's
error handler. Thus, in the case of an error, we were basically blind. Let's
log all errors so we can investigate them later.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
All tests in /usr/libexec/tests/osbuild-composer should be able to run without
any arguments. This was not a case of libvirt.sh - it required two arguments
set by some Jenkinsfile logic.
This commit moves test/cases/libvirt.sh to tools/libvirt_test.sh and extracts
the logic controlling the test case from Jenkinsfile to test/cases/libvirt.sh.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
