This no longer pulls in systemd/worker, saving space and makes it
suitable for use in a UBI container, where qemu-img is not available.
This drops support for --inbuilt-worker from entrypoint.py. The script
could be simplified further in a future commit, or folded into the
main binary.
koji and ansible are not in RHEL repositories. Depending on them breaks RHEL
gating (see OSCI-1541): It tries to build a custom image with -tests package
in it but in the build environment there's no EPEL.
This commit makes the RPM independent from EPEL. However, we still need koji
and ansible, so the provision script now enables EPEL and installs the packages
from there. This is not nice but we have to live with that until OSCI-1541 is
solved.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
koji-osbuild-tests depends on koji which is not available in RHEL. As we need
to get rid of EPEL from deploy.sh (see the following commit), we need a
mechanism to preinstall EPEL before koji-osbuild-tests is installed. This
commit introduces pre_install_packages to Schutzfile - a simple way to
install packages before ${PROJECT}-tests is installed.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add a Dockerfile that creates a container based on Fedora with
osbuild-composer deployed. Create a suitable entrypoint that runs
osbuild-composer in the container and creates the required sockets
without systemd.
To test this, build the container via:
docker build ./containers/osbuild-composer
Then create your certificates in /etc/osbuild-composer/. Then run
composer with something like:
docker run --rm -v /etc/osbuild-composer:/etc/osbuild-composer <id>
(Where <id> is the container ID returned by `docker build`.)
I thought rand in Go is auto-seeded but I was wrong, see [1].
This commit adds seed initialization.
[1]: https://golang.org/pkg/math/rand/#Seed
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Imagine this situation: You have a RHEL system booted from an image produced
by osbuild-composer. On this system, you want to use osbuild-composer to
create another image of RHEL.
However, there's currently something funny with partitions:
All RHEL images built by osbuild-composer contain a root xfs partition. The
interesting bit is that they all share the same xfs partition UUID. This might
sound like a good thing for reproducibility but it has a quirk.
The issue appears when osbuild runs the qemu assembler: it needs to mount all
partitions of the future image to copy the OS tree into it.
Imagine that osbuild-composer is running on a system booted from an imaged
produced by osbuild-composer. This means that its root xfs partition has this
uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
When osbuild-composer builds an image on this system, it runs osbuild that
runs the qemu assembler at some point. As I said previously, it will mount
all partitions of the future image. That means that it will also try to
mount the root xfs partition with this uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
Do you remember this one? Yeah, it's the same one as before. However, the xfs
kernel driver doesn't like that. It contains a global table[1] of all xfs
partitions that forbids to mount 2 xfs partitions with the same uuid.
I mean... uuids are meant to be unique, right?
This commit changes the way we build RHEL 8.4 images: Each one now has a
unique uuid. It's now literally a unique universally unique identifier. haha
[1]: a349e4c659/fs/xfs/xfs_mount.c (L51)
Now that we have an abstract partition table definition, we can use it to
generate org.osbuild.fstab stage options.
This is extremely nice because it removes magic contains.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Using osbuild.QEMUAssemblerOptions to encode a partition table was weird.
This commit introduces a disk package that contains data types for defining
partition tables. Also, there's a handy function to convert the abstact
partition table to osbuild.QEMUAssemblerOptions.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the partition table definition was defined inside an assembler.
This has an issue though: The partitions and filesystems are needed at several
other places, e.g. grub2 stage and fstab stage. As the partition table was
basically hardcoded, this didn't matter - we could just use constants
in these stages. Not ideal but it worked.
This commit changes the behaviour: A partition table is firstly created and
then it's passed to the assembler function where complete assembler options
are created out of it.
To make this change as small as possible, osbuild.QEMUAssemblerOptions type
is used to encode the partition table for now.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
require.JSONEqf cannot handle diffs of such a big entity as a manifest is.
It just prints an empty string.
This commit unmarshalls the manifests instead and then uses the cmp library
to make a very nice and readable diff.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The kernel options are updated to remove the read only option "ro" from
the qcow2 and amazon image types. Also, the qcow2's kernel options are
updated to only set console=ttyS0 once. It was declared twice which is
redundant so now it is set for both tty0 and ttyS0.
Build the osbuild-composer(7) man-page as part of the RPM build and
include it in the osbuild-composer RPM. Previously the man-page was not
shipped at all in any of the produced RPMs.
make is currently included in the Fedora buildroot, however there is a
plan to remove it since F34. Since make is now used in the %build
section, it should be listed explicitly as a BuildRequires.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
rng-tools is no longer included as a package in the RHEL 8.4 image. This
package is both removed from being an included package and also
specifically declared as excluded. The test manifests are updated.
The RHEL 8.4 specific runner was introduced in osbuild 22, released
on the 8th of October 2020. It should by now be in relevant Fedora
releases and RHEL 8.4.
Adapt the corresponding test cases.
64432c70 promised to remove it but this didn't actually happen. We don't
use the submodule anymore so let's indeed drop it this time.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the compose status returned failure as soon as possible.
koji-osbuild considers the job as done when its status == failure and proceeds
with uploading the logs to koji and marking the job as failed. However, not
all osbuild-composer jobs might be done at this point so the logs might be
incomplete making the debugging hard.
This commit changes the behaviour: Now, the compose status is pending until
ALL jobs belonging to it are finished.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, we had no clue what errors were catched by the default echo's
error handler. Thus, in the case of an error, we were basically blind. Let's
log all errors so we can investigate them later.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
All tests in /usr/libexec/tests/osbuild-composer should be able to run without
any arguments. This was not a case of libvirt.sh - it required two arguments
set by some Jenkinsfile logic.
This commit moves test/cases/libvirt.sh to tools/libvirt_test.sh and extracts
the logic controlling the test case from Jenkinsfile to test/cases/libvirt.sh.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit enables:
- mockbuild for F33 aarch64
- base tests for F33 aarch64
- ami image test for F33 aarch64
This is mostly a MVP - enabling more tests would require more work than I have
a capacity for right now. Still, it gives us at least some coverage for
alt-arches. Most importantly, base tests build qcow2 and image tests build ami
so we have at least some trust that image building for aarch64 works.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
%#v was my bad understanding of Go's error formatting. Let's use the standard
%v that gives saner and human-readable error messages.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Test cases shouldn't be smart - if kvm is not available and the boot test
is still run, the test should indeed fail. It's up the test runner to decide
whether the test should be run. (It's currently not, so this is not
a functional change).
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Our current testing environment cannot run VMs for aarch64, therefore we
cannot test the qcow2 image. Also, we don't have an openstack account with
aarch64 so also we cannot test openstack image type. Additionaly, we cannot
selectively disable boot testing for now.
Let's do the smallest change possible and just add ami test case - we can boot
test these images in AWS.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Hybrid boot, i.e. MBR+UEFI, is only supported on x86_64, but not
on aarch64, which only supports UEFI. Therefore, disable legacy,
i.e. MBR, boot for aarch64 and remove the corresponding BIOS boot
partition.
Since hybrid boot is always used on x86_64, the "uefi" parameter
is removed completely, together with the "legacy only" option on
x86_64. Thus we are left with a simple one to one mapping of
architecture to qemu assembler stage options in `qemuAssembler`.
22c9f6a introduced a new environment variable to api.sh,
AWS_API_TEST_SHARE_ACCOUNT, but only set it in the RHEL 8 tests.
Seems like the corresponding branch was not rebased to a version which
already had those 8.4 tests:
https://github.com/osbuild/osbuild-composer/pull/1098
Previously, baseurl was required in openapi.yaml. In order to add support
for metalink and mirrorlist repos as well, make all optional, since openapi
does not support mutually exclusive parameters. Instead, enforce this logic
in server.go, and if no repo has been specified, return a 400 bad request error.
Previously, the checks that dependencies were successful were all over the
Run() method. This led to a issue #1101 (lovely binary number btw).
This commit rewrites the Run() method to:
1) Extract dynamic args. Return an error if they cannot be unmarshalled.
2) Check if dependencies were successful. If not, call kojiFail, update the
job and return.
3) Create the CGImport metadata and call kojiImport.
Fixes#1101
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
test.API is actually just http.Handler, let's use this interface directly
instead of defining our own one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
These packages are architecture-specific. Including them in the generic
package list broke image builds for everything but x86_64.
The corresponding architecture-dependant packages were already included
in `arch.bootloaderPackages` (some via dependencies).
Fixes#1103
osbuild output should be always JSON-parsable. However, if a user has a weird
installation of osbuild, it can return malformed JSON. In this case, it's
very hard to debug what happened because the worker doesn't provide any
useful information. This commit adds the non-parsable osbuild output to the
error so there's at least some clue in the logs what happened.
Let me reiterate: In 99% these cases, this shouldn't be needed. It will give
us some hints in these 1% cases though.
You may want to ask if using json.Decoder isn't better because it doesn't
buffer the input. However, it does buffer. Firstly, json.Decoder.Decode()
buffers the whole JSON value (it contains kinda cool state machine to do it)
and in the second run, it parses the buffer. Therefore, the only thing that
this commit actually does is that it moves the buffer out of json.Decoder.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This is a developer tool. Allowing setting QEMU_EXTRA_ARGS so that
developers can add arguments that make sense on their machines and for
their workflows.
Just a few tweaks were necessary to add support for macOS:
1. /usr/bin/bash → /bin/bash, which is a link on Linux as well.
2. Use hdiutil instead of genisoimage to make the cloud-init iso.
3. Ask qemu to fall back to macOS' hypervisor hvf.
Instead, append `write_files: <JSON>` to the end of the file. This
works, because JSON is valid YAML.
For two reasons:
1. The generated user-data was hard to read, because python3-pyyaml
outputs weird syntax. Keeping the file as written makes it easier to
recognize when debugging an issue.
2. The tool now only depends on modules that python3 ships, making it
easier to run on a pristine system.
