Add "image_name" and "stream_optimized" fields to the osbuild job as
replacement for the local target options. The former signifies the name
of the uploaded artifact and whether an artifact should be uploaded at
all (only weldr API). The latter will be deprecated at some point, when
osbuild itself can make streamoptimized vmdk images.
This change separates what have always been two distinct concepts:
artifacts that are reported back to the composer node (in practice
always running on the same machine), and upload targets to clouds and
such. Separating them makes it easier to add job types that only allow
one upload target while keeping artifacts.
Keep the local target around, so that jobs that are scheduled can still
be run after an upgrade.
The server hasn't used common.ImageBuildState to mark a job as
successful or failed for a long time. Instead, it's using the job's
return argument for that. (Jobs don't have a high-level concept of
failing).
Drop the check in the server, and always send "FINISHED" from the client
for backwards compatibility.
osbuild reports failing builds in two ways: it sets the "success" field
in its output to `false` and it returns with a non-zero exit status. The
worker used both, returning an `OSBuildError` when osbuild return
non-zero, but also forwarding the resulting object with the "success"
field.
Change this to only use the "success" field and ignore the return value.
The latter is useful for people running osbuild in a terminal or script,
but is redundant for this use-case.
This makes error reporting more consistent: `RunOSBuild` only returns an
error when *running* osbuild failed, not when the build fails.
Previous commits introduces a new way to generate all X.509 certificates
needed for testing. This commit reuses the same method for auth tests.
This has two benefits:
1) The new code generates certificates with Subject Alternative Name which
means we can use it on systems with Go 1.15 (Fedora 33, RHEL 8.4).
2) The new code generates much saner certificates.
The Koji test in Github actions was always a bit quick and dirty solution.
I think it's much nicer solution to run it on Schutzbot.
Therefore, this commit moves the koji_test.go to a new osbuild-koji-tests
executable. This new test isn't run in the base test suite as one would
anticipate but inside the koji.sh test. This is needed because
osbuild-koji-tests requires a running koji instance. This might change
in the future but I think it works for now.
We have the same thing for AWS. The AWS target also specifies under what name
should be the image available in EC2.
As requested by Brew maintainers Tomáš Kopeček and Lubomír Sedlář.
osbuild runs directly on the host, there's no intermediate container,
therefore we should set the container type to none.
As suggested by Brew maintainers Tomáš Kopeček and Lubomír Sedlář.
Attempt to clarify the structure of our tests. Each test case is now
encapsulated in a script in `test/cases`. Each of these scripts should
be runnable on a pristine machine and be independent of each other. It
is up to the test-orchestractor to decide if they should be run
consequtively instance, or in parallel on separate instances. Each
script can execute several tests and call whatever helper binaries
is desired. However, each case should be assumed to always run as one.
We no longer release into F31, and the right specfile was anyway not
being tested.
This allows us to remove a workaround that updates the VMs during
deploy, and other fedora-31 specific hacks.
Add an additional integration test, which builds a fedora-32 image via
the newly moved composer API.
This adds a new certificate for clients to authenticate, with
CN=client.osbuild.org.
Also stop allowing certificates with `CN=worker.osbuild.org` in the
`osbuild-composer.toml` used in tests and amend `osbuild-auth-tests` to
use `CN=client.osbuild.org` for accessing the koji API.
Heavily based on an earlier version of this test in `test/cmd/aws.sh` by
Sanne Raymaekers.
This removes the osbuild-composer-cloud package, binary, systemd units,
the (unused) test binary, and the (only-run-on-RHEL) test in aws.sh.
Instead, move the cloud API into the main package, using the same
socket as the koji API, osbuild-composer-api.socket. Expose it next to
the koji API on route `/api/composer/v1`.
This is a backwards incompatible change, but only of the -cloud parts,
which have been marked as subject to change.
Instead, call it osbuild-composer-api.socket, but provide a symlink for
backwards compatibility. Change `schutzbot/provision.sh` to only enable
osbuild-composer-api.socket.
In the future, this new socket is the only API socket, which provides
both the "cloud" API and the one for koji.
This means that the koji API is always enabled.
The issue comes from the fact that the PR introducing it was very old
and meanwhile the variable used for image name creation has changed.
This patch makes sure both functions are the same.
In case the job fails completely, resources might leak. This patch
together with the one introducing predictable names for CI artifacts
will make sure we wipe all resources after each CI run.
These had the generic name osbulid-tests, rename the binary to more
closely describe what is under test: weldr API as composed by
composer-cli.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Fedora 33 images can now be built and test cases are added for the new
images. The fedora 33 qcow2 and vmdk images are based off of the
official images and their kickstarters found here:
https://pagure.io/fedora-kickstarts. The fedora 33 iot image is based
off of the the config found here: https://pagure.io/fedora-iot/ostree.
The openstack, azure, and amazon image types have changes made to them
based off of the changes made to the qcow2. The changes between fedora
32 and fedora 33 are as follows:
Grub now loads its kernel command line options from
etc/kernel/cmdline, /usr/lib/kernel/cmdline, and /proc/cmdline instead
of from grub env. This is addressed by adding kernelCmdlineStageOptions
to use osbuild's kernel-cmdline stage to set these options. Alongside
`ro biosdevname=0 net.ifnames=0`, we also set `no_timer_check
console=tty1 console=ttyS0,115200n8` per what is set in the official
qcow2. For azure and amazon, the kernelOptions are still set as they
were in fedora 32.
The timezone is now set to UTC if a user does not set a timezone in the
blueprint customizations. Also, the hostname is set to
localhost.localdomain if the hostname isn't set in the blueprint.
Finally, the following packages have been removed:
polkit
geolite2-city
geolite2-country
zram-generator-defaults
Split the actual service into its own type `Composer` in composer.go.
main.go now (more or less) contains only collecting configuration from
the environment and the file system, as well as activation file
descriptors.
Aside from making the code easier to grok, this is a first step towards
running composer in a different environment than the one set up by
systemd.
The default values of fields in both ComposerConfig.Koji and
ComposerConfig.Worker are well-suited for how they're used.
The nil-checks in main.go only checked that the sections exist. This is
quite a weak check for validity, because the sections could be empty. If
anything is required for composer to function, we could add proper
validation in the future.
Do the same for the CA fields, which contain file names. Go has lots of
precedent for using empty strings to denote "no value" in the standard
library. Use it for CA files, too, instead of pointers.
The configuration file is API. Let's give it a bit more prominence to
help people treat it as such, and a chance to test it. A basic test is
included in this commit.
Also, this cuts down on the noise in main.go a bit.
The cloud API will be moved to `/api/composer/v1` in the future.
Mention this in the `servers` section of the openapi.yml (relative URLs
are allowed) too, even though our generator does not consider it.
Prior this commit, /etc/osbuild-composer/ca-crt.pem certificate was
used as an authority to validate client certificates.
After this commit, the host's trusted certificates are used to do
the validation. Ability to override this behaviour is also introduced:
In osbuild-composer config file, under koji and worker sections, a new CA
option is now available. If set, osbuild-composer uses it as a path
to certificate used to validate client certificates instead of the
default ones.
With this feature, it's possible to restore the validation behaviour
used before this change. Just put following lines in
/etc/osbuild-composer/osbuild-composer.toml:
[koji]
ca = "/etc/osbuild-composer/ca-crt.pem"
[worker]
ca = "/etc/osbuild-composer/ca-crt.pem"
This commit introduces a new test binary responsible for testing TLS
authentication.
Currently, it covers both remote worker API and Koji API. It tests that
the server refuses certificates issued by an untrusted CA or self-signed ones.
Also, it tests that the certificate is issued for an allowed domain.
TODO: certs with subject alternative name are currently not used in tests.
They should work just right, but a proper testing requires more tinkering with
OpenSSL than I'm willing to accept at this time
This commit adds a domain allowlist which works the same way as the one
for remote workers.
To accept just w1.osbuild.org and w2.osbuild.org, use:
[koji]
domain_allowlist = [ "w1.osbuild.org", "w2.osbuild.org" ]
Prior this change, the structure was following:
[koji.localhost.kerberos]
This change modifies it to:
[koji.servers.localhost.kerberos]
This allows us to put more config options under the koji section. See
following commits, they use this new possibility.
There's need for control which certificates to accept. This commit introduces
the domain allowlist. The basic idea is that composer accepts only
certificates issued to domain names specified in osbuild-composer config file.
It allows multiple domains to be specified.
To accept just w1.osbuild.org and w2.osbuild.org, use:
domain_allowlist = [ "w1.osbuild.org", "w2.osbuild.org" ]
as @teg suggests in
https://github.com/osbuild/osbuild-composer/issues/888#issuecomment-662942314
this is ON by default so we can be alerted for missing cloud
credentials in CI!
If you want to disable it then -fail-local-boot=off
Note: special case the qemu image types which always need to be
booted locally.
Follow the worker API so we standardise on one library. This simplifies
the code quite a bit.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
CHANGE_ID is set for PRs, which is why it worked during the review
process, but in the master branch it is not set. BRANCH_NAME is set for
both PRs and master branch. In case of PRs it is in form `PR-<pull
request number>`.
In the same way we require authentication for the worker API, require
clients of the koji API to authenticate using SSL client certificates.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Contrary to our assumption, we cannot initialize the build with the
link to the task. We can only update the link once the build has
completed.
This seems like a bug in koji, but we keep it like this for now.
