These had the generic name osbulid-tests, rename the binary to more
closely describe what is under test: weldr API as composed by
composer-cli.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Fedora 33 images can now be built and test cases are added for the new
images. The fedora 33 qcow2 and vmdk images are based off of the
official images and their kickstarters found here:
https://pagure.io/fedora-kickstarts. The fedora 33 iot image is based
off of the the config found here: https://pagure.io/fedora-iot/ostree.
The openstack, azure, and amazon image types have changes made to them
based off of the changes made to the qcow2. The changes between fedora
32 and fedora 33 are as follows:
Grub now loads its kernel command line options from
etc/kernel/cmdline, /usr/lib/kernel/cmdline, and /proc/cmdline instead
of from grub env. This is addressed by adding kernelCmdlineStageOptions
to use osbuild's kernel-cmdline stage to set these options. Alongside
`ro biosdevname=0 net.ifnames=0`, we also set `no_timer_check
console=tty1 console=ttyS0,115200n8` per what is set in the official
qcow2. For azure and amazon, the kernelOptions are still set as they
were in fedora 32.
The timezone is now set to UTC if a user does not set a timezone in the
blueprint customizations. Also, the hostname is set to
localhost.localdomain if the hostname isn't set in the blueprint.
Finally, the following packages have been removed:
polkit
geolite2-city
geolite2-country
zram-generator-defaults
Split the actual service into its own type `Composer` in composer.go.
main.go now (more or less) contains only collecting configuration from
the environment and the file system, as well as activation file
descriptors.
Aside from making the code easier to grok, this is a first step towards
running composer in a different environment than the one set up by
systemd.
The default values of fields in both ComposerConfig.Koji and
ComposerConfig.Worker are well-suited for how they're used.
The nil-checks in main.go only checked that the sections exist. This is
quite a weak check for validity, because the sections could be empty. If
anything is required for composer to function, we could add proper
validation in the future.
Do the same for the CA fields, which contain file names. Go has lots of
precedent for using empty strings to denote "no value" in the standard
library. Use it for CA files, too, instead of pointers.
The configuration file is API. Let's give it a bit more prominence to
help people treat it as such, and a chance to test it. A basic test is
included in this commit.
Also, this cuts down on the noise in main.go a bit.
We expect to merge this into the main osbuild-composer package in the
future and change the port it is exposed on, so for now communicate that
this is not stable API.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The cloud API will be moved to `/api/composer/v1` in the future.
Mention this in the `servers` section of the openapi.yml (relative URLs
are allowed) too, even though our generator does not consider it.
Until now, all jobs were put as "osbuild" jobs into the job queue and
the worker API hard-coded sending an osbuild manifest and upload
targets.
Change the API to take a "type" and "args" keys, which are equivalent to
the job-queue's type and args. Workers continue to support only osbuild
jobs, but this makes other jobs possible in the future.
Prior this commit, /etc/osbuild-composer/ca-crt.pem certificate was
used as an authority to validate client certificates.
After this commit, the host's trusted certificates are used to do
the validation. Ability to override this behaviour is also introduced:
In osbuild-composer config file, under koji and worker sections, a new CA
option is now available. If set, osbuild-composer uses it as a path
to certificate used to validate client certificates instead of the
default ones.
With this feature, it's possible to restore the validation behaviour
used before this change. Just put following lines in
/etc/osbuild-composer/osbuild-composer.toml:
[koji]
ca = "/etc/osbuild-composer/ca-crt.pem"
[worker]
ca = "/etc/osbuild-composer/ca-crt.pem"
This commit introduces a new test binary responsible for testing TLS
authentication.
Currently, it covers both remote worker API and Koji API. It tests that
the server refuses certificates issued by an untrusted CA or self-signed ones.
Also, it tests that the certificate is issued for an allowed domain.
TODO: certs with subject alternative name are currently not used in tests.
They should work just right, but a proper testing requires more tinkering with
OpenSSL than I'm willing to accept at this time
1) generating a private key and signing request was merged into one command
2) -sha256 was dropped, let openssl decide which digest should be used
3) signing request is deleted after the it's signed
The following commits will add testing of authentication on this socket.
This commit also sets up the config for composer other it won't start
with remote socket enabled.
This commit adds a domain allowlist which works the same way as the one
for remote workers.
To accept just w1.osbuild.org and w2.osbuild.org, use:
[koji]
domain_allowlist = [ "w1.osbuild.org", "w2.osbuild.org" ]
Prior this change, the structure was following:
[koji.localhost.kerberos]
This change modifies it to:
[koji.servers.localhost.kerberos]
This allows us to put more config options under the koji section. See
following commits, they use this new possibility.
There's need for control which certificates to accept. This commit introduces
the domain allowlist. The basic idea is that composer accepts only
certificates issued to domain names specified in osbuild-composer config file.
It allows multiple domains to be specified.
To accept just w1.osbuild.org and w2.osbuild.org, use:
domain_allowlist = [ "w1.osbuild.org", "w2.osbuild.org" ]
When remote worker socket was enabled, this was happening:
e := echo.New()
go func() {
e.Listener = listener1
e.Start("")
}()
e.Listener = listener2
e.Start("")
Yeah, this is a race condition. None of the echo's Start methods cannot safely
handle multiple listeners.
This commit fixes this issue by using Echo only as a router for standard
http.Server which handles multiple listeners in a non-racy way.
as @teg suggests in
https://github.com/osbuild/osbuild-composer/issues/888#issuecomment-662942314
this is ON by default so we can be alerted for missing cloud
credentials in CI!
If you want to disable it then -fail-local-boot=off
Note: special case the qemu image types which always need to be
booted locally.
Follow the worker API so we standardise on one library. This simplifies
the code quite a bit.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This job uses shellcheck to analyze all our bash scripts except for
those in /vendor directory.
It also allows sourcing /etc/os-release without checking it, because
it doesn't make any sense to check that the CI machine has correct
/etc/os-release file. Also allow useless cats because I don't think they
cause any harm.
Running mock with verbose enabled made sense a while back when we were
still unsure if things would work, but it's generating a ton of logs now
that eventually clogs Jenkins' disk.
Disable the verbose flag to reduce mock's log output.
Signed-off-by: Major Hayden <major@redhat.com>
Rather than using the arbitrary port 8701, use the standard 443. The
worker API will remain on a separate port, and as long as the two APIs
are exposed by the same binary that will have to remain separate at
8700.
Move the test instance of koji on localhost from 443 to 4343, to avoid a
conflict.
In a follow-up we should also give this API a prefix, so the cloud API
can share the same port with it.
Signed-off-by: Tom Gundersen <teg@jklm.no>
CHANGE_ID is set for PRs, which is why it worked during the review
process, but in the master branch it is not set. BRANCH_NAME is set for
both PRs and master branch. In case of PRs it is in form `PR-<pull
request number>`.
This sets up containers running koji and supporting infrastructure, and
calls the osbuild-composer-koji API to build and image and push it into
our testing instance.
koji-compose.py and various fixes by Christian Kellner.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In the same way we require authentication for the worker API, require
clients of the koji API to authenticate using SSL client certificates.
Signed-off-by: Tom Gundersen <teg@jklm.no>
