In order to add the newly supported sysconfig stage, the osbuild
dependency needs to be updated to version 24. The version is updated to
24 in the specfile and README. Also, for testing purposes, the Schutzfile
now points to the commit hash for osbuild version 24 for rhel-8.3 tests.
Fedora 33 and rawhide got an updated version of the azblob library. Sadly, it
introduced a non-compatible API change. This commit does the same thing as
a67baf5a did for kolo/xmlrpc:
We now have two wrappers around the affected part of the API. Fedora 32 uses
the wrapper around the old API, whereas Fedora 33 and 34 (and RHEL with its
vendored deps) use the wrapper around the new API. The switch is implemented
using go build flags and spec file magic.
See a67baf5a for more thoughts.
Also, there's v0.11.1-0.20201209121048-6df5d9af221d in go.mod, why?
The maintainers of azblob probably tagged a wrong commit with v0.12.0 which
breaks go. The long v0.11.1-.* version is basically the proper v0.12.0 commit.
See https://github.com/Azure/azure-storage-blob-go/issues/236 for more
information.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
koji and ansible are not in RHEL repositories. Depending on them breaks RHEL
gating (see OSCI-1541): It tries to build a custom image with -tests package
in it but in the build environment there's no EPEL.
This commit makes the RPM independent from EPEL. However, we still need koji
and ansible, so the provision script now enables EPEL and installs the packages
from there. This is not nice but we have to live with that until OSCI-1541 is
solved.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Build the osbuild-composer(7) man-page as part of the RPM build and
include it in the osbuild-composer RPM. Previously the man-page was not
shipped at all in any of the produced RPMs.
make is currently included in the Fedora buildroot, however there is a
plan to remove it since F34. Since make is now used in the %build
section, it should be listed explicitly as a BuildRequires.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
All tests in /usr/libexec/tests/osbuild-composer should be able to run without
any arguments. This was not a case of libvirt.sh - it required two arguments
set by some Jenkinsfile logic.
This commit moves test/cases/libvirt.sh to tools/libvirt_test.sh and extracts
the logic controlling the test case from Jenkinsfile to test/cases/libvirt.sh.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Everybody hates the local workers. The first step of getting rid of them
is to split their socket out of osbuild-composer.socket - we need to keep
this one to support the Weldr API but the local worker socket can live in
its own file.
The behaviour should be the same for now: osbuild-composer.service always
starts the local worker socket.
However, this split allows the osbuild-composer executable to be run without
the Weldr API activated. The following commit explores this option more
in depth.
Note that the new socket can be used by root only because workers are always
run as root.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Not a functional change. I think the previous state was a result of code rotting
and a suboptimal rebase.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
osbuild-composer doesn't actually require osbuild. osbuild-composer-worker
does. Let's remove the dependency from osbuild-composer and depend on the
right version of osbuild in the worker.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
#1070 removed osbuild-composer-koji and introduced a new Obsoletes field.
However, rpmlint doesn't like unversioned obsoletes. This commit fixes that
by adding the last version of osbuild-composer that had the koji subpackage
to the Obsoletes field.
Fixes#1076
Remove both the package osbuild-composer-koji, and the only file it
shipped: osbuild-composer-koji.socket.
It's been deprecated since 835b556, but the backwards-compatible
solution in that commit never worked, because osbuild-composer only
checks for "osbuild-composer-api.socket" when starting up.
Since this has been meant to be deprecated for a while, just remove it
outright.
Add an "Obsoletes:" for the package, so that it gets uninstalled on
existing systems.
The certificate generation is based on work by Lars Karlitski in our osbuild
CA. The server and client certs now contains Subject Alternative Name making
Python's request module and Go 1.15 happy (they deprecated certificates
without SAN).
Several reasons why we want to switch to the certificate generation:
1) The pre-generated certificates are not documented. If someone wants
to inspect them, he must know the right openssl incantation. This way,
you are able to see what's inside the certificates in a plain text.
2) The pre-generated certificates are going to expire at one point and
someone will be surprised.
3) Shipping private keys in RPMs is iffy. I know, it's just for testing but
still...
4) Auth tests are generating their own certificates. To achieve consistency,
we have two options:
a) Ship also all certificates for auth tests. That's extra 8 ones or
something like that.
b) Generate all certificates on fly. This commit does that.
5) The setup introduced by this commit is very similar to the one in our CA
making the test environment very similar to what's running in production.
tl;dr: I think this is a good step forward.
The Koji test in Github actions was always a bit quick and dirty solution.
I think it's much nicer solution to run it on Schutzbot.
Therefore, this commit moves the koji_test.go to a new osbuild-koji-tests
executable. This new test isn't run in the base test suite as one would
anticipate but inside the koji.sh test. This is needed because
osbuild-koji-tests requires a running koji instance. This might change
in the future but I think it works for now.
This was introduced in osbuild 23, so we also need to bump the dependency
in the spec file and also the submodule.
The test is also modified and a typo in its name is fixed.
The test package should be self-contained and contain all the configuration
necessary for a known-good test run (minus secrets). This moves repo
overrides from the test orchestration into the test package.
We want all the external sources (including boot isos and repositories) to
be pinned by their content hash, and never use anything that is not strictly
defined. This moves us in the right direction, but we still have some tests
to update to use these shipped repos rather than official mirrors.
One remaining challenge is that we must make sure all our test runners have
access to the RHEL snapshots, which is not currently the case for the runners
in EC2, but a solution is in the works.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Attempt to clarify the structure of our tests. Each test case is now
encapsulated in a script in `test/cases`. Each of these scripts should
be runnable on a pristine machine and be independent of each other. It
is up to the test-orchestractor to decide if they should be run
consequtively instance, or in parallel on separate instances. Each
script can execute several tests and call whatever helper binaries
is desired. However, each case should be assumed to always run as one.
Add an additional integration test, which builds a fedora-32 image via
the newly moved composer API.
This adds a new certificate for clients to authenticate, with
CN=client.osbuild.org.
Also stop allowing certificates with `CN=worker.osbuild.org` in the
`osbuild-composer.toml` used in tests and amend `osbuild-auth-tests` to
use `CN=client.osbuild.org` for accessing the koji API.
Heavily based on an earlier version of this test in `test/cmd/aws.sh` by
Sanne Raymaekers.
This removes the osbuild-composer-cloud package, binary, systemd units,
the (unused) test binary, and the (only-run-on-RHEL) test in aws.sh.
Instead, move the cloud API into the main package, using the same
socket as the koji API, osbuild-composer-api.socket. Expose it next to
the koji API on route `/api/composer/v1`.
This is a backwards incompatible change, but only of the -cloud parts,
which have been marked as subject to change.
Instead, call it osbuild-composer-api.socket, but provide a symlink for
backwards compatibility. Change `schutzbot/provision.sh` to only enable
osbuild-composer-api.socket.
In the future, this new socket is the only API socket, which provides
both the "cloud" API and the one for koji.
This means that the koji API is always enabled.
Fedora 33 ships the new API so let's do the switch now.
But... this would break older Fedoras because they only have the old API,
right?
We have the following options:
1) Ship xmlrpc compat package to Fedora 33+. This would mean that we delay the API switch till F32 EOL. This would be the most elegant solution, yet it has two issues: a) We will surely not be able to deliver the compat package before F33 Final Freeze. b) It's an extra and annoying work.
2) Downstream patch. No.
3) Use build constraints and have two versions of our code for both different
API.
I chose solution #3. It has an issue though:
%gobuild macro already passes -tags argument to go build. Therefore the
following line fails because it's not possible to use -tags more than once:
%gobuild -tags kolo_xmlrpc_oldapi ...
Therefore I had to come up with manual tinkering with the build constraints
in the spec file. This is pretty ugly but I like that:
1) Go code is actually clean, no weird magic is happening there.
2) We can still ship our software to Fedora/RHEL as we used to
(no downstream patches)
3) All downstreams can use the upstream spec file directly.
Note that this doesn't affect RHEL in any way as it uses vendored libraries.
%gotest ./... was failing on ELN because it couldn't find the vendored
dependencies.
Let me explain why this worked before:
In Fedora, we don't use vendored deps, therefore we had no issue.
In RHEL8, we use vendored deps and -mod=vendor with GO111MODULE unset.
In this case, vendored deps were indeed used.
However, ELN added GO111MODULE=off. When modules are disabled, -mod=vendor
is ignored[1]. It turns out that go with disabled modules uses vendor directory
only in case when $(pwd) is inside GOPATH. Don't ask me why.
Therefore, this fix uses cd to ensure that pwd inside GOPATH (even though
technically we're still in the same directory).
Note that in %build we already had this magic cd command, therefore it wasn't
an issue there.
[1]: Actually `GO111MODULE=off go build -mod=vendor` returns an error. However,
`GO111MODULE=off GOFLAGS=-mod=vendor go build` just ignores the go flag from
env.
Fixes #rhbz1884383
In case the job fails completely, resources might leak. This patch
together with the one introducing predictable names for CI artifacts
will make sure we wipe all resources after each CI run.
Lorax-composer won't be shipped into Fedora 34. To upgrade users from it to
osbuild-composer we need to obsolete it in our spec file.
Fixes RHBZ#1886405
The -test subpackage should depend on all that it needs, and we should
never need to install anything explicitly from the test scripts.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Also install it is part of he tests subpackage. This a helper-tool, not
golang code, so should not live in `internal`. We need access to this
from the integration tests, so install it onto the tests system.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is obviously not a good long-term solution, but at least the fedora
version of the binary has a build-id so RPM will not complain.
Signed-off-by: Tom Gundersen <teg@jklm.no>
No tests should be run directly from git, but should rather be installed
onto the test system using rpm and run from there. This moves towards
unifying our two types of test cases.
The new structure of is now:
`test/cmd`: the executors, one for each test-case. This is installed
into `/usr/libexec/test/osbuild-composer`.
`test/data`: data and config used by the tests. This is installed into
`/usr/share/tests/osbuild-composer`.
`schutzbot`: configuration of the actual test run. In particular, this
is where the distros and repositories to test against are
configured.
This is very much still work-in-progress, and is only the first step
towards simplifying schutzbot. Apart from moving files around, this
should be a noop.
Signed-off-by: Tom Gundersen <teg@jklm.no>
These had the generic name osbulid-tests, rename the binary to more
closely describe what is under test: weldr API as composed by
composer-cli.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This commit introduces a new test binary responsible for testing TLS
authentication.
Currently, it covers both remote worker API and Koji API. It tests that
the server refuses certificates issued by an untrusted CA or self-signed ones.
Also, it tests that the certificate is issued for an allowed domain.
TODO: certs with subject alternative name are currently not used in tests.
They should work just right, but a proper testing requires more tinkering with
OpenSSL than I'm willing to accept at this time
Follow the worker API so we standardise on one library. This simplifies
the code quite a bit.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
