If called from within the source directory, i.e. the local plugin
exists, copy those to the share directory so they can be picked
up by the entry point scripts, in case the rpms are not found.
In case `TEST_PATH` was not specified as command line argument,
it was falling back to `test`. Make the latter an absolute path,
by pre-pending `PWD`, otherwise podman complains about the name
of the volume.
Instead of just using the "latest" container everywhere, which will
change every time a new release is made, add a build argument to
specify the version and then match that version to the host in all
the build scripts. This will make it possible to use the tests for
gating, and ensure that we test the plugins on the OS version that
is targeted.
Instead of building on the existing quay.io/osbuild/koji:v1, and
then replacing a lot of it (entry point), move the packages and
the dnf.conf change over from the former base and then directly
depend on Fedora. This gives us more control, especially over
what Fedora version is being used.
This is similar to how other osbuild packages are testing: everything
that's needed for testing is included in the tests package or a
dependency of it. The test runner then runs every executable in
/usr/libexec/tests/<packagename>. This gives a simple test API to
projects depending on this package (notably osbuild-composer).
The local development workflow described in HACKING.md is meant to
continue to work. To ensure this, all relevant scripts gained a
TEST_DATA variable, which defaults to `./test`, but is set from $1 to
the installed path from integration.sh.
Instead of taking podman-plugins from the source directory, use the one
that will be released into RHEL 8.3.1.
This will simplify moving tests into an rpm.
De-serialize the koji init and import logs, required fields in the
ComposeLogs, and if non-empty, attach them to the task.
Update the tests to check for the presence of these logs.
Instead of getting the `koji_build_id` from the direct reply of
the compose request call, use the one returned in the compose
status.
The reason behind this is that composer was changed so that the
CGInitBuild call to koji is now being done by a worker and not
composer itself. This means that once the compose request call
returns, the build id is not yet known. In composer release 24,
the compose request call internally waits for the worker that
does the CGInitBuild API call, but that will be changed, and
the koji_build_id will then not be returned from the compose
request API call anymore. This prepares for that. The tests are
also adapted to simulate the new behavior.
NB: this makes composer 24 a dependency, since the build id is
taken from the ComposeStatus, which was only added there.
Add a new command line option `--skip-tag` that will skip tagging
after a successful build. The help text is copied from the same
option of other sub-commands in the koji client. The hub plugin's
jsonschema was updated accordingly, and the builder plugin will
skip the tag if the option was requested.
Tests were added or augmented for all three plugins to test the
new option.
For each compose, mock also the "compose/<id>/logs" API endpoint
and just return some string. Add a feature to be able to control
the http status of the route though, so we can simulate failures
during log fetching.
In the builder unit test that checks a successful compose, use
more than one architecture, and check that an image request was
created for all of those. This should expose bugs multi-arch
specific bugs.
Generate the certificate to be used for koji in make-certs.sh by
the same CA that also generates the composer and client certs.
Create a single certificate that uses the SubjectAltName (SAN)
extension to cover two domains: localhost, org.osbuild.koji.koji,
which previously was done via two separate certificates; this is
the legacy usage which stopped working with go 1.15 (see previous
commit). As a consequence the apache config is modified to use
only one virtual host with a ServerAlias directive.
Rework the generation of certificates, in order to make use of
the SubjectAltName (SAN) extension, that is required for modern
usage of TLS (see RFC 2818, or [1]) and now enforced by default
for go version 1.15[2] (Fedora 33). For this a different config
file is used, originally written by Lars, and assign SANs to
the server and client certificates. Additionally, the correct
extensions are used for each of those, so that their usage is
limited to the server or client use case. The client certificate
is renamed from "worker" to "client".
The lifetime of the certificates is increased, as a side effect of
the new config file.
[1] https://github.com/urllib3/urllib3/issues/497
[2] https://golang.org/doc/go1.15#commonname
Co-authored-by: Lars Karlitski <lars@karlitski.net>
The official postgres container image is fetched from docker.io,
which recently introduced rate limiting, which makes introduces
possible CI error if we run into this limit.
Instead use a custom mirror[1] of the official image, hosted on
quay.io[2]. As a side effect this updates the posgres version
from 12 to 13.
[1] osbuild/containers@7db3c68
[2] quay.io/osbuild/postgres:v1
In the basic invocation test, specify the expected argument values
not only their type. With the exception of the default image type
they correspond to the various command line arguments given.
Add more supported build architectures to the standard build tag.
This will check the logic for (un-)supported build architectures
in the builder plugin. If we only have one arch, which is the
requested arch, it will not catch logic errors, that invert the
supported vs requested architectures.
On RHEL a pre-compiled version is currently installed from the
vendor directory (see the README.md there). On Fedora it is
packaged in podman-plugins and that used to be installed
without explicitly asking for it, but that seems to have changed.
Explicitly install it on Fedora to ensure it is present.
Symptom of the dns plugin missing is that the koji hub container
will not start because it will block waiting for the postgres
container forever.
Change the use of DATA_DIR across all scripts so that it will
only be set if it does not yet exist. This makes it possible
to change the location of it across all scripts at once.
Change the use of SHARE_DIR across all scripts so that it will
only be set if it does not yet exist. This makes it possible
to change the location of it across all scripts at once.
Move the container directory, containing the container definitions
for all the test containers, to test/, where all the other test-
related files are located (with the exception of `Schutzbot`).
Use `test/build-container.sh` to build the container, instead of
replicating that in `test-integration.sh`.
Add a new helper method that can be used to create an instance of
OSBuildImage. Optionally, the session and options can be passed
in as well as a custom config data, otherwise the default session
and object are used. If custom configuration data was specified,
a temporary configuration file with the config data is created
and set plugin-wide so it is picked up by the object constructor.
Upstream composer has introduce a few changes that we need to
adapt for:
- the koji composer API is now exposed on the standard https
port (443). Thus koji hub and web need to move to a different
pair: 8080 (http) and 4343 (https). Change the scripts and
tests for that
- the koji API gained a prefix 'api/composer-koji/v1/'. Change
client and unit tests to use that prefix. Use urljoin to
create new APIs
- composer configuration format (osbuild-composer.toml) has
changed and now also includes configuration for the CA
and allowed domains
- update the composer RPM repositories to the commit for the
21 upstream release.
Create a MockHost class that mocks a small subset of the HostExport
koji class, i.e. the builder specific XML RCP interface. All the
methods needed for tagging a build, including waiting for tasks,
are mocked. MockHost will keep a list of tagged builds, to check
in the unit test functions.
Assert, via the new UploadTracker, that the "compose-request.json"
is always saved, especially in the case where composer refuses the
compose (via bad request).
Replace the "fast_incremental_upload" of the plugin with a custom
one that will keeps track of all uploaded files through it. Can
be used to ensure that certain uploads happen.
NB: this assumes that "fast_incremental_upload" was or will be
directly imported into the plugin namespace.
When returning the result from the task handler function, return
a more complete and structured object in all cases. The name of
the sub-object is named after the service the item belongs to;
specifically composer is used for what belongs to (osbuild)-composer.
