The Cloud API supports passing in a variety of image customizations,
like e.g. extra packages or pre-defining users.
Add a new command line option to the client `--customizations` which
takes a path to a JSON file which contains the customziations; they
will be passed via the existing `opts` argument to the hub.
Add support for `customizations` to the `opts`/`options` arguments
to the hub plugin. No validation to the object is done. Instead we
rely in Composer for the validation of the content.
Add support for `customizations` the image `ComposeRequest` in the
builder plugin. All specified values are just passed through to
composer as-is.
Add tests for the respective plugins.
We need koji-osbuild-builder to be able to connect to composer via a proxy
because koji builders in our internal deployment cannot reach
api.openshift.com directly. This commit adds a new option `proxy` to the
builder plugin config that controls whether a proxy is used to route all
requests to composer.
configparser treats quotes literally:
client_id = "koji"
translates to the following python string:
'"koji"'
This doesn't matter in the test because the oauth mock from osbuild-composer
actually ignores the credentials and return the token in every case.
But if you want to take this config as an example, you will probably put the
credentials automatically inside the quotes which will not work because a real
oauth server will refuse the client ID surrounded by quotes.
Note that .conf works different than .toml:
TOML requires to have strings in quotes. Yeah, consistency. 🤷
Use the `self.secret` and not `self.id` for the secret. Doh. Mea culpa.
Fix the corresponding test as well, which also checked for the wrong
thing.
Reported-By: Ondřej Budai <ondrej@budai.cz>
This adds support for specifing the package sets for repositories;
on the command line this can be done via `--repo-package-set` with
and argument of `;` separated package set names. This will result
in repo information being transported via dict instead of plain
strings. Thus the hub plugin's schema was modified accordingly.
Last but not least, the builder plugin now can decode these dicts
and setup the repos accordingly.
Test were added for plugins as well as the integration test changed
to use this new feature.
The first upstream commit that supports this feature is pinned.
When we are on an RHEL 8.x host we need to supply `rhel-8x` as
target distro. The previous code just used the major version so
we always built `rhel-8`, i.e. RHEL 8.3.
OStree compose requests need special options, like the `ref` the
`parent` and the `url`. Add support for those options to all three
plugins:
The command line plugin now takes `--ostree-{parent,ref,url}`
and passes it to koji via the existing options dictionary.
The JSON schemata in the hub plugin was adjusted to allow these
new options.
Finally the builder plugin will look for the new `ostree` dict
inside the options, create an `OSTreeOptions` object from it,
and attach it to each image request.
NB: since the ostree options are per image request and are thus
architecture dependent we support a "$arch" substition in the
`parent` and `ref` options that will be resolved by the plugin;
this allows to builds arch specific commits for with a single
compose request.
Add the respective unit tests.
Map the image types used by the koji API to the image types used
by the cloud api. This should allow for a smooth transition when
the plugin is upgraded, i.e. the pungi configuration can be used
unmodified. After all the plugins are upgraded the pungi config
should be changed to use the native image types and then this
mapping could be removed again.
Take the current list of valid image types currently supported by
the cloud api and validdate it during the compose request. Also
allow a test "image_type" image type which is used all over the
place in the testing code.
Composer now[1] has integrated the koji API into the "cloud API"
and thus we can use this more general purpose and powerful API
instead of using the specialized koji API endpoint.
Adapt the request and response structures as well as the unit
tests to use that.
[1] PR #2214, commit 11e2ae45284bfb0d89ef1c1e0d2aa4ae230ea573
Switch the authentication method that the koji builder plugin
uses from mTLS to SSO. Since we are using the mock openid
server from the `osbuild-composer-tests` package. Make this
package a dependency of `koji-osbuild-tests`.
Implement support for authentication via OAuth2 using the client
credentials "Client Credentials Grant" flow (4.4 of RFC 6749).
For this a new configuration section is added to the config file,
where the client_id, client_secret and token_url have to be
specified.
The impelmention does currently not support "refresh tokens", but
does support refreshing the token if an `expires_in` is present
in the token itself.
Corresponding unit tests have been added.
[1] https://datatracker.ietf.org/doc/html/rfc6749#section-4.4
Change the integration test so that it adapts to the host, i.e. it
will take the name, version and architecture for the tags and urls
for the repos from the host it is running on. This should make it
more future proof since we now have a central place where this is
configured: the ci configuration, i.e. `.gitlab-ci.yml`.
When the builder is used as an command line binary: have `repo`
be a proper command line argument. It was currently specified
before arch which can be passed multiple times, but so does
repo; hence one of them needs to be optional.
when building the containers the initial `dnf upgrade` will download
content from the latest nightly trees which turns the container into a
Beta system and subsequent `dnf isntall` gets confused!
Podman 2.2.0 doesn't create a gateway by default. See:
https://github.com/containers/podman/issues/8748
This commits introduces a workaround: specifying the gateway manually.
Note that the gateway is used in test/run-builder.sh
If called from within the source directory, i.e. the local plugin
exists, copy those to the share directory so they can be picked
up by the entry point scripts, in case the rpms are not found.
In case `TEST_PATH` was not specified as command line argument,
it was falling back to `test`. Make the latter an absolute path,
by pre-pending `PWD`, otherwise podman complains about the name
of the volume.
Instead of just using the "latest" container everywhere, which will
change every time a new release is made, add a build argument to
specify the version and then match that version to the host in all
the build scripts. This will make it possible to use the tests for
gating, and ensure that we test the plugins on the OS version that
is targeted.
Instead of building on the existing quay.io/osbuild/koji:v1, and
then replacing a lot of it (entry point), move the packages and
the dnf.conf change over from the former base and then directly
depend on Fedora. This gives us more control, especially over
what Fedora version is being used.
This is similar to how other osbuild packages are testing: everything
that's needed for testing is included in the tests package or a
dependency of it. The test runner then runs every executable in
/usr/libexec/tests/<packagename>. This gives a simple test API to
projects depending on this package (notably osbuild-composer).
The local development workflow described in HACKING.md is meant to
continue to work. To ensure this, all relevant scripts gained a
TEST_DATA variable, which defaults to `./test`, but is set from $1 to
the installed path from integration.sh.
Instead of taking podman-plugins from the source directory, use the one
that will be released into RHEL 8.3.1.
This will simplify moving tests into an rpm.
De-serialize the koji init and import logs, required fields in the
ComposeLogs, and if non-empty, attach them to the task.
Update the tests to check for the presence of these logs.
Instead of getting the `koji_build_id` from the direct reply of
the compose request call, use the one returned in the compose
status.
The reason behind this is that composer was changed so that the
CGInitBuild call to koji is now being done by a worker and not
composer itself. This means that once the compose request call
returns, the build id is not yet known. In composer release 24,
the compose request call internally waits for the worker that
does the CGInitBuild API call, but that will be changed, and
the koji_build_id will then not be returned from the compose
request API call anymore. This prepares for that. The tests are
also adapted to simulate the new behavior.
NB: this makes composer 24 a dependency, since the build id is
taken from the ComposeStatus, which was only added there.
Add a new command line option `--skip-tag` that will skip tagging
after a successful build. The help text is copied from the same
option of other sub-commands in the koji client. The hub plugin's
jsonschema was updated accordingly, and the builder plugin will
skip the tag if the option was requested.
Tests were added or augmented for all three plugins to test the
new option.
For each compose, mock also the "compose/<id>/logs" API endpoint
and just return some string. Add a feature to be able to control
the http status of the route though, so we can simulate failures
during log fetching.
In the builder unit test that checks a successful compose, use
more than one architecture, and check that an image request was
created for all of those. This should expose bugs multi-arch
specific bugs.
Generate the certificate to be used for koji in make-certs.sh by
the same CA that also generates the composer and client certs.
Create a single certificate that uses the SubjectAltName (SAN)
extension to cover two domains: localhost, org.osbuild.koji.koji,
which previously was done via two separate certificates; this is
the legacy usage which stopped working with go 1.15 (see previous
commit). As a consequence the apache config is modified to use
only one virtual host with a ServerAlias directive.
Rework the generation of certificates, in order to make use of
the SubjectAltName (SAN) extension, that is required for modern
usage of TLS (see RFC 2818, or [1]) and now enforced by default
for go version 1.15[2] (Fedora 33). For this a different config
file is used, originally written by Lars, and assign SANs to
the server and client certificates. Additionally, the correct
extensions are used for each of those, so that their usage is
limited to the server or client use case. The client certificate
is renamed from "worker" to "client".
The lifetime of the certificates is increased, as a side effect of
the new config file.
[1] https://github.com/urllib3/urllib3/issues/497
[2] https://golang.org/doc/go1.15#commonname
Co-authored-by: Lars Karlitski <lars@karlitski.net>
The official postgres container image is fetched from docker.io,
which recently introduced rate limiting, which makes introduces
possible CI error if we run into this limit.
Instead use a custom mirror[1] of the official image, hosted on
quay.io[2]. As a side effect this updates the posgres version
from 12 to 13.
[1] osbuild/containers@7db3c68
[2] quay.io/osbuild/postgres:v1
In the basic invocation test, specify the expected argument values
not only their type. With the exception of the default image type
they correspond to the various command line arguments given.
Add more supported build architectures to the standard build tag.
This will check the logic for (un-)supported build architectures
in the builder plugin. If we only have one arch, which is the
requested arch, it will not catch logic errors, that invert the
supported vs requested architectures.
