While it is technically possible to build more than one image type
as part of a Koji compose in osbuild-composer, this option is not used
in reality and it also makes very little sense. If the user wants to
build more than one image type, they should submit multiple Koji builds.
Adjust affected unit tests.
Previously, the image status in the compose status uploaded to the koji
build was represented as a single string describing the overall status.
All information related to the image upload or error details were
thrown away.
Refactor `ImageStatus` to contain all the information about the image,
its upload targets and potential errors, as they are returned by
composer.
This will improve the experience when debugging compose failures. In
addition, including all the data will be important once images
will be uploaded directly to the cloud, because it will contain
information to locate the image in the cloud environment.
Refactor the test case implementation to make it easier to reuse
existing code when adding a new testing scenario.
The common pieces were moved under a new `SutInfo` class, which provides
information and convenience methods related to the system on which the
test is running.
This change will make it easier to later add a new scenario testing
upload to the cloud.
Use latest osbuild and osbuild-composer releases. This is important to
be later able to test direct uploading to the cloud for Koji composes.
Also the mock openid server had to be enhanced to be usable for testing
`koji-osbuild`.
Modify used osbuild-worker configuration to use JWT for authentication
with composer. Update configurations of both, composer and worker to
handle multi-tenancy.
Do not start any services in `schutzbot/deploy.sh`, because at that
point, none of the configuration files are in place.
Ensure that the correct units are started and masked by
`test/integration.sh` to simulate the Service scenario more closely.
This means that the local worker is masked and only remote worker is
started.
Co-authored-by: Jakub Rusz <jrusz@redhat.com>
The status calls are sometimes failing on:
upstream connect error or disconnect/reset before headers. reset reason:
connection termination
Since all requests are going through the company proxy, I think that the
networking isn't working 100% reliably. This commit adds a retry mechanism
provided by the urllib3 library. It will retry on all networking issues and
also on some 5xx errors that makes sense to retry (like gateway failures).
A test is added that runs the compose waiting code against a mock server
that fails every second request. This is imho sufficient to mimick a flaky
networking.
RHEL is now shipping podman 4 that encodes the networks differently, let's
implement a small dirty hack to fix it on RHEL before
https://github.com/osbuild/koji-osbuild/pull/100 can land and fix it properly.
The release number in the RPMland canonically starts at 1, not at 0. I think
it technically doesn't cause any harm but let's stick to what all other packages
do.
The Cloud API supports passing in a variety of image customizations,
like e.g. extra packages or pre-defining users.
Add a new command line option to the client `--customizations` which
takes a path to a JSON file which contains the customziations; they
will be passed via the existing `opts` argument to the hub.
Add support for `customizations` to the `opts`/`options` arguments
to the hub plugin. No validation to the object is done. Instead we
rely in Composer for the validation of the content.
Add support for `customizations` the image `ComposeRequest` in the
builder plugin. All specified values are just passed through to
composer as-is.
Add tests for the respective plugins.
We need koji-osbuild-builder to be able to connect to composer via a proxy
because koji builders in our internal deployment cannot reach
api.openshift.com directly. This commit adds a new option `proxy` to the
builder plugin config that controls whether a proxy is used to route all
requests to composer.
We decided to do this late change to Cloud API in order to keep the field name
same for all the APIs:
https://github.com/osbuild/osbuild-composer/pull/2479
I strongly believe that gpgkeys are not used in any of the current koji-osbuild
deployments yet so this change should be safe.
koji-osbuild-tests is dependent on osbuild-composer-tests but this package
is not shipped into Fedora because it provides zero value to users.
Let's do the same for koji-osbuild. koji-osbuild-tests will be built only
in our upstream CI but not built in the official Koji.
configparser treats quotes literally:
client_id = "koji"
translates to the following python string:
'"koji"'
This doesn't matter in the test because the oauth mock from osbuild-composer
actually ignores the credentials and return the token in every case.
But if you want to take this config as an example, you will probably put the
credentials automatically inside the quotes which will not work because a real
oauth server will refuse the client ID surrounded by quotes.
Note that .conf works different than .toml:
TOML requires to have strings in quotes. Yeah, consistency. 🤷
Use the `self.secret` and not `self.id` for the secret. Doh. Mea culpa.
Fix the corresponding test as well, which also checked for the wrong
thing.
Reported-By: Ondřej Budai <ondrej@budai.cz>
This is necessary once so we can enable the upstream release bot workflow,
which consists of just pushing a tag and bumping the version number
directly after the release is done.
This adds support for specifing the package sets for repositories;
on the command line this can be done via `--repo-package-set` with
and argument of `;` separated package set names. This will result
in repo information being transported via dict instead of plain
strings. Thus the hub plugin's schema was modified accordingly.
Last but not least, the builder plugin now can decode these dicts
and setup the repos accordingly.
Test were added for plugins as well as the integration test changed
to use this new feature.
The first upstream commit that supports this feature is pinned.
When we are on an RHEL 8.x host we need to supply `rhel-8x` as
target distro. The previous code just used the major version so
we always built `rhel-8`, i.e. RHEL 8.3.
It similar to BuildTask - so it doesn't do much on the builder and
doesn't use a lot of resources. It makes sense to have much lower weight
for this type of task.
OStree compose requests need special options, like the `ref` the
`parent` and the `url`. Add support for those options to all three
plugins:
The command line plugin now takes `--ostree-{parent,ref,url}`
and passes it to koji via the existing options dictionary.
The JSON schemata in the hub plugin was adjusted to allow these
new options.
Finally the builder plugin will look for the new `ostree` dict
inside the options, create an `OSTreeOptions` object from it,
and attach it to each image request.
NB: since the ostree options are per image request and are thus
architecture dependent we support a "$arch" substition in the
`parent` and `ref` options that will be resolved by the plugin;
this allows to builds arch specific commits for with a single
compose request.
Add the respective unit tests.
Map the image types used by the koji API to the image types used
by the cloud api. This should allow for a smooth transition when
the plugin is upgraded, i.e. the pungi configuration can be used
unmodified. After all the plugins are upgraded the pungi config
should be changed to use the native image types and then this
mapping could be removed again.
Take the current list of valid image types currently supported by
the cloud api and validdate it during the compose request. Also
allow a test "image_type" image type which is used all over the
place in the testing code.
Composer now[1] has integrated the koji API into the "cloud API"
and thus we can use this more general purpose and powerful API
instead of using the specialized koji API endpoint.
Adapt the request and response structures as well as the unit
tests to use that.
[1] PR #2214, commit 11e2ae45284bfb0d89ef1c1e0d2aa4ae230ea573
Switch the authentication method that the koji builder plugin
uses from mTLS to SSO. Since we are using the mock openid
server from the `osbuild-composer-tests` package. Make this
package a dependency of `koji-osbuild-tests`.
Implement support for authentication via OAuth2 using the client
credentials "Client Credentials Grant" flow (4.4 of RFC 6749).
For this a new configuration section is added to the config file,
where the client_id, client_secret and token_url have to be
specified.
The impelmention does currently not support "refresh tokens", but
does support refreshing the token if an `expires_in` is present
in the token itself.
Corresponding unit tests have been added.
[1] https://datatracker.ietf.org/doc/html/rfc6749#section-4.4
