This update reflects the migration of composer from a single instance to
running in Openshift. All the essentials are still there and very few
details omitted in favor of clarity.
Previously it the `repo` schema didn't mark the `baseurl` property as
required, although Cloud API "technically" requires it and the builder
code also made assumptions about it being always provided.
Note on Cloud API:
Cloud API requires that at least one of `baseurl`, `metalink` or
`mirrorlist` is specified for a repo. Since the plugin schema specifies
only `baseurl` as a property, it is as a result mandatory in order for
the Cloud API request to be valid and succeed.
Mark `baseurl` property as required in the `repo` schema and add unit
test for it.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Reformat the code to be easier to read.
An empty dict was sometimes passed to `osbuildImage()` as the `priority`
argument, because `opts` were already part of `args`. Fix it.
Restructure `test_input_validation()` test to allow multiple test cases.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit needs to do three things:
- update the terraform definition
- add fedora-37 to the gitlab-ci file
- bump osbuild and osbuild-composer versions, so we have F37 RPMs for them
I also removed the commented out aarch64 runners to clean the file a bit.
CGImport seems to be slow in certain cases. Let's increase the timeout,
otherwise CGImport for the 10 GiB RAW AMI might exceed it.
Fun fact: Fedora's Koji has the timeout configured to 10800 seconds. 🤯
It turned out, that the upload options for AWS EC2 or GCP with just the
required properties specified, would match both schemas. This is causing
the validation fail with `oneOf` used for `upload_options`.
This will be fixed in osbuild-composer PR#3018. However, we can't use
the same approach for koji-osbuild, while keeping the schema backward
compatible and sane.
Another discussed option would be to define `upload_options` as an empty
object with `additionalProperties` set to `True`. This would
effectively mean no validation of `upload_options`. None of the plugins
actually modify the `upload_options` in any way. It is passed as
provided to `osbuild-composer`.
I think that the change in this commit is a compromise. The validation
of the `upload_options` schema is kept, but it is relaxed to `anyOf`.
[1] https://github.com/osbuild/osbuild-composer/pull/3018
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The Azure Resource Group Location is no longer required and
osbuild-composer can determine the correct Location from the
provided Resource Group.
Related to https://github.com/osbuild/osbuild-composer/pull/3093.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The builder plugin could produce a traceback when image build failed,
but it didn't contain any details.
```
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/koji/daemon.py", line 1468, in runTask
response = (handler.run(),)
File "/usr/lib/python3.6/site-packages/koji/tasks.py", line 335, in run
return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
File "/usr/lib/python3.6/site-packages/koji/util.py", line 271, in call_with_argcheck
return func(*args, **kwargs)
File "/usr/lib/koji-builder-plugins/osbuild.py", line 719, in handler
status = client.wait_for_compose(cid, callback=self.on_status_update)
File "/usr/lib/koji-builder-plugins/osbuild.py", line 483, in wait_for_compose
status = self.compose_status(compose_id)
File "/usr/lib/koji-builder-plugins/osbuild.py", line 454, in compose_status
return ComposeStatus.from_dict(res.json())
File "/usr/lib/koji-builder-plugins/osbuild.py", line 251, in from_dict
ImageStatus.from_dict(s) for s in data["image_statuses"]
File "/usr/lib/koji-builder-plugins/osbuild.py", line 251, in <listcomp>
ImageStatus.from_dict(s) for s in data["image_statuses"]
File "/usr/lib/koji-builder-plugins/osbuild.py", line 228, in from_dict
error = ComposeStatusError(error_id=error_id, **error)
TypeError: __init__() missing 1 required positional argument: 'details'
```
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Log both the entrypoint and the return value from adding a task to
Koji's database. We can measure both to ensure a task has been
successfully added to the database as a service level indicator.
Each 'Task id' corresponds to a 'Compose id' in case everything works as
expected. In order to be able to track both in Splunk to measure our
first service level indicator (SLI) we need to explicitly log the 'Task
id' when it is received by the plugin.
The support for specifying multiple `image_types` for a single compose
has been removed by [1]. This turned out to be problematic, because e.g.
Pungi uses the array type when triggering image builds via osbuild.
Bring back the support for specifying the `image_type` as an array, but
restrict it to a single item. This will cover the Pungi use-case, since
it is always passing a single `image_type` in the array. The array is
then converted to a string in the Hub plugin and passed as such to the
Builder plugin.
Extend unit tests covering the introduced compatibility layer.
[1] c725265081
Extend the integration test with a new case, testing that direct upload
to the cloud works for Koji composes. Test this using a single cloud
provider, specifically AWS.
The test case submits a new osbuild-image build using Koji CLI,
determines the image information once the build finishes and then checks
that such image exists in AWS. The image is then deleted as part of the
test case tear-down.
The AWS credentials are now configured in the worker's configuration, if
the appropriate environment variables are set.
Update the SPEC file with a new test dependency and update the required
osbuild-composer version.
The upload options are expected to be provided as a JSON file. The same
options will be used for all image type and architecture combinations,
similarly as it is done for ostree options.
Extend unit tests to cover the newly added functionality.
While it is technically possible to build more than one image type
as part of a Koji compose in osbuild-composer, this option is not used
in reality and it also makes very little sense. If the user wants to
build more than one image type, they should submit multiple Koji builds.
Adjust affected unit tests.
Previously, the image status in the compose status uploaded to the koji
build was represented as a single string describing the overall status.
All information related to the image upload or error details were
thrown away.
Refactor `ImageStatus` to contain all the information about the image,
its upload targets and potential errors, as they are returned by
composer.
This will improve the experience when debugging compose failures. In
addition, including all the data will be important once images
will be uploaded directly to the cloud, because it will contain
information to locate the image in the cloud environment.
Refactor the test case implementation to make it easier to reuse
existing code when adding a new testing scenario.
The common pieces were moved under a new `SutInfo` class, which provides
information and convenience methods related to the system on which the
test is running.
This change will make it easier to later add a new scenario testing
upload to the cloud.
Use latest osbuild and osbuild-composer releases. This is important to
be later able to test direct uploading to the cloud for Koji composes.
Also the mock openid server had to be enhanced to be usable for testing
`koji-osbuild`.
Modify used osbuild-worker configuration to use JWT for authentication
with composer. Update configurations of both, composer and worker to
handle multi-tenancy.
Do not start any services in `schutzbot/deploy.sh`, because at that
point, none of the configuration files are in place.
Ensure that the correct units are started and masked by
`test/integration.sh` to simulate the Service scenario more closely.
This means that the local worker is masked and only remote worker is
started.
Co-authored-by: Jakub Rusz <jrusz@redhat.com>
The status calls are sometimes failing on:
upstream connect error or disconnect/reset before headers. reset reason:
connection termination
Since all requests are going through the company proxy, I think that the
networking isn't working 100% reliably. This commit adds a retry mechanism
provided by the urllib3 library. It will retry on all networking issues and
also on some 5xx errors that makes sense to retry (like gateway failures).
A test is added that runs the compose waiting code against a mock server
that fails every second request. This is imho sufficient to mimick a flaky
networking.
RHEL is now shipping podman 4 that encodes the networks differently, let's
implement a small dirty hack to fix it on RHEL before
https://github.com/osbuild/koji-osbuild/pull/100 can land and fix it properly.
The release number in the RPMland canonically starts at 1, not at 0. I think
it technically doesn't cause any harm but let's stick to what all other packages
do.
The Cloud API supports passing in a variety of image customizations,
like e.g. extra packages or pre-defining users.
Add a new command line option to the client `--customizations` which
takes a path to a JSON file which contains the customziations; they
will be passed via the existing `opts` argument to the hub.
Add support for `customizations` to the `opts`/`options` arguments
to the hub plugin. No validation to the object is done. Instead we
rely in Composer for the validation of the content.
Add support for `customizations` the image `ComposeRequest` in the
builder plugin. All specified values are just passed through to
composer as-is.
Add tests for the respective plugins.
We need koji-osbuild-builder to be able to connect to composer via a proxy
because koji builders in our internal deployment cannot reach
api.openshift.com directly. This commit adds a new option `proxy` to the
builder plugin config that controls whether a proxy is used to route all
requests to composer.
We decided to do this late change to Cloud API in order to keep the field name
same for all the APIs:
https://github.com/osbuild/osbuild-composer/pull/2479
I strongly believe that gpgkeys are not used in any of the current koji-osbuild
deployments yet so this change should be safe.
