Previously, we had a webhook relay. It received a notification from Github
and sent it to AWS SQS. Now, the webhook is dead. The new method (already used
in osbuild-composer and image-builder) is to send the notification directly
from a github action to AWS SQS.
We want to use `dnf` in the same way as our users do. This means we want the
modular repositories and weak deps enabled. Fastestmirror is fine, it doesn't
change the content set nor depsolving.
Also, this is a workaround for rhbz#1908352, tl;dr: installing podman without
weak deps makes it unusable on Fedora 32.
Repository URLs are predictable. There's no need to use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
Change the repository path on S3 to a more predictable one, mirroring
the pattern we're using for osbuild-composer.
Notably, don't use short commit ids. The length of these is not
predictable. It depends on the shortest unique prefix in the repository
and git configuration.
For example, koji-osbuild commit $SHA for fedora-33 on x86_64 will
result in this URL:
koji-osbuild/fedora-33/x86_64/$SHA
This is similar to how other osbuild packages are testing: everything
that's needed for testing is included in the tests package or a
dependency of it. The test runner then runs every executable in
/usr/libexec/tests/<packagename>. This gives a simple test API to
projects depending on this package (notably osbuild-composer).
The local development workflow described in HACKING.md is meant to
continue to work. To ensure this, all relevant scripts gained a
TEST_DATA variable, which defaults to `./test`, but is set from $1 to
the installed path from integration.sh.
Instead of taking podman-plugins from the source directory, use the one
that will be released into RHEL 8.3.1.
This will simplify moving tests into an rpm.
Now that osbuild and osbuild-composer use predictable destinations
for their dnf repositories, those can be used and auto-generated
from the environment and a specific git commit (identified by
the hash).
This will make updating easier, because the only thing we need to
change is the commit hash.
For osbuild-composer the latest release is used, for osbuild the
predictable repos were introduced after the latest release, so
the commit that introduced the feature is used.
Instead of installing and enabling the koji socket, use the new
api socket. The koji socket is not properly wired up anymore
and any attempt to actually use it with hang.
Also update the Fedora CI to osbuild 23, so that it is in sync
with the RHEL ci. This is important since the koji socket has
been deprecated by the api socket and the koji socket actually
does not work anymore.
Upstream composer has introduce a few changes that we need to
adapt for:
- the koji composer API is now exposed on the standard https
port (443). Thus koji hub and web need to move to a different
pair: 8080 (http) and 4343 (https). Change the scripts and
tests for that
- the koji API gained a prefix 'api/composer-koji/v1/'. Change
client and unit tests to use that prefix. Use urljoin to
create new APIs
- composer configuration format (osbuild-composer.toml) has
changed and now also includes configuration for the CA
and allowed domains
- update the composer RPM repositories to the commit for the
21 upstream release.
This is right when PR-952 landed, which is what we are updating
from. This will use SSL certification and also re-enable auto
kerberos authorization via config files for composer.
Try testing a compose via the koji, which involves creating using
the koji command line plugin to make the XMLRPC call to koji hub,
where the osbuild koji hub plugin verifies the parameters and then
creates the task. The osbuild koji plugin for the builder is then
picking up the task, and uses composer's koji API to request a
compose. Once this is successful it will be imported by composer
into koji via the 'CGImport' method.
The `koji osbuild-image` command waits for all this and reports
whether the task was successful or not via its exit code (and
on stdout).
This uses a fleet of containers: a database one, a kerberos kdc
one, another one for the koju hub and finally one for the koji
builder. The pre-build RPMs are used to install the plugins.
NB: On RHEL we need to manually install the `dnsname` podman
plugin, since it is missing, but required so that containers
can address each other by hostnames.
See [schutzbot/vendor/README.md](schutzbot/vendor/README.md)
The CI is in two stages, for each supported distro.
First the RPMs are generated from the spec file in the repo for the
given distro and architecture.
Once all the RPM builds have succeeded successfully, a test machine is
provisioned with osbulid-composer installed, and koji API enabled.
The repository containing the RPMs of the code being tested is also
enabled on the test machine, and the cli client is installed.
Finally, the test/integration.sh script is executed, which currently
does nothing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
