Previously:
We used images built from pre-mass-rebuild composes but installed packages
from post-mass-rebuild composes. This caused weird stuff like sshd crashing
when installing non-related packages via dnf.
Now:
Both the image and repositories are post-mass-rebuild ones. This should solve
these weird issues.
Let's explain how RPMs for RHEL are built:
We use a subscribed RHEL 8.x machine and mock build these on it. Mock
initializes its own buildroot based on the latest RHEL 8 CDN content, see[1].
This means that the minor version of the buildroot is independent of the minor
version of the host.
However, we currently upload RPMs to a directory whose name consists also of
the minor version of the host. Our hosts are currently running RHEL 8.3 so
the RPMs are uploaded into rhel-8.3 directory despite them being built in the
RHEL 8.4 buildroot (RHEL 8 CDN buildroot specifically). This means that
we cannot guarantee that they are installable on RHEL 8.3 which is weird.
This commit adds a special case for hosts that run on subscribed RHEL and
thus build RPMs in a buildroot constructed from RHEL CDN. These RPMs are
now uploaded into rhel-8-cdn directory. This change more accurately reflects
the way we build our RPMs and removes some confusion.
[1]: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/rhel-8.tpl#L37
s3cmd sync actually downloads metadata for all objects in a s3 bucket.
We have built a lot of RPMs, thus this takes 5 minutes on AWS and 25 minutes
on my laptop (!!!).
Let's use recursive put instead. This doesn't delete any files on the remote
side. As we upload RPMs only once, this also shouldn't fail on "the
object already exists". Using this method, we should be able to upload the
RPMs in seconds.
The same patch was applied in osbuild-composer cf73edd2
RHEL 8.4 is now GA, so we don't need any extra tests for it. This should also
make the CI more reliable because having two distros with the same DISTRO_CODE
caused some tests to fail randomly (they used the same intermediate
artifacts).
instead of duplicating the same script here!
Specifies needed cloud credentials.
NOTE: don't start osbuild-composer in deploy.sh because this is
now done by /usr/libexec/osbuild-composer-test/provision.sh. Otherwise
leads to errors because the socket is already taken.
This builds osbuild in F33aarch64 as well as RHEL8.4 and CentOS8. No tests are currently run.
With the mockbuilds in place we will be able to run composer CI against osbuild master, when necessary.
Previously, we had a webhook relay. It received a notification from Github
and sent it to AWS SQS. Now, the webhook is dead. The new method (already used
in osbuild-composer and image-builder) is to send the notification directly
from a github action to AWS SQS.
Pin the osbuild-composer that schutzbot runs a reverse dependency test
against. This allows to control which exact version to test against, and
ensures that PRs against osbuild always run against the same version.
Now that osbuild-composer's CI uploads RPMs to a predictable destination
(the same one that osbuild uses), we can use that instead of rebuilding
osbuild-composer on every CI run. This should speed up the mockbuild
stage considerably.
Pin it to v24 now.
Drop setting fastestmirror, disabling weak dependencies, and removal of
modular repositories.
Try to install as close to what people do in production, which means
sticking to the defaults.
It was only used once, to retry dnf. This is not necessary, because dnf
already has retrying logic. We're also not using `retry` on any of the
other calls to dnf in this script.
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
The length of these is not predictable. It depends on the shortest
unique prefix in the repository and git configuration.
Just use the full one, which also makes it easier to copy the id from
`git log` or GitHub.
Change the repository path on S3 to a more predictable one. We really
only need the name of the project (static osbuild for this repository),
the name of the distro (use the same as osbuild-composer's API for
consistency) and the commit SHA.
In particular, drop the PR number / branch name. Also don't remove the
dots from version numbers. All places we're using them in (paths and
URLs) support dots.
For example, osbuild commit xxxxxxx for fedora-33 on x86_64 will result
in this URL:
osbuild/fedora-33/x86_64/xxxxxxx
Jenkins has been configured to use the latest commit on a pull request
(instead of merging to master) for a long time now. Rename the variable
to reflect that.
osbuild composer upstream has dropped support for Fedora 31 since
this is very close to end-of-life. Since we use composer master
in our CI this change actually broke it.
s3cmd does not work properly with python 3.9 (used on Fedora 33):
Problem: <class 'AttributeError: 'xml.etree.ElementTree.Element'
object has no attribute 'getchildren'
S3cmd: 2.1.0
python: 3.9.0rc1 (default, Aug 12 2020, 00:00:00)
Use the one from the official repositories, which in case of F33
has a downstream fix for the error.
See rhbz#1884607 and s3tools/s3cmd#1137
Patch based on osbuild-composer commit 1a69a891 by Ondřej Budai.
The osbuild-composer-tests package recently started to list its actual
dependencies, which include packages from EPEL. Enable EPEL in
deploy.sh.
Based on this patch by Ondřej Budai <obudai@redhat.com>:
https://github.com/osbuild/osbuild-composer/pull/1022
Jenkins' declarative pipelines have interesting requirements around when
you can use traditional groovy scripting in the pipeline and some items
in `post` require special handling.
Signed-off-by: Major Hayden <major@redhat.com>
The whole rcm subpackage was removed in osbuild-composer's commit fbfa191.
Unfortunately, this broke osbuild's schutzbot because it tries to start
the rcm socket.
This commit removes enabling of the not-anymore-existing socket unit.
