Let's explain how RPMs for RHEL are built:
We use a subscribed RHEL 8.x machine and mock build these on it. Mock
initializes its own buildroot based on the latest RHEL 8 CDN content, see[1].
This means that the minor version of the buildroot is independent of the minor
version of the host.
However, we currently upload RPMs to a directory whose name consists also of
the minor version of the host. Our hosts are currently running RHEL 8.3 so
the RPMs are uploaded into rhel-8.3 directory despite them being built in the
RHEL 8.4 buildroot (RHEL 8 CDN buildroot specifically). This means that
we cannot guarantee that they are installable on RHEL 8.3 which is weird.
This commit adds a special case for hosts that run on subscribed RHEL and
thus build RPMs in a buildroot constructed from RHEL CDN. These RPMs are
now uploaded into rhel-8-cdn directory. This change more accurately reflects
the way we build our RPMs and removes some confusion.
[1]: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/rhel-8.tpl#L37
s3cmd sync actually downloads metadata for all objects in a s3 bucket.
We have built a lot of RPMs, thus this takes 5 minutes on AWS and 25 minutes
on my laptop (!!!).
Let's use recursive put instead. This doesn't delete any files on the remote
side. As we upload RPMs only once, this also shouldn't fail on "the
object already exists". Using this method, we should be able to upload the
RPMs in seconds.
The same patch was applied in osbuild-composer cf73edd2
RHEL 8.4 is now GA, so we don't need any extra tests for it. This should also
make the CI more reliable because having two distros with the same DISTRO_CODE
caused some tests to fail randomly (they used the same intermediate
artifacts).
This builds osbuild in F33aarch64 as well as RHEL8.4 and CentOS8. No tests are currently run.
With the mockbuilds in place we will be able to run composer CI against osbuild master, when necessary.
Pin the osbuild-composer that schutzbot runs a reverse dependency test
against. This allows to control which exact version to test against, and
ensures that PRs against osbuild always run against the same version.
Now that osbuild-composer's CI uploads RPMs to a predictable destination
(the same one that osbuild uses), we can use that instead of rebuilding
osbuild-composer on every CI run. This should speed up the mockbuild
stage considerably.
Pin it to v24 now.
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
The length of these is not predictable. It depends on the shortest
unique prefix in the repository and git configuration.
Just use the full one, which also makes it easier to copy the id from
`git log` or GitHub.
Change the repository path on S3 to a more predictable one. We really
only need the name of the project (static osbuild for this repository),
the name of the distro (use the same as osbuild-composer's API for
consistency) and the commit SHA.
In particular, drop the PR number / branch name. Also don't remove the
dots from version numbers. All places we're using them in (paths and
URLs) support dots.
For example, osbuild commit xxxxxxx for fedora-33 on x86_64 will result
in this URL:
osbuild/fedora-33/x86_64/xxxxxxx
Jenkins has been configured to use the latest commit on a pull request
(instead of merging to master) for a long time now. Rename the variable
to reflect that.
s3cmd does not work properly with python 3.9 (used on Fedora 33):
Problem: <class 'AttributeError: 'xml.etree.ElementTree.Element'
object has no attribute 'getchildren'
S3cmd: 2.1.0
python: 3.9.0rc1 (default, Aug 12 2020, 00:00:00)
Use the one from the official repositories, which in case of F33
has a downstream fix for the error.
See rhbz#1884607 and s3tools/s3cmd#1137
Patch based on osbuild-composer commit 1a69a891 by Ondřej Budai.
Build the RPMs in a mock using a simple script so that ansible-osbuild
can focus fully on deployment rather than compiling RPMs.
Signed-off-by: Major Hayden <major@redhat.com>
