s3cmd sync actually downloads metadata for all objects in a s3 bucket.
We have built a lot of RPMs, thus this takes 5 minutes on AWS and 25 minutes
on my laptop (!!!).
Let's use recursive put instead. This doesn't delete any files on the remote
side. As we upload RPMs only once, this also shouldn't fail on "the
object already exists". Using this method, we should be able to upload the
RPMs in seconds.
The same patch was applied in osbuild-composer cf73edd2
Let's explain how RPMs for RHEL are built:
We use a subscribed RHEL 8.x machine and mock build these on it. Mock
initializes its own buildroot based on the latest RHEL 8 CDN content, see[1].
This means that the minor version of the buildroot is independent of the minor
version of the host.
However, we currently upload RPMs to a directory whose name consists also of
the minor version of the host. Our hosts are currently running RHEL 8.3 so
the RPMs are uploaded into rhel-8.3 directory despite them being built in the
RHEL 8.4 buildroot (RHEL 8 CDN buildroot specifically). This means that
we cannot guarantee that they are installable on RHEL 8.3 which is weird.
This commit adds a special case for hosts that run on subscribed RHEL and
thus build RPMs in a buildroot constructed from RHEL CDN. These RPMs are
now uploaded into rhel-8-cdn directory. This change more accurately reflects
the way we build our RPMs and removes some confusion.
Also, we need to bump osbuild commit so we have a version that already has
the rhel-8-cdn change in it.
This also bumps all deps so we have rhel-8-cdn repos everywhere.
[1]: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/rhel-8.tpl#L37
Repository URLs are predictable. There's no need to use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
Change the repository path on S3 to a more predictable one, mirroring
the pattern we're using for osbuild-composer.
Notably, don't use short commit ids. The length of these is not
predictable. It depends on the shortest unique prefix in the repository
and git configuration.
For example, koji-osbuild commit $SHA for fedora-33 on x86_64 will
result in this URL:
koji-osbuild/fedora-33/x86_64/$SHA
The CI is in two stages, for each supported distro.
First the RPMs are generated from the spec file in the repo for the
given distro and architecture.
Once all the RPM builds have succeeded successfully, a test machine is
provisioned with osbulid-composer installed, and koji API enabled.
The repository containing the RPMs of the code being tested is also
enabled on the test machine, and the cli client is installed.
Finally, the test/integration.sh script is executed, which currently
does nothing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
