Fix RHEL 10 mock confings not being created if the machine was subscribed and fix RHEL 9 being used in the RHEL 10 mock confings for a repo path.
These issues caused the rpm jobs for RHEL 10 GA to fail.
Add workarounds needed for mockbuild.sh to work on c10s / el10, due to
non-existent EPEL-10 and issues on c10s.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The pulp client is very large and defines a lot of symbols in on
package, which causes very large memory usage on el8 aarch64 (presumably
because of 64k page sizes).
Adding a 1 GiB swapfile fixes the issue in our CI runners.
DNF has more elaborate locking system and can wait for other instances of
itself when installing packages. Using rpm directly to install local
package is causing failures in CI due to it not being able to acquire
lock on `/var/lib/rpm/.rpm.lock`.
Using DNF should improve the situation, although there is no good
documentation to link and support this claim for sure.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
RPM Spec
--------
Remove all Go dependecies
Add Start and End marker comments for bundling information
Add '-k' to goprep to preserve the vendor directory
tools
-----
Add script to update the RPM spec file to generate the indication lines
based on vendor/modules.txt
Packit
------
Run the new script as a post-upstream-clone hook
Makefile
--------
Run the new script on the generated spec file before generating the RPM
mockbuild.sh
------------
Run the new script before creating the RPM
- name mangling from centos to centos-stream was enabled also for
centos 9
- CS9 mock configs from mock-core-configs-36.3-1.fc35.noarch were added
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The commonly used 'greenprint' function now adds a date + timestamp to
each message for debugging and tracking the duration of segments of each
scripts.
We are no longer using the variable names that s3cmd uses by default
so we need to configure it explicitly.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Let's explain how RPMs for RHEL are built:
We use a subscribed RHEL 8.x machine and mock build these on it. Mock
initializes its own buildroot based on the latest RHEL 8 CDN content, see[1].
This means that the minor version of the buildroot is independent of the minor
version of the host.
However, we currently upload RPMs to a directory whose name consists also of
the minor version of the host. Our hosts are currently running RHEL 8.3 so
the RPMs are uploaded into rhel-8.3 directory despite them being built in the
RHEL 8.4 buildroot (RHEL 8 CDN buildroot specifically). This means that
we cannot guarantee that they are installable on RHEL 8.3 which is weird.
This commit adds a special case for hosts that run on subscribed RHEL and
thus build RPMs in a buildroot constructed from RHEL CDN. These RPMs are
now uploaded into rhel-8-cdn directory. This change more accurately reflects
the way we build our RPMs and removes some confusion.
Also, we need to bump osbuild commit so we have a version that already has
the rhel-8-cdn change in it.
[1]: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/rhel-8.tpl#L37
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
EXTRA_REPO_PATH_SEGMENT was needed when both Jenkins and Gitlab CI
were running in parallel (so they don't override their artifacts).
Jenkins is now decommissioned so we can drop the variable.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
s3cmd sync actually downloads metadata for all objects in a s3 bucket.
We have built a lot of RPMs, thus this takes 5 minutes on AWS and 25 minutes
on my laptop (!!!).
Let's use recursive put instead. This doesn't delete any files on the remote
side. As we upload RPMs only once, this also shouldn't fail on "the
object already exists". Using this method, we should be able to upload the
RPMs in seconds.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
GitLab CI builds its own rpms and thus it must be use a different path.
This commit modifies mockbuild.sh and deploy.sh to be able to add an
extra path segment into the path so GitLab can use a different path.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
by defining the COMPOSE_URL environment variable! This will allow
testing more flavors of internal releases.
The rest is renaming files and variables to reflect the fact that
we're running tests against internal trees, not only nightlies.
Mockbuild using systemd-nspawn currently fails on Fedora 34. The
workaround is to use "simple" isolation method - the traditional
chroot() call.
Reported as: https://bugzilla.redhat.com/show_bug.cgi?id=1931452
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The image definition is shared with the latest RHEL 8.y one (8.4 currently).
I expect that we the introduction of 8.5 support, we point the centos 8
distro at it.
The test repositories and manifests use the official CentOS composes. From
what I can tell, they are persistent. This is not guaranteed though, so we
might need to switch to RPMRepo at some point.
The "classic" CentOS 8 should also be buildable but due to the chicken and egg
issue (this commit will get into Centos "8.4" but Centos "8.4" isn't a thing
yet), we cannot test it and therefore it might be broken.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
rhel 8.4 tests are added. The configs are based off of those used for
rhel 8.3. The Schutzbot Mockbuild, Base, Image, Integration, and OSTree
tests are added for 8.4. Repo overrides are added for the rhel 8.4 tests
so that the tests use rpmrepo snapshots.
The mockbuild uses the jenkins rhel84-nightly-repo credential to
override the rhel mock template's repos with rhel 8.4 nightly repos.
These repos are stored in a credential because they are internal links.
The image tests and koji tests need a special distro selector since the
rhel-8 test cases are only for rhel 8 versions less than 8.4. The rhel
8.4 tests are named with the rhel-84 pattern whereas the other rhel 8
versions have the rhel-8 pattern.
Also, instead of having only rhel-8 and rhel-8-beta repo configs for the
tests, we now have a specific repo config for each rhel release we test.
The repo is also now pulled from an rpmrepo snapshot. For whichever
distro is being tested, the approriate repo config will be copied to
/etc/osbuild-composer/repositories as rhel-8 and rhel-8-beta since this
is the naming osbuild-composer looks for. For testing purposes, the
rhel-8 and rhel-8-beta repo should be the same since eventually all rhel
releases will go from beta to not beta. The fedora repo overrides are
already done in tools/provision.sh so the rhel override is set there as
well. Currently, only rhel 8.4 requires an override.
Running `make srpm` on the host can have different results from building
the SRPM in mock.
Stop using the Makefile altogether for this, which is meant for
developer convenience only. The interface is clear: a tarball generated
from `git archive` and defining the `commit` variable. Less stuff to
install on the host, too.
To make sure we test against what we will release with, use osbuild from
the target distro rather than rebuilding it.
The only exception is in RHEL8.3, where we are not rebasing any longer,
but expect users who use new osbulid-composer from upstream to use it
with new osbuild from upstream too. In this case, use the RPM from
osbuild upstream, for now pinned to the same version as was in the
submodule.
This introduces a new configuration file: Schutzfile, which is meant
to contain the full test-matrix. For now it only points to the upstream
osbuild commit to add to the distro we are testing against (only relevant
for 8.3).
The submodule is now unused and is therefore removed. The produced repos
now only contain osbuild-composer, osbuild is never built as part of
osbuild-composer CI.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
The length of these is not predictable. It depends on the shortest
unique prefix in the repository and git configuration.
Just use the full one, which also makes it easier to copy the id from
`git log` or GitHub.
Change the repository path on S3 to a more predictable one. We really
only need the name of the project (static osbuild-composer for this
repository), the name of the distro (use the same as osbuild-composer's
API for consistency) and the commit SHA.
In particular, drop the PR number / branch name. Also don't remove the
dots from version numbers. All places we're using them in (paths and
URLs) support dots.
For example, osbuild composer commit xxxxxxx for fedora-33 on x86_64
will result in this URL:
osbuild-composer/fedora-33/x86_64/xxxxxxx
