Pin the osbuild-composer that schutzbot runs a reverse dependency test
against. This allows to control which exact version to test against, and
ensures that PRs against osbuild always run against the same version.
Now that osbuild-composer's CI uploads RPMs to a predictable destination
(the same one that osbuild uses), we can use that instead of rebuilding
osbuild-composer on every CI run. This should speed up the mockbuild
stage considerably.
Pin it to v24 now.
Drop setting fastestmirror, disabling weak dependencies, and removal of
modular repositories.
Try to install as close to what people do in production, which means
sticking to the defaults.
It was only used once, to retry dnf. This is not necessary, because dnf
already has retrying logic. We're also not using `retry` on any of the
other calls to dnf in this script.
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
The length of these is not predictable. It depends on the shortest
unique prefix in the repository and git configuration.
Just use the full one, which also makes it easier to copy the id from
`git log` or GitHub.
Change the repository path on S3 to a more predictable one. We really
only need the name of the project (static osbuild for this repository),
the name of the distro (use the same as osbuild-composer's API for
consistency) and the commit SHA.
In particular, drop the PR number / branch name. Also don't remove the
dots from version numbers. All places we're using them in (paths and
URLs) support dots.
For example, osbuild commit xxxxxxx for fedora-33 on x86_64 will result
in this URL:
osbuild/fedora-33/x86_64/xxxxxxx
Jenkins has been configured to use the latest commit on a pull request
(instead of merging to master) for a long time now. Rename the variable
to reflect that.
Explicit re-raise the BufferError exception in recv from the orignal
JSONDecodeError, so the latter gets recorded as the underlying cause.
Uncovered by pylint 2.6.0: W0707: "Not using raise from makes the
traceback inaccurate, because the message implies there is a bug in
the exception-handling code itself, which is a separate situation
than wrapping an exception."
All runners stopped calling `api.setup_stdio` (commit c40b414), and
thus all output of runners and also modules is now redirected to a
pipe (created via Popen and subprocess.PIPE for stdout).
Text was read from that pipe via `stdout.read(4096)`, which means
that it is now buffered in chunks of 4096, where it previously was
line buffered in the case that osbuild was run in the terminal and
--json was not specified. This is very annoying for anyone wanting
to follow osbuild's output in real-time.
Restore the previous behavior by using `os.read`, which should be
a small wrapper around read(3), which does not block until all the
requested data is available but returns early (short reads). This
means, new text will be forwarded as soon is it is available in the
pipe. Increase the read buffer to 32768 while at it, which is what
Popen is using in Python 3.9.
Add two simple tests to check that the osbuild executable fails with the
right exit codes when passed an invalid manifest or checkpoint.
This reuses test.OSBuild, which is extended to raise CalledProcessError
if needed.
osbuild_cli() sometimes returned an exit code, but at the end called
sys.exit() directly. The idea was probably to always return the code
with which the executable should exit.
Make this consistent and call sys.exit() in __main__.py, with the value
returned by osbuild_cli().
Highlights are metadata fields are return for packages in the rpm
stage, the spec file has been fixed to not mangle shebangs anymore.
Generally, CI and testing has been improved.
Metadata information can easily become very big, like in the case
of the package metadata of the org.osbuild.rpm stage, quite likely
exceeding the configured maximum package length of the underlying
socket. To avoid potential issues here, transfer the actual data
by writing it to a temporary file and sending a open fd over.
osbuild composer upstream has dropped support for Fedora 31 since
this is very close to end-of-life. Since we use composer master
in our CI this change actually broke it.
s3cmd does not work properly with python 3.9 (used on Fedora 33):
Problem: <class 'AttributeError: 'xml.etree.ElementTree.Element'
object has no attribute 'getchildren'
S3cmd: 2.1.0
python: 3.9.0rc1 (default, Aug 12 2020, 00:00:00)
Use the one from the official repositories, which in case of F33
has a downstream fix for the error.
See rhbz#1884607 and s3tools/s3cmd#1137
Patch based on osbuild-composer commit 1a69a891 by Ondřej Budai.
Instead of hard-coding the use of the "org.osbuild.linux" runner,
use the new `osbuild.pipeline.detect_host_runner` function to
dynamically detect the runner for the host system. That should fix
the tests on RHEL systems, where python3 is by default not present
and even if it is manually installed, is an indirection via
alternatives (i.e. a link to /etc/alternatives), which must be
explicitly configured in the build root container for the host.
In `test_output`, require that the test string is found in the
output, not equal to the output. Other warnings or log messages
might be printed by the runner.
As of commit 1e3c0ae "unified libdir handling", the libdir argument
for the BuildRoot constructor is not a keyword argument anymore.
Adapt the argument accordingly.
Extract the existing code that creates the runner for the host
build container into a small helper method, so it can be re-used
in other places, like the tests.
Stop the rpm post install script from mangling shebangs for runners,
stages, and assemblers on RHEL. This breaks running them in osbuild
build roots, which might contain a different operating system than the
host, and thus not have platform-python.
This was always the intended behavior, and cb265e8b5 was a step in that
direction.
In effect, this changes a requirement on build roots, because when
building RHEL on RHEL, all current stages and assemblers now have a
python3 shebang. Thus, this means all build roots must have python3
explicitly installed.
Fixes#354
Also, upgrade the test to fedora-32 from https://osbuild.org/rpmrepo.
Changed a.json to not include a zipl stage with empty options. This
breaks the test, because the osbuild run of b.json is relying on a
checkpoint of the result of a.json. If that's not present, the rpm
database (and other generated files) are marked as differing, because
the whole pipeline is generated again. This might have worked before,
because the previous package list happened to be deterministic.
Also, upgrade the test to fedora-32 from https://osbuild.org/rpmrepo.
Changed a.json to not include a kernel-cmdline stage with empty options.
This breaks the test, because the osbuild run of b.json is relying on a
checkpoint of the result of a.json. If that's not present, the rpm
database (and other generated files) are marked as differing, because
the whole pipeline is generated again. This might have worked before,
because the previous package list happened to be deterministic.
