Metadata information can easily become very big, like in the case
of the package metadata of the org.osbuild.rpm stage, quite likely
exceeding the configured maximum package length of the underlying
socket. To avoid potential issues here, transfer the actual data
by writing it to a temporary file and sending a open fd over.
osbuild composer upstream has dropped support for Fedora 31 since
this is very close to end-of-life. Since we use composer master
in our CI this change actually broke it.
s3cmd does not work properly with python 3.9 (used on Fedora 33):
Problem: <class 'AttributeError: 'xml.etree.ElementTree.Element'
object has no attribute 'getchildren'
S3cmd: 2.1.0
python: 3.9.0rc1 (default, Aug 12 2020, 00:00:00)
Use the one from the official repositories, which in case of F33
has a downstream fix for the error.
See rhbz#1884607 and s3tools/s3cmd#1137
Patch based on osbuild-composer commit 1a69a891 by Ondřej Budai.
Instead of hard-coding the use of the "org.osbuild.linux" runner,
use the new `osbuild.pipeline.detect_host_runner` function to
dynamically detect the runner for the host system. That should fix
the tests on RHEL systems, where python3 is by default not present
and even if it is manually installed, is an indirection via
alternatives (i.e. a link to /etc/alternatives), which must be
explicitly configured in the build root container for the host.
In `test_output`, require that the test string is found in the
output, not equal to the output. Other warnings or log messages
might be printed by the runner.
As of commit 1e3c0ae "unified libdir handling", the libdir argument
for the BuildRoot constructor is not a keyword argument anymore.
Adapt the argument accordingly.
Extract the existing code that creates the runner for the host
build container into a small helper method, so it can be re-used
in other places, like the tests.
Stop the rpm post install script from mangling shebangs for runners,
stages, and assemblers on RHEL. This breaks running them in osbuild
build roots, which might contain a different operating system than the
host, and thus not have platform-python.
This was always the intended behavior, and cb265e8b5 was a step in that
direction.
In effect, this changes a requirement on build roots, because when
building RHEL on RHEL, all current stages and assemblers now have a
python3 shebang. Thus, this means all build roots must have python3
explicitly installed.
Fixes#354
Also, upgrade the test to fedora-32 from https://osbuild.org/rpmrepo.
Changed a.json to not include a zipl stage with empty options. This
breaks the test, because the osbuild run of b.json is relying on a
checkpoint of the result of a.json. If that's not present, the rpm
database (and other generated files) are marked as differing, because
the whole pipeline is generated again. This might have worked before,
because the previous package list happened to be deterministic.
Also, upgrade the test to fedora-32 from https://osbuild.org/rpmrepo.
Changed a.json to not include a kernel-cmdline stage with empty options.
This breaks the test, because the osbuild run of b.json is relying on a
checkpoint of the result of a.json. If that's not present, the rpm
database (and other generated files) are marked as differing, because
the whole pipeline is generated again. This might have worked before,
because the previous package list happened to be deterministic.
Stage tests (test/run/test_stage.py) currently use hard-coded pipelines
from `test/data/stages`. These could benefit from the mpp tools to make
them easier to understand and update to newer versions of Fedora.
Future commits will do so.
Use dnf's per-user cache when `--dnf-cache` is not given. This speeds up
repeated invocations of `mpp-depsolve` considerably, without having to
specify a cache location.
Technically, a few extra "/"s don't matter, but in practice, it
sometimes does. In particular, rpmrepo returns `500 Internal Server
Error` when a path contains additional "/"s.
This simplifies the `test-data` rules in `Makefile` considerably. Also,
it allows adding `*.mpp.json` files in other directories without needing
to copy rules. (make's pattern-based rules only allow a single `%`).
Adjust test/data/README.md accordingly.
Create a cache directory with the scope of the stage test suit and
after each test is run cache the downloaded files for the `files`
source via the new `OSBuild.copy_source_data` method. Initialize
OSBuild with that cache directory, so previously downloaded files
get used between different stage tests.
Add a new method that will copy the downloaded data for a specified
source to a directory, creating a directory structure so that the
target directory can then be used to initialize the osbuild cache,
i.e. the target directory be used for the `cache_from` parameter in
the `test.OSBuild` constructor.
The main reason why this is done per source, and not for all sources,
is that not all source can be copied via a plain `cp` operation or
easily added to, in case that the `target` directory already contains
data. The `org.osbuild.ostree` source is such an example.
Wrap all calls to the various setup functions in the new exception
handler provided by `osbuild.api`. This will make sure that any
exception is properly printed to stderr, as well as communicated
to osbuild in a structured and machine readable way.
Use `traceback.print_tb()` to serialize the exceptions' backtrace.
The previously used expression `str(e.__traceback__)` will just
give `<traceback object at 0x…>`, which is not very helpful.
Add a test to check that the method name that raises the exception,
also called `exception`, is in the traceback.
When using `str(type(exception))` this ends up to be something like
`<class 'ValueError'>` for a `ValueError` exception. Get the vanilla
name of the exception type via `type(exception).__name__`.
Add a test to ensure that we encode this properly.
Rename the `API.exception` member to `API.error`, to make it more
generic, so it can also be used for other sort of errors in the
future. Also add a layer of additional structure with `type` and
`data` members so different types of errors apart. Currently only
`exception` is used.
Adapt the tests in test/mod/test_api.py to check for the new
structure and its content.
The `api.exception` method, which is internally also used by
`api.exception_handler` helper, calls `sys.exit` and this no
statement after a call to either method should be reached.
Add an assertion to make sure of that.
The osbuild-composer-tests package recently started to list its actual
dependencies, which include packages from EPEL. Enable EPEL in
deploy.sh.
Based on this patch by Ondřej Budai <obudai@redhat.com>:
https://github.com/osbuild/osbuild-composer/pull/1022
