Add a new `info` property that holds the `meta.ModuleInfo` info
for the stage. This gives each instance of a stage access to
meta (or class) information about it, i.e. its schema, docs but,
more importantly, also its name and path to the executable.
Thefore the `name` property is coverted into a transient property
which access the `name` member of `info`.
Change the `formats/v1` load mechanism to carry a new `index`
argument which is used to load the `ModuleInfo` for each stage.
Adapt all tests to load the info as well when creating stages.
All tests and invocations of `add_stage` actually pass a valid
options dictionary. Thefore move the `options` args before
the `sources` arg and remove the default value (`None`).
Add the python3-iniparse package to f32-build.mpp.json, which is a dependency
of the new 'org.osbuild.rhsm' stage. This is needed in order to be able to
test the 'org.osbuild.rhsm' stage.
Regenerate affected manifests in test/data/manifest using 'make --always-make
test-data'.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Instead of having build pipelines nested within the pipeline it is
the build pipeline for, the nested structure is transferred into a
flat list of pipelines. As a result the recursion is gone and all
the pipelines and trees are build one after the other. This is now
possible since floating objects are kept alive by the store itself
and all trees that are being built are transparently via them.
The immediate result dictionary changed accordingly. To keep the
JSON output of osbuild the same, the result is now routed through
a format specific converter.
Additionally, the v1 format module gained a function to retrieve
the global tree_id and output_id. With the new models those global
ids will go away eventually and thus need to go through the format
specific code.
Instead of passing the store directory to Pipeline.run, pass an
already initialized ObjectStore object. This binds the lifetime
of the store and its (temporary) objects to the run of osbuild
not the run of the pipeline.
This prepares re-using the stores with multiple (non-nested)
pipelines.
Instead of a pipeline, describe now takes a Manifest instance.
The reason is that a manifest fully describes the build, which
includes the sources. Now that the describe function takes the
manifest, the sources can be included as well.
Adapt the tests to refelect that change.
The 'Manifest' class represents what to build and the necessary
sources to do so. For now thus it is just a combination of the
pipeline the source options.
The description of a pipeline is format dependent and thus needs
to be located at the specific format module.
Temporarily remove two tests; they should be added back to a format
specific test suit.
Instead of having the pipeline and the source option as separate
arguments, the load function now takes the full manifest, which
has those two items combined.
Instead of importing the load, load_build functions into the osbuild
namespace and using it via that, use the load function via the module
that provides them, i.e. the formats.v1 module.
The validation of the manifest descritpion is eo ipso format
specific and thus belongs into the format specific module.
Adapt all usages throughout the codebase to directly use the
version 1 specific function.
Add two simple tests to check that the osbuild executable fails with the
right exit codes when passed an invalid manifest or checkpoint.
This reuses test.OSBuild, which is extended to raise CalledProcessError
if needed.
Instead of hard-coding the use of the "org.osbuild.linux" runner,
use the new `osbuild.pipeline.detect_host_runner` function to
dynamically detect the runner for the host system. That should fix
the tests on RHEL systems, where python3 is by default not present
and even if it is manually installed, is an indirection via
alternatives (i.e. a link to /etc/alternatives), which must be
explicitly configured in the build root container for the host.
In `test_output`, require that the test string is found in the
output, not equal to the output. Other warnings or log messages
might be printed by the runner.
As of commit 1e3c0ae "unified libdir handling", the libdir argument
for the BuildRoot constructor is not a keyword argument anymore.
Adapt the argument accordingly.
Also, upgrade the test to fedora-32 from https://osbuild.org/rpmrepo.
Changed a.json to not include a zipl stage with empty options. This
breaks the test, because the osbuild run of b.json is relying on a
checkpoint of the result of a.json. If that's not present, the rpm
database (and other generated files) are marked as differing, because
the whole pipeline is generated again. This might have worked before,
because the previous package list happened to be deterministic.
Also, upgrade the test to fedora-32 from https://osbuild.org/rpmrepo.
Changed a.json to not include a kernel-cmdline stage with empty options.
This breaks the test, because the osbuild run of b.json is relying on a
checkpoint of the result of a.json. If that's not present, the rpm
database (and other generated files) are marked as differing, because
the whole pipeline is generated again. This might have worked before,
because the previous package list happened to be deterministic.
This simplifies the `test-data` rules in `Makefile` considerably. Also,
it allows adding `*.mpp.json` files in other directories without needing
to copy rules. (make's pattern-based rules only allow a single `%`).
Adjust test/data/README.md accordingly.
Create a cache directory with the scope of the stage test suit and
after each test is run cache the downloaded files for the `files`
source via the new `OSBuild.copy_source_data` method. Initialize
OSBuild with that cache directory, so previously downloaded files
get used between different stage tests.
Add a new method that will copy the downloaded data for a specified
source to a directory, creating a directory structure so that the
target directory can then be used to initialize the osbuild cache,
i.e. the target directory be used for the `cache_from` parameter in
the `test.OSBuild` constructor.
The main reason why this is done per source, and not for all sources,
is that not all source can be copied via a plain `cp` operation or
easily added to, in case that the `target` directory already contains
data. The `org.osbuild.ostree` source is such an example.
Use `traceback.print_tb()` to serialize the exceptions' backtrace.
The previously used expression `str(e.__traceback__)` will just
give `<traceback object at 0x…>`, which is not very helpful.
Add a test to check that the method name that raises the exception,
also called `exception`, is in the traceback.
When using `str(type(exception))` this ends up to be something like
`<class 'ValueError'>` for a `ValueError` exception. Get the vanilla
name of the exception type via `type(exception).__name__`.
Add a test to ensure that we encode this properly.
Rename the `API.exception` member to `API.error`, to make it more
generic, so it can also be used for other sort of errors in the
future. Also add a layer of additional structure with `type` and
`data` members so different types of errors apart. Currently only
`exception` is used.
Adapt the tests in test/mod/test_api.py to check for the new
structure and its content.
The `api.exception` method, which is internally also used by
`api.exception_handler` helper, calls `sys.exit` and this no
statement after a call to either method should be reached.
Add an assertion to make sure of that.
Create a new api endpoint called exception, that communicates
exception backtraces separately back to osbuild, as opposed to
dumping them into the normal log. Additionally, add a corresponding
test to check that a call to api.exception correctly sets
API.exception.
Access metadata.api only after `api` has exited the context and
thus the event loop has stopped and all incoming messages, like
the one setting the metadata, have been processed.
See commit 803433fb62 for a lecture
about the internals and all the details involved.
