Pipelines encode which source content they need in the form of
repository metadata checksums (or rpm checksums). In addition, they
encode where they fetch that source content from in the form of URLs.
This is overly specific and doesn't have to be in the pipeline's hash:
the checksum is enough to specify an image.
In practice, this precluded using alternative ways of getting at source
packages, such as local mirrors, which could speed up development.
Introduce a new osbuild API: sources. With it, a stage can query for a
way to fetch source content based on checksums.
The first such source is `org.osbuild.dnf`, which returns repository
configuration for a metadata checksum. Note that the dnf stage continues
to verify that the content it received matches the checksum it expects.
Sources are implemented as programs, living in a `sources` directory.
They are run on the host (i.e., uncontained) right now. Each source gets
passed options, which are taken from a new command line argument to
osbuild, and an array of checksums for which to return content.
This API is only available to stages right now.
A pipeline run only returned logs in the `StageFailed` and
`AssemblerFailed` exceptions. Remove those and always return structured
data instead.
It only returns data for stages that actually ran (i.e., didn't come
from the cache). This is similar to the output in interactive mode.
Also change osbuildtest to be able to deal with output that is larger
than the pipe buffer by using subprocess.communicate().
We've been using a generic `osbuild-run`, which sets up the build
environment (and works around bugs) for all build roots. It is already
getting unwieldy, because it tries to detect the OS for some things it
configures. It's also about to cause problems for RHEL, which doesn't
currently support a python3 shebang without having /etc around.
This patch changes the `build` key in a pipeline to not be a pipeline
itself, but an object with `runner` and `pipeline` keys. `pipeline` is
the build pipeline, as before. `runner` is the name of the runner to
use. Runners are programs in the `runners` subdirectory.
Three runners are included in this patch. They're copies of osbuild-run
for now (except some additions for rhel82). The idea is that each of
them only contains the minimal setup code necessary for an OS, and that
we can review what's needed when updating a build root.
Also modify the `--build-pipeline` command line switch to accept such a
build object (instead of a pipeline) and rename it accordingly, to
`--build-env`.
Correspondingly, `OSBUILD_TEST_BUILD_PIPELINE` → `OSBUILD_TEST_BUILD_ENV`.
When the test runner receives SIGINT, osbuild's mounts stay around.
osbuild handles SIGING correctly, but it doesn't have time before being
killed because it's parent went away.
Fix this by waiting on it explicitly in the test runner.
Fixes#119
For stages testing it is too slow to rebuild the a image containing
@Core packages every time. Let's just reuse the a image for all tests.
This should speed up the test running time a LOT.
The stage testing is based on an output from the tree-diff tool. During
one test two pipelines are run and their outputs are compared using
tree-diff. The diff is then compared with expected diff included in
the repository.
The testosbuild.TestCase class creates a fresh store for each test,
because tests should run independent of each other.
This can lead to long waiting times while developing a new test case.
Allow overriding the store used with OSBUILD_TEST_STORE. This should
never be used where tests are actually run. It is a development-only
feature.
Use the unittest module from the standard library. Also, ensure that
separate runs of this test don't share a osbuild store and clean up
after themselves.
With contributions from Ondřej Budai and Tom Gundersen.
