This new manifests creates an boot iso which contains an ostree
commit that is build in the same manifest. The boot iso can then
be installed via the ISO.
This is to test the functionality of the new `saved_entry` grub2
config file. Ideally an integration test would install a new non-
default kernel and check that it does not get selected. Something
for the future.
In addition to the required base layer, provided via the the input
of the same name, the oci-archive stage now accepts up to nine
additional layers that get added on top of each other, sorted in
ascending order, i.e. `layer.1` to `layer.9`.
Adapt the `fedora-ostree-container` example manifest so that the
ostree commit is now in a separate layer, which makes it possible
to share the base layer between different commits container.
Add a new basic test for the dracut stage. It uses a osbuild
pipeline to build an initrd and inspects it via the initrd.py
module. The content is compared to a reference located in the
same directory as the pipeline (test/data/stages/dracut/).
Add a new module that contains a utilities to inspect an initrd.
It contains a `Initrd` class that loads a given initrd and can
then be used get the list of files via Initrd.filelist, the dracut
modules via `Initrd.modules` and the kernel modules via the
`Initrd.kmods` properties. Another top level function `read_initrd`
returns a dict that contains the name of the initrd as key and as
value another dictionary with `modules`, `kmods` as well as the
`compression` & `early_cpio` metadata.
The initrd.py can also use as a script that will print the dict
from `read_initrd` as JSON.
The implementation is a heavily based on dracut's `lsinitrd` bash
script and additionally contains a python port of the skipcpio.c
utility.
Change the test manifests that use containers to not include
docs when installing. Also don't install docs in the build root
for those manifests. Since the fedora-ostree-container.mpp is
being built in CI, this also tests that the new option.
The `org.osbuild.files` source provides files, but might in the
future not be the only one that does. Therefore rename it to
match the internal tool that is being used to fetch the files.
This is done for most other osbuild modules that target tools.
The format v1 loader is adapted to make this change transparent
for users of the v1 format, so we are backwards compatible.
Change the MPP depsolve preprocessor so that for format v2 based
manifest `org.osbuild.curl` source is used. Also rename the
corresponding source test. Adapt the format v2 mod test to use
the curl source.
Add a v2 manifest to builds an ostree commit, then builds a small
container with a webserver and puts that commit into it.
Start a new test suit where this manifest is built and then checked
that the artifact exists. In the future the artifact should also be
inspected and run by a container engine and the commit be pulled
via ostree.
Add a basic check to verify that loading and then describing the
pipeline results in the same description that was put in. This
test is esp. valuable because it checks the runner mapping and
name, id mappings.
Add a new test to check that validation works for the basic test
pipeline. This needs to be extended in the future to check that
invalid data is being caught properly, but it is a start.
Add a simple noop pipeline, with a noop stage and a noop input,
which all in all does nothing much; but it will validate, load
exectue the pipeline, stage and inputs. So maybe not really
"nothing" in the strictest sense.
Add an example that builds a small container, based on the minimal
container fedora kickstart (see [1, 2]). There is a deviation
because osbuild currently does not support removing
random files like it is done in [2].
The build pipeline "build" in f32-build-v2.json) is a port of the
existing build pipeline in f32-build.json.
This manifest is currently not used in any automated tests (yet),
but is there as a showcase for the new format. It does however
test the format version 2 support for mpp, since that is called
in CI.
[1] https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-common.ks
[2] https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base-minimal.ks
Change the `ModuleInfo.schema` propertly into a `get_schema`
method call. This is in preparation to allow for different
schemata versions to be supported.
Commit d028ea5b16 introduced bug when introducing the `store`
argument to `Stage.run`, instead of passing `var=var`, i.e.
`var` is being passed as keyword argument, it is now being
passed as a positional one. Since the `path=/run/osbuild`
keyword argument comes before the `var=/var/tmp` argument,
`var` is now being passed as `path` instead of var.
Since `var` is always being passed in throughout the entire
codebase, make it a positional argument, and move it before
`path`.
Adapt the tests to pass `var` as positional argument.
Instead of using `org.osbuild.test` stages, so `org.osbuild.noop`
stages so that the options we pass are actually valid and we can
use the manifest in schema validation checking.
Add a new `add_source` method that will add an individual `Source`
to a `Manifest` give its `ModuleInfo` and options. The dictionary
of source options in the manifest is replaced with a list of such
`Sources` and `add_source` will append to it. Adap the version 1
format code to use `add_source` and reconstruct the source options
from the list of source on `describe`.
Remove the `sources_options` constructor parameter for `Manifest`
and adapt all the source base for this.
The stage was structured in a way so that it supports many different
inputs, like, `archive` and in the future maybe trees, git checkouts.
This does not fit very well into the new input model, where instead
seperate `copy.tree` and `copy.files` stages might make more sense.
Since the stage is not been used anywhere, and mainly got added
as a way to get the ignition dracut module via an archive of a git
repo, remove the stage for now. A replacement will be brought back
in the future.
The sysconfig directory contains a variety of system configuration files. The
values found in each of the configuration files may need to be specified
before first boot.
Currently, only the kernel and network configs can be modified and will
override existing files.
In the test case for the tar assembler, actually verify the
content by un-tar-ing the result again and comparing it to
the tree. This would have spotted missing SELinux labels.
Convert the assembler phase of the main pipeline in the old format
into a new Pipeline that as the assembler as a stage, where the
input of that stage is the main pipeline. This removes the need of
having "assemblers" as special concepts and thus the corresponding
code in `Pipeline` is removed. The new assembler pipeline is marked
as exported, but the pipeline that builds the tree is not anymore.
Adapt the `describe` and `output` functions of the `v1` format to
handle the assembler pipeline. Also change the tests accordingly.
NB: The id reported for the assembler via `--inspect` and the result
will change as a result of this, since the assembler stage is now
the first and only stage of a new pipeline and thus has no base
anymore.
Every pipeline that gets added to the `Manifest` now need to have
a unique name by which it can be identified. The version 1 format
loader is changed so that the main pipeline that builds the tree
is always called `tree`. The build pipeline for it will be called
`build` and further recursive build pipelines `build-build`, where
the number of repetitions of `build` corresponds to their level of
nesting. An assembler, if it exists, will be added as `assembler`.
The `Manifest.__getitem__` helper is changed so it will first try
to access pipeline via its name and then fall back to an id based
search. NB: in the degenrate case of multiple pipelines that have
exactly the same `id`, i.e. same stages, with the same options and
same build pipeline, only the first one will be return; but only
the first one here will be built as well, so this is in practice
not a problem.
The formatter uses this helper to get the tree pipeline via its
name wherever it is needed.
This also adds an `__iter__` method `Manifest` to ease iterating
over just the pipeline values, a la `for pipeline in manifet`.
The current description test took a hand crafted Manifest. Since
the internal `Manifest` representation and the format (version 1)
representation are becoming more and more different, it might
soon not be possible to go from `Manifest` to the format specific
description unless it was previously loaded and thus an internal
mapping table can aid the serialization. Thus the description
test is re-worked to load a manifest and then describe it and
compare that the result is the same.
Also rename the test from `test_pipeline` to `test_describe`
as this is a better description of the test.
The netns() function sets up a new namespace for tests. The function is
also used to determine whether those tests can be run (using
unittest.skipUnless()). A bug in the function made the changes stick if
the function failed early. Specifically, when the "ip link" line fails,
the function exits without reverting to the old namespace.
Since the code is used in "skipUnless()", it's run during test
collection, which means that even if the relevant tests aren't selected,
they affect the environment for other tests.
Now that assemblers are represented via the `Stage` class, the
Assembler class is not needed anymore. Adjust the monitor method
to take an `pipeline.Stage` for the `assembler` method as well.
Instead of using the `Assemblers` class to represent assemblers,
use the `Stage` class: The `Pipeline.add_assembler` method will
now instantiate and `Stage` instead of an `Assembler`. The tree
that the pipeline built is converted to an Input (while loading
the manifest description in `format/v1.py`) and all existing
assemblers are converted to use that input as the tree input.
The assembler run test is removed as the Assembler class itself
is not used (i.e. run) anymore.
Inputs are modules like Stages, Assemblers and Sources. Add them
as a new module klass to the various functions. Include them in
the schema test, so the schema of all inputs is validated.
Also sort the module classes alphabetically in the class mapping
and class list.
Add a new `info` property that holds the `meta.ModuleInfo` info
for the stage. This gives each instance of a stage access to
meta (or class) information about it, i.e. its schema, docs but,
more importantly, also its name and path to the executable.
Thefore the `name` property is coverted into a transient property
which access the `name` member of `info`.
Change the `formats/v1` load mechanism to carry a new `index`
argument which is used to load the `ModuleInfo` for each stage.
Adapt all tests to load the info as well when creating stages.
All tests and invocations of `add_stage` actually pass a valid
options dictionary. Thefore move the `options` args before
the `sources` arg and remove the default value (`None`).
Add the python3-iniparse package to f32-build.mpp.json, which is a dependency
of the new 'org.osbuild.rhsm' stage. This is needed in order to be able to
test the 'org.osbuild.rhsm' stage.
Regenerate affected manifests in test/data/manifest using 'make --always-make
test-data'.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Instead of having build pipelines nested within the pipeline it is
the build pipeline for, the nested structure is transferred into a
flat list of pipelines. As a result the recursion is gone and all
the pipelines and trees are build one after the other. This is now
possible since floating objects are kept alive by the store itself
and all trees that are being built are transparently via them.
The immediate result dictionary changed accordingly. To keep the
JSON output of osbuild the same, the result is now routed through
a format specific converter.
Additionally, the v1 format module gained a function to retrieve
the global tree_id and output_id. With the new models those global
ids will go away eventually and thus need to go through the format
specific code.
Instead of passing the store directory to Pipeline.run, pass an
already initialized ObjectStore object. This binds the lifetime
of the store and its (temporary) objects to the run of osbuild
not the run of the pipeline.
This prepares re-using the stores with multiple (non-nested)
pipelines.
