Commit d028ea5b16 introduced bug when introducing the `store`
argument to `Stage.run`, instead of passing `var=var`, i.e.
`var` is being passed as keyword argument, it is now being
passed as a positional one. Since the `path=/run/osbuild`
keyword argument comes before the `var=/var/tmp` argument,
`var` is now being passed as `path` instead of var.
Since `var` is always being passed in throughout the entire
codebase, make it a positional argument, and move it before
`path`.
Adapt the tests to pass `var` as positional argument.
All sources fetch various types of `items`, the specific nature
of which is dependent on the source type, but they are all
identifyable by a opaque identifier. In order for osbuild to
check that all the inputs that a stage needs are are indeed
contained in the manifest description, osbuild must learn what
ids are fetched by what source. This is done by standarzing
the common "items" part, i.e. the "id" -> "options for that id"
mapping that is common to all sources.
For the version 1 of the format, extract the files and ostree
the item information from the respective options.
Adapt the sources (files, ostree) so that they use the new items
information, but also fall back to the old style; the latter is
needed since the sources tests still uses the SourceServer.
Now that meta.Index provides a way to detect the format given the
input data, use that method instead of the hard-coded use of the
version 1 format via the input. This should make the main entry
point format independent.
Introdcue a `FormatInfo` class that, very much like `ModuleInfo`
can be used to obtain meta information about a format. Methods
are added to `Index` to allow the enumeration of available formats,
getting the `FormatInfo` for a format given its name and to detect
a format via the manifest description data.
Change the top-level documentation to reflect the changes. Also
remove an outdated section about validation of the schema; this
was moved to the format specific code some time ago.
Convert the `org.osbuild.ostree` stage to use inputs instead of
sources. In the format (version 1) loading code, convert the
stage to use an input based on the existing stage options.
Add a new input that can handle ostree commits. For source origins,
it will pull any number of commits specified via `references`. For
pipeline origins it will use the compose.json to obtain the commit
id and pull that. Either way the data passed onto the stage is a
dictionary of commit ids and commit metadata, such as the `ref`
that is an ostree ref that is optionally created in case the `ref`
option for that reference was specified.
Instead of manually constructing and appending the input for
stages (here the stages that replace the assembler), use the
new `Stage.add_input` method.
In much the same way has `Pipeline` has `add_stage` and `Manifest`
has `add_pipeline`, introduce an `add_input` method to `Stage` to
be able add `Inputs` to stages.
The `info` parameter for Input constructor is of type `PModuleInfo`,
which is located in `meta`. This in turn imports jsonschema. Ergo,
importing importing `inputs` will create a dependency on jsonschema.
At the same time the `osbuild` package globally imports `Pipeline`,
via `__init__.py`, and `osbuild.api` is used in the runners and
stages, which are run inside the buildroot. If one now wanted to
use `inputs` from `Pipeline`, it would lead to jsonschema being
imported (via `meta`) which might not be available on the build-
root and it is a rather random dependency to have.
On obvious solution would be to use a construct with `TYPE_CHECKING`,
a la:
if TYPE_CHECKING:
from .meta import ModuleInfo
Sadly, pylint will now complain about it. This could be fixed with:
if TYPE_CHECKING:
from .meta import ModuleInfo
else:
ModuleInfo = "osbuild.meta.ModuleInfo"
But this is just gross. So we will have to accept that Python is,
well, Python und omit the type information for the `info` param.
Add a new jsoncomm rpc method call, `source`, that will return
the directory within the store where resources for that specific
type of resource, like e.g. tree, files, or ostree can be found
or stored.
Currently all options for inputs are totally opaque to osbuild
itself. This is neat from a seperation of concerns point of view
but has one major downside: osbuild can not verify the integrity
of the pipeline graph, i.e. if all inputs that need pipelines or
sources do indeed exists. Therefore intrdouce two generic fields
for inputs: `origin` and `references`. The former can either be
a source or a pipeline. The latter is an array of identifiers or
a dictionary where the keys are the identifiers and the values
are additional options for that id. The identifiers then refer
to either resources obtained via a source or a pipeline that has
already been built.
Instead of using `org.osbuild.test` stages, so `org.osbuild.noop`
stages so that the options we pass are actually valid and we can
use the manifest in schema validation checking.
Add a new `add_source` method that will add an individual `Source`
to a `Manifest` give its `ModuleInfo` and options. The dictionary
of source options in the manifest is replaced with a list of such
`Sources` and `add_source` will append to it. Adap the version 1
format code to use `add_source` and reconstruct the source options
from the list of source on `describe`.
Remove the `sources_options` constructor parameter for `Manifest`
and adapt all the source base for this.
Very much like, stages and inputs, a new `Source` calls represents
a source type with its options. A `download` method on the `Source`
class can be used to only donwload content to the cache.
Split the internal logic into two parts: 1) fetching the commit
into the internal cache repo and then 2) exporting that commit,
i.e. a local pull from the cache repo to the output directory.
If no `output` directory was specified, only fetch the commit,
do not attempt to export it.
NB: this commit changes at what point the gpg verification is
done. Previously the check was on export. Now, we are checking
the signature on import only. The export step will be replaced
by an ostree `Input` that will have the ability to verify
commits a second time.
Split the internal logic of the stage in two parts: 1) downloading
files to the internal cache and 2) exporting the downloaded files
from said cache to the output directory. Additionally, ff no such
`output` directory was specified, i.e. it is empty or `None`, only
download files but do not attempt to export them.
The stage was structured in a way so that it supports many different
inputs, like, `archive` and in the future maybe trees, git checkouts.
This does not fit very well into the new input model, where instead
seperate `copy.tree` and `copy.files` stages might make more sense.
Since the stage is not been used anywhere, and mainly got added
as a way to get the ignition dracut module via an archive of a git
repo, remove the stage for now. A replacement will be brought back
in the future.
The sysconfig directory contains a variety of system configuration files. The
values found in each of the configuration files may need to be specified
before first boot.
Currently, only the kernel and network configs can be modified and will
override existing files.
Add the ability to opt out of preserving the ACLs, SELinux
contexts and extended attributes. It is opt out instead of
opt in since the assembler by default tries to preserve as
much as possible.
In the test case for the tar assembler, actually verify the
content by un-tar-ing the result again and comparing it to
the tree. This would have spotted missing SELinux labels.
All of these options, i.e. SELinux labels, ACLs and extended
attributes (xattrs), are opt-in and thus were currently ignored.
This lead to trees that had their selinux labels missing and
were thus incorrect.
Detect the case that one file has a SELinux label but the other
file does not have any label at all. This was currently not
possible to detect because both calls to get the labels were
wrapped in one try-except block and any failure in one of the
two calls, like a missing label, would lead to an early return
of the function, with a success value.
This source has been declared obsolete some time ago and is not
support anymore. We wont support it in the upcoming new manifest
format, therefore drop it now.
Include all inputs of a stage during the calculation of its id,
since they determine, very much like options, the content the
stage produces; thus different inputs should lead to different
ids.
Add an `id` property that, like `Stage.id`, can be used to uniquely
identify an input based on its name and options. Two stages with
the same name and options will have the same `id`.
Now that `Pipelines` have no assemblers anymore and thus only one
identifier, i.e. the one corresponding to the tree (`tree_id`),
the `id` and `tree_id` are now the same. Therefore replace the
usage of `tree_id` with `id` and drop the former. Add some extra
documentation including some caveats about the uniquness of `id`.
Convert the assembler phase of the main pipeline in the old format
into a new Pipeline that as the assembler as a stage, where the
input of that stage is the main pipeline. This removes the need of
having "assemblers" as special concepts and thus the corresponding
code in `Pipeline` is removed. The new assembler pipeline is marked
as exported, but the pipeline that builds the tree is not anymore.
Adapt the `describe` and `output` functions of the `v1` format to
handle the assembler pipeline. Also change the tests accordingly.
NB: The id reported for the assembler via `--inspect` and the result
will change as a result of this, since the assembler stage is now
the first and only stage of a new pipeline and thus has no base
anymore.
Since `__iter__` is return an iterator over the `Pipeline` objects,
the `"name" in manifest` check would not work for name or ids. Thus
provide an implemention of `__contains__` that does exactly that.
Add a new helper helper, `Manfifest.get` that will return a
pipeline give a name or an id or `None` if no pipeline could
be found with either. The implementation is taken from the
existing `__getitem__` method and the latter as now based on
the new `get` method.
Every pipeline that gets added to the `Manifest` now need to have
a unique name by which it can be identified. The version 1 format
loader is changed so that the main pipeline that builds the tree
is always called `tree`. The build pipeline for it will be called
`build` and further recursive build pipelines `build-build`, where
the number of repetitions of `build` corresponds to their level of
nesting. An assembler, if it exists, will be added as `assembler`.
The `Manifest.__getitem__` helper is changed so it will first try
to access pipeline via its name and then fall back to an id based
search. NB: in the degenrate case of multiple pipelines that have
exactly the same `id`, i.e. same stages, with the same options and
same build pipeline, only the first one will be return; but only
the first one here will be built as well, so this is in practice
not a problem.
The formatter uses this helper to get the tree pipeline via its
name wherever it is needed.
This also adds an `__iter__` method `Manifest` to ease iterating
over just the pipeline values, a la `for pipeline in manifet`.
Add a new `export` property to the `Pipeline` object that indicates
whether a the result, i.e. the tree after the pipelines has been
built, should be exported, i.e. copied to the output directory.
In the current format (v1), the main pipeline, gets marked as such
by the corresponding loader.
Instead of passing all pre-created pipelines to the Manifest
constructor, add a `add_pipeline` method, analogous to the
existing `Pipeline.add_{stage, assembler}` methods. Convert
the format loading code to use that and remove the constructor
parameter.
