Add a `version` keyword argument to `Index.get_schema` which
will in turn look for `osbuild<version>.json` in case of the
schema for the manifest is requested and otherwise forward
the version argument to the `get_schema` method for the
respective `ModuleInfo`.
When loading the schema information via the source code of a
module, look for a `SCHEMA_2` global variable, representing
the schema version 2. Extend the `get_schema` method so in
takes a `version` keyword argument. Rework the code so that
if version 2 for the format is specified but no dedicated
schema data is found, a fallback based on the version 1 is
provided. This makes it easy to use all existing stages
without explicitly duplicating all schema information.
NB: The code is not very pretty, the hope is that in the
future, the module, being an executable, could be called
with a command line switch, a la `--schema <version>` and
this would return the schema data. So that hackery code
we currently have will hopefully vanish soon. I am sorry
though for this mess.
Change the `ModuleInfo.schema` propertly into a `get_schema`
method call. This is in preparation to allow for different
schemata versions to be supported.
Commit d028ea5b16 introduced bug when introducing the `store`
argument to `Stage.run`, instead of passing `var=var`, i.e.
`var` is being passed as keyword argument, it is now being
passed as a positional one. Since the `path=/run/osbuild`
keyword argument comes before the `var=/var/tmp` argument,
`var` is now being passed as `path` instead of var.
Since `var` is always being passed in throughout the entire
codebase, make it a positional argument, and move it before
`path`.
Adapt the tests to pass `var` as positional argument.
All sources fetch various types of `items`, the specific nature
of which is dependent on the source type, but they are all
identifyable by a opaque identifier. In order for osbuild to
check that all the inputs that a stage needs are are indeed
contained in the manifest description, osbuild must learn what
ids are fetched by what source. This is done by standarzing
the common "items" part, i.e. the "id" -> "options for that id"
mapping that is common to all sources.
For the version 1 of the format, extract the files and ostree
the item information from the respective options.
Adapt the sources (files, ostree) so that they use the new items
information, but also fall back to the old style; the latter is
needed since the sources tests still uses the SourceServer.
Now that meta.Index provides a way to detect the format given the
input data, use that method instead of the hard-coded use of the
version 1 format via the input. This should make the main entry
point format independent.
Introdcue a `FormatInfo` class that, very much like `ModuleInfo`
can be used to obtain meta information about a format. Methods
are added to `Index` to allow the enumeration of available formats,
getting the `FormatInfo` for a format given its name and to detect
a format via the manifest description data.
Change the top-level documentation to reflect the changes. Also
remove an outdated section about validation of the schema; this
was moved to the format specific code some time ago.
Convert the `org.osbuild.ostree` stage to use inputs instead of
sources. In the format (version 1) loading code, convert the
stage to use an input based on the existing stage options.
Add a new input that can handle ostree commits. For source origins,
it will pull any number of commits specified via `references`. For
pipeline origins it will use the compose.json to obtain the commit
id and pull that. Either way the data passed onto the stage is a
dictionary of commit ids and commit metadata, such as the `ref`
that is an ostree ref that is optionally created in case the `ref`
option for that reference was specified.
Instead of manually constructing and appending the input for
stages (here the stages that replace the assembler), use the
new `Stage.add_input` method.
In much the same way has `Pipeline` has `add_stage` and `Manifest`
has `add_pipeline`, introduce an `add_input` method to `Stage` to
be able add `Inputs` to stages.
The `info` parameter for Input constructor is of type `PModuleInfo`,
which is located in `meta`. This in turn imports jsonschema. Ergo,
importing importing `inputs` will create a dependency on jsonschema.
At the same time the `osbuild` package globally imports `Pipeline`,
via `__init__.py`, and `osbuild.api` is used in the runners and
stages, which are run inside the buildroot. If one now wanted to
use `inputs` from `Pipeline`, it would lead to jsonschema being
imported (via `meta`) which might not be available on the build-
root and it is a rather random dependency to have.
On obvious solution would be to use a construct with `TYPE_CHECKING`,
a la:
if TYPE_CHECKING:
from .meta import ModuleInfo
Sadly, pylint will now complain about it. This could be fixed with:
if TYPE_CHECKING:
from .meta import ModuleInfo
else:
ModuleInfo = "osbuild.meta.ModuleInfo"
But this is just gross. So we will have to accept that Python is,
well, Python und omit the type information for the `info` param.
Add a new jsoncomm rpc method call, `source`, that will return
the directory within the store where resources for that specific
type of resource, like e.g. tree, files, or ostree can be found
or stored.
Currently all options for inputs are totally opaque to osbuild
itself. This is neat from a seperation of concerns point of view
but has one major downside: osbuild can not verify the integrity
of the pipeline graph, i.e. if all inputs that need pipelines or
sources do indeed exists. Therefore intrdouce two generic fields
for inputs: `origin` and `references`. The former can either be
a source or a pipeline. The latter is an array of identifiers or
a dictionary where the keys are the identifiers and the values
are additional options for that id. The identifiers then refer
to either resources obtained via a source or a pipeline that has
already been built.
Instead of using `org.osbuild.test` stages, so `org.osbuild.noop`
stages so that the options we pass are actually valid and we can
use the manifest in schema validation checking.
Add a new `add_source` method that will add an individual `Source`
to a `Manifest` give its `ModuleInfo` and options. The dictionary
of source options in the manifest is replaced with a list of such
`Sources` and `add_source` will append to it. Adap the version 1
format code to use `add_source` and reconstruct the source options
from the list of source on `describe`.
Remove the `sources_options` constructor parameter for `Manifest`
and adapt all the source base for this.
Very much like, stages and inputs, a new `Source` calls represents
a source type with its options. A `download` method on the `Source`
class can be used to only donwload content to the cache.
Split the internal logic into two parts: 1) fetching the commit
into the internal cache repo and then 2) exporting that commit,
i.e. a local pull from the cache repo to the output directory.
If no `output` directory was specified, only fetch the commit,
do not attempt to export it.
NB: this commit changes at what point the gpg verification is
done. Previously the check was on export. Now, we are checking
the signature on import only. The export step will be replaced
by an ostree `Input` that will have the ability to verify
commits a second time.
Split the internal logic of the stage in two parts: 1) downloading
files to the internal cache and 2) exporting the downloaded files
from said cache to the output directory. Additionally, ff no such
`output` directory was specified, i.e. it is empty or `None`, only
download files but do not attempt to export them.
The stage was structured in a way so that it supports many different
inputs, like, `archive` and in the future maybe trees, git checkouts.
This does not fit very well into the new input model, where instead
seperate `copy.tree` and `copy.files` stages might make more sense.
Since the stage is not been used anywhere, and mainly got added
as a way to get the ignition dracut module via an archive of a git
repo, remove the stage for now. A replacement will be brought back
in the future.
The sysconfig directory contains a variety of system configuration files. The
values found in each of the configuration files may need to be specified
before first boot.
Currently, only the kernel and network configs can be modified and will
override existing files.
Add the ability to opt out of preserving the ACLs, SELinux
contexts and extended attributes. It is opt out instead of
opt in since the assembler by default tries to preserve as
much as possible.
In the test case for the tar assembler, actually verify the
content by un-tar-ing the result again and comparing it to
the tree. This would have spotted missing SELinux labels.
All of these options, i.e. SELinux labels, ACLs and extended
attributes (xattrs), are opt-in and thus were currently ignored.
This lead to trees that had their selinux labels missing and
were thus incorrect.
Detect the case that one file has a SELinux label but the other
file does not have any label at all. This was currently not
possible to detect because both calls to get the labels were
wrapped in one try-except block and any failure in one of the
two calls, like a missing label, would lead to an early return
of the function, with a success value.
This source has been declared obsolete some time ago and is not
support anymore. We wont support it in the upcoming new manifest
format, therefore drop it now.
Include all inputs of a stage during the calculation of its id,
since they determine, very much like options, the content the
stage produces; thus different inputs should lead to different
ids.
Add an `id` property that, like `Stage.id`, can be used to uniquely
identify an input based on its name and options. Two stages with
the same name and options will have the same `id`.
Now that `Pipelines` have no assemblers anymore and thus only one
identifier, i.e. the one corresponding to the tree (`tree_id`),
the `id` and `tree_id` are now the same. Therefore replace the
usage of `tree_id` with `id` and drop the former. Add some extra
documentation including some caveats about the uniquness of `id`.
Convert the assembler phase of the main pipeline in the old format
into a new Pipeline that as the assembler as a stage, where the
input of that stage is the main pipeline. This removes the need of
having "assemblers" as special concepts and thus the corresponding
code in `Pipeline` is removed. The new assembler pipeline is marked
as exported, but the pipeline that builds the tree is not anymore.
Adapt the `describe` and `output` functions of the `v1` format to
handle the assembler pipeline. Also change the tests accordingly.
NB: The id reported for the assembler via `--inspect` and the result
will change as a result of this, since the assembler stage is now
the first and only stage of a new pipeline and thus has no base
anymore.
