In format version 2, the source specific keys for the sources,
here "urls", is replaced by a generic `items` key, common to
all sources. Express that in the schema.
Port the `org.osbuild.ostree.commit` assembler to a new assembler
like stage. The notable differences are: the fs tree preparation
now happens in the `ostree.preptree` stage and there is no option
to create a tarball, since that should be handled by a tar stage.
This new stage is a combination of the existing `rpm-ostree` stage
and the old `ostree.commit` assembler. It first does prepare a new
OSTree compliant root fs (previously done in `ostree.commit`) and
then uses `rpm-ostree tree postprocess` to perform various post-
processing on the tree to make it fully OSTree compliant.
Add an example that builds a small container, based on the minimal
container fedora kickstart (see [1, 2]). There is a deviation
because osbuild currently does not support removing
random files like it is done in [2].
The build pipeline "build" in f32-build-v2.json) is a port of the
existing build pipeline in f32-build.json.
This manifest is currently not used in any automated tests (yet),
but is there as a showcase for the new format. It does however
test the format version 2 support for mpp, since that is called
in CI.
[1] https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-common.ks
[2] https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base-minimal.ks
Support for dep-solving in format version 2. Instead of adding
the checksums to a key in the options, it will be added to the
references inside an input. The urls for the items are added
in the sources dict via the new format, i.e. in the `items`
dict instead of the `urls` dict.
Rename the parsing and process functions to have a format version
specific suffix. This should make it easy to add support for the
format version 2. Logic should be unchanged.
Add support for foramt version 2. This is slightly easier than
version 1, since there is no recursion. In addition to the
path of the manifest to import the pipeline from, the pipeline
identifier needs to be specified.
Source merging is also different since in format version two,
there is a generic "items" key, which means we can merge
other sources than "org.osbuild.files".
This new command line argument can be used to explicitly request
the exporting of built artifcats. They will be store in the
output directory in a subdirectory with the name or id that was
requested.
In format version 2, the source specific keys for the sources,
here "urls", is replaced by a generic `items` key, common to
all sources. Express that in the schema.
Explicitly declare the schema for version 2. Notable differences
are the declaration of the `packages` input as well is the lack
of that property in `options`.
Add support for format version 2. The laoding fits to the manifest
description schema in `schema/osbuild2.json`. Loading, describing
and validation are fully functional. The `output` method resturns
results in a new format too but is very much work in progress and
quite a hack right now. The output format will definitely change
again.
Add a `version` keyword argument to `Index.get_schema` which
will in turn look for `osbuild<version>.json` in case of the
schema for the manifest is requested and otherwise forward
the version argument to the `get_schema` method for the
respective `ModuleInfo`.
When loading the schema information via the source code of a
module, look for a `SCHEMA_2` global variable, representing
the schema version 2. Extend the `get_schema` method so in
takes a `version` keyword argument. Rework the code so that
if version 2 for the format is specified but no dedicated
schema data is found, a fallback based on the version 1 is
provided. This makes it easy to use all existing stages
without explicitly duplicating all schema information.
NB: The code is not very pretty, the hope is that in the
future, the module, being an executable, could be called
with a command line switch, a la `--schema <version>` and
this would return the schema data. So that hackery code
we currently have will hopefully vanish soon. I am sorry
though for this mess.
Change the `ModuleInfo.schema` propertly into a `get_schema`
method call. This is in preparation to allow for different
schemata versions to be supported.
Commit d028ea5b16 introduced bug when introducing the `store`
argument to `Stage.run`, instead of passing `var=var`, i.e.
`var` is being passed as keyword argument, it is now being
passed as a positional one. Since the `path=/run/osbuild`
keyword argument comes before the `var=/var/tmp` argument,
`var` is now being passed as `path` instead of var.
Since `var` is always being passed in throughout the entire
codebase, make it a positional argument, and move it before
`path`.
Adapt the tests to pass `var` as positional argument.
All sources fetch various types of `items`, the specific nature
of which is dependent on the source type, but they are all
identifyable by a opaque identifier. In order for osbuild to
check that all the inputs that a stage needs are are indeed
contained in the manifest description, osbuild must learn what
ids are fetched by what source. This is done by standarzing
the common "items" part, i.e. the "id" -> "options for that id"
mapping that is common to all sources.
For the version 1 of the format, extract the files and ostree
the item information from the respective options.
Adapt the sources (files, ostree) so that they use the new items
information, but also fall back to the old style; the latter is
needed since the sources tests still uses the SourceServer.
Now that meta.Index provides a way to detect the format given the
input data, use that method instead of the hard-coded use of the
version 1 format via the input. This should make the main entry
point format independent.
Introdcue a `FormatInfo` class that, very much like `ModuleInfo`
can be used to obtain meta information about a format. Methods
are added to `Index` to allow the enumeration of available formats,
getting the `FormatInfo` for a format given its name and to detect
a format via the manifest description data.
Change the top-level documentation to reflect the changes. Also
remove an outdated section about validation of the schema; this
was moved to the format specific code some time ago.
Convert the `org.osbuild.ostree` stage to use inputs instead of
sources. In the format (version 1) loading code, convert the
stage to use an input based on the existing stage options.
Add a new input that can handle ostree commits. For source origins,
it will pull any number of commits specified via `references`. For
pipeline origins it will use the compose.json to obtain the commit
id and pull that. Either way the data passed onto the stage is a
dictionary of commit ids and commit metadata, such as the `ref`
that is an ostree ref that is optionally created in case the `ref`
option for that reference was specified.
Instead of manually constructing and appending the input for
stages (here the stages that replace the assembler), use the
new `Stage.add_input` method.
In much the same way has `Pipeline` has `add_stage` and `Manifest`
has `add_pipeline`, introduce an `add_input` method to `Stage` to
be able add `Inputs` to stages.
The `info` parameter for Input constructor is of type `PModuleInfo`,
which is located in `meta`. This in turn imports jsonschema. Ergo,
importing importing `inputs` will create a dependency on jsonschema.
At the same time the `osbuild` package globally imports `Pipeline`,
via `__init__.py`, and `osbuild.api` is used in the runners and
stages, which are run inside the buildroot. If one now wanted to
use `inputs` from `Pipeline`, it would lead to jsonschema being
imported (via `meta`) which might not be available on the build-
root and it is a rather random dependency to have.
On obvious solution would be to use a construct with `TYPE_CHECKING`,
a la:
if TYPE_CHECKING:
from .meta import ModuleInfo
Sadly, pylint will now complain about it. This could be fixed with:
if TYPE_CHECKING:
from .meta import ModuleInfo
else:
ModuleInfo = "osbuild.meta.ModuleInfo"
But this is just gross. So we will have to accept that Python is,
well, Python und omit the type information for the `info` param.
Add a new jsoncomm rpc method call, `source`, that will return
the directory within the store where resources for that specific
type of resource, like e.g. tree, files, or ostree can be found
or stored.
Currently all options for inputs are totally opaque to osbuild
itself. This is neat from a seperation of concerns point of view
but has one major downside: osbuild can not verify the integrity
of the pipeline graph, i.e. if all inputs that need pipelines or
sources do indeed exists. Therefore intrdouce two generic fields
for inputs: `origin` and `references`. The former can either be
a source or a pipeline. The latter is an array of identifiers or
a dictionary where the keys are the identifiers and the values
are additional options for that id. The identifiers then refer
to either resources obtained via a source or a pipeline that has
already been built.
Instead of using `org.osbuild.test` stages, so `org.osbuild.noop`
stages so that the options we pass are actually valid and we can
use the manifest in schema validation checking.
