For some reasons I forgot to fix those in the previous runs. Fix a
linter and pep8 warning.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Avoid using `os` or `input` as symbol names, since they shadow
previously defined symbols.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Make sure to correctly implement virtual functions and raise
NotImplementedError, as suggested by the standard library. Pylint will
otherwise complain about unavailable functions being accessed.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
The `problem` member is only defined if a problem-mark is present. If
the mark is not set, we get no details on the error whatsoever and have
to just print a generic error message.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
The __getitem__() function incorrectly returns `None` if the
partition-type is not found. Make sure to always return IndexError if
the requested value is outside the supported range.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
This mergest the handling of process_stages() and process_format() into
just one process_format(), which incrementally tracks the call stack
of the formating, which allows it to detect when it is hitting a stage
and can call _process_stage().
This means it is possible to mix things like mpp-if and mpp-join with
stages.
For example, you can do complex combinations like:
pipelines:
- name: rootfs
stages:
mpp-join:
- - type: org.ostree.foo
- mpp-if: use_bar
then:
type: org.osbuild.bar
- mpp-if: use_extra_stages
then:
mpp-eval: extra_stages
This is particularly useful if you included something and you want
to mpp-join something that was set in a variable.
Both file embedding and depsolves start by iterating over all stages,
and we want to add another similar one for container installs, so
break out the iteration over the containers so that it is done in
one place only.
This introduces a new dependency resolver to osbuild-mpp for Arch Linux
which uses the pacman package manager. The used solver is determined by
the `solver` field in the `mpp-depsolve` object inside the manifest
file, if it does not exists it falls back to the DepSolver for dnf/rpm.
Co-Authored-By: Jelle van der Waa <jvanderwaa@redhat.com>
This moves the handling of includes to the manifest loader, thus
supporting nested includes. search_dirs is moved to a property of the
Manifest so that it can be tracked during loads.
In addition we need to fix Manifest.path to the actual path that was
loaded instead of whatever the parent include said, so that relative
includes are handled from the proper location of the loaded manifest.
In process_embed_files(), it assumed that a stage had a "type" field,
which breaks if a stage is e.g. a `mpp-if` node, so use .get() instead
of raw dict lookups.
yaml files are essentially compatible with json files, although
they have some advantages, like allowing comments and being easier
for humans to read/write.
This changes the reading of the file to use a yaml parser instead of
a json parser, but still produces json at the end. I tried manually
converting a json file to yaml and running osbuild-mpp, and it produced
an identical file.
By default the yaml parser doesn't respect order so i had to tweak
the loader a bit to use OrderedDict.
Extract the code that finds a file and opens it from the existing
method that find manifests and opens them. This is so that the
former code can be re-used.
We now allow there to be default variables (none so far) which gets
installed as the initial value of manifest.vars.
Additionally, when including a file the default comes from the current
state of the including manifest, allowing the included manifest to use
variables from the base file.
We expand variables in various places during load, like when importing
files, etc. This means that the current approach of merging the
overrides into `manifest.vars` at the end doesn't work.
Instead we track overrides completely separate, and when we do the
expansion that always replaces whatever is in `manifest.vars`.
This:
{"mpp-eval": "foo+bar"}
Is essentially the same as using mpp-format with a trivial format
string:
{"mpp-format-int": "{foo+bar}"}
However, it is less to type, clearer to read, and supports
returning more complex types. For example, you can have a variable
that is a dict and expand that using eval.
This avoid having the local_vars dictionary getting filled with
a large number of built-in variable that are of no use for this
dictionary.
This commit was created by from Alexander Larsson.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
Using this we can now define variables in the variable section using
basic expression that are evaluated as f-string.
For example, you can use the syntax:
```
"mpp-vars": {
"rootfs_uuid": {"mpp-format-string": "{__import__('uuid').uuid1()}"},
"bootfs_uuid": "156f0420-627b-4151-ae6f-fda298097515"
},
```
This will automatically call uuid.uuid1() for rootfs_uuid, thus allowing
to dynamically set the uuid for the rootfs variable.
This variable being able to be overridden via the -D argument of the
osbuild-mpp tool.
In addition, you can also define variable based on variables defined
above, for example:
```
"mpp-vars": {
"rootfs_size": 4294967296,
"homefs_size": {"mpp-format-string": "{rootfs_size}"}
},
```
For context, we're in particular interested in this for the uuid
generation, as we have had issues with xfs failing to mount a filesystem
if the uuid is already mounted. In particular, if we built e.g. a
raspberry pi image using osbuild and then we use that raspberry pi for building
the same manifest (typically a later version of it) the osbuild fails in the
loopback mount case. Being able to easily support reproducible fixed-uuid
images, as well as ones with a real unique uuid makes a lot of sense.
This commit was created with a lot of inputs from Alexander Larsson.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
Since `.sort()` returns None, we were checking that None == None
which is not what we aimed to do.
Quick reproducer:
```
>>> assert [1,2,3].sort() == [2,3,4].sort()
>>> assert sorted([1,2,3]) == sorted([2,3,4])
Traceback (most recent call last):
File "<input>", line 1, in <module>
assert sorted([1,2,3]) == sorted([2,3,4])
AssertionError
```
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
We create the urls for the packages after dep-solving. During the
big refactoring (802f4010) the code that would expand the base url
during that step got lost. Re-introduce that so that local repos
work correctly again.
Also make sure the path is absolute before calling `as_uri` since
it is not guranteed that the result will be a `file:///` uri.
Reported-by: Stephen Smoogen <smooge@smoogespace.com>
Allow the manifest variables, defined via mpp-vars, to be used from
within the mpp blocks. For this template strings are used, where
variables are marked via `$`. We cannot use the `mpp-format` logic
easily there, since that is processed after other mpp directives
have been processed.
As a result remove the built-in substitution from support from dnf
dep-solving, since we had to post-process the resulting urls with
variable substitution afterwards. Now that is covered with this
more generic mechanism.
Introduce a helper method that will return the mpp node of a given
name or `None` if it does not exist. Delete it from the parent
dictionary if it did.
Reorder the typing imports as well to be sorted below the non-
`from` imports.
It is now possible to define a partition layout via `mpp-define-image`.
The defined layout is actually written to a temporary sparse file and
read back via `sfdisk`, so that all partition data like `size` and
`start` include actual padding and such. The `image` variable will be
defined with `size` and `layout` keys, the latter containing the
partition layout data. It can be accessed via the "String expansion"
mechanism.
