We expand variables in various places during load, like when importing
files, etc. This means that the current approach of merging the
overrides into `manifest.vars` at the end doesn't work.
Instead we track overrides completely separate, and when we do the
expansion that always replaces whatever is in `manifest.vars`.
This:
{"mpp-eval": "foo+bar"}
Is essentially the same as using mpp-format with a trivial format
string:
{"mpp-format-int": "{foo+bar}"}
However, it is less to type, clearer to read, and supports
returning more complex types. For example, you can have a variable
that is a dict and expand that using eval.
This avoid having the local_vars dictionary getting filled with
a large number of built-in variable that are of no use for this
dictionary.
This commit was created by from Alexander Larsson.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
Using this we can now define variables in the variable section using
basic expression that are evaluated as f-string.
For example, you can use the syntax:
```
"mpp-vars": {
"rootfs_uuid": {"mpp-format-string": "{__import__('uuid').uuid1()}"},
"bootfs_uuid": "156f0420-627b-4151-ae6f-fda298097515"
},
```
This will automatically call uuid.uuid1() for rootfs_uuid, thus allowing
to dynamically set the uuid for the rootfs variable.
This variable being able to be overridden via the -D argument of the
osbuild-mpp tool.
In addition, you can also define variable based on variables defined
above, for example:
```
"mpp-vars": {
"rootfs_size": 4294967296,
"homefs_size": {"mpp-format-string": "{rootfs_size}"}
},
```
For context, we're in particular interested in this for the uuid
generation, as we have had issues with xfs failing to mount a filesystem
if the uuid is already mounted. In particular, if we built e.g. a
raspberry pi image using osbuild and then we use that raspberry pi for building
the same manifest (typically a later version of it) the osbuild fails in the
loopback mount case. Being able to easily support reproducible fixed-uuid
images, as well as ones with a real unique uuid makes a lot of sense.
This commit was created with a lot of inputs from Alexander Larsson.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
Since `.sort()` returns None, we were checking that None == None
which is not what we aimed to do.
Quick reproducer:
```
>>> assert [1,2,3].sort() == [2,3,4].sort()
>>> assert sorted([1,2,3]) == sorted([2,3,4])
Traceback (most recent call last):
File "<input>", line 1, in <module>
assert sorted([1,2,3]) == sorted([2,3,4])
AssertionError
```
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
We create the urls for the packages after dep-solving. During the
big refactoring (802f4010) the code that would expand the base url
during that step got lost. Re-introduce that so that local repos
work correctly again.
Also make sure the path is absolute before calling `as_uri` since
it is not guranteed that the result will be a `file:///` uri.
Reported-by: Stephen Smoogen <smooge@smoogespace.com>
Allow the manifest variables, defined via mpp-vars, to be used from
within the mpp blocks. For this template strings are used, where
variables are marked via `$`. We cannot use the `mpp-format` logic
easily there, since that is processed after other mpp directives
have been processed.
As a result remove the built-in substitution from support from dnf
dep-solving, since we had to post-process the resulting urls with
variable substitution afterwards. Now that is covered with this
more generic mechanism.
Introduce a helper method that will return the mpp node of a given
name or `None` if it does not exist. Delete it from the parent
dictionary if it did.
Reorder the typing imports as well to be sorted below the non-
`from` imports.
It is now possible to define a partition layout via `mpp-define-image`.
The defined layout is actually written to a temporary sparse file and
read back via `sfdisk`, so that all partition data like `size` and
`start` include actual padding and such. The `image` variable will be
defined with `size` and `layout` keys, the latter containing the
partition layout data. It can be accessed via the "String expansion"
mechanism.
Previous versions of mpp would already set the arch and basearch
substitution, which would work for dep-solving itself, but not
properly re-write the resulting URLs which means that the manifest
was broken. Fix this by properly replacing the substitutions in
the URL. Also support official 'releasever' substitution.
Instead of passing dictionaries around that are inconvenient to
use in code and even more in the `mpp-format-*` directives, use
a simple class to represent package information. Use that in
the `pkginfo` dict that can be accessed via `mpp-format-*`. Use
the `evra` property instead of string manipulation in the
`fedora-boot.mpp.json` and `-ostree-bootiso.mpp.json` manifest.
This lets a mpp manifest define some variables and then use
these variables inside python f-strings to expand things in a
flexible way. This allows a single value such as `rootfs_size` to
be expanded in various places, including using computations to
e.g. define the partition offsets/sizes.
Move all the global code into a `main` function and call that. This
fixes a lot of pylint warnings where variable names were re-used
from the main, and thus global, context.
Create a DepSolver class that carries the global state such as dirs
and subscription information, as well as local state, like the
repositories and basedir. The latter can be reset so the class can
easily be re-used for all dep-solve sections.
This avoids having any global state.
Sort the checksum: urls dictionary via the url, so that no matter
where the urls came from (import or any dep-solve section), the
checksum: url dict is the same for the same set of urls.
Add support for additional paths that are searched when trying to
load a manifest. Currently only the path of the manifest that has
the include is searched. With this changed additional directories
will be included after that, in the order they were given ton the
command line.
Both v1 and v2 code paths used the same logic to add the resolved
dependencies to the `source_urls` dict, which is already stored
in the base class. Move the logic into the base class too.
Rewrite image pre-processor to single tool so that it is easier to use.
Now also supports `ignore-weak-deps` when dep-solving and supports
relative paths for local files.
Also create a symlink to the osbuild package, so that the tools can be
run from the source checkout and have access to the osbuild package.
The logic to sort urls was added globally in `mpp-import-pipeline`
but only the in the v1 code path was the `state.manifest_urls`
variable set and thus for v2 the actual sorting did not happen.
Fix this and set the `manifest_urls` to the `org.osbuild.curl`
items, which makes sense because we only know how to sort those.
In both mpp-depsolve and mpp-import-pipeline, sort the packages to
url dictionary before writing the JSON. This makes it easier to
look for packages but more importantly ensures that the resulting
set of packages has the same ordering in the sources section
independently of how it was assembled.
Add a simple tool that will spit out a valid org.osbuild.inline
source entry that encodes the given file. Currently always uses
base64 as encoding and sha256 for the hashing.
The pylorax implementation of the template running code supports
globbing, as well as `--exclude` and `--optional` commands. These
are handled independently for each `installpkg` command, so that
requesting the installation of firmware packages in one command
together with an exclude of `*alsa` does indeed only exclude any
alsa firmware packages but not any other alsa packages. The
previous version of this script would just build an global list
for excludes which has a drastically different result because
a global exclude of "*alsa" would result in a global exclusion of
all matching packages and probably a dependency error.
Therefore, add support for dnf based filtering of packages on a
per `installpkg` command bases, very much like pylorax does.
Add a simple helper that is meant to gather the list of packages
to be installed via a lorax template that uses the "installpkg"
directives. A prominent example is the 'runtime-install.tmpl'
script from lorax-templates-generic, used to create boot isos.
Although MPP supported having multiple repo entries for each
depsolve block, in reality that didn't actually work because
the same (global) baseurl was used for all packages.
Now the basurl can still be (otionally) specified globally,
but also overridden by the repo block and the package paths
will be relative to that.
The `org.osbuild.files` source provides files, but might in the
future not be the only one that does. Therefore rename it to
match the internal tool that is being used to fetch the files.
This is done for most other osbuild modules that target tools.
The format v1 loader is adapted to make this change transparent
for users of the v1 format, so we are backwards compatible.
Change the MPP depsolve preprocessor so that for format v2 based
manifest `org.osbuild.curl` source is used. Also rename the
corresponding source test. Adapt the format v2 mod test to use
the curl source.
Support for dep-solving in format version 2. Instead of adding
the checksums to a key in the options, it will be added to the
references inside an input. The urls for the items are added
in the sources dict via the new format, i.e. in the `items`
dict instead of the `urls` dict.
Rename the parsing and process functions to have a format version
specific suffix. This should make it easy to add support for the
format version 2. Logic should be unchanged.
Add support for foramt version 2. This is slightly easier than
version 1, since there is no recursion. In addition to the
path of the manifest to import the pipeline from, the pipeline
identifier needs to be specified.
Source merging is also different since in format version two,
there is a generic "items" key, which means we can merge
other sources than "org.osbuild.files".
