Extract the code that finds a file and opens it from the existing
method that find manifests and opens them. This is so that the
former code can be re-used.
We now allow there to be default variables (none so far) which gets
installed as the initial value of manifest.vars.
Additionally, when including a file the default comes from the current
state of the including manifest, allowing the included manifest to use
variables from the base file.
We expand variables in various places during load, like when importing
files, etc. This means that the current approach of merging the
overrides into `manifest.vars` at the end doesn't work.
Instead we track overrides completely separate, and when we do the
expansion that always replaces whatever is in `manifest.vars`.
This:
{"mpp-eval": "foo+bar"}
Is essentially the same as using mpp-format with a trivial format
string:
{"mpp-format-int": "{foo+bar}"}
However, it is less to type, clearer to read, and supports
returning more complex types. For example, you can have a variable
that is a dict and expand that using eval.
This avoid having the local_vars dictionary getting filled with
a large number of built-in variable that are of no use for this
dictionary.
This commit was created by from Alexander Larsson.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
Using this we can now define variables in the variable section using
basic expression that are evaluated as f-string.
For example, you can use the syntax:
```
"mpp-vars": {
"rootfs_uuid": {"mpp-format-string": "{__import__('uuid').uuid1()}"},
"bootfs_uuid": "156f0420-627b-4151-ae6f-fda298097515"
},
```
This will automatically call uuid.uuid1() for rootfs_uuid, thus allowing
to dynamically set the uuid for the rootfs variable.
This variable being able to be overridden via the -D argument of the
osbuild-mpp tool.
In addition, you can also define variable based on variables defined
above, for example:
```
"mpp-vars": {
"rootfs_size": 4294967296,
"homefs_size": {"mpp-format-string": "{rootfs_size}"}
},
```
For context, we're in particular interested in this for the uuid
generation, as we have had issues with xfs failing to mount a filesystem
if the uuid is already mounted. In particular, if we built e.g. a
raspberry pi image using osbuild and then we use that raspberry pi for building
the same manifest (typically a later version of it) the osbuild fails in the
loopback mount case. Being able to easily support reproducible fixed-uuid
images, as well as ones with a real unique uuid makes a lot of sense.
This commit was created with a lot of inputs from Alexander Larsson.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
Since `.sort()` returns None, we were checking that None == None
which is not what we aimed to do.
Quick reproducer:
```
>>> assert [1,2,3].sort() == [2,3,4].sort()
>>> assert sorted([1,2,3]) == sorted([2,3,4])
Traceback (most recent call last):
File "<input>", line 1, in <module>
assert sorted([1,2,3]) == sorted([2,3,4])
AssertionError
```
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
We create the urls for the packages after dep-solving. During the
big refactoring (802f4010) the code that would expand the base url
during that step got lost. Re-introduce that so that local repos
work correctly again.
Also make sure the path is absolute before calling `as_uri` since
it is not guranteed that the result will be a `file:///` uri.
Reported-by: Stephen Smoogen <smooge@smoogespace.com>
Allow the manifest variables, defined via mpp-vars, to be used from
within the mpp blocks. For this template strings are used, where
variables are marked via `$`. We cannot use the `mpp-format` logic
easily there, since that is processed after other mpp directives
have been processed.
As a result remove the built-in substitution from support from dnf
dep-solving, since we had to post-process the resulting urls with
variable substitution afterwards. Now that is covered with this
more generic mechanism.
Introduce a helper method that will return the mpp node of a given
name or `None` if it does not exist. Delete it from the parent
dictionary if it did.
Reorder the typing imports as well to be sorted below the non-
`from` imports.
It is now possible to define a partition layout via `mpp-define-image`.
The defined layout is actually written to a temporary sparse file and
read back via `sfdisk`, so that all partition data like `size` and
`start` include actual padding and such. The `image` variable will be
defined with `size` and `layout` keys, the latter containing the
partition layout data. It can be accessed via the "String expansion"
mechanism.
Previous versions of mpp would already set the arch and basearch
substitution, which would work for dep-solving itself, but not
properly re-write the resulting URLs which means that the manifest
was broken. Fix this by properly replacing the substitutions in
the URL. Also support official 'releasever' substitution.
Instead of passing dictionaries around that are inconvenient to
use in code and even more in the `mpp-format-*` directives, use
a simple class to represent package information. Use that in
the `pkginfo` dict that can be accessed via `mpp-format-*`. Use
the `evra` property instead of string manipulation in the
`fedora-boot.mpp.json` and `-ostree-bootiso.mpp.json` manifest.
This lets a mpp manifest define some variables and then use
these variables inside python f-strings to expand things in a
flexible way. This allows a single value such as `rootfs_size` to
be expanded in various places, including using computations to
e.g. define the partition offsets/sizes.
Move all the global code into a `main` function and call that. This
fixes a lot of pylint warnings where variable names were re-used
from the main, and thus global, context.
Create a DepSolver class that carries the global state such as dirs
and subscription information, as well as local state, like the
repositories and basedir. The latter can be reset so the class can
easily be re-used for all dep-solve sections.
This avoids having any global state.
Sort the checksum: urls dictionary via the url, so that no matter
where the urls came from (import or any dep-solve section), the
checksum: url dict is the same for the same set of urls.
Add support for additional paths that are searched when trying to
load a manifest. Currently only the path of the manifest that has
the include is searched. With this changed additional directories
will be included after that, in the order they were given ton the
command line.
Both v1 and v2 code paths used the same logic to add the resolved
dependencies to the `source_urls` dict, which is already stored
in the base class. Move the logic into the base class too.
Rewrite image pre-processor to single tool so that it is easier to use.
Now also supports `ignore-weak-deps` when dep-solving and supports
relative paths for local files.
Also create a symlink to the osbuild package, so that the tools can be
run from the source checkout and have access to the osbuild package.
The logic to sort urls was added globally in `mpp-import-pipeline`
but only the in the v1 code path was the `state.manifest_urls`
variable set and thus for v2 the actual sorting did not happen.
Fix this and set the `manifest_urls` to the `org.osbuild.curl`
items, which makes sense because we only know how to sort those.
In both mpp-depsolve and mpp-import-pipeline, sort the packages to
url dictionary before writing the JSON. This makes it easier to
look for packages but more importantly ensures that the resulting
set of packages has the same ordering in the sources section
independently of how it was assembled.
