Sort the checksum: urls dictionary via the url, so that no matter
where the urls came from (import or any dep-solve section), the
checksum: url dict is the same for the same set of urls.
Add support for additional paths that are searched when trying to
load a manifest. Currently only the path of the manifest that has
the include is searched. With this changed additional directories
will be included after that, in the order they were given ton the
command line.
Both v1 and v2 code paths used the same logic to add the resolved
dependencies to the `source_urls` dict, which is already stored
in the base class. Move the logic into the base class too.
Rewrite image pre-processor to single tool so that it is easier to use.
Now also supports `ignore-weak-deps` when dep-solving and supports
relative paths for local files.
Also create a symlink to the osbuild package, so that the tools can be
run from the source checkout and have access to the osbuild package.
The logic to sort urls was added globally in `mpp-import-pipeline`
but only the in the v1 code path was the `state.manifest_urls`
variable set and thus for v2 the actual sorting did not happen.
Fix this and set the `manifest_urls` to the `org.osbuild.curl`
items, which makes sense because we only know how to sort those.
In both mpp-depsolve and mpp-import-pipeline, sort the packages to
url dictionary before writing the JSON. This makes it easier to
look for packages but more importantly ensures that the resulting
set of packages has the same ordering in the sources section
independently of how it was assembled.
Add a simple tool that will spit out a valid org.osbuild.inline
source entry that encodes the given file. Currently always uses
base64 as encoding and sha256 for the hashing.
The pylorax implementation of the template running code supports
globbing, as well as `--exclude` and `--optional` commands. These
are handled independently for each `installpkg` command, so that
requesting the installation of firmware packages in one command
together with an exclude of `*alsa` does indeed only exclude any
alsa firmware packages but not any other alsa packages. The
previous version of this script would just build an global list
for excludes which has a drastically different result because
a global exclude of "*alsa" would result in a global exclusion of
all matching packages and probably a dependency error.
Therefore, add support for dnf based filtering of packages on a
per `installpkg` command bases, very much like pylorax does.
Add a simple helper that is meant to gather the list of packages
to be installed via a lorax template that uses the "installpkg"
directives. A prominent example is the 'runtime-install.tmpl'
script from lorax-templates-generic, used to create boot isos.
Although MPP supported having multiple repo entries for each
depsolve block, in reality that didn't actually work because
the same (global) baseurl was used for all packages.
Now the basurl can still be (otionally) specified globally,
but also overridden by the repo block and the package paths
will be relative to that.
The `org.osbuild.files` source provides files, but might in the
future not be the only one that does. Therefore rename it to
match the internal tool that is being used to fetch the files.
This is done for most other osbuild modules that target tools.
The format v1 loader is adapted to make this change transparent
for users of the v1 format, so we are backwards compatible.
Change the MPP depsolve preprocessor so that for format v2 based
manifest `org.osbuild.curl` source is used. Also rename the
corresponding source test. Adapt the format v2 mod test to use
the curl source.
Support for dep-solving in format version 2. Instead of adding
the checksums to a key in the options, it will be added to the
references inside an input. The urls for the items are added
in the sources dict via the new format, i.e. in the `items`
dict instead of the `urls` dict.
Rename the parsing and process functions to have a format version
specific suffix. This should make it easy to add support for the
format version 2. Logic should be unchanged.
Add support for foramt version 2. This is slightly easier than
version 1, since there is no recursion. In addition to the
path of the manifest to import the pipeline from, the pipeline
identifier needs to be specified.
Source merging is also different since in format version two,
there is a generic "items" key, which means we can merge
other sources than "org.osbuild.files".
Detect the case that one file has a SELinux label but the other
file does not have any label at all. This was currently not
possible to detect because both calls to get the labels were
wrapped in one try-except block and any failure in one of the
two calls, like a missing label, would lead to an early return
of the function, with a success value.
Use dnf's per-user cache when `--dnf-cache` is not given. This speeds up
repeated invocations of `mpp-depsolve` considerably, without having to
specify a cache location.
Technically, a few extra "/"s don't matter, but in practice, it
sometimes does. In particular, rpmrepo returns `500 Internal Server
Error` when a path contains additional "/"s.
Support excluding packages when dep-solving; needed for e.g. the
ostree commits to not create a rescue kernel by excluding the
'dracut-config-rescue' package.
Move the `tree-diff` tool into ./tools, which is our new place for tools
used by the test-suite or during development.
The only hard-coded user is the TestBase, so fix its path to the tool
so the test-suite will continue to find it.
This adds a new MPP which supports importing pipelines from another
file. It simply looks for "mpp-import-pipeline" tags at the same
position where we would expect a "pipeline" tag. It then uses the "path"
attribute in it to find a manifest. From this manifest, the "sources"
are merged back into the original sources, and the "pipeline" is taken
verbatim to replace the "mpp-import-pipeline".
The idea is to allow importing build-pipelines from other files into our
test manifests, without duplicating the build-pipeline everywhere.
This adds F32 manifests in ./test/data/. To avoid magically deducing the
package list out of the void, this adds a ManifestPreProcessor (MPP)
called `./tools/mpp-depsolve.py`. What this does is it takes a manifest
on stdin, modifies it, and produces a manifest on stdout.
The `mpp-depsolve.py` preprocessor takes a manifest and modifies all the
`org.osbuild.rpm` stages. It parses a new option to that stage called
`mpp-depsolve`, which contains a package-list, a repo-list, and dnf
metadata. It then drops this `mpp-depsolve` option (since it would be an
invalid manifest otherwise), depsolves the packages, inserts a proper
"packages" option as well as appends the correct paths to the sources
entry.
With this in place, this adds `mpp-f32-base.json` and
`mpp-f32-build.json` in ./test/data/manifests/. These will then be used
as base F32 manifests for our test-suite.
Lastly, this adds `./test/data/README.md` as a place to document the
files we place in `./test/data/`, since most of the files do not allow
for comments.
