When iterating partitions to mount, skip any with filesystem type
"swap". This is done in two places:
1. When mounting partitions to find /etc/fstab.
2. When mounting partitions and volumes to analyse the tree.
When iterating through partitions, store the fstype along with the other
information. This will be useful for identifying btrfs partitions,
which we will need to scan for subvolumes, and for identifying swap
partitions, so we can avoid trying to mount them.
Run isort for imports.
Pylint: wrong-import-order / C0411
Solves the following linter warnings:
- standard import "pathlib" should be placed before third party import
"yaml"
- standard import "collections.OrderedDict" should be placed before
third party imports "yaml", "jsonschema"
- standard import "typing.Dict" should be placed before third party
imports "yaml", "jsonschema"
Fix default arg values.
Pylint: dangerous-default-value / W0102
- Using mutable default values ([]) for function arguments is considered
dangerous.
Rename format variable.
Pylint: redefined-builtin / W0622
- 'format' is a built-in function.
Use f-strings instead of formatting where possible.
Pylint: consider-using-f-string / C0209
Remove unnecessary else after returns.
Pylint: no-else-return / R1705
Remove unnecessary else after continue.
Pylint: no-else-continue / R1724
Set the encoding (utf-8) for all calls to open().
Pylint: unspecified-encoding / W1514
Disable the too-many-branches and too-many-statements warnings for
append_partitions() and append_filesystem(). We can refactor the
functions to make them smaller later, but for now we're addressing only
the simpler issues.
Initialise with dict literal instead of call to function.
Pylint: use-dict-literal / R1735
Use implicit truthiness for glob instead of len().
Pylint: use-implicit-booleaness-not-len / C1802
Rename ambiguous variable 'l' to 'line'.
pycodestyle: ambiguous-variable-name (E741)
Merge comparisons with 'in'.
Pylint: consider-using-in / R1714
`read_boot_entries()` could previously fail when trying to split lines
in bootloader entries, which contained only "\n" and became empty
string after stripping whitespace characters. This is the case e.g. on
F41 images.
Moreover, bootloader entries can contain comments as lines starting with
"#", which were previously not ignored by the function and would end up
in the parsed entry and could potentially fail to be split.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add unit test for osbuild-image-info's `read_boot_entries()` function,
to ensure that it can handle various situations that can happen in the
real world.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add the image-info tool as osbuild-image-info from the manifest-db
repository [0]. This is an exact copy without any changes.
[0] 8e05a898d4/tools/image-info
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Enable generating SBOM documents for depsolved transactions when using
DNF5. Enable SBOM testing with DNF5 in unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The change in commit ed33869430 to
use hashlib.file_digest breaks with older python, because
it was added in 3.11.
This change reverts back to hashing all the data in the case
where file_digest doesn't exist.
This allows using e.g. mpp-eval in the resolve-image operation, similar
to how it is now possible in the mpp-depsolve handling.
We want this so we can inject the list of images from a list variable
(that can then be mpp-join:ed, etc).
For the DNF4 version, we actually use 'dnf' package and not 'libdnf'.
Fix the SPEC file dependencies and also the check in unit test.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Currently if one uses `mpp-embed` with URLs, osbuild-mpp still wants to
download the full file just so it can hash it. Make this more efficient
by hashing from the stream instead, which `hashlib` natively supports.
This also makes osbuild-mpp work with large artifacts in environments
that may not have enough space to temporarily save the data.
Extend osbuild-depsolve-dnf, to return JSON with SPDX SBOM that
corresponds to the depsolved package set, if it has been requested.
For now, only DNF4 is supported.
Cover the new functionality with unit test.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This will allow validating request arguments in the solver method in a
different way for dnf4 and dnf5 and raising an exception if needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The DNF5 implementation has known issues, which are not yet fixed and in
some cases, they can't be fixed due to the limitations and state of the
DNF5 implementation itself. Skip them for now.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case for installation of a package, which is excluded in a
different transaction. This is a common scenario in osbuild-composer,
where the image base package set is depsolved in the first transaction
and can contain package excludes. The user must be able to install these
excluded packages when specified explicitly in the Blueprint.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case for depsolving a package group with specific optional
packages of the group being excluded. This is a common scenario in
many image definitions.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a test case, that the repositories not enabled for a transaction are
not really used to depsolve its package specs. This is a common use case
for osbuild-composer, where the custom repositories specified by the
user are enabled only for the second transaction and not for the first
one (when depsolving the image base package set). This is important, so
that the user can't possibly replace important packages shipped by
the "system" repositories (e.g. kernel, systemd, etc.) with versions
from their custom repositories.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Fix a bug in the `osbuild-depsolve-dnf`, which would cause the
`RepoError` to not be handled and producing a traceback.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Implement the generator for repo config combinations, instead of
hard-coding it. The motivation is to be able to optionally add e.g.
q broken repo to the list to test `RepoError`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Porting the latests osbuild version in `osbuild/images` revealed a
regression in error messages returned by the tool in case a non-existent
package is requested in the depsolve request.
Test that requesting a non-existent package results in `MarkingErrors`,
which was the original behavior of the DNF4-based
`osbuild-depsolve-dnf`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Remove one test case, which does not really add value or extend the test
coverage, since the same thing is already tested by previous test case.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Until now, the unit test tested only very simple case with a single
transaction, which happens only for vanilla images. Any user
customization would result in multiple transactions in the depsolve
request. This case is not yet tested at all.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Modify the function that executes the depsolve command, so that does not
produce exception in case the command fails. Instead, return the message
returned by `osbuild-depsolve-dnf` and the exit code. This will allow to
test also other scenarios (such as depsolve errors) in the test case.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit includes the used sovler in the dnfjson reply. This
is mostly information (e.g. in service logs) but also useful in
tests to ensure that the expected solver was really run.
Note that this needs https://github.com/osbuild/images/pull/723
first.
This moves the dnf and dnf5 code into a new osbuild module called
solver. The dnf specific code is in dnf.py and dnf5 is in dnf5.py
At runtime the osbuild-depsolve-dnf script reads a config file from
/usr/lib/osbuild/solver.json and imports the selected solver. This
currently just contains a 'use_dnf5' bool but can be extended to support
other configuration options or depsolvers.
At build time a config file is selected from tools/solver-dnf.json or
tools/solver-dnf5.json and installed. Currently dnf5 is not installed,
it will be added when dnf5 5.2.1.0 becomes available in rawhide (Fedora
41).
The error messages have been normalized since the top level functions in
osbuild-depsolve-dnf do not know which version of dnf is being used.
This commit allows to the user to define the "container-transport"
variable when they are defining a container image to be resolved by the
osbuild-mpp tool. When the user defines the "container-storage" as
"container-transport" osbuild-mpp will look for the image in the local
container-storage.
Signed-off-by: Roberto Majadas <rmajadas@redhat.com>
The `tree-diff` tool is used by the `gen-stage-diff` tool to generate
the stage `diff.json` files. A few of these stages have timestamped
files that need to be edited after the tool is run to replace the
`sha256` content hash with `null` so the stage diff tests ignore these
files.
This commit updates the `tree-diff` tool to check through a list of the
files that contain timestamps and to use the `null` value rather than
the `sha256` content hash so the stage tests don't fail.
The optional_metadata_types option isn't available in older versions of
dnf. We could version-guard the option, but let's be more explicit and
check if it exists instead.
If it doesn't we can safely ignore it and rely on dnf's default
behaviour.
When the filelists are enabled in the optional metadata, the local cache
for the depsolve will include a filelist file for each repository.
Count the files matching *filelists* using glob() and compare them with
the number of repositories when the option is enabled.
When the option is not enabled, there should be no filelists.
We originally enabled filelists unconditionally in dnf5. Let's make it
optional as part of the 'optional-metadata' option in the arguments so
that the caller can decide if it's needed, for example when using dnf5
with older repositories that have packages with file dependencies, but
don't download them when they're not needed, since the filelists are
quite large.
Filelist repo metadata is required when using newer versions of libdnf
with old repositories or packages that specify dependencies on files.
For example, depsolving with RHEL 9.3 repos on Fedora 40 fails to
resolve platform-python with the message
nothing provides /usr/libexec/platform-python needed by platform-python-...
Not loading filelists is beneficial because it greatly reduces the size
of the metadata to download. Filelists were downloaded by default for
repositories in the past, but with newer versions of dnf/libdnf (in
Fedora 40), they are downloaded on-demand (e.g. when running a query
that requires file information). Newer package guidelines prohibit
depending on file paths, so filelists are not needed for package
depsolving with newer distro repositories.
Add an 'optional-metadata' property to the 'arguments' object of the
depsolve request, so that callers can enable any optional metadata they
need.
Further reading:
- https://libdnf.readthedocs.io/en/stable/tutorial-py/#case-for-loading-the-filelists
- https://github.com/rpm-software-management/dnf/releases/tag/4.19.0
- https://dnf.readthedocs.io/en/stable/user_faq.html#starting-with-fedora-40-i-noticed-repository-metadata-is-synchronized-much-faster-what-happened
Also fix the test, it wasn't updated to use sslverify instead of
ignoressl. This now correctly passes on -dnf and fails on -dnf5 which
will be fixed in the next commit.
