The name 'cache' is more recognisable and conveys the purpose of the
option just as well (if not better) than 'store'.
Also, the option that controls the size of this directory is called
'cache-max-size', so we're already referring to it as a cache.
This commit tweaks build() to be mypy clean without the need to
call assert. This drops the map() and instead we use the existing
dict-like access of the manifest to get the pipeline. In practise
this should not happen but lets be prepared.
Note that a small tweak for the error is needed to make it clear
what is happening.
Thanks to Simon for raising this.
This commit makes the returns of the of {Build,Download}Result.as_dict()
explicit. Ideally this would just be a dataclass and dataclass.asdict()
but because we need to support python3.6 this is not possible today.
This commit adds error reporting from source download errors
to the monitor. It reuses the `BuildResult` for symmetry but
we probably want to refactor this a bit to make source handling
a bit more similar to stages.
In order to avoid having to rely on the output of `osbuild --json`
when using `--progress=JSONSeqMonitor` the monitor needs to include
the `osbuild.pipeline.BuildResult` for each individual stage.
This commit adds those to the montior.
Using a metalink or mirrorlist along with the package paths and
checksums allows them to be reliably downloaded even when mirrors are
not all in sync. It will retry with a new mirror until it succeeds, or
has tried all of the mirrors.
It turned out that in many cases, stages need to join two absolute
paths, the pipeline tree path and the path on a booted system. However,
the standard `os.path.join()` function can't handle such situation as
just prepending the root to the subsequent paths.
Add a new helper function, which is able to join any paths together,
regardless if any of them is absolute or not. If the root is not
absolute, the result will be made absolute to the filesystem root `/`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit adds a tiny unit test for the new `shlex` based
os-release parsing and tweaks the error message in a small
and non-functional way (just because it's slightly nicer
for a user). The test checks for three keys NAME which is
quoted with `"`, ID which is not quoted and OSTREE_VERSION
which is quoted with `'`.
- Replaced string stripping with `shlex.split()` to properly
handle values in the os-release file;
- This ensures cleaner and more accurate key-value assignments,
follwing a broader set of shell-like parsing rules;
- Add os-release file for Fedora CoreOS 40 for testing.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
- Add optional bind_mounts parameter to __init__ method;
- Enhanced methods to accept an optional `bind_mounts`.
This allows for more flexible for configurations when setting
up bind mounts.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
Enable generating SBOM documents for depsolved transactions when using
DNF5. Enable SBOM testing with DNF5 in unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add functions for transforming package sets depsolved using libdnf5 to
the SBOM standard-agnostic model. Cover the function with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Disable the newly reported pylint issue W0135
(contextmanager-generator-missing-cleanup), because as far as I was able
to understand the motivation behind it, it should not apply to the code
and it should be a false positive. We do not use context manager inside
a generator, but inside another context manager. So the execution flow
should return sequentially through the stack and all context manager
cleanups should be executed as expected.
The reported issue:
osbuild/util/containers.py:184:4: W0135: The context used in function 'container_source' will not be exited. (contextmanager-generator-missing-cleanup)
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Fix:
osbuild/pipeline.py:101:12: R1737: Use 'yield from' directly instead of yielding each element one by one (use-yield-from)
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
New fucntion returns tuple of 'root' and relative 'file path', which could be
useful in contexts, where knowing 'root' is required, for example setting
selinux labels.
Michael Vogt pointed out that testcases start failing when we describe
the new format. Let's add a test case and fix the describe to include
the metadata.
Metadata is freeform in the `Manifest` instance but it is stored on it
during loading (at which time its properties are validated) and returned
as-is on describe.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
This change allows use the more advanced features of bash like
array operations (e.g. `${@:2}` to drop the first two arguments
or similar. On fedora/rhel this is a no-op as it is already using
sh -> bash (afaik).
We currently use the absolute path of these binaries in the
helper. This has some advantages but given that we control the
inputs for PATH in general it seems unnecessary.
We are also slightly inconsistent about this in the codebase but
favor the non absolute path version. A quick count:
```
$ git grep '"chroot"'|wc -l
13
$ git grep '"/usr/sbin/chroot"'|grep -v test_|wc -l
8
```
for `mount` and `umount` it seems this is the only place that uses
the absolute path.
It's not an important change but it has the nice property that it
allows us to use e.g. `testutil.mock_command()` in our tests and
it would be nice to be consistent.
This commit moves the joining of path fragements from f-strings
to pathlib and simplifies some of the map/filter/lambda expressions
into more standard list comprehensions.
'_hawkey.Reldep' object has no attribute 'name' in the version shipped
on RHEL-8. Add code to handle this situation in case it happens.
Default to using named attributes if these are available.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Since the `with_sbom` variable was used only in a single place, we can
simplify the code (and remove one extra line of it) to just directly use
the if condition.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend osbuild-depsolve-dnf, to return JSON with SPDX SBOM that
corresponds to the depsolved package set, if it has been requested.
For now, only DNF4 is supported.
Cover the new functionality with unit test.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This will allow validating request arguments in the solver method in a
different way for dnf4 and dnf5 and raising an exception if needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add implementation of standard-agnostic model for SBOM, and simple SPDX
v2.3 model. Also add convenience functions for converting DNF4 package
set to the standard-agnostic model and for converting it to SPDX model.
Cover the functionality with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
It's sometimes useful to set up a loop device for an already formatted
disk/filesystem image to derive new artifacts from it. In that case, we
want to make sure it's impossible to modify its contents in any way in
that process, both for our own purposes and for other stages operating
on it.
Notably, mounting some filesystems read-only still seem to touch the
disk (like XFS).
The existing jsoncomm is a work of beautiy. For very big arguments
however the used `SOCK_SEQPACKET` hits the limitations of the
kernel network buffer size (see also [0]). This lead to various
workarounds in #824,#1331,#1836 where parts of the request are
encoded as part of the json method call and parts are done via
a side-channel via fd-passing.
This commit changes the code so that the fd channel is automatically
and transparently created and the workarounds are removed. A test
is added that ensures that very big messages can be passed.
[0] https://github.com/osbuild/osbuild/pull/1833
The current error message when an export is not found could be
improved by printing what exports are actually availalble to make
it easier for the user to e.g. spot typos.
