It turned out that in many cases, stages need to join two absolute
paths, the pipeline tree path and the path on a booted system. However,
the standard `os.path.join()` function can't handle such situation as
just prepending the root to the subsequent paths.
Add a new helper function, which is able to join any paths together,
regardless if any of them is absolute or not. If the root is not
absolute, the result will be made absolute to the filesystem root `/`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit adds a tiny unit test for the new `shlex` based
os-release parsing and tweaks the error message in a small
and non-functional way (just because it's slightly nicer
for a user). The test checks for three keys NAME which is
quoted with `"`, ID which is not quoted and OSTREE_VERSION
which is quoted with `'`.
- Replaced string stripping with `shlex.split()` to properly
handle values in the os-release file;
- This ensures cleaner and more accurate key-value assignments,
follwing a broader set of shell-like parsing rules;
- Add os-release file for Fedora CoreOS 40 for testing.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
- Add optional bind_mounts parameter to __init__ method;
- Enhanced methods to accept an optional `bind_mounts`.
This allows for more flexible for configurations when setting
up bind mounts.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
Enable generating SBOM documents for depsolved transactions when using
DNF5. Enable SBOM testing with DNF5 in unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add functions for transforming package sets depsolved using libdnf5 to
the SBOM standard-agnostic model. Cover the function with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Disable the newly reported pylint issue W0135
(contextmanager-generator-missing-cleanup), because as far as I was able
to understand the motivation behind it, it should not apply to the code
and it should be a false positive. We do not use context manager inside
a generator, but inside another context manager. So the execution flow
should return sequentially through the stack and all context manager
cleanups should be executed as expected.
The reported issue:
osbuild/util/containers.py:184:4: W0135: The context used in function 'container_source' will not be exited. (contextmanager-generator-missing-cleanup)
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Fix:
osbuild/pipeline.py:101:12: R1737: Use 'yield from' directly instead of yielding each element one by one (use-yield-from)
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
New fucntion returns tuple of 'root' and relative 'file path', which could be
useful in contexts, where knowing 'root' is required, for example setting
selinux labels.
Michael Vogt pointed out that testcases start failing when we describe
the new format. Let's add a test case and fix the describe to include
the metadata.
Metadata is freeform in the `Manifest` instance but it is stored on it
during loading (at which time its properties are validated) and returned
as-is on describe.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
This change allows use the more advanced features of bash like
array operations (e.g. `${@:2}` to drop the first two arguments
or similar. On fedora/rhel this is a no-op as it is already using
sh -> bash (afaik).
We currently use the absolute path of these binaries in the
helper. This has some advantages but given that we control the
inputs for PATH in general it seems unnecessary.
We are also slightly inconsistent about this in the codebase but
favor the non absolute path version. A quick count:
```
$ git grep '"chroot"'|wc -l
13
$ git grep '"/usr/sbin/chroot"'|grep -v test_|wc -l
8
```
for `mount` and `umount` it seems this is the only place that uses
the absolute path.
It's not an important change but it has the nice property that it
allows us to use e.g. `testutil.mock_command()` in our tests and
it would be nice to be consistent.
This commit moves the joining of path fragements from f-strings
to pathlib and simplifies some of the map/filter/lambda expressions
into more standard list comprehensions.
'_hawkey.Reldep' object has no attribute 'name' in the version shipped
on RHEL-8. Add code to handle this situation in case it happens.
Default to using named attributes if these are available.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Since the `with_sbom` variable was used only in a single place, we can
simplify the code (and remove one extra line of it) to just directly use
the if condition.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend osbuild-depsolve-dnf, to return JSON with SPDX SBOM that
corresponds to the depsolved package set, if it has been requested.
For now, only DNF4 is supported.
Cover the new functionality with unit test.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This will allow validating request arguments in the solver method in a
different way for dnf4 and dnf5 and raising an exception if needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add implementation of standard-agnostic model for SBOM, and simple SPDX
v2.3 model. Also add convenience functions for converting DNF4 package
set to the standard-agnostic model and for converting it to SPDX model.
Cover the functionality with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
It's sometimes useful to set up a loop device for an already formatted
disk/filesystem image to derive new artifacts from it. In that case, we
want to make sure it's impossible to modify its contents in any way in
that process, both for our own purposes and for other stages operating
on it.
Notably, mounting some filesystems read-only still seem to touch the
disk (like XFS).
The existing jsoncomm is a work of beautiy. For very big arguments
however the used `SOCK_SEQPACKET` hits the limitations of the
kernel network buffer size (see also [0]). This lead to various
workarounds in #824,#1331,#1836 where parts of the request are
encoded as part of the json method call and parts are done via
a side-channel via fd-passing.
This commit changes the code so that the fd channel is automatically
and transparently created and the workarounds are removed. A test
is added that ensures that very big messages can be passed.
[0] https://github.com/osbuild/osbuild/pull/1833
The current error message when an export is not found could be
improved by printing what exports are actually availalble to make
it easier for the user to e.g. spot typos.
Current osbuild will always print some non output even
when run with `--monitor=JSONSeqMonitor` because of the
unconditional `print/sys.stdout.write()` in `main_cli.py`.
This commit adds a new `-q` option to silence this so that something
like osbuild-composer can run `osbuild -q --monitor=JSONSeqMonitor`
to get pure json-seq output during the build.
The use-case is to run `osbuild --monitor-fd` from e.g. bib and
osbuild-composer so that we get pure json from the monitor-fd
and anything that goes on std{out,err} can be logged as it is
most likely error output.
For consistency, use subprocess.run() with check=True for the calls that
were previously using subprocess.check_call().
Update the affected tests to match.
If one of the chroot mounts fails to unmount, keep iterating so that we
don't stop and continue to unmount the rest.
Print an error message with the failed mounts, but don't fail the build.
Since failing to unmount doesn't fail the exiting of the context, and
the context itself doesn't know what will be running in the chroot,
do a lazy unmount.
The libdir is passed down for sources but it is never used in
any of our sources. As this is confusing and we want to eventually
support multiple libdirs remove this code.
It looks like the libdir for soruces was added a long time ago in 8423da3
but there is no indication if/how it is/was supposed to get used and
AFACT from going over the git history it was very used.
SourceService:dispatch() never sends "libdir" to the actual sources,
so it is not an even technically an API break.
Add a new util module called host which is used for functions that are
meant for interactions with the host. These functions should not be
used in stages.
The containers.get_host_storage() function is renamed to
host.get_container_storage() for clarity, since it is no longer
namespaced under containers.
