Show the stage name (if one is set) when failing the stage in the
validator. This closes#1007, example output:
```
€ python3 -m osbuild supakeen-os.json
supakeen-os.json has errors:
pipelines[0].stages[0]
could not find schema information for 'org.osbuild.rpmb'
.pipelines[0].stages[0].inputs.packages:
could not find schema information for 'org.osbuild.filesz'
```
The generic ways of checking if an object is in the cache does not apply
for ostree as the internal structure of a repo is quite specific. Thus
we need to use the ostree executable to ask it to explore its repo for
us.
Before, the download method was defined in the inherited class of each
program. With the same kind of workflow redefined every time. This
contribution aims at making the workflow more clear and to generalize
what can be in the SourceService class.
The download worklow is as follow:
Setup -> Filter -> Prepare -> Download
The setup mainly step sets up caches. Where the download data will be
stored in the end.
The filter step is used to discard some of the items to download based
on some criterion. By default, it is used to verify if an item is
already in the cache using the item's checksum.
The Prepare step goes from each element and let the overloading step the
ability to alter each item before downloading it. This is used mainly
for the curl command which for rhel must generate the subscriptions.
Then the download step will call fetch_one for each item. Here the
download can be performed sequentially or in parallel depending on the
number of workers selected.
Introduce a new class member `content_type` that specifies what type of
items the source will store in the cache. Use that to generalize the
setup step, which is shared across all sources.
This warning was globally disabled in commit c124ab2, due to dynamic
attributes of the `LoopInfo` class. This false positive is silenced
locally now. Some actual positives have meanwhile made it into the
code base, but have fixed via previous commits so we can now enable
W0201/attribute-defined-outside-init again.
Silence pylint warning W0201 (attribute-defined-outside-init) in
`set_status`; it sets dynamic attributes on the LoopInfo class
which pylint does not recognize.
When merging user and group database from individual commits also merge
the corresponding /etc/sub{u,g}id database. These are created when the
users are added via `useradd` and thus also need to be imported with
the corresponding users.
Add a new class `SubIdsDB` as a database of subordinate Ids, like the
ones in `/etc/subuid` and `/etc/subgid`. Methods to read and write
data from these two files are provided.
Add corresponding unit tests.
Drop `CAP_MAC_ADMIN` from the default capabilities which is needed
to write and read(!) unknown SELinux labels. Adjust the stages
that need to read or write SELinux labels accordingly.
Drop CAP_{NET_ADMIN,SYS_PTRACE} from the default capabilities which
are only needed to run bwrap from inside a stage which is done by
the `ostree.commit` and `ostree.preptree` stages, so retain them
directly there.
Add new stage metadata `CAPABILITIES` where stages can request
additional capabilities that are not in the default set.
Currently this is not used by any stage since the default set
contains the sum of all needed capabilities.
Drop all capabilities that are not required by any of the stages.
N.B. at least one stage (`ostree.preptree`) itself executes bwrap
itself, which in turn needs `CAP_SYS_PTRACE` and `CAP_NET_ADMIN`.
Add a new member variable `caps` that if not `None` indicates the
capabilities to retain, i.e. all other capabilities not specified
will be dropped via `bubblewrap` (`--cap-drop`).
Add corresponding tests.
LVM2 introduced system.devices as an alternative way to filter
devices. Since we create devices in a stage the devices won't be
added to the /etc/lvm/devices/system.devices file since /etc/ is
inside the container. As a result the we can't see these devices
and will fail with "Could not find parent device".
Therefore we add support for managing our own per-service devices
file, iff a `system.devices` is present.
This extends the possible ways of passing references to inputs. The
current ways possible are:
1) "plain references", an array of strings:
["ref1", "ref2", ...]
2) "object references", a mapping of keys to objects:
{"ref1": { <options> }, "ref2": { <options> }, ...}
This patch adds a new way:
3) "array of object references":
[{"id": "ref1", "options": { ... }}, {"id": ... }, ]
While osbuild promises to preserves the order for "object references"
not all JSON serialization libraries preserve the order since the
JSON specification does leave this up to the implementation.
The new "array of object references" thus allows for specifying the
references together with reference specific options and this in a
specific order.
Additionally this paves the way for specifying the same input twice,
e.g. in the case of the `org.osbuild.files` input where a pipeline
could then be specified twice with different files. This needs core
rework though, since internally we use dictionaries right now.
Specifically this test checks that the order given in the manifest is
preserved when loaded, i.e. the internal dict has the keys ordered in
the same way, independently in which way they were specified -- list
or object.
When using "plain refs", that is, when using an array of strings,
we did not enforce the constraints of exactly one reference. This
was done for dictionary references.
This is a left-over from the time when `systemd-nspawn` was used,
which only retained a limited set of capabilities which did not
include `CAP_MAC_ADMIN`[1]. Bubblewrap, on the other hand, retains
all currently capabilities if the process is run as root[2].
[1] see e.g. src/nspawn/nspawn.c#L147 of commit c52950c
[2] see commit abc56644566a6095bb72a5bf70fcee7dd90e9447
mkfs.xfs already has `additionalProperties: true` for the devices
section, as this is necessary for example when creating lvm2 setups.
This should be possible for other filesystem types too.
