Port sources to also use the host services infrastructure that is
used by inputs, devices and mounts. Sources are a bit different
from the other services that they don't run for the duration of
the stage but are run before anything is built. By using the same
infrastructure we re-use the process management and inter process
communcation. Additionally, this will forward all messages from
sources to the existing monitoring framework.
Adapt all existing sources and tests.
Since the `sources.SourcesServer` has been removed, nothing is
using the export functionality anymore. Inputs are now used to
make content in the store available to stages. Remove all the
export logic from org.osbuild.ostree.
Instead of using stderr for the ostree subprocess command
capture its output so that in the case of an error we get
properly return the error output. With the old behavior
all the `ostree` command output would land in the journal
of the worker.
Source, for compatability reasons, have two modes: download only
and download and export. The difference is the arguments that
are passed to the source: For download only, the `output` param
is empty. In this case also `checksums` *can* be empty and if so
it means everything, i.e. the commits, should be fetched. The
latter was not properly handled so far. Adjust the logic, which
now closely mimics that of the `org.osbuild.curl` source to fix
this case.
Also catch exceptions invoking `ostree` and properly return them
via the json error messaging.
Instead of supplying the full cache dir, i.e. the directory in
the store where the source will place the fetched resources, to
the source, only supply the root folder of the cache and let
the source itself create the desired sub-directory. This allows
the source to determine what type of resource it provides. This
makes the final directory independent of the name of the source:
a `org.osbuild.curl` source can place file-like resource in the
`org.osbuild.files` sub-directory. Then the `org.osbuild.files`
input can be used to get those from the cache directory.
In format version 2, the source specific keys for the sources,
here "urls", is replaced by a generic `items` key, common to
all sources. Express that in the schema.
All sources fetch various types of `items`, the specific nature
of which is dependent on the source type, but they are all
identifyable by a opaque identifier. In order for osbuild to
check that all the inputs that a stage needs are are indeed
contained in the manifest description, osbuild must learn what
ids are fetched by what source. This is done by standarzing
the common "items" part, i.e. the "id" -> "options for that id"
mapping that is common to all sources.
For the version 1 of the format, extract the files and ostree
the item information from the respective options.
Adapt the sources (files, ostree) so that they use the new items
information, but also fall back to the old style; the latter is
needed since the sources tests still uses the SourceServer.
Split the internal logic into two parts: 1) fetching the commit
into the internal cache repo and then 2) exporting that commit,
i.e. a local pull from the cache repo to the output directory.
If no `output` directory was specified, only fetch the commit,
do not attempt to export it.
NB: this commit changes at what point the gpg verification is
done. Previously the check was on export. Now, we are checking
the signature on import only. The export step will be replaced
by an ostree `Input` that will have the ability to verify
commits a second time.
Fix all occurrences of format-strings without any interpolation. pylint
warns about those (and for some reason did not do so for our modules).
A followup will fix the pylint tests, so make sure all the warnings are
resolved.
Make sure access to the shared ostree metadata is locked properly. This
is the default since 2018.5, but lets be explicit here. This also makes
sure that the option exists and the local version supports locked and
protected access.
It is unclear whether the `ostree init` honors that as well. It really
should, and if it doesn't we can always report it upstream.
Instead of verifying the gpg signature when pull from the actual
remote source into the local cache, verify the commit when it is
being pulled from the local cache into the output directory. This
ensures that the signatures are checked against the provided keys
even when the commit was already in the cache and at that time
the key might have been different.
NB: ostree expects the signature to be present on the remote at
the *target* repository, i.e. in our case the output repository.
The keys are therefore attached to a temporary remote that is
created at the output repository with the same name/id that is
used for the actual remote.
Add a new `gpgkeys` option that, if set, must contain a list of
public keys. These keys will then be used by ostree to verify
signed commits when pulling from the remote. If the `gpgkeys`
option is missing, no verification will be attempted.
This source can be used to fetch ostree commits. The commits are
accessed via their commit is. The only option currently is `url`,
given for each commit, that will be used as the location of the
remote. A cache repository, that will be created if necessary,
acts as an intermediary, so remotes will be added with `name` as
the identifier to it and commits are pulled into that. In the
output directory another repository will be created as 'repo' and
the requested commit pulled into that from the cache repository via
a local pull.
