Fix:
assemblers/org.osbuild.qemu:310:36: E0606: Possibly using variable 'prep_type' before assignment (possibly-used-before-assignment)
inputs/org.osbuild.tree:85:15: E0606: Possibly using variable 'path' before assignment (possibly-used-before-assignment)
stages/org.osbuild.sfdisk:58:36: E0606: Possibly using variable 'prep_type' before assignment (possibly-used-before-assignment)
stages/org.osbuild.systemd.unit:23:16: E0606: Possibly using variable 'unit_dropins_dir' before assignment (possibly-used-before-assignment)
test/mod/test_meta.py:219:29: E0606: Possibly using variable 'schema_part' before assignment (possibly-used-before-assignment)
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new util module called host which is used for functions that are
meant for interactions with the host. These functions should not be
used in stages.
The containers.get_host_storage() function is renamed to
host.get_container_storage() for clarity, since it is no longer
namespaced under containers.
The current `make_container()` helper is a bit silly (which is
entirely my fault). It requires a container tag as input but all
tests end up creating a random number for this input. So instead
just remove the input and return the container_id from the podman
build in the contextmanager and use that.
This commit tweaks the excellent work from Gianluca in PR#1550 a
little bit. Now that the container inputs are their own input
type some of the code that used to be part of the original
`inputs/org.osbuild.containers` can be simplified.
An input that pairs with the org.osbuild.containers-storage source. It
makes the host container storage available to a stage and a container ID
for the stage to use.
Unlike all the other input types, this input does not refer to static
objects in the osbuild sources store but a dynamically mounted container
store that depends on the host storage configuration and the
availability of specific containers in that store.
Co-authored-by: Achilleas Koutsou <achilleas@koutsou.net>
Co-authored-by: Gianluca Zuccarelli <gzuccare@redhat.com>
The format so far was assumed to be `docker-archive` if the container
was coming from a source and `oci-archive` if it was coming from a
pipeline. The source format will now be changed to `dir` instead of
`docker-archive`. The pipeline format remains `oci-archive`.
With the new archive format being `dir`, the source can't be linked into
the build root and is bind mounted instead with the use of a MountGuard
created with the instance of the service, and torn down when the service
is stopped.
The _data field is removed from the map functions. It was unused and
these functions aren't part of the abstract class so they don't need to
have consistent signatures.
Update the skopeo stage with support for the newly supported `dir`
format.
Apply the isort modifications to the entire source tree, not just the
selected python files of test-src.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
This fixes pylint warnings on our modules that are currently not part of
CI-pylint. The fixes should all be straightforward.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
This extends the possible ways of passing references to inputs. The
current ways possible are:
1) "plain references", an array of strings:
["ref1", "ref2", ...]
2) "object references", a mapping of keys to objects:
{"ref1": { <options> }, "ref2": { <options> }, ...}
This patch adds a new way:
3) "array of object references":
[{"id": "ref1", "options": { ... }}, {"id": ... }, ]
While osbuild promises to preserves the order for "object references"
not all JSON serialization libraries preserve the order since the
JSON specification does leave this up to the implementation.
The new "array of object references" thus allows for specifying the
references together with reference specific options and this in a
specific order.
Additionally this paves the way for specifying the same input twice,
e.g. in the case of the `org.osbuild.files` input where a pipeline
could then be specified twice with different files. This needs core
rework though, since internally we use dictionaries right now.
Specifically this test checks that the order given in the manifest is
preserved when loaded, i.e. the internal dict has the keys ordered in
the same way, independently in which way they were specified -- list
or object.
When using "plain refs", that is, when using an array of strings,
we did not enforce the constraints of exactly one reference. This
was done for dictionary references.
We hardcode oci-archive for now and use the first file in the pipeline
tree as the archive. Long term we may want to get this info from the
metadata of the oci-archive stage instead.
This adds a stage called org.osbuild.skopeo that installs docker and
oci archive files into the container storage of the tree being
constructed.
The source can either be a file from another pipeline, for example one
created with the existing org.osbuild.oci-archive stage, or it can
be using the new org.osbuild.skopeo source and org.osbuild.containers
input, which will download an image from a registry and install that.
There is an optional option in the install stage that lets you
configure a custom storage location, which allows the use of the
additionalimagestores option in the container storage.conf
to use a read-only image stores (instead of /var/lib/container).
Note: skopeo fails to start if /etc/containers/policy.json is
not available, so we bind mount it from the build tree to the
buildroot if available.
The `refs` variable is used as parameter to indicate which commits
to checkout, but it also was used as result variable to store the
ids of the commits that actually got checked out; naturally, it
was initialized to the empty array. This of course meant nothing
was ever actually checked out. Doh.
New input type that takes a ostree repo and checks out any number
of commits inside that repo to a temporary directory. Each commit
will be checked out to a separate sub-directory. The name of the
dir is the commit id of the corresponding commit.
This input can thus be used to access files and directories of
commits in stages.
Currently the `org.osbuild.files` input only supports the source origin.
Extend support to mapping files from pipelines, using the recently added
sub-tree reading capability of `ObjectStore.reat_at`. Restructure the
JSON schema to keep is as readable as possible.
The `org.osbuild.files` input provides individual files to a stage.
Change the `refs` key in the returned dict to `files` to better
reflect that fact. Also adapt the documentation to indicate that
the keys actually paths and not necessarily checksums. This prepares
for future extension of the `files` input to pipeline origins.
Create a `InputService` class with an abstract method called `map`,
meant to be implemented by all inputs. An `unmap` method may be
optionally overridden by inputs to cleanup resources.
Instantiate a `host.ServiceManager` in the `Stage.run` section and
pass the to the host side input code so it can be used to spawn the
input services.
Convert all existing inputs to the new service framework.
Instead of bind-mounting each individual input into the container,
create a temporary directory that is used by all inputs and bind-
mount this to the well known location ("/run/osbuild/inputs"). The
temporary directory is then passed to the input so that it can
make the requested resources available relative to that directory.
This is enforced by the common input handling code.
Additionally, pass the well known input path via a new "paths" key
to the arguments dictionary passed to the stage.
Use `ostree-ouput` instead of `files-output` as prefix for the
temporary directory prefix.
Also fix the description of the "origin" to reflect that
pipeline and source origins are supported.
Additionally remove some dead code.
Add a new input that can handle ostree commits. For source origins,
it will pull any number of commits specified via `references`. For
pipeline origins it will use the compose.json to obtain the commit
id and pull that. Either way the data passed onto the stage is a
dictionary of commit ids and commit metadata, such as the `ref`
that is an ostree ref that is optionally created in case the `ref`
option for that reference was specified.
Currently all options for inputs are totally opaque to osbuild
itself. This is neat from a seperation of concerns point of view
but has one major downside: osbuild can not verify the integrity
of the pipeline graph, i.e. if all inputs that need pipelines or
sources do indeed exists. Therefore intrdouce two generic fields
for inputs: `origin` and `references`. The former can either be
a source or a pipeline. The latter is an array of identifiers or
a dictionary where the keys are the identifiers and the values
are additional options for that id. The identifiers then refer
to either resources obtained via a source or a pipeline that has
already been built.
This represents the tree-type input, which is the simplest form
of input a stage can consume. For the stage it is just a file
system tree at a specific location and the contents itself is
mostly opaque. An obvious thing to do with such a content is
carbon copy it to a different location.
Currently this input only supports the pipelines as origins,
which are identified via their id.
