Add support for embedding container images via the cloud API. For
this the container resolve job was plumbed into the cloud api's
handler and the API specification updated with a new `containers`
section that mimics the blueprint section with the same name.
Dependency errors are not set by the workers, they're not set directly
in the job result. They are added by the worker server in case the job
error indicates it's a dependency error.
The ellipsis operator was used as a hack to not need to pass any details
as an argument, but it makes what the end object will actually look like
less obvious. It also makes it impossible to pass an array to details
without getting a nested array.
Fixes#2874
If the user does not pass a name, use the distribution as a name
A provided tag is used only if name is provided. It
The tag's default is a generated using UUID to avoid collisions
Worker
------
Add configuration for the default container registry.
Use the default container registry if not provided as part
of the image name.
When using the default registry use the configured values
Return the image url as part of the result.
Composer Worker API
-------------------
Add `ContainerTargetResultOptions` to return the image url
Composer API
------------
Add UploadOptions to allow setting of the image name and tag
Add UploadStatus to return the url of the uploaded image
Co-Developed-By: Christian Kellner <christian@kellner.me>
This issue was found by Coverity:
Error: DEADCODE (CWE-561): [#def1]
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:386: cond_null: Condition "err != nil", taking false branch. Now the value of "err" is "nil".
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:390: null: At condition "err != nil", the value of "err" must be "nil".
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:390: dead_error_condition: The condition "err != nil" cannot be true.
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:391: dead_error_line: Execution cannot reach this statement: "<temporary>.$0 = container....".
389|
390| if err != nil {
391|-> return resolvedIds{}, nil
392| }
393|
The nvdimm module is required for booting the image via UEFI HTTP.
The rest are added for feature parity with the official RHEL 9.0 ISO.
Fixes rhbz#2030730
Most filesystems entries in fstab don't have a PassNo, which makes the
order of those entries dependent on the sorting algorithm. Changes in
the algorithm can introduce changes in the sort order, which we don't
like.
Add a secondary sorting key, the Path, which is guaranteed unique, to
guarantee stable ordering.
Since the `jobStatus` functions return a `JobInfo`
struct that contains the `JobStatus`, it makes sense
to rename the function names for the sake of consistency.
The osbuild jobtype currently contains the
architecture as a suffix. Since the arch
is now being supplied as a label, the
`arch` suffix can be removed.
The number of return values from the `jobStatus`
function was growing and getting out of hand. Not
all return values were being used in all cases
and so returning a single struct with the information
and status of a job makes more sense. Then in each case
the resulting fields can be used as needed.
Add the architecture label to build jobs
which will enable filtering and monitoring
build jobs by architecture. Build job results
contain the `arch` field in the results struct,
this is then used to pass to the metrics, where
there is a value, otherwise it is set to an
empty string.
Remove a duplicate call to the `DequeueJobMetrics`
function in the worker server. This duplicate call
resulted in negative numbers for pending jobs in
the prometheus metrics.
Instead of using a cached result `GetDefaultAuthFile`, always
do call the function when a new `Client` is created, since at
least `/run/containers` can get created as a side-effect by
one of the container. Now that we check eagerly and often the
path check function was reworked to only return paths that do
exist and are accessible.
Also check if `REGISTRY_AUTH_FILE` is set and if so, and it
is accessible use that.
To check accessability, use `unix.Access` instead of `os.Stat`,
since On Fedora/RHEL 9 `os.Stat` is implemented via `statx` and
will indeed return `EACCES` for inaccessible paths. But on RHEL
8 `lstat` is used and that will return `ENOENT` but then later
when trying to open the file we will get `EPERM`.
Add support for embedding containers in OSTree commits by
storing them in `/usr/share/containers/storage`. The storage
engine is configured accordingly so that this extra location
is automatically taken into account by e.g. `podman`.
Add support for embedding containers in OSTree commits by
storing them in `/usr/share/containers/storage`. The storage
engine is configured accordingly so that this extra location
is automatically taken into account by e.g. `podman`.
We never want an empty path but always force a specific auth
file location, even if the location does not actually exist,
due to the peculiarities mentioned in the comment of the
`container.GetDefaultAuthFile` function.
The main reason is that there should be only one place where the
container resolution is happening, which is the worker, so that
we only have one central place to configure aspects of it, like
container credentials.
Implement all of Fedora in terms of this new abstraction. What used to be the
manifest functions (and before that the pipeline functions) are now the image
functions, whose purpose is to instantiate the right image kind structs from the
image type definitions we currently have in the distro definition.
This abstracts away the manifest instantiation. The idea is that we define one
of these image kind types to represent a group of image types that are
sufficiently similar. Each image kind will have a struct with with all the
properties that can be customised for the image and a function to turn that into
an actual manifest. This is similar to how distro/fedora/manifest.go and
cmd/osbuild-playground works today, and aspires to move these closer together
and to eventually make the distro definitions simpler.
For now cmd/osbuild-playground is moved over to using the new abstraction.
For now this encapsulates osbuild export and filename in that
exported tree. In the future we could add MIME type.
For now this is a concrete type, but should probably be an
interface, so the consumer of artefacts know they are the right
type. Enforcing we only push AMIs to EC2, etc.
Similarly to how checkpoints work, each pipeline can be marked for
export, and the manifest can return all the names of the exported
pipelines, to be passed to osbuild. Additionally, the Export
function returns an artefact object, which can be used to know how
to access the exports once osbuild is done. For now, this is unused.
If Checkpoint() is called on a pipeline, it is marked for
checkpointing. Calling GetCheckpoints() returns the names of all
its pipelines that are marked for checkpointing as a slice of
strings. This can be passed to osbuild by the caller, in which case
the trees produced by each of these pipelines will be checkpointed
to speed up future builds.
Before this can be used in production we need a mechanism for
automatically cleaning up the cache.
This is meant to encapsulate the tweaks we do to the OS tree
orthogonally to anything else. For now it still contains some
configuration that only sometimes applies, but this should
continue being reworked until all the fields in this struct
always apply to any artefact that is using it.
At the same time, stop instantiating with default values, as the
empty values should work. This is not a functional change as the
caller always sets these now.
Add support to handle upload options in image requests for Koji
composes. The image is always uploaded to Koji, but now it can be
uploaded to the cloud environment in addition to Koji as part of the
build.
The image name used for Koji image can't be used as is for uploading to
the cloud, because each cloud provider has its own requirements for the
valid characters. For now, let the Cloud API implementation generate a
random image name. The name is always returned in the compose status's
upload status, so it should be possible to attach it to the Koji build
to allow users to find the image.
Enhance the `koji-finalize` job implementation to be able to cope with
multiple upload targets being specified for an `OSBuildJob`.
Implement a convenience method `OSBuildJobResult.TargetResultsByName()`
for filtering the target results attached to the job result by their
name. Cover the method with an unit test. And lastly use this method in
the `koji-finalize` job to find the appropriate Koji upload target
results.
This is a preparation for enabling cloud uploads for Koji composes.
Enhance the `koji-finalize` job implementation to use deferred function
to ensure that the job status is always reported back to the composer.
In addition, if the `JobError` is set, also fail the Koji job.
Previously, composer and Koji were not updated in some corner cases when
the job would fail.
Support for adding containers in non-ostree images. The reason we
don't support OSTree artefacts just yet is that the default storage
location for container is `/var/lib/containers/storage`. But for
OSTree images all content in `/var` is discarded, since that is
deployment specific data. We therefore need to store the containers
somewhere else, e.g. `/usr/share/containers/storage`, but then also
need to configure the system to find containers in that location.
osbuild only recently gained the corresponding stage to do so and
thus this will be done in a follow up.
Support for adding containers in non-ostree images. The reason we
don't support OSTree artefacts just yet is that the default storage
location for container is `/var/lib/containers/storage`. But for
OSTree images all content in `/var` is discarded, since that is
deployment specific data. We therefore need to store the containers
somewhere else, e.g. `/usr/share/containers/storage`, but then also
need to configure the system to find containers in that location.
osbuild only recently gained the corresponding stage to do so and
thus this will be done in a follow up.
Add a new `containers` section that can be used to request the
embedding of containers into images. The only requirement is
the source property to specify where to fetch the container from.
This suppports specifying the digest of the container or the tag.
In case none is given it defaults to the `latest` tag. The `Name`
field can be used to optionally specify a name to use inside the
image.
NB: currently no tools or apis support container resolution yet.
This follows in the next commits.
