Add the information about the image boot mode to the OSBuildJob options
as well as to the OSBuildJobResult options. The intention is that the
worker will simply copy the value from the job options to job result,
so that the `KojiFinalize` job can then access this information and
upload it to Koji as extra metadata.
This information is required in Koji in order for Red Hat's SP tooling
to know how to import image to the cloud environment in case the boot
mode affects the import parameters.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
The duration middleware should come after the tenant channel middleware,
otherwise the tenant in the context will be empty. The status middleware
can come beforehand because it queries the request context right before
sending a response.
Change the OSTreeResolveSpec to match the ostree SourceSpec by removing
the Parent field.
Change OSTreeResolveResultSpec to match the CommitSpec by adding the
Secrets field. The RHSM field is kept for backwards compatibility with
older workers.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
Return manifest.Manifest from the Manifest() function without
serializing. The caller then has to call the manifest.Serialize()
function using the depsolved packages.
This moves towards changing the order of actions required to generate a
manifest. With this change, the manifest creation and depsolving can be
done independently, but this still requires instantiating the manifest
object twice (InstantiateManifest() is called in PackageSets() and
Manifest()), which we don't want to have to do.
Removing the dependence of the manifest package on the distro package to
import manifest into distro.
Wherever arch names are needed, we use the enums from the platform
package instead.
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].
Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.
[1] See dnf.config
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.
Copy the value whenever a spec is moved to a different representation.
This changes the `Manifest` function of the `ImageType`
interface so that any warnings detected during the
`checkOptions` step of the manifest initialization can
be propagated back to the Weldr-API (see next commit).
Signed-off-by: Irene Diez <idiez@redhat.com>
Only add the arch label for osbuild job types, as the finish metrics
behave similarly. Having arch labels on dequeue metrics for any other
job type (but not on the finish metrics) would produce weird results.
Register the custom middleware function to the worker
server. This function is responsible for recording all
the status codes of all the server's endpoints.
Due to a bug with echo/v4, a request to an endpoint using
the incorrect method should return a `405` error but returns
a `404` error instead when a middleware function is registered.
The worker `server_test` has been updated to reflect this.
If a job is unresponsive the worker has most likely crashed or been shut
down and the in-progress job been lost.
Instead of failing these jobs, requeue them up to two times. Once a job is lost
a third time it fails. This avoids infinite loops.
This is implemented by extending FinishJob to RequeuOrFinish job. It takes a
max number of requeues as an argument, and if that is 0, it has the same
behavior as FinishJob used to have.
If the maximum number of requeues has not yet been reached, then the running
job is returned to pending state to be picked up again.
Add support for embedding container images via the cloud API. For
this the container resolve job was plumbed into the cloud api's
handler and the API specification updated with a new `containers`
section that mimics the blueprint section with the same name.
The ellipsis operator was used as a hack to not need to pass any details
as an argument, but it makes what the end object will actually look like
less obvious. It also makes it impossible to pass an array to details
without getting a nested array.
Fixes#2874
Since the `jobStatus` functions return a `JobInfo`
struct that contains the `JobStatus`, it makes sense
to rename the function names for the sake of consistency.
The osbuild jobtype currently contains the
architecture as a suffix. Since the arch
is now being supplied as a label, the
`arch` suffix can be removed.
The number of return values from the `jobStatus`
function was growing and getting out of hand. Not
all return values were being used in all cases
and so returning a single struct with the information
and status of a job makes more sense. Then in each case
the resulting fields can be used as needed.
Add the architecture label to build jobs
which will enable filtering and monitoring
build jobs by architecture. Build job results
contain the `arch` field in the results struct,
this is then used to pass to the metrics, where
there is a value, otherwise it is set to an
empty string.
Remove a duplicate call to the `DequeueJobMetrics`
function in the worker server. This duplicate call
resulted in negative numbers for pending jobs in
the prometheus metrics.
Enhance the `koji-finalize` job implementation to be able to cope with
multiple upload targets being specified for an `OSBuildJob`.
Implement a convenience method `OSBuildJobResult.TargetResultsByName()`
for filtering the target results attached to the job result by their
name. Cover the method with an unit test. And lastly use this method in
the `koji-finalize` job to find the appropriate Koji upload target
results.
This is a preparation for enabling cloud uploads for Koji composes.
Enhance the `koji-finalize` job implementation to use deferred function
to ensure that the job status is always reported back to the composer.
In addition, if the `JobError` is set, also fail the Koji job.
Previously, composer and Koji were not updated in some corner cases when
the job would fail.
This is the first step to support embedding container images. Here
we add the `containers []container.Spec` argument to supply images
with resolved container specifications. For now all distros will
return an error in case a container is actually supplied since none
of them currently support embedding containers. NB: also no apis or
tools will actually resolve containers.
Koji API removed by the previous commit was the last user of osbuild-koji job.
Let's remove it since nothing uses it. This also removes all of the
compatibility code in Cloud API, see concerns below:
Compatibility concerns:
- the internal deployment was moved to a completely different composer
instance, thus there are no old jobs
- Fedora deployment is still unused in prod, thus we don't care about keeping
backward compatibility of the old jobs
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The test_distro Manifest, which is used in tests across multiple
packages, was using the old structure. Updated to the v2 structure and
adapted all tests.
The osbuild export is specific to the upload target and different
targets may require using a different export. While osbuild-composer
still does not support multiple exports for osbuild jobs, this prepares
the ground for such support in the future.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
message is always a super-set of the old and new way of providing the
exports to osbuild job.
A backward compatibility code handling the conversion of VMDK image to
stream-optimized sub-format has been kept in the implementation since
PR#2529 [1] merged on May 4th 2022. Since this change, no API
implementation is submitting jobs, which would hit this conversion code,
because VMDK images are already being produced in the desired
sub-format.
On-premise deployments are expected to use the same composer and worker
versions. There are no composer / worker instances in production, which
are not running the modified code.
Delete the backward compatibility code.
[1] https://github.com/osbuild/osbuild-composer/pull/2529
Add a new worker client error type `ErrorTargetError` representing that
at least one of job targets failed. The actual target errors are added
to the job detail.
Add a new `OSBuildJobResult.TargetErrors()` method for gathering a slice
of target errors contained within an `OSBuildJobResult` instance. Cover
the method with unit test.
The `TargetErrors` is not used any more since PR#2192 [1] and there is
no need to keep the backward compatibility any more, because there are
no composer / worker instances in production, which are not running the
modified code.
In addition, delete unit tests covering this legacy error handling.
[1] https://github.com/osbuild/osbuild-composer/pull/2192
