With the new images library changes to move the pipeline
roles out of the image types into the manifest we need
to tweak osbuild-composer to use the new way of getting
the payload and build pipelines.
This commit implements that new method based on the
manifest instead of the image type. See images pr#1766
for the rational of the change.
This removes all artifact directories, and their contents, if there
isn't an associated Job. This is used to clean up local artifacts after
the compose job has been deleted.
Related: RHEL-60120
The original compose request contains useful details that are not
preserved when it is converted to a manifest. Things like the
distribution, arch, image type, blueprint or customizations are useful
when examining builds later.
This saves the original request json using the job id and a new
directory (ComposeRequest) under the artifacts directory. The original
request, if present, is then added to the compose/<id>/metadata response
alongside the package list.
Related: RHEL-60120
These fields are not set by the depsolve job, they are only set and used
in tests so remove them. Errors are reported in the result.JobError
Related: Related: RHEL-60125
This adds support for sending a search job to the worker client,
gathering results, and handling errors.
The errors returned are the same as for the Depsolve job, since they
both use the osbuild-depsolve-dnf script via images/pkg/dnfjson.
Related: RHEL-60136
This is similar to the depsolve job, and it shares the solver (which
supports locking, as does DNF itself). This will allow searching for
specific package names, names with globs, or names as substrings of
other names using * as the wildcard.
Related: RHEL-60136
This updates composer to use the updated API in images around the
seed handling for manifests, see images PR#1107 for details.
Note that this has no semantic changes yet. We could now simplfy
some things because images will auto-seed but that is for a followup.
Fixes the special case that if no worker is available and we
generate an internal timeout and cancel the depsolve including all
followup jobs, no error was propagated.
When requeuing a job the next worker requesting the job would decrement
pending counter, but the pending counter only ever got incremented once,
when the job was first enqueued. Thus make sure to increment the pending
counter when a job is requeued.
For Koji composes, all files are uploaded to Koji as part of the osbuild
job (specifically as part of handling the Koji target). So in order to
be able to upload SBOM documents to Koji as part of Koji compose, the
osbuild job needs to to be able to access the depsolve job result, which
contains the SBOM documents. For this, the osbuild job must depend on
the depsolve job.
For Koji composes, make sure that osbuild job depends on the depsolve
job and set the DepsolveDynArgsIdx.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When the osbuild worker cannot register itself with the server
on startup the worker will "crash". This is inconsistent with the
existing behavior in `workerHeartbeat()` which deals with connectivity
or other server issue gracefully and retries periodically.
To unify the behavior this commit changes the behavior and only
issues a `logrus.Warnf` instead of the previous `Falalf` when
the registration fails.
Co-authored-by: Florian Schüller <florian.schueller@redhat.com>
I find it slightly eaiser to read this code when
`api.BasePath = conf.BasePath` is right at the top as it's
unrelated to the parsing code below.
Note that the code itself is problematic:
- api.BasePath is global but client is not, this means that
multiple client with different configs will result in
api.BasePath being potentially wrong
- api.BasePath is set in a non-thread safe manner
Changing is a bigger job but we might consider it (IMHO).
The current name `Client.server` feels a bit misleading as it is
unclear if this is an abstraction for a "server" object or an
URL. This rename makes this unambiguous.
The usual convention to create new object is to prefix `New*` so
this commit renames the `WorkerClientError`. Initially I thought
it would be `NewWorkerClientError()` but looking at the package
prefix it seems unneeded, i.e. `clienterrors.New()` already
provides enough context it seems and it's the only error we
construct.
We could consider renaming it to `clienterror` (singular) too
but that could be a followup.
I would also like to make `clienterror.Error` implement the
`error` interface but that should be a followup to make this
(mechanical) rename trivial to review.
Drop `common.CurrentArch()` implementation and use
`arch.Current().String()` from the osbuild/images instead.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update the osbuild/images to the version which introduces "dot notation"
for distro release versions.
- Replace all uses of distroregistry by distrofactory.
- Delete local version of reporegistry and use the one from the
osbuild/images.
- Weldr: unify `createWeldrAPI()` and `createWeldrAPI2()` into a single
`createTestWeldrAPI()` function`.
- store/fixture: rework fixtures to allow overriding the host distro
name and host architecture name. A cleanup function to restore the
host distro and arch names is always part of the fixture struct.
- Delete `distro_mock` package, since it is no longer used.
- Bump the required version of osbuild to 98, because the OSCAP
customization is using the 'compress_results' stage option, which is
not available in older versions of osbuild.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Sends a status update to the worker server every 5 minutes.
Also fixes a bug where the body the worker client sent would be empty if
it had to refresh the JWT token. Instead of io.Reader use io.ReadSeeker
so the body can be reread to create the second request (after the token
refresh).
Unresponsive workers (>=1 hour of no status update) are cleaned up.
Several things are enabled by keeping track of workers, in future the
worker server could:
- keep track of how many workers are active
- see if a worker for a specific architecture is available
This will make it easier to reproduce image builds using the same
tooling which was used by the worker.
This change will enable to include the osbuild version in Koji build
metadata.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Include the osbuild/images module version in the Manifest job result.
The module has direct impact on image definitions and the content of
produced manifest, therefore including this information in the Manifest
job result is very helpful for various purposes (debugging,
traceability).
This will enable to embed this information in the Koji build metadata.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
