It turned out that the upload target was never adopted by the service,
thus we are removing it as part of upload code consolidation.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Delete the `internal/upload/koji` package and replace it with
`pkg/upload/koji` package provided by `osbuild/images`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The upload/koji package functions were creating a logger and then were
using it. This is not ideal for a library implementation.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the Koji target handling in the osbuild job, to also upload SBOM
documents attached to the related depsolve job result.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the worker was determining the GCE image guest OS Features
on its own, based on the OS name. This caused problems, in case the
osbuild-composer was of a newer version than the worker.
Example:
osbuild-composer contained support for c10s GCE image type and its
implementation also contained the proper guest OS Features list for it.
However, when the worker got the osbuild job, it built it and tried to
fetch the guest OS Features for the distro. Since its implementation was
too old, it didn't contain the code that added the actual support for
c10s GCE images and got no guest OS features list (which is the default
for unsupported distros). The image was successfully uploaded and
shared, but it does not boot in GCP, because it does not know that it
should use UEFI to boot it.
This behavior could be considered a bug. The worker should be dumb. It
should not be making decisions about the image features, but instead it
should take them from the upload target options. And composer should be
the authoritative source of truth for this. Because otherwise, we
basically have two components that need to be updated in sync to add
support for GCE images on a new distro.
Move the GCE image guest OS features to the GCP upload target options.
The worker will just take what is specified there and use it when
importing the image to GCP. As a compatibility layer for the case when
the composer would be older than the worker (unlikely, but still),
worker will try to determine the image guest OS features in case the
list in the upload target options is empty.
Extend the GCP functional tests to check that the imported image has at
least some guest OS features set.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The usual convention to create new object is to prefix `New*` so
this commit renames the `WorkerClientError`. Initially I thought
it would be `NewWorkerClientError()` but looking at the package
prefix it seems unneeded, i.e. `clienterrors.New()` already
provides enough context it seems and it's the only error we
construct.
We could consider renaming it to `clienterror` (singular) too
but that could be a followup.
I would also like to make `clienterror.Error` implement the
`error` interface but that should be a followup to make this
(mechanical) rename trivial to review.
This is an alternative/complementary fix for PR#4137. It is very
simple so should be uncontroverisal.
It fixes an issue that @schuellerf discovered, i.e. that when an error
interface is passed into clienterrors.Error.Details the details get
lost because the json.Marshaler will not know how to handler an
error interface.
To find the problematic uses of `error` a custom vet checker was
build in https://github.com/mvo5/osbuild-cvet. With that the
result is:
```
$ go run github.com/mvo5/osbuild-cvet@latest ./...
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-depsolve.go:93:26: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:404:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:519:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:556:31: do not pass '[]error' to WorkerClientError() details, use []string instead
```
and once this commit is in no more errors.
Just like PR#4137 this is not perfect because it will not do a
recursive check for the passed argument.
We need the ability to use different CloudWatch group for the
osbuild-executor on Fedora workers in staging and production
environment.
Extend the worker confguration to allow configuring the CloudWatch group
name used by the osbuild-executor. Extend the secure instance code to
instruct cloud-init via user data to create /tmp/cloud_init_vars file
with the CloudWatch group name in the osbuild-executor instance, to make
it possible for the executor to configure its logging differently based
on the value.
Cover new changes by unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add any errors to job error details when an osbuild build fails.
Otherwise these won't show up in the worker log, which makes
debugging issues harder.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Drop `common.CurrentArch()` implementation and use
`arch.Current().String()` from the osbuild/images instead.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Drop `common.GetHostDistroName()` implementation and use
`distro.GetHostDistroName()` from the osbuild/images instead.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Indent the osbuild manifest before uploading it to Koji. This will make
it much nicer for reading by humans.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This will make it easier to reproduce image builds using the same
tooling which was used by the worker.
This change will enable to include the osbuild version in Koji build
metadata.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Copy the Manifest info data from the Manifest job result to the Koji
target result, so that this information can be then imported to Koji
build metadata by the koji-finalize job.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add the information about osbuid artifact to the target result.
Specifically the name of the osbuild pipeline which was exported for the
specific target, and the filename of the exported file.
This will later enable embedding this information in Koji build metadata
to make it easy to reproduce the image build using the attached
manifest.
Import osbuild manifest and build log to the Koji build as outputs. Also
note the respective filenames in the image output extra metadata.
Note that the osbuild manifest is imported as a log file for now. Koji
has very limited set of output types defined and I still need to
determine the best way to use a custom output type in Koji instances (as
other content generators do).
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the Koji target handling in the OSBuild job implementation and
upload also the osbuild manifest and osbuild output log to Koji. Add all
the necessary metadata to the Koji target result options, so that
KojiFinalize job can then import them to the build.
Note that none of these files is yet imported by the KojiFinalize job.
Also note that the osbuild output log is still a JSON, which is not
great to read by humans. Adjustments to make it nice will be done in
following commits.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `KojiTargetResultOptions` previously contained information only
about the uploaded image file. And even then, some information, such as
the filename, were scattered in other structures such as
`KojiFinalizeJob` struct.
Since the plan is to start uploading also osbuild manifest and osbuild
build log to Koji, we need to extend the result options structure to
hold more information and also make it specific to which file is the
information related.
Rework the `KojiTargetResultOptions` to contain information about:
- the built image
- build log
- osbuild manifest
Information about each file contains:
- filename
- checksum type
- file checksum
- file size
For now, only the built image information is set and consumed by the
worker.
Add custom JSON (un)marshaler for `KojiTargetResultOptions` to handle
backward compatibility when old version of worker or composer server
interact with each other. Cover them with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Uploads an OCI image to OCI object storage, and generates a
pre-authenticated request for the object, which can be used to import it
into custom images.
Previously, the worker would happily report success if osbuild succeeded,
there was no JobError, but the job actually panicked in the meantime.
Let's fix this by adding a recovery mechanism.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
