When the osbuild worker cannot register itself with the server
on startup the worker will "crash". This is inconsistent with the
existing behavior in `workerHeartbeat()` which deals with connectivity
or other server issue gracefully and retries periodically.
To unify the behavior this commit changes the behavior and only
issues a `logrus.Warnf` instead of the previous `Falalf` when
the registration fails.
Co-authored-by: Florian Schüller <florian.schueller@redhat.com>
I find it slightly eaiser to read this code when
`api.BasePath = conf.BasePath` is right at the top as it's
unrelated to the parsing code below.
Note that the code itself is problematic:
- api.BasePath is global but client is not, this means that
multiple client with different configs will result in
api.BasePath being potentially wrong
- api.BasePath is set in a non-thread safe manner
Changing is a bigger job but we might consider it (IMHO).
The current name `Client.server` feels a bit misleading as it is
unclear if this is an abstraction for a "server" object or an
URL. This rename makes this unambiguous.
The usual convention to create new object is to prefix `New*` so
this commit renames the `WorkerClientError`. Initially I thought
it would be `NewWorkerClientError()` but looking at the package
prefix it seems unneeded, i.e. `clienterrors.New()` already
provides enough context it seems and it's the only error we
construct.
We could consider renaming it to `clienterror` (singular) too
but that could be a followup.
I would also like to make `clienterror.Error` implement the
`error` interface but that should be a followup to make this
(mechanical) rename trivial to review.
Drop `common.CurrentArch()` implementation and use
`arch.Current().String()` from the osbuild/images instead.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update the osbuild/images to the version which introduces "dot notation"
for distro release versions.
- Replace all uses of distroregistry by distrofactory.
- Delete local version of reporegistry and use the one from the
osbuild/images.
- Weldr: unify `createWeldrAPI()` and `createWeldrAPI2()` into a single
`createTestWeldrAPI()` function`.
- store/fixture: rework fixtures to allow overriding the host distro
name and host architecture name. A cleanup function to restore the
host distro and arch names is always part of the fixture struct.
- Delete `distro_mock` package, since it is no longer used.
- Bump the required version of osbuild to 98, because the OSCAP
customization is using the 'compress_results' stage option, which is
not available in older versions of osbuild.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Sends a status update to the worker server every 5 minutes.
Also fixes a bug where the body the worker client sent would be empty if
it had to refresh the JWT token. Instead of io.Reader use io.ReadSeeker
so the body can be reread to create the second request (after the token
refresh).
Unresponsive workers (>=1 hour of no status update) are cleaned up.
Several things are enabled by keeping track of workers, in future the
worker server could:
- keep track of how many workers are active
- see if a worker for a specific architecture is available
This will make it easier to reproduce image builds using the same
tooling which was used by the worker.
This change will enable to include the osbuild version in Koji build
metadata.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Include the osbuild/images module version in the Manifest job result.
The module has direct impact on image definitions and the content of
produced manifest, therefore including this information in the Manifest
job result is very helpful for various purposes (debugging,
traceability).
This will enable to embed this information in the Koji build metadata.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the Manifest job result structure to hold information about
osbuild-composer version, which produced the manifest. This will be
useful for other job types which depend on it and can then push this
information further as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add any non-Koji upload target results attached to an OSBuild result, to
the image extra metadata. This will make it easy to locate any image
from Koji uploaded to cloud, in the target cloud environment.
The rationale behind including only non-Koji target results is that one
can find it only in Koji, so there is no added value in including the
Koji target results at all.
Extend the `koji.sh` to check the target results in image metadata when
testing Koji scenario with cloud upload.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `KojiTargetResultOptions` previously contained information only
about the uploaded image file. And even then, some information, such as
the filename, were scattered in other structures such as
`KojiFinalizeJob` struct.
Since the plan is to start uploading also osbuild manifest and osbuild
build log to Koji, we need to extend the result options structure to
hold more information and also make it specific to which file is the
information related.
Rework the `KojiTargetResultOptions` to contain information about:
- the built image
- build log
- osbuild manifest
Information about each file contains:
- filename
- checksum type
- file checksum
- file size
For now, only the built image information is set and consumed by the
worker.
Add custom JSON (un)marshaler for `KojiTargetResultOptions` to handle
backward compatibility when old version of worker or composer server
interact with each other. Cover them with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Uploads an OCI image to OCI object storage, and generates a
pre-authenticated request for the object, which can be used to import it
into custom images.
Previously, the worker would happily report success if osbuild succeeded,
there was no JobError, but the job actually panicked in the meantime.
Let's fix this by adding a recovery mechanism.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add the information about the image boot mode to the OSBuildJob options
as well as to the OSBuildJobResult options. The intention is that the
worker will simply copy the value from the job options to job result,
so that the `KojiFinalize` job can then access this information and
upload it to Koji as extra metadata.
This information is required in Koji in order for Red Hat's SP tooling
to know how to import image to the cloud environment in case the boot
mode affects the import parameters.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
The duration middleware should come after the tenant channel middleware,
otherwise the tenant in the context will be empty. The status middleware
can come beforehand because it queries the request context right before
sending a response.
Change the OSTreeResolveSpec to match the ostree SourceSpec by removing
the Parent field.
Change OSTreeResolveResultSpec to match the CommitSpec by adding the
Secrets field. The RHSM field is kept for backwards compatibility with
older workers.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
Return manifest.Manifest from the Manifest() function without
serializing. The caller then has to call the manifest.Serialize()
function using the depsolved packages.
This moves towards changing the order of actions required to generate a
manifest. With this change, the manifest creation and depsolving can be
done independently, but this still requires instantiating the manifest
object twice (InstantiateManifest() is called in PackageSets() and
Manifest()), which we don't want to have to do.
Removing the dependence of the manifest package on the distro package to
import manifest into distro.
Wherever arch names are needed, we use the enums from the platform
package instead.
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].
Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.
[1] See dnf.config
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.
Copy the value whenever a spec is moved to a different representation.
This changes the `Manifest` function of the `ImageType`
interface so that any warnings detected during the
`checkOptions` step of the manifest initialization can
be propagated back to the Weldr-API (see next commit).
Signed-off-by: Irene Diez <idiez@redhat.com>
