This will allow us to use the service accounts which work against
identity.api.openshift.com. These are much easier to manage, especially
with the new multi-tenancy, as there's a single page to create/expire
them across an account.
They also have the added benefit of not expiring automatically when
they're not used like offline tokens, and immediate expiration when
desired.
This was causing:
Feb 21 06:08:06 ip-10-30-18-43.us-east-1.aws.redhat.com osbuild-worker[20798]: time="2022-02-21T06:08:06Z" level=warning msg="Error reporting job result: error setting job status: 404 — Token not found (IMAGE-BUILDER-WORKER-5)"
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Defaults according to https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config:
Defaults to a chain of credential providers to search for credentials in
environment variables, shared credential file, and EC2 Instance Roles.
If nothing is specified fall back to whatever instance role.
Move `OSBuildMetadataToRPMs()` and `PackageMetadataToSignature()`
functions from the `rpmmd` package to `osbuild2` package to prevent
import cycles while de-duplicating `rpmStageInputs()` function from
`stage_inputs.go` of distro definitions.
Rename `PackageMetadataToSignature()` to
`RPMPackageMetadataToSignature()`, since it takes specifically
`RPMPackageMetadata` type as an argument.
Adjust affected parts of code (unit tests, cloudapi, worker).
Signed-off-by: Tomas Hozza <thozza@redhat.com>
We have two fields, `Repos` and `PackageSets`. Renaming
`PackageSetsRepositories` to `PackageSetsRepos` for consistency.
The struct is for internal use only so the rename has no impact as long
as the serialised name is the same (json tag).
Also it's shorter.
Added docstring to the struct that explains the arguments in the same
way as they are described for the `depsolve()` function.
Changing the name of the argument in the internal `depsolve()` function
for the same reasons.
Change order of arguments for depsolve function:
Put the two similar arguments (repos and packageSetsRepositories) next
to each other since they serve similar purposes.
Add docstring for depsolve function:
It is useful to clarify how the arguments are used even if it's an
unexported function.
Validate the results for a koji build job
and check for empty osbuild output. This commit
ensures that there aren't any gaps in the error
handling. Additionally this commit adds a few
extra error cases rather than a generic error.
Before accessing a field of the `OSBuildOutput`, which itself is a
field of the `osbuildKojiResults` struct, check if it is actually
is set (non-nill), otherwise dereferencing it will crash the
worker.
The field will be null if osbuild has not been invoked at all or
if osbuild crashed or refused to accept the input.
The manifest is of type distro.Manifest, which is an alias for a
byte array, i.e. it is already in marshalled form. There is no
need to marshal it again before passing it to osbuild.
Extend the compose endpoints to have minimal koji support.
This is intended to replace the current koji API so that it
can be consumed through api.openshift.com.
This check got inverted during the work on "Worker errors backwards
compatibility". As a consequence, osbuild was never run and the
result structure `buildResult.OSBuildOutput` was `nil` Since the
overall status reporting is not complete, and does not take this,
i.e. `buildResult.OSBuildOutput`, being `nil` as an error case,
the overall status was reported as "success". See the function
`composeStatusFromJobStatus` in `internal/kojiapi/server.go`.
Implement the structured errors as defined by the worker client.
Every error for each of the job types now returns a structured
error with a reason and a specific error code. This will make
it possible to differentiate between 4xx errors and 5xx errors.
This commit refactors the way errors are implemented in the workers,
but maintains backwards compatability in composer by checking for
both kinds of errors.
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
The format is now always 'JOB_ID' (JOB_TYPE). This means that we also know
the job type when a job is finished or when it failed.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Running the job in this case is basically undefined, so let's just skip it
in order to not break anything.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Pipeline names are added to each job before adding to the queue. When a
job is finished, the names are copied to the Result object as well. This
is done for both OSBuild and Koji jobs.
The pipeline names in the result are primarily used to separate package
lists into build and payload/image packages in two cases:
1. Koji builds: for reporting the build root and image package lists to
Koji (in Koji finalize).
2. Cloud API (v1 and v2): for reporting the payload packages in the
metadata request.
The pipeline names are also used to print the system log output in the
order in which pipelines are executed. This still isn't used when
printing the OSBuild Result (osbuild2.Result.Write()) and we still rely
on sorting by pipeline name
(see https://github.com/osbuild/osbuild-composer/pull/1330).
- koji-finalize:
Use v2 result type to collect RPM metadata.
The separation between the "build" pipeline and the rest is based on the
pipeline name, which isn't completely reliable since pipeline names can
be arbitrary.
Koji will fail a build if it specifies duplicate packages, so the RPM
lists are deduplicated. The "build" pipeline package list is also
deduplicated in case there are multiple build stages in the same
pipeline.
- osbuild:
Use v2 result type for printing build result to log.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
This is backwards compatible, as long as the timeout is 0 (never
timeout), which is the default.
In case of the dbjobqueue the underlying timeout is due to
context.Canceled, context.DeadlineExceeded, or net.Error with Timeout()
true. For the fsjobqueue only the first two are considered.
Allow depsolving to be done in a worker through the job queue rather
than synchronously in composer.
The benefit this might unlock include:
- no more blocking calls in the cloud/koji APIs
- only workers accessing repositoires
- no VPN access from composer
- composer not needing to be subscribed to CDN, etc
- no dnf cache managment in composer
Potential problems:
- the version of composer (so the distro definitions) that
triggered a depsolve, may not be the same that uses the
result to generate a manfiset
Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2 configurations for the listeners are now possible:
- enableJWT=false with client ssl auth
- enableJWT=true with https
Actual verification of the tokens is handled by
https://github.com/openshift-online/ocm-sdk-go.
An authentication handler is run as the top level handler, before any
routing is done. Routes which do not require authentication should be
listed as exceptions.
Authentication can be restricted using an ACL file which allows
filtering based on JWT claims. For more information see the inline
comments in ocm-sdk/authentication.
As an added quirk the `-v` flag for the osbuild-composer executable was
changed to `-verbose` to avoid flag collision with glog which declares
the `-v` flag in the package `init()` function. The ocm-sdk depends on
glog and pulls it in.
To help along with debugging, this commit makes the worker able to print
the status of the different stages with a oneliner for each successfull
stages and a detailed message for failed ones.
Sample output:
Jul 23[..]: Build stages results:
Jul 23[..]: org.osbuild.rpm success
Jul 23[..]: org.osbuild.selinux success
Jul 23[..]: Stages results:
Jul 23[..]: org.osbuild.rpm success
Jul 23[..]: org.osbuild.fix-bls success
Jul 23[..]: org.osbuild.fstab success
Jul 23[..]: org.osbuild.grub2 success
Jul 23[..]: org.osbuild.locale success
Jul 23[..]: org.osbuild.timezone success
Jul 23[..]: org.osbuild.users failure:
Jul 23[..]: [/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
Jul 23[..]: All rules containing unresolvable specifiers will be skipped.
Jul 23[..]: Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
Jul 23[..]: useradd: group 'toto' does not exist
Fixes#1584
Move the OSBuildStagesToRPMs function, associated test, and RPM type
from the worker into the rpmmd subpackge. We will use this function in
the cloud API to compile the NEVRAs for the new metadata endpoint.
If a user uses a temporary access key for login, a session token is also
needed.
This commit adds support for it to the internal aws library and also
to the osbuild-upload-aws helper. Note that this doesn't affect the main
osbuild-composer executable nor the worker. Everything here should work
as before and session tokens are not supported. Something for a follow up
if anyone needs it.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Koji image request handling now reads the exports defined by each image
type. All APIs now support reading the exports defined by each image
type. The worker still falls back to "assembler" in case the call comes
from an older version of composer.
