Before accessing a field of the `OSBuildOutput`, which itself is a
field of the `osbuildKojiResults` struct, check if it is actually
is set (non-nill), otherwise dereferencing it will crash the
worker.
The field will be null if osbuild has not been invoked at all or
if osbuild crashed or refused to accept the input.
The manifest is of type distro.Manifest, which is an alias for a
byte array, i.e. it is already in marshalled form. There is no
need to marshal it again before passing it to osbuild.
Extend the compose endpoints to have minimal koji support.
This is intended to replace the current koji API so that it
can be consumed through api.openshift.com.
We may need to use several SSO providers, so extend our
configuration to allow that.
Based on PoC from Sanne:
```
package main
import (
"net/http"
"log"
"github.com/openshift-online/ocm-sdk-go/authentication"
"github.com/openshift-online/ocm-sdk-go/logging"
)
type H struct{}
func (h *H) ServeHTTP(w http.ResponseWriter, r *http.Request) {
log.Println("HURRAY")
}
func main() {
logBuilder := logging.NewGoLoggerBuilder()
logger, err := logBuilder.Build()
if err != nil {
panic(err)
}
aH, err := authentication.NewHandler().
KeysURL("https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/certs").
KeysURL("https://identity.api.openshift.com/auth/realms/rhoas/protocol/openid-connect/certs").
Logger(logger).Next(&H{}).Build()
if err != nil {
panic(err)
}
log.Fatal(http.ListenAndServe(":8080", aH))
}
```
This check got inverted during the work on "Worker errors backwards
compatibility". As a consequence, osbuild was never run and the
result structure `buildResult.OSBuildOutput` was `nil` Since the
overall status reporting is not complete, and does not take this,
i.e. `buildResult.OSBuildOutput`, being `nil` as an error case,
the overall status was reported as "success". See the function
`composeStatusFromJobStatus` in `internal/kojiapi/server.go`.
Implement the structured errors as defined by the worker client.
Every error for each of the job types now returns a structured
error with a reason and a specific error code. This will make
it possible to differentiate between 4xx errors and 5xx errors.
This commit refactors the way errors are implemented in the workers,
but maintains backwards compatability in composer by checking for
both kinds of errors.
Because of the way the gcp secrets are stored for the workers, and how
the mapping from vault to openshift works (unable to map a multiple key
secret into a single json file), there's a bit of juggling required to
get the gcp credentials in the right format.
These tests can be run only in our CI environment, and not as pure unit tests
launched by go test ./... Let's make sure that all files that belong to these
tests are properly marked with build constraints.
This should fix the following error in Coverity Scan runs:
# github.com/osbuild/osbuild-composer/cmd/osbuild-auth-tests
runtime.main_main·f: function main is undeclared in the main package
Warning: Build command go build -o bin/ ./... exited with code 2. Please verify that the build completed successfully.
See https://github.com/osbuild/osbuild-composer/runs/4427232580?check_suite_focus=true
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When backed by a DB, composer has no need of a queue directory.
This also addresses "Error moving artifacts for job" logging noise.
Signed-off-by: sanne <sanne.raymaekers@gmail.com>
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
The format is now always 'JOB_ID' (JOB_TYPE). This means that we also know
the job type when a job is finished or when it failed.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Generated image test case manifests for all supported distros, arches and
image-types are being tested as part of distro unit tests. However due
to time constrains, the unit test does not depsolve the image's default
package sets and thus does not check if they changed in the internal
osbuild-composer's representation, compared to the generated image test
case.
Extend the `TestDistro_Manifest()` function used by the unit test to
allow depsolving image's package sets.
Introduce a new test case binary `osbuild-composer-manifest-tests`
allowing to check the manifests generated by composer for all supported
combinations of images against generated manifests, including depsolving
image's default package sets.
Introduce a new CI test case `manifest_tests.sh` executing the
`osbuild-composer-manifest-tests` binary and testing all existing image
test cases. Run it in CI on RHEL-9 runner.
Modify SPEC file to ship the newly added test case.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Running the job in this case is basically undefined, so let's just skip it
in order to not break anything.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Mistakenly removed in 4577ac0717. Composer
itself does the authentication, not the gateway, therefore we do need
the auth exclude.
Added a comment to explain why it's attached to the api socket and not a
separate listener.
Pipeline names are added to each job before adding to the queue. When a
job is finished, the names are copied to the Result object as well. This
is done for both OSBuild and Koji jobs.
The pipeline names in the result are primarily used to separate package
lists into build and payload/image packages in two cases:
1. Koji builds: for reporting the build root and image package lists to
Koji (in Koji finalize).
2. Cloud API (v1 and v2): for reporting the payload packages in the
metadata request.
The pipeline names are also used to print the system log output in the
order in which pipelines are executed. This still isn't used when
printing the OSBuild Result (osbuild2.Result.Write()) and we still rely
on sorting by pipeline name
(see https://github.com/osbuild/osbuild-composer/pull/1330).
- koji-finalize:
Use v2 result type to collect RPM metadata.
The separation between the "build" pipeline and the rest is based on the
pipeline name, which isn't completely reliable since pipeline names can
be arbitrary.
Koji will fail a build if it specifies duplicate packages, so the RPM
lists are deduplicated. The "build" pipeline package list is also
deduplicated in case there are multiple build stages in the same
pipeline.
- osbuild:
Use v2 result type for printing build result to log.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
This is backwards compatible, as long as the timeout is 0 (never
timeout), which is the default.
In case of the dbjobqueue the underlying timeout is due to
context.Canceled, context.DeadlineExceeded, or net.Error with Timeout()
true. For the fsjobqueue only the first two are considered.
Allow depsolving to be done in a worker through the job queue rather
than synchronously in composer.
The benefit this might unlock include:
- no more blocking calls in the cloud/koji APIs
- only workers accessing repositoires
- no VPN access from composer
- composer not needing to be subscribed to CDN, etc
- no dnf cache managment in composer
Potential problems:
- the version of composer (so the distro definitions) that
triggered a depsolve, may not be the same that uses the
result to generate a manfiset
Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Because there's only a few combinations of upload types and image types
that make sense, enforce correct combinations by eliminating upload
types.
Fixes#1775
The image is not available via Weldr API, because it requires RHUI
client RPMs.
The content and configuration is based on RHEL-8.6 EC2 SAP image, since
there is no definition for the RHEL-9 SAP image yet.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The content of `/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf` is
dynamically created and its lines are not ordered deterministically.
This causes the rpm-ostree based image test cases to fail often on
Fedora, because the diff of the expected and actual image-info report
always produces some different lines.
Add a new option `-skip-tmpfilesd-path` to `osbuild-image-tests`
accepting a tmpfiles.d configuration path, which should be ignored when
comparing the expected and actual image-info report. The option can be
specified multiple times and all paths will be ignored.
Modify the `image_tests.sh` test case to use the new option and ignore the
`/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf` file when testing Fedora
images.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
