The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
Return manifest.Manifest from the Manifest() function without
serializing. The caller then has to call the manifest.Serialize()
function using the depsolved packages.
This moves towards changing the order of actions required to generate a
manifest. With this change, the manifest creation and depsolving can be
done independently, but this still requires instantiating the manifest
object twice (InstantiateManifest() is called in PackageSets() and
Manifest()), which we don't want to have to do.
Expose the Directory and File customizations in the Cloud API. Also
validate the provided customizations while processing the compose
request by trying to convert them to internal representations
`fsnode.File` and `fsnode.Directory`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Support for creating multiple amis from a single compose. It uses the
AWSEC2* jobs to push images to new regions, and share them with new
accounts.
The compose it depends upon has to have succeeded.
Dependency errors are not set by the workers, they're not set directly
in the job result. They are added by the worker server in case the job
error indicates it's a dependency error.
The ellipsis operator was used as a hack to not need to pass any details
as an argument, but it makes what the end object will actually look like
less obvious. It also makes it impossible to pass an array to details
without getting a nested array.
Fixes#2874
This test tested two things:
1) Invalid route - this is already covered by TestUnknownRoute
2) Invalid UUID in the compose status route - this is now covered by
TestComposeStatusInvalidUUID
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The test_distro Manifest, which is used in tests across multiple
packages, was using the old structure. Updated to the v2 structure and
adapted all tests.
The `TargetErrors` is not used any more since PR#2192 [1] and there is
no need to keep the backward compatibility any more, because there are
no composer / worker instances in production, which are not running the
modified code.
In addition, delete unit tests covering this legacy error handling.
[1] https://github.com/osbuild/osbuild-composer/pull/2192
Error handling is structured in such a way that typically, a ServiceCodeError is passed
through the echo HTTP error, in reference to internally defined errors. We want to be able
to obtain and return specific external errors, for example during validation from openapi3.
Add a 'details' field to the serviceError struct, to contain extra / externally defined
information. Modify HTTPErrorHandler to anticipate either a string or a ServiceErrorCode
from echo, and respond accordingly. Edit the affected tests to expect the appropriate response.
Support the composes/<id>/manifests API endpoint for non-koji builds.
The endpoint will have to anyway handle `osbuild` job results once Koji
composes will start using `osbuild` job type for builds.
The endpoint previously contained a bug. If the `osbuild-koji` job had
an empty manifest attached as a static job argument (this is the default
type value), then this empty manifest was added to the endpoint
response. Since Cloud API uses the depsolve and manifest jobs, the
actual manifest was never attached to the job as a static argument. As a
result, the endpoint was always returning an empty manifest for any koji
compose. Fixing this required also adjusting unit tests, which was
relying on the buggy behavior.
Extend the unit test testing a successful compose to test the logs
endpoint.
Support the composes/<id>/logs API endpoint for non-koji builds. The
endpoint will have to anyway handle `osbuild` job results once Koji
composes will start using `osbuild` job type for builds.
Extend the unit test testing a successful compose to test the logs
endpoint.
Define supported job type names as constants and use them in all places,
instead of string literals.
There are multiple benefits of this approach. Using constants removed
the room for typos in the string literals. One can use autocompletion in
IDE for job types. Using constant makes it easier to find all references
where it is used and thus all places that are handling a specific job
type.
This value is set in the worker config. In future it might also be
passed through the api to upload into target accounts, but it should
never be set in composer.
Validate incoming requests with openapi3. Remove unsupported
uuid format from the openapi spec. Similarly, change url to uri as
uri is a supported format and url is not.
Co-authored-by: Ondřej Budai <obudai@redhat.com>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add depsolve job error dependency test cases for
regular composes and koji composes. The error furthest
up the chain should be returned in the details field
of the job error.
When composer exits, it doesn't wait for the manifest generation goroutines
to finish. This is generally a bad practice so let's introduce a bit of
syncing and a new Shutdown method to prevent this.
This also prevents the manifest generation goroutine from creating weird
states when interrupted on a random line of code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This allows us to use t.TempDir() everywhere. This was not possible before
because t.TempDir() fails the test if something was written in the temp dir
after the test case is finished. So by waiting on the depsolving goroutine
to finish, we can be sure that nothing is written in the tempdir anymore.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the goroutine was never stopped because it was all the time
calling RequestJob that returned a TimeoutError.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add an error object to the ComposeStatus.ImageStatus.
The error object contains a human-readable error reason
and optional details in the case of an error.
This commit adds a very in-depth test for multi-tenancy. It queues several
composes and then runs all jobs belonging to them while checking that
they are run by the correct tenant.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit implements multi-tenancy. A tenant is defined based on a value
from JWT claims. The key of this value must be specified in the configuration
file. This allows us to pick different values when using multiple SSOs.
Let me explain more in depth how this works:
Cloud API gets a new compose request. Firstly, it extracts a tenant name from
JWT claims. The considered claims are configured as an array in
cloud_api.jwt.tenant_provider_fields in composer's config file. The channel
name for all jobs belonging to this compose is created by `"org-" + tenant`.
Why is the channel prefixed by "org-"? To give us options in the future. I can
imagine the request having a channel override. This basically means that
multiple tenants can share a channel. A real use-case for this is multiple
Fedora projects sharing one pool of workers.
Why this commit adds a whole new cloud_api section to the config? Because the
current config is a mess and we should stop adding new stuff into the koji
section. As the Koji API is basically deprecated, we will need to remove it
soon nevertheless.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Channels are a concept similar to job types. Callers must specify a channel
name when queueing a new job. A list of channels is also specified when
dequeueing a job. The dequeued job's channel will always be from one of the
specified channel. Of course, the job types are also respected. The dequeued
job will also always be from one of the specified type.
Currently, all calls to jobqueue were changed so all queue operations use
an empty channel name and all dequeue operations use a list containing
an empty channel.
Thus, this is a non-functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
rpmmd_mock fixture are complex and unneeded in the context of cloudapi, let's
just copy 3 lines from them and drop the dependency.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
In addition to individual image status, have an
overall status that captures success or failure
of the compose as a whole.
This is not as fine grained, and only distinguishes
between "pending", "failure" and "success".
This captures other jobs than the image builds, which
is relevant for the koji composes, which consists also
of koji-init and koji-finalize, in addition to the build
jobs.
For now upload requests are required if and only if we are not
using koji. When using the koji integration the produced artifacts
are uploaded to koji only. In the future we may want to support
also uploading to the cloud providers.
Since the workers will use structured error messages
going forward, it is necessary to maintain backwards
compatability for there errors in composer. Tests have
been added to the various apis to ensure that each api
checks for both kinds of errors, old and new.
