The osbuild export is specific to the upload target and different
targets may require using a different export. While osbuild-composer
still does not support multiple exports for osbuild jobs, this prepares
the ground for such support in the future.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
message is always a super-set of the old and new way of providing the
exports to osbuild job.
The `TestKojiJobTypeValidation` unit test used the `OSBuildJob` and
Koji Target fields in the wrong way, however without any impact on the
testing itself. The reason is that no actual images were built as part
of the test nor the created jobs were ever picked up by a worker.
This change was forgotten in PR#2758.
[1] https://github.com/osbuild/osbuild-composer/pull/2758
The filename of the image as produced by osbuild for a given export is
currently set in each target options type in the `Filename` struct
member. However, the value is not really specific to any target type,
but to the specific export used for the target. For this reason move the
value form target type options to the `Target` struct inside a new
struct `OsbuildArtifact` under the name`ExportFilename`.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
object is always a super-set of the old and new way of providing the
export filename in the Target.
The `TargetErrors` is not used any more since PR#2192 [1] and there is
no need to keep the backward compatibility any more, because there are
no composer / worker instances in production, which are not running the
modified code.
In addition, delete unit tests covering this legacy error handling.
[1] https://github.com/osbuild/osbuild-composer/pull/2192
Error handling is structured in such a way that typically, a ServiceCodeError is passed
through the echo HTTP error, in reference to internally defined errors. We want to be able
to obtain and return specific external errors, for example during validation from openapi3.
Add a 'details' field to the serviceError struct, to contain extra / externally defined
information. Modify HTTPErrorHandler to anticipate either a string or a ServiceErrorCode
from echo, and respond accordingly. Edit the affected tests to expect the appropriate response.
When the Koji target support was added to the osbuild job, based on the
osbuild-koji job, the meaning of target option values got messed up.
The side effect of the issue is that when Koji composes are
submitted via Cloud API the resulting image is currently always uploaded
back to the worker server.
`OsBuildKoji` job
-----------------
- `OSBuildKojiJob.ImageName` is set to the filename of the image as
exported by osbuild.
- `OSBuildKojiJob.KojiFilename` is set to the desired filename which
should be used when uploading the image to Koji.
`OsBuild` job + `KojiTargetOptions` before
------------------------------------------
- `OSBuildJob.ImageName` is set to the filename of the image as exported
by osbuild. This is done only by the Cloud API code for Koji composes.
Cloud API does not set this for regular composes and any other target.
The variable is set in common case only by Weldr API code with the
same meaning and it is used by the `OsBuild` job implementation as an
indication that the image should be uploaded back to the worker server.
- `Target.ImageName` is not set at all. Other targets use it for the
desired filename which should be used when uploading the image to the
target environment.
- `KojiTargetOptions.Filename` is set to the desired filename which
should be used when uploading the image to Koji. All other target
types use `Filename` variable in their options for the filename of the
image as exported by osbuild.
`OsBuild` job + `KojiTargetOptions` after
-----------------------------------------
- `OSBuildJob.ImageName` is still set to the filename of the image as
exported by osbuild. This is kept for a backward compatibility of new
composer with older workers.
- `Target.ImageName` is set to the desired filename which should be used
when uploading the image to Koji.
- `KojiTargetOptions.Filename` is set to the filename of the image as
exported by osbuild.
This change is backward incompatible, meaning that old worker won't be
able to handle Koji compose requests submitted via Cloud API using a new
composer and also a new worker won't be able to handle Koji compose
requests submitted by a new composer. This is intentional, because after
discussion with Ondrej Budai, the Cloud API Koji integration is
currently not used anywhere in production.
Use the `EnsureJobChannel()` middleware in all `compose/<id>` endpoints.
Specifically in the:
- status
- metadata
- manifests
- logs
As a result, these endpoints now return `404` in case the server has JWT
enabled and the channel associated with the request does not match the
channel associated with the requested compose (job).
Extend the multi-tenancy unit test to ensure that these endpoints behave
as expected in case of match and mismatch between the request and
compose channels.
Add `EnsureJobChannel()` middleware method, intended for `compose/<id>`
endpoints. Its purpose is to ensure that the tenant channel set in
the request `echo.Context` matches the tenant channel associated with
the compose. In case of mismatch, `404` is returned.
Add `JobChannel()` method to the worker server implementation for
requesting channel associated with the job.
Extract the determination of tenant channel into a helper function.
This will simplify handler and middleware methods, which won't have
to implement the same logic by themselves.
Fix the multi-tenancy unit test to pass the appropriate context when
querying compose statuses, because the server that is being use has JWT
enabled and expects the tenant to be set in it.
Switch to using `osbuild` job type with `koji` upload target for Koji
build jobs, instead of using `osbuild-koji` job type.
Modify unit tests accordingly.
Previously, only a subset from all Koji Compose unit test cases were
run. Remove this limitation and run all defined unit tests, which were
copied from `kojiapi`.
In addition, fix unit tests and relevant cloudapi methods to make unit
tests pass.
Add `TestRouteWithReply()` to `test/helpers.go` to allow getting the
compose ID when submitting a new compose. This is needed to make some
unit tests deterministic.
Do not delete values from `fields` slice in `dropFields()` in
`test/helpers.go`. The behavior was previously not consistent.
If the top-level map contained the value, it was deleted from it, but
the nested maps also contained the value, it was not deleted. On the
other hand, if the top level map didn't contain the value, but nested
maps did contain it, the value was deleted from all nested maps.
Support the composes/<id>/manifests API endpoint for non-koji builds.
The endpoint will have to anyway handle `osbuild` job results once Koji
composes will start using `osbuild` job type for builds.
The endpoint previously contained a bug. If the `osbuild-koji` job had
an empty manifest attached as a static job argument (this is the default
type value), then this empty manifest was added to the endpoint
response. Since Cloud API uses the depsolve and manifest jobs, the
actual manifest was never attached to the job as a static argument. As a
result, the endpoint was always returning an empty manifest for any koji
compose. Fixing this required also adjusting unit tests, which was
relying on the buggy behavior.
Extend the unit test testing a successful compose to test the logs
endpoint.
Support the composes/<id>/logs API endpoint for non-koji builds. The
endpoint will have to anyway handle `osbuild` job results once Koji
composes will start using `osbuild` job type for builds.
Extend the unit test testing a successful compose to test the logs
endpoint.
Define supported job type names as constants and use them in all places,
instead of string literals.
There are multiple benefits of this approach. Using constants removed
the room for typos in the string literals. One can use autocompletion in
IDE for job types. Using constant makes it easier to find all references
where it is used and thus all places that are handling a specific job
type.
Change the definition of `EnqueueOSBuildAsDependency()` function to
accept a slice of job IDs on which the OSBuild job depends. Previously,
only the manifest job ID was accepted as the only possible dependency.
This change will be needed in order to enqueue OSBuild jobs for Koji,
which depends on two jobs.
Move package set chain collation to the distro package and add
repositories to the package sets while returning the package sets from
their source, i.e., the ImageType.PackageSets() method.
This also removes the concept of "base repositories". There are no
longer repositories that are added implicitly to all package sets but
instead each package set needs to specify *all* the repositories it will
be depsolved against.
This paves the way for the requirement we have for building RHEL 7
images with a RHEL 8 build root. The build root package set has to be
depsolved against RHEL 8 repositories without any "base repos" included.
This is now possible since package sets and repositories are explicitly
associated from the start and there is no implicit global repository
set.
The change requires adding a list of PackageSet names to the core
rpmmd.RepoConfig. In the cloud API, repositories that are limited to
specific package sets already contain the correct package set names and
these are now copied to the internal RepoConfig when converting types in
genRepoConfig().
The user-specified repositories are only associated with the payload
package sets like before.
This value is set in the worker config. In future it might also be
passed through the api to upload into target accounts, but it should
never be set in composer.
Validate incoming requests with openapi3. Remove unsupported
uuid format from the openapi spec. Similarly, change url to uri as
uri is a supported format and url is not.
Co-authored-by: Ondřej Budai <obudai@redhat.com>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
oneOf means that the body is valid against exactly ONE schema. There's an
issue with AWS EC2 upload options though: It requires region and
share_with_accounts fields. Such a request is also valid AWS S3 upload though
(this one only require region). This means that AWS EC2 upload options will be
always valid against two schemas which violates the oneOf rule.
Let's switch to anyOf and explain this in the openAPI spec.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
It was never required, never used. I honestly think that this was a copy-paste
error, I don't see any reason why a user would have an object reference.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I think that we can spare the users of clouadpi of writing "rhsm": "false"
into the requests so I decided to make this property optional and default
to false.
This is nice because it matches the behaviour of Weldr repositories and
sources so we can also use test/data/repositories without any changes after
openapi validation is enabled.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The VMDK image is already produced as stream-optimized. Therefore stop
setting the `StreamOptimized` option in `OSBuildJob` structure by both,
Weldr and Cloud APIs.
Keep the handling of the option in worker for backward compatibility,
in case an older instance of Composer server is used, which does not
produce VMDK manifests as stream-optimized. In such case, the worker
needs to convert the image.
The generated gcp name had an invalid `.tar.gz` extension. This
extension still needs to be supplied for the object name however.
The integration tests supply the image name rather than relying in the
generated one, which is why this slipped through.
Add depsolve job error dependency test cases for
regular composes and koji composes. The error furthest
up the chain should be returned in the details field
of the job error.
If an osbuild or koji-osbuild job has failed, add
a check to see if it is a result of the build jobs
dependencies and return the dependency failure job
error furthest up the chain of errors & add this
error to the details filed of the build job error.
When composer exits, it doesn't wait for the manifest generation goroutines
to finish. This is generally a bad practice so let's introduce a bit of
syncing and a new Shutdown method to prevent this.
This also prevents the manifest generation goroutine from creating weird
states when interrupted on a random line of code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I think it's a natural fit, they access some fields from the server so let's
find a new home for them there.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This allows us to use t.TempDir() everywhere. This was not possible before
because t.TempDir() fails the test if something was written in the temp dir
after the test case is finished. So by waiting on the depsolving goroutine
to finish, we can be sure that nothing is written in the tempdir anymore.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the goroutine was never stopped because it was all the time
calling RequestJob that returned a TimeoutError.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The VMDK image must be in stream-optimized format in order to be
imported to VSphere. osbuild-composer does not produce VMDK by default
as stream-optimized. Instead, it is converted on the fly when the image
build job has been submitted via Weldr API.
Since we are aiming mainly for the VSphere use case with the VMDK image
in the service, the image should be ready for importing to VSphere.
Implement a temporary workaround for the Cloud API and AWS S3 target to
upload stream-optimized VMDK image.
Adjust the `api.sh` test case to not convert the VMDK image downloaded
form S3, before importing it to VSphere.
If a user requires that packages from a certain repository are checked using
a GPG key, they should specify it. Now, this is enforced to catch this issue
earlier than in osbuild.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
