The generated gcp name had an invalid `.tar.gz` extension. This
extension still needs to be supplied for the object name however.
The integration tests supply the image name rather than relying in the
generated one, which is why this slipped through.
Add depsolve job error dependency test cases for
regular composes and koji composes. The error furthest
up the chain should be returned in the details field
of the job error.
If an osbuild or koji-osbuild job has failed, add
a check to see if it is a result of the build jobs
dependencies and return the dependency failure job
error furthest up the chain of errors & add this
error to the details filed of the build job error.
When composer exits, it doesn't wait for the manifest generation goroutines
to finish. This is generally a bad practice so let's introduce a bit of
syncing and a new Shutdown method to prevent this.
This also prevents the manifest generation goroutine from creating weird
states when interrupted on a random line of code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I think it's a natural fit, they access some fields from the server so let's
find a new home for them there.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This allows us to use t.TempDir() everywhere. This was not possible before
because t.TempDir() fails the test if something was written in the temp dir
after the test case is finished. So by waiting on the depsolving goroutine
to finish, we can be sure that nothing is written in the tempdir anymore.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the goroutine was never stopped because it was all the time
calling RequestJob that returned a TimeoutError.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The VMDK image must be in stream-optimized format in order to be
imported to VSphere. osbuild-composer does not produce VMDK by default
as stream-optimized. Instead, it is converted on the fly when the image
build job has been submitted via Weldr API.
Since we are aiming mainly for the VSphere use case with the VMDK image
in the service, the image should be ready for importing to VSphere.
Implement a temporary workaround for the Cloud API and AWS S3 target to
upload stream-optimized VMDK image.
Adjust the `api.sh` test case to not convert the VMDK image downloaded
form S3, before importing it to VSphere.
If a user requires that packages from a certain repository are checked using
a GPG key, they should specify it. Now, this is enforced to catch this issue
earlier than in osbuild.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Oh no, we made a mistake here: Both our json repositories and repo files in
/etc/yum.repos.d have the GPG key in a field named `gpgkey`. Unfortunately,
cloudapi uses a field named `gpg_key`. One consequence of this issue is that
our api.sh test is meant to pass GPG keys in the compose request but since
it's using a bad field name (`gpgkey`), the key is actually not used.
I've decided to fix this in cloudapi: The `gpg_key` field is now renamed to
`gpgkey`. This is a breaking change but no one is using this API anyway so
we think it's better to do this now than introducing weird backward
compatible hacks.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We have to do a small hack to enable edge-commit on Fedora because its name
is different. We can also change this in the image definition but I want to
iterate quickly on the Fedora Integration MVP and don't want to run in
any conflicts with
https://github.com/osbuild/osbuild-composer/pull/2461
This commit also enables a test for Fedora IoT built through the API.
While enabling the test, I also simplified our decision logic for SSH_USER
and DISTRO.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add an error object to the ComposeStatus.ImageStatus.
The error object contains a human-readable error reason
and optional details in the case of an error.
This commit adds a very in-depth test for multi-tenancy. It queues several
composes and then runs all jobs belonging to them while checking that
they are run by the correct tenant.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit implements multi-tenancy. A tenant is defined based on a value
from JWT claims. The key of this value must be specified in the configuration
file. This allows us to pick different values when using multiple SSOs.
Let me explain more in depth how this works:
Cloud API gets a new compose request. Firstly, it extracts a tenant name from
JWT claims. The considered claims are configured as an array in
cloud_api.jwt.tenant_provider_fields in composer's config file. The channel
name for all jobs belonging to this compose is created by `"org-" + tenant`.
Why is the channel prefixed by "org-"? To give us options in the future. I can
imagine the request having a channel override. This basically means that
multiple tenants can share a channel. A real use-case for this is multiple
Fedora projects sharing one pool of workers.
Why this commit adds a whole new cloud_api section to the config? Because the
current config is a mess and we should stop adding new stuff into the koji
section. As the Koji API is basically deprecated, we will need to remove it
soon nevertheless.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Channels are a concept similar to job types. Callers must specify a channel
name when queueing a new job. A list of channels is also specified when
dequeueing a job. The dequeued job's channel will always be from one of the
specified channel. Of course, the job types are also respected. The dequeued
job will also always be from one of the specified type.
Currently, all calls to jobqueue were changed so all queue operations use
an empty channel name and all dequeue operations use a list containing
an empty channel.
Thus, this is a non-functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
rpmmd_mock fixture are complex and unneeded in the context of cloudapi, let's
just copy 3 lines from them and drop the dependency.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add support for building images for the Azure marketplace: add a
new image type "azure-rhui" that can be used to build images
tailored to the Azure marketplace.
Add two sample manifests for 8.5 and 8.6, but note that even the
8.5 is using the 8.6 distro definitions. Also no image-info is
included since `image-info` cannot (yet) handle LVM setups and
the azure marketplace images use the LVM setup.
Customizations being nil isn't the same as an empty object. The
edge-installer types require customizations equalling nil.
This partially reverts bb486bcccb.
Shorter and clearer error types.
- InvalidParameterError is now RefError. It is returned when a ref (the
new ref or the parent ref) is invalid.
- The ResolveRefError is not renamed. It is returned when a ref can't
be resolved (e.g., ref not found at URL).
New error type ParameterComboError. It is returned when a parent ref is
specified but no URL, which is an invalid parameter combination.
Use the ostree package error types to keep the existing distinction
between Ref- and URL-related errors.
Introduce a new error condition for a general InvalidOSTreeParams
failure.
The customizations struct contains different kinds of customizations,
such as users and filesystems. If both filesystems and users are
configured, the current code overwrites the user configuration.
Change this by unconditionally allocating the customizations struct
when creating the empty blueprint, and ammending it for each
customization we find.
This means that the customizations will be non-nil, also when it is
empty, which was not the case in the past. This should not change
the behavior of the code.
Move `OSBuildMetadataToRPMs()` and `PackageMetadataToSignature()`
functions from the `rpmmd` package to `osbuild2` package to prevent
import cycles while de-duplicating `rpmStageInputs()` function from
`stage_inputs.go` of distro definitions.
Rename `PackageMetadataToSignature()` to
`RPMPackageMetadataToSignature()`, since it takes specifically
`RPMPackageMetadata` type as an argument.
Adjust affected parts of code (unit tests, cloudapi, worker).
Signed-off-by: Tomas Hozza <thozza@redhat.com>
- Any repository without package_sets is added to the general `Repos`
field of the DepsolveJob, just like before.
- Repositories with package_sets are added to the `PackageSetsRepos`
map, indexed by the package set names.
- Repositories defined in the customizations as `PayloadRepositories`
are considered to be associated only with the `PayloadPackageSets`
names from the image type and are added to the `PackageSetsRepos`
under the payload sets.
The repository collection and conversion of repository structs (from
Repository to RepoConfig) has been moved to a separate function.
