This commit adds a very in-depth test for multi-tenancy. It queues several
composes and then runs all jobs belonging to them while checking that
they are run by the correct tenant.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit implements multi-tenancy. A tenant is defined based on a value
from JWT claims. The key of this value must be specified in the configuration
file. This allows us to pick different values when using multiple SSOs.
Let me explain more in depth how this works:
Cloud API gets a new compose request. Firstly, it extracts a tenant name from
JWT claims. The considered claims are configured as an array in
cloud_api.jwt.tenant_provider_fields in composer's config file. The channel
name for all jobs belonging to this compose is created by `"org-" + tenant`.
Why is the channel prefixed by "org-"? To give us options in the future. I can
imagine the request having a channel override. This basically means that
multiple tenants can share a channel. A real use-case for this is multiple
Fedora projects sharing one pool of workers.
Why this commit adds a whole new cloud_api section to the config? Because the
current config is a mess and we should stop adding new stuff into the koji
section. As the Koji API is basically deprecated, we will need to remove it
soon nevertheless.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Channels are a concept similar to job types. Callers must specify a channel
name when queueing a new job. A list of channels is also specified when
dequeueing a job. The dequeued job's channel will always be from one of the
specified channel. Of course, the job types are also respected. The dequeued
job will also always be from one of the specified type.
Currently, all calls to jobqueue were changed so all queue operations use
an empty channel name and all dequeue operations use a list containing
an empty channel.
Thus, this is a non-functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
rpmmd_mock fixture are complex and unneeded in the context of cloudapi, let's
just copy 3 lines from them and drop the dependency.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add support for building images for the Azure marketplace: add a
new image type "azure-rhui" that can be used to build images
tailored to the Azure marketplace.
Add two sample manifests for 8.5 and 8.6, but note that even the
8.5 is using the 8.6 distro definitions. Also no image-info is
included since `image-info` cannot (yet) handle LVM setups and
the azure marketplace images use the LVM setup.
Customizations being nil isn't the same as an empty object. The
edge-installer types require customizations equalling nil.
This partially reverts bb486bcccb.
Shorter and clearer error types.
- InvalidParameterError is now RefError. It is returned when a ref (the
new ref or the parent ref) is invalid.
- The ResolveRefError is not renamed. It is returned when a ref can't
be resolved (e.g., ref not found at URL).
New error type ParameterComboError. It is returned when a parent ref is
specified but no URL, which is an invalid parameter combination.
Use the ostree package error types to keep the existing distinction
between Ref- and URL-related errors.
Introduce a new error condition for a general InvalidOSTreeParams
failure.
The customizations struct contains different kinds of customizations,
such as users and filesystems. If both filesystems and users are
configured, the current code overwrites the user configuration.
Change this by unconditionally allocating the customizations struct
when creating the empty blueprint, and ammending it for each
customization we find.
This means that the customizations will be non-nil, also when it is
empty, which was not the case in the past. This should not change
the behavior of the code.
Move `OSBuildMetadataToRPMs()` and `PackageMetadataToSignature()`
functions from the `rpmmd` package to `osbuild2` package to prevent
import cycles while de-duplicating `rpmStageInputs()` function from
`stage_inputs.go` of distro definitions.
Rename `PackageMetadataToSignature()` to
`RPMPackageMetadataToSignature()`, since it takes specifically
`RPMPackageMetadata` type as an argument.
Adjust affected parts of code (unit tests, cloudapi, worker).
Signed-off-by: Tomas Hozza <thozza@redhat.com>
- Any repository without package_sets is added to the general `Repos`
field of the DepsolveJob, just like before.
- Repositories with package_sets are added to the `PackageSetsRepos`
map, indexed by the package set names.
- Repositories defined in the customizations as `PayloadRepositories`
are considered to be associated only with the `PayloadPackageSets`
names from the image type and are added to the `PackageSetsRepos`
under the payload sets.
The repository collection and conversion of repository structs (from
Repository to RepoConfig) has been moved to a separate function.
Tags is an array of strings that associates repositories with package
sets. A repository tagged with a package set name will be used only for
the named package sets.
We have two fields, `Repos` and `PackageSets`. Renaming
`PackageSetsRepositories` to `PackageSetsRepos` for consistency.
The struct is for internal use only so the rename has no impact as long
as the serialised name is the same (json tag).
Also it's shorter.
Added docstring to the struct that explains the arguments in the same
way as they are described for the `depsolve()` function.
Changing the name of the argument in the internal `depsolve()` function
for the same reasons.
For the brew use-case, we also need to build AWS images containing RHUI. This
commit is thus adding them.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This is for now only supported for koji builds.
A lot of code moved around for this, but functionally not much changed. PostCompose() now only parses the input, and queueing of all the jobs have been factored out in separate functions. PostCompose() is mostly agnostic to koji/non-koji requests.
Replace Job() and JobStatus() with typesafe versions, and introduce JobType()
for the rare instances where we don't know the type up front.
Additionally, catch a few more error cases:
- if OSBuildResult is nil, then we failed to invoke osbuild
- make sure the same JobResult handling is done for osbuild-koji, as for osbuild
This only extends the API, the backend can still only deal with composes of a single build.
I aimed to keep the API practically backwards compatible, i.e., no current consumer of it should notice the change. I hope I didn't mess that up.
fixup: image statuses
In addition to individual image status, have an
overall status that captures success or failure
of the compose as a whole.
This is not as fine grained, and only distinguishes
between "pending", "failure" and "success".
This captures other jobs than the image builds, which
is relevant for the koji composes, which consists also
of koji-init and koji-finalize, in addition to the build
jobs.
For now upload requests are required if and only if we are not
using koji. When using the koji integration the produced artifacts
are uploaded to koji only. In the future we may want to support
also uploading to the cloud providers.
Extend the compose endpoints to have minimal koji support.
This is intended to replace the current koji API so that it
can be consumed through api.openshift.com.
Since the workers will use structured error messages
going forward, it is necessary to maintain backwards
compatability for there errors in composer. Tests have
been added to the various apis to ensure that each api
checks for both kinds of errors, old and new.
Implement the structured errors as defined by the worker client.
Every error for each of the job types now returns a structured
error with a reason and a specific error code. This will make
it possible to differentiate between 4xx errors and 5xx errors.
This commit refactors the way errors are implemented in the workers,
but maintains backwards compatability in composer by checking for
both kinds of errors.
