Do not create files directly in `/tmp`, but use `$WORKDIR`, which is a
temporary directory for transient files, which gets cleaned up when the
test case finishes. Without this change, running `api.sh` twice fails
the second time.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This commit implements multi-tenancy. A tenant is defined based on a value
from JWT claims. The key of this value must be specified in the configuration
file. This allows us to pick different values when using multiple SSOs.
Let me explain more in depth how this works:
Cloud API gets a new compose request. Firstly, it extracts a tenant name from
JWT claims. The considered claims are configured as an array in
cloud_api.jwt.tenant_provider_fields in composer's config file. The channel
name for all jobs belonging to this compose is created by `"org-" + tenant`.
Why is the channel prefixed by "org-"? To give us options in the future. I can
imagine the request having a channel override. This basically means that
multiple tenants can share a channel. A real use-case for this is multiple
Fedora projects sharing one pool of workers.
Why this commit adds a whole new cloud_api section to the config? Because the
current config is a mess and we should stop adding new stuff into the koji
section. As the Koji API is basically deprecated, we will need to remove it
soon nevertheless.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We (mistakenly) don't enforce a minimum size for /var,
so setting it to 1024 (1kB) causes the image build to fail.
CI does not expose this in a helpful way at the moment,
so this is a bit tricky to debug.
Also skip customizations for the AWS.S3 upload type. Not all the
image types with this upload type support filesystem customizations
and that's as expected. We could make a more fine-grained test in
the future, but testing with a coulpe of targets should be
sufficient.
We no longer test Cloud API on Fedora and Fedora 33 is EOL anyway.
Remove all Fedora 33 related lines from the test case.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When backed by a DB, composer has no need of a queue directory.
This also addresses "Error moving artifacts for job" logging noise.
Signed-off-by: sanne <sanne.raymaekers@gmail.com>
Script is run with the image type to build as the argument.
The target / cloud service is selected based on the image type
specified. This is how the API actually works now: Only an image type
can be specified.
The script now supports all the blobby image types for testing:
- edge-commit
- edge-container
- edge-installer
- image-installer
- guest-image (qcow2)
- vsphere (vmdk)
These are image types that are uploaded to S3 and provided to the user
as an object to download rather than a VM image on a cloud provider.
To verify the cloud api compose request options for the qcow2 and vmdk
image types, download the object and inspect it using image-info.
Checks if postgresql is installed and that user1 and user2 exist in the
passwd file.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
By default, pgxpool.Pool has 4 connections (or number of cpus if higher).
Currently, we have 3 replicas, that means max 3*4=12 DB connections.
The dequeue operation is actually blocking - when a worker is waiting for
a job, one connection is blocked. My theory is that with 16 workers, we just
don't have enough connections that causes all sorts of weird slowdowns.
This commit bumps the number of connection from one replica to 10, therefore
we should be at 30 connections in total.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
RHEL 9.0 AWS API test is failing with
Host key verification failed.
This is probably due to a recent change in openssh deprecating rsa host
keys (or likely rsa keys in general).
- turn off StrictHostKeyChecking when checking groups
- use 'ed25519' type for user ssh keys
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Workers now depsolve in parallel to image builds, so we can
again move depsolivng to the workers. This will help us deal
with increases in traffic as we currently only have one
depsolve handler per pod. It would also avoid any issues with
composer running out of disk space due to dnf metadata caches.
This reverts commit c65b1e9b26.
It no longer makes sense because:
- we don't make any changes to 8.5
- we don't regenerate test manifests for 8.5
- osbuild-composer for 8.5 is in the rhel-8.5.0 branch
Also, the latest-8.5.0 symlink was removed, which broke the CI.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
V2 is compliant with api.openshift.com design guidelines.
Errors are predefined, have codes, and are queryable.
All requests have an operationId set: a unique identifier which is
sortable by time. This is added to the response in case of an error.
All returned objects have the href, id, and kind field set.
2 configurations for the listeners are now possible:
- enableJWT=false with client ssl auth
- enableJWT=true with https
Actual verification of the tokens is handled by
https://github.com/openshift-online/ocm-sdk-go.
An authentication handler is run as the top level handler, before any
routing is done. Routes which do not require authentication should be
listed as exceptions.
Authentication can be restricted using an ACL file which allows
filtering based on JWT claims. For more information see the inline
comments in ocm-sdk/authentication.
As an added quirk the `-v` flag for the osbuild-composer executable was
changed to `-verbose` to avoid flag collision with glog which declares
the `-v` flag in the package `init()` function. The ocm-sdk depends on
glog and pulls it in.
The 'google-cloud-sdk' RPM built by Google for RHEL, which provides
the 'gcloud' command, is built only with Python 2. Since Python 2.7
is already EOL in upstream and not available in CentOS Stream 9, we
can not use 'gcloud' from the 'google-cloud-sdk' RPM.
The 'awscli' is not available in RHEL-9 repositories.
The Azure CLI 'az' available in official upstream repositories has
broken dependencies on RHEL-9 and can not be successfully installed. To
workaround the issue, run the tool from the official container image
provided by Microsoft.
Use the `quay.io/osbuild/cloud-tools` F34-based container image instead
of locally installed cloud CLI tools.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The RHEL-8.5 and RHEL-9.0 `ami` images are now based on the official
RHEL EC2 images. As a result, they use a different default user -
`ec2-user`.
Fix the `api.sh` test case to use the correct user when testing RHEL-9
`ami` images.
Fix#1632
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use TEST_ID for any resources created in Azure. Also create all
necessary vm network resources in advance to have predictable names
using TEST_ID as well.
RHEL 9.0 isn't yet in .gitlab-ci.yml so this actually doesn't change in test
runs but it should make enabling of the tests easier.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Redefine the `ami` image type in RHEL-8.5 to be based on RHEL
ec2 images. The pipeline has different default settings, therefore the
common "os" pipeline is not used. The RHEL ec2 images have a different
default size than the original `ami` image definition. The RHEL ec2
images use a different default partitioning scheme. Their configuration
is slightly different for each architecture and the x86_64 version
of the image does not support UEFI.
Update rpmrepo snapshots used to generate RHEL-8.5 x86_64 and aarch64
image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Cloud api now exposes user customization that let a customer able to add
a new user with a set of groups and a ssh key.
Testing:
* adds 2 users to the AWS image, accessible with a temp ssh key.
* the first one is in the group wheel, the other is not
Fixes#1574
