Refactor test/cases/api.sh to incorporate testing of cloudapi with
multiple cloud providers as the target. Since all variables in Bash are
by default global, don't declare them as empty in advance. The only
place where underclared variables can be potentially expanded are the
cleanup functions. Ensure that there are no unbound variables expanded
inside cleanup functions. Rename all AWS-specific variables to
contain "AWS_" prefix to make their purpose explicit.
Modify provision.sh to append the GCP credentials file path to the
worker configuration.
Add GCP api.sh test case to integration tests in Jenkins and run it only
if the appropriate GCP credentials environment variable is defined. Run
the GCP test case for RHEL images.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The tools/provision.sh script is sourced by all test cases and it sets
up the system and software for running test cases. As part of the setup,
it copied over the whole content of test/data/composer/ to
/etc/osbuild-composer. However the source directory contains not only
osbuild-composer's configuration, but also configuration for the worker.
The worker however expects its configuration in /etc/osbuild-worker.
The fact that provision.sh does not copy the worker configuration to the
correct directory didn't affect the CI, because the only test case that
relied on it is koji.sh, which copies the worker configuration
explicitly.
Move osbuild-worker test configuration to a separate 'test/data/worker/'
subdirectory. Also install the osbuild-worker test configuration to its
own subdirectory in the "-test" RPM.
Move the copying of worker configuration to the correct destination
directory from koji.sh to provision.sh, so that all test cases can rely
on the system being set up properly. Do not use wildcard for copying
osbuild-{composer,worker} configuration files, but explicitly copy each
file to its respective destination directory.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for GCP as an upload target to the internal API.
Extend the cloudapi to allow GCP as an upload target in the compose
request. Regenerate the cloudapi go code. Added GCP-specific upload
result component in the API definition, similar to AWS. It is not yet
used, but it will be once returning a target-specific result from
worker is supported.
Add support for GCP upload target to the worker job implementation.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new internal upload target for Google Cloud Platform and
osbuild-upload-gcp CLI tool which uses the API.
Supported features are:
- Authenticate with GCP using explicitly provided JSON credentials
file or let the authentication be handled automatically by the
Google cloud client library. The later is useful e.g. when the worker
is running in GCP VM instance, which has associated permissions with
it.
- Upload an existing image file into existing Storage bucket.
- Verify MD5 checksum of the uploaded image file against the local
file's checksum.
- Import the uploaded image file into Compute Node as an Image.
- Delete the uploaded image file after a successful image import.
- Delete all cache files from storage created as part of the image
import build job.
- Share the imported image with a list of specified accounts.
GCP-specific image type is not yet added, since GCP supports importing
VMDK and VHD images, which the osbuild-composer already supports.
Update go.mod, vendor/ content and SPEC file with new dependencies.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The openstack boot test often ruins our days with:
Waiting for instance 63ac19be-2e19-44e2-8bef-9770d68a190c to become Active
failed: A timeout occurred
I decided to investigate. It turns out the first boot of an image can take
up to 18 minutes. The subsequent ones are usually much faster (but don't rely
on this fact, I saw 15 minutes there).
This commit bumps the timeout to 30 minutes. This should be plenty of time
for the instance to spin up and get into the ACTIVE state.
Honestly, I'm not very happy with the solution but it should help with the
failing Schutzbot. As a follow up, I will reach to the PSI OpenStack team
and ask them if we could somehow speed up the process (maybe by using another
flavor, ci.m1.medium.ephemeral just might be slow for some reason, I don't
know).
Anyway, this should help us in the short term because I strongly believe that
a slow test is still better than a failing one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
/etc/containers/registries.conf.d/rhel-shortnames.conf shipped in
containers-common-1:1.2.2-1.module+el8.4.0+10073+30e5ea69 has a wrong
shortname for ubi8-minimal:
"ubi8-minimal" = "registry.access.redhat.com/repository/ubi8-minimal"
resulting in `name unknown: Repo not found` when trying to pull the image
via its short name.
Related: rhbz#1931785
Use the new tags of the `osbuild/containers` repository. The
infrastructure was simplified a lot and the new tags are much easier to
handle without any conflicts when building other images.
Note that this change uses the `latest` tag of all images. As an
alternative, we can also switch to the `latest-<date>` tags, which would
use a pinned immutable image tagged by the given date. However, these
tags are burried somewhere deep down in the ./tools/ directory and are
easy to forget upgrading. So for now use `latest` until we have better
synchronization infrastructure in place.
However, if these images ever break and we want to get back stable CI
behavior, we can always switch back to older builds by picking those
immutable tags instead of `latest`.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Explicitly set the kernel to boot into.
Also change the blueprint/kernenl handling:
Rather than only falling back to the default kernel name for
getting the package list, let GetKernel() always return the
correct result so we can rely on this being consistent.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In order to land a PR before osbuild reaches the stable repositories
we pin the verson to test against (which has been pushed out).
This is an exception from our usual procedures as we would otherwise
not be able to land this bug fix for RHEL8.4.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Let's keep this on the same filesystem as the osbuild store, and
in particular stay away from /var/tmp and its scary semantics.
We are not aware of any issues caused by /var/tmp, but getting
rid of it means we don't have to think about that when debugging,
if nothing else.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We are gaining new ostree features that overlap to a great deal
with the current ones. We still need to keep the current features
for backwards compatibility, so add another test run that does the
same but using new API.
For now this simply uses the `url` parameter rather than `parent`
to build update commits. Further changes will be made in follow-up
commits.
Use `curl` rather than `composer-cli` as we have a chicken-and-egg
problem where we can't land this feature without tests, but
`composer-cli` can't add support for it without having it first in
`composer`.
For now this is simply used to resolve the parent commit, in case
one is not provided. In the future it will be used by new image
types to actually pull content from.
This extends the weldr API, so that future work does not have to
modify that.
The logic we now implement for the ostree commit image types is:
If the URL is provided, but the parent commit is not. The parent
commit is taken to be the current HEAD of the ostree repo at the
given url, with the given (or default) ref.
This only provides a small optional convenience, but we will
soon introduce image types where the URL of the repository is
required.
This commit still needs testing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than setting this automagically, expose it to the caller. For
now the only caller we have simply passes it back in, so this is a
noop.
In follow-up commits this will be used to resolve the parent commit.
This is tested by verifying that the generated manifests do not
change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We aim at shrinking our deps eventually but we need subman for the time
being. This patch basically un-exclude subman which was introduced by
https://github.com/osbuild/osbuild-composer/pull/893
Signed-off-by: Antonio Murdaca <runcom@linux.com>
VMDK image has default name 'disk.vmdk' and there is no option to change the name when uploading to vSphere,
so I'm using symlink so that uploaded image has the name user specified instead of the default one.
New upload target for VMWare, similar to the ones for AWS and Azure,
allowing users to set credentials for their vSphere instance.
Commit also includes function that performs the actual upload.
Two new test cases added to format-request-map and test cases are
generated.
1. kernel-rt for RHEL images:
Requires new package repositories for RHEL 8.3 and 8.4.
Creates an OSTree commit with the `kernel-rt` as a customization.
2. kernel-debug for Fedora images:
kernel-rt isn't included in the official fedora repositories. Using
kernel-debug at least tests the feature with the fedora-iot-commit type.
The kernel now comes from the blueprint packages even when it's not
specified. Removing from the base packages of the image types avoids
duplication and allows for alternative kernels to be specified without
also including the default.
The latter is necessary for RHEL for Edge and Fedora IoT images (ostree
commits) that fail to build when multiple kernels are installed.
ImageType tests modified to fix expected package order.
Blueprints can now be used to specify a kernel as part of the kernel
customizations. Specifying a kernel adds it to the package list.
If no known kernel is specified (neither in the customizations nor the
package list), the default "kernel" is included automatically.
If kernels are specified in both the package list and the
customizations, both are added (even if they're duplicates).
https://github.com/osbuild/osbuild-composer/pull/1228 was merged with
a failing Schutzbot's pipeline. The failure is caused because `var` apparently
isn't a right Groovy syntax. Let's use the right word `def` instead.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
s390x isn't supported on Centos.
rhel-commit-edge sounds just wrong for Centos. We can revert this change any
time. The thing is that I wasn't able to find something like CentOS IoT and
we don't want to be in a position of defining a new distribution spin.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I'm sorry, I need to extend this condition and my brain isn't powerful enough
to reason about complex negative conditions.
Not a functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The image definition is shared with the latest RHEL 8.y one (8.4 currently).
I expect that we the introduction of 8.5 support, we point the centos 8
distro at it.
The test repositories and manifests use the official CentOS composes. From
what I can tell, they are persistent. This is not guaranteed though, so we
might need to switch to RPMRepo at some point.
The "classic" CentOS 8 should also be buildable but due to the chicken and egg
issue (this commit will get into Centos "8.4" but Centos "8.4" isn't a thing
yet), we cannot test it and therefore it might be broken.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When unit tests succeed, no one cares about them.
When unit tests fail, it's hard to find which one failed. This commit removes
the verbose flag, so it's easy to spot what failed.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
