by defining the COMPOSE_URL environment variable! This will allow
testing more flavors of internal releases.
The rest is renaming files and variables to reflect the fact that
we're running tests against internal trees, not only nightlies.
Make the handling of GCP credentials more consistent with what is being
done e.g. for Azure. Make the GCP section in worker's configuration a
pointer so that it does not show up in the printed worker's
configuration during start up if it was not specified in the actual
configuration file.
Load the GCP credentials file, if provided, during the worker start up to
prevent failure later on while processing a job with GCP upload target.
Pass the loaded GCP credentials as []byte to the OSBuildJobImpl.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Reduce the code related to Compute Node v1 API calls in a similar way as
it is done in the API usage examples.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Modify worker's job implementation to try to share GCP image only if the
provided list of accounts is not empty.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Originally, the internal GCP library in `internal/upload/gcp` was
logging various information and errors. Refactor the code to move all
logging to callers of the library. As a result, some methods now return
additional information to preserve the same amount of information being
logged for GCP.
Refactor methods to have only single purpose and not do any extra work,
such as storage cleanup. Methods which create new resources now don't do
any cleanup at all. The caller is responsible to check for any errors
and perform any cleanup necessary. Necessary methods to perform cleanup
are provided.
Modify worker's job implementation and GCP CLI tool to explicitly do all
necessary cleanup, including in case of errors.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The same test is run in distro/distro_test.go. The redundancy was probably
caused by a bitrot in several commits.
I decided to remove the test from distro implementations to reduce the amount
of duplicated code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit adds NewDefault() method to distroregistry that returns a slice
with all distributions supported by osbuild-composer. This way, there's only
one place where a distribution needs to be defined while its support
is being added to composer.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
My goal is to add a method to distroregistry to return Registry with
all supported distributions. This way, all supported distributions
would be defined only on one place.
To achieve this, the Registry must live outside the distro package
because the distro implementation depends on it and this would create
a circular dependency unsupported by Go.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This replaces Packages() and BuildPackages() by returning a map of
package sets, the semantics of which is up to the distro to define.
They are meant to be depsolved and the result returned back as a
map to Manifest(), with the same keys.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use `generate-test-cases` to generate the qcow2 image types with
customization for fedora 32, 33 and CentOS 8 and RHEL 8 and 8.4.
The previous commit changed the customization blueprint but the
changes to the resulting manifest and test data was done by hand.
Use en_US.UTF-8 as default for LANG, which is what previously was
used and is also needed to properly work on non-us/latin setups[1].
In the customization tests, use a different value than the default
one to check that the customization does in fact work.
[1] http://git.app.eng.bos.redhat.com/git/spin-kickstarts.git/tree/rhel8/rhel-8.2-kvm-x86_64.ks#n4
Co-authored-by: Achilleas Koutsou <achilleas@koutsou.net>
Since kernel upgrading from 'default' to 'rt kernel' has SSH
connection issue, 'install' kernel and 'upgrade' kernel must
have 'kernel-rt' included.
https://github.com/osbuild/osbuild-composer/issues/1222
Only RHEL 8.4 repo has 'rt kernel' repo, 'rt kernel' test will be
RHEL 8.4 only
Note that this doesn't actually test for the ostree fields, I'm not sure
if that's possible with this test framework. But it does make sure that
a test compose won't try to fetch the url.
We have integration tests that boot most of the image types, let
the image tests focus on the actual content of the images, shaving
off considerable CI time.
The only missing one is OpenStack (which should be ported over) and
tar (which we need to define the integration test for).
Signed-off-by: Tom Gundersen <teg@jklm.no>
A few recent changes in image type definitions haven't been reflected in
the test cases yet. This also acts as a check to make sure that the
changes in composer don't affect the old behaviour.
Causes for (some) changes:
- Kernel modules added to package lists:
Prior to PR #1175 image types defined the kernel package in their
package list. Some only included `kernel-core` and not the `kernel`
metapackage. Now images default to having the `kernel` metapackage
included which also adds `kernel-modules` and `alsa-sof-firmware`.
- New package source for rt kernel.
Move the container build to the same phase as the RPM builds. This does not make a huge difference, but should
shave off about two minutes of total CI runtime.
This commit adds support for uploading images directly to Azure using the
cloud API.
The UploadStatus part is currently not implemented and will be added in a
follow-up PR.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit adds and implements org.osbuild.azure.image target.
Let's talk about the already implemented org.osbuild.azure target firstly:
The purpose of this target is to authenticate using the Azure Storage
credentials and upload the image file as a Page Blob. Page Blob is basically
an object in storage and it cannot be directly used to launch a VM. To achieve
that, you need to define an actual Azure Image with the Page Blob attached.
For the cloud API, we would like to create an actual Azure Image that is
immediately available for new VMs. The new target accomplishes it.
To achieve this, it must use a different authentication method: Azure OAuth.
The other important difference is that currently, the credentials are stored
on the worker and not in target options. This should lead to better security
because we don't send the credentials over network. In the future, we would
like to have credential-less setup using workers in Azure with the right
IAM policies applied but this requires more investigation and is not
implemented in this commit.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This file contains a client for Azure Storage API. As we soon introduce the
client for Azure API, we need a distinction here.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The UploadImage method doesn't actually create an image. It creates a Page
Blob. Blob is something like S3 object but in the Azure terminology. Page
Blob means that's optimized for random access and it's the only blob type
that can be used to create images.
This commit cleans up the terminology so it's less confusing.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
If the image size isn't aligned to 512 bytes, the Azure API returns very hard
to understand error message. Let's do this check ourselves early so we can
return a sane error.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add NEWS section documenting changes done to Cloud API related to
UploadStatus.
Fix one typo in `rhel84-grub2-saved-entry.md`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Return GCP-specific target results form the worker, similar as it is
done for AWS.
Extend Cloud API to allow GCP-specific upload Options.
Modify Cloud API to return UploadOptions as part of the UploadStatus.
Modify Cloud API integration test to check returned upload Options and
upload Type.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
