Reduce the code related to Compute Node v1 API calls in a similar way as
it is done in the API usage examples.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Modify worker's job implementation to try to share GCP image only if the
provided list of accounts is not empty.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Originally, the internal GCP library in `internal/upload/gcp` was
logging various information and errors. Refactor the code to move all
logging to callers of the library. As a result, some methods now return
additional information to preserve the same amount of information being
logged for GCP.
Refactor methods to have only single purpose and not do any extra work,
such as storage cleanup. Methods which create new resources now don't do
any cleanup at all. The caller is responsible to check for any errors
and perform any cleanup necessary. Necessary methods to perform cleanup
are provided.
Modify worker's job implementation and GCP CLI tool to explicitly do all
necessary cleanup, including in case of errors.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The same test is run in distro/distro_test.go. The redundancy was probably
caused by a bitrot in several commits.
I decided to remove the test from distro implementations to reduce the amount
of duplicated code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit adds NewDefault() method to distroregistry that returns a slice
with all distributions supported by osbuild-composer. This way, there's only
one place where a distribution needs to be defined while its support
is being added to composer.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
My goal is to add a method to distroregistry to return Registry with
all supported distributions. This way, all supported distributions
would be defined only on one place.
To achieve this, the Registry must live outside the distro package
because the distro implementation depends on it and this would create
a circular dependency unsupported by Go.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This replaces Packages() and BuildPackages() by returning a map of
package sets, the semantics of which is up to the distro to define.
They are meant to be depsolved and the result returned back as a
map to Manifest(), with the same keys.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use `generate-test-cases` to generate the qcow2 image types with
customization for fedora 32, 33 and CentOS 8 and RHEL 8 and 8.4.
The previous commit changed the customization blueprint but the
changes to the resulting manifest and test data was done by hand.
Use en_US.UTF-8 as default for LANG, which is what previously was
used and is also needed to properly work on non-us/latin setups[1].
In the customization tests, use a different value than the default
one to check that the customization does in fact work.
[1] http://git.app.eng.bos.redhat.com/git/spin-kickstarts.git/tree/rhel8/rhel-8.2-kvm-x86_64.ks#n4
Co-authored-by: Achilleas Koutsou <achilleas@koutsou.net>
Since kernel upgrading from 'default' to 'rt kernel' has SSH
connection issue, 'install' kernel and 'upgrade' kernel must
have 'kernel-rt' included.
https://github.com/osbuild/osbuild-composer/issues/1222
Only RHEL 8.4 repo has 'rt kernel' repo, 'rt kernel' test will be
RHEL 8.4 only
Note that this doesn't actually test for the ostree fields, I'm not sure
if that's possible with this test framework. But it does make sure that
a test compose won't try to fetch the url.
We have integration tests that boot most of the image types, let
the image tests focus on the actual content of the images, shaving
off considerable CI time.
The only missing one is OpenStack (which should be ported over) and
tar (which we need to define the integration test for).
Signed-off-by: Tom Gundersen <teg@jklm.no>
A few recent changes in image type definitions haven't been reflected in
the test cases yet. This also acts as a check to make sure that the
changes in composer don't affect the old behaviour.
Causes for (some) changes:
- Kernel modules added to package lists:
Prior to PR #1175 image types defined the kernel package in their
package list. Some only included `kernel-core` and not the `kernel`
metapackage. Now images default to having the `kernel` metapackage
included which also adds `kernel-modules` and `alsa-sof-firmware`.
- New package source for rt kernel.
Move the container build to the same phase as the RPM builds. This does not make a huge difference, but should
shave off about two minutes of total CI runtime.
This commit adds support for uploading images directly to Azure using the
cloud API.
The UploadStatus part is currently not implemented and will be added in a
follow-up PR.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit adds and implements org.osbuild.azure.image target.
Let's talk about the already implemented org.osbuild.azure target firstly:
The purpose of this target is to authenticate using the Azure Storage
credentials and upload the image file as a Page Blob. Page Blob is basically
an object in storage and it cannot be directly used to launch a VM. To achieve
that, you need to define an actual Azure Image with the Page Blob attached.
For the cloud API, we would like to create an actual Azure Image that is
immediately available for new VMs. The new target accomplishes it.
To achieve this, it must use a different authentication method: Azure OAuth.
The other important difference is that currently, the credentials are stored
on the worker and not in target options. This should lead to better security
because we don't send the credentials over network. In the future, we would
like to have credential-less setup using workers in Azure with the right
IAM policies applied but this requires more investigation and is not
implemented in this commit.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This file contains a client for Azure Storage API. As we soon introduce the
client for Azure API, we need a distinction here.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The UploadImage method doesn't actually create an image. It creates a Page
Blob. Blob is something like S3 object but in the Azure terminology. Page
Blob means that's optimized for random access and it's the only blob type
that can be used to create images.
This commit cleans up the terminology so it's less confusing.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
If the image size isn't aligned to 512 bytes, the Azure API returns very hard
to understand error message. Let's do this check ourselves early so we can
return a sane error.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add NEWS section documenting changes done to Cloud API related to
UploadStatus.
Fix one typo in `rhel84-grub2-saved-entry.md`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Return GCP-specific target results form the worker, similar as it is
done for AWS.
Extend Cloud API to allow GCP-specific upload Options.
Modify Cloud API to return UploadOptions as part of the UploadStatus.
Modify Cloud API integration test to check returned upload Options and
upload Type.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The previous version referred to lorax-composer as the definition of
what osbuild-composer does. This worked fine while osbuild-composer was
considered an alternative for it. Now that osbuild-composer is the
default one, it should describe what it does without references to
lorax. Furthemore, composer is now able to build OSTree commits as well
as VM images, to the previous description was slightly incomplete.
This commit introduces description which is up-to-date and does not
refer to lorax any more.
Previously, the test merged stderr and stderr and treated it as json. This was
obviously wrong. osbuild's stderr only returns python exception that are
of course not json.
The behaviour is now different:
1) stderr and stdout are treated separately
2) json from stdout is indented to prevent long lines. If it fails the test
just prints the stdout as returned from osbuild.
3) stderr is printed as returned from osbuild
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add the TargetResult struct to OSBuildJobResult. Include the 'options'
interface on TargetResult to contain target-specific information,
for example amiID and region from AWS. Expose 'options' on a status
call as an UploadStatus field. Add logic to support AWS within this
format, which can be used as a template for other targets.
Extend the Cloud API integration test for GCP to also create a Compute
Node instance using the image, boot it and ssh to it. This brings the
GCP integration testing on par with AWS testing.
Use VHD image type for GCP integration testing, because it by default
contains `cloud-init` installed, unlike the VMDK image.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
