Fix a bug in the worker job implementation and GCP CLI upload tool,
which causes the code to report wrong error instance in case the image
import failed for some reason.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
By default, a when block is evaluated after an agent is started. I discovered
this randomly: I opened a pipeline and saw that it was stuck on "Prepare EL8
internal 🤔" stage even though the pipeline should have even run it.
This commit fixes it by adding "beforeAgent true" to all when blocks. It
changes the behaviour to more sane "if when is true, allocate an agent".
See https://www.jenkins.io/doc/book/pipeline/syntax/#evaluating-when-before-entering-agent-in-a-stage
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add StorageListObjectsByMetadata() to internal GCP library. The method
allows one to search specific Storage bucket for objects based on
provided metadata.
Extend cloud-cleaner to search for any Storage objects related to the
image import, using custom metadata set on the object. Delete all found
objects.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend StorageObjectUpload() to allow setting custom metadata on the
uploaded object.
Modify worker's osbuild job implementation and GCP CLI upload tool to
set the chosen image name as a custom metadata on the uploaded object.
This will make it possible to connect Storage objects to specific
images.
Add News entry about image name being added as metadata to uploaded GCP
Storage object as part of worker job.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend internal GCP library to allow deleting Compute Node image and
instance. In addition provide function to load service account
credentials file content from the environment.
Change names used for GCP image and instance in `api.sh` integration
test to make them predictable. This is important, so that cloud-cleaner
can identify potentially left over resources and clean them up. Use the
same approach for generating predictable, but run-specific, test ID as
in GenerateCIArtifactName() from internal/test/helpers.go. Use SHA224
to generate a hash from the string, because it can contain characters
not allowed by GCP for resource name (specifically "_" e.g. in "x86_64").
SHA-224 was picked because it generates short enough output and it is
future proof for use in RHEL (unlike MD5 or SHA-1).
Refactor cloud-cleaner to clean up GCP resources and also to run cleanup
for each cloud in a separate goroutine.
Modify run_cloud_cleaner.sh to be able to run in environment in which
AZURE_CREDS is not defined.
Always run cloud-cleaner after integration tests for rhel8, rhel84 and
cs8, which test GCP.
Define DISTRO_CODE for each integration testing stage in Jenkinsfile.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Split the GCP library into multiple files:
- compute.go - code interacting mainly with the Compute Node resources
- storage.go - code interacting mainly with the Cloud Storage resources
- gcp.go - common code (e.g. authentication with GCP)
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The internal GCP library was originally placed into `internal/upload`
directory, since its purpose was mainly to upload and import built
images to GCP.
Functionality for other cloud-provider-specific libraries is broader,
however scattered around the `internal/` directory based on purpose (e.g. in
`internal/boot` and `internal/upload`). Since all parts of provider-specific
library usually share some common pieces (e.g. authentication), it makes
sense to consolidate them into a single package (e.g. in
`internal/cloud/<provider>`).
Create `internal/cloud` directory, where all cloud-provider-specific
internal libraries should be consolidated. Start with GCP.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Most images share a build root, checkpoint the build root to speed up tests
at the potential cost of some disk space (in the cases the build roots are not the same).
Afetr a run, a store contains downloaded rpm's and whatever pipelines
were checkpointed. We want to reuse these for subsequent runs, so
don't delet the store.
This risks minimally increasing the disk space usage, but should speed things up significantly.
cloud-init not longer depends on net-tools, so we need to add in the dependency explicitly.
We aimed for the qcow2 to be equivalent to the RHEL8.4 qcow2 produced by imagefactory. However, we missed this unintended change from RHEL8.3.
by defining the COMPOSE_URL environment variable! This will allow
testing more flavors of internal releases.
The rest is renaming files and variables to reflect the fact that
we're running tests against internal trees, not only nightlies.
Make the handling of GCP credentials more consistent with what is being
done e.g. for Azure. Make the GCP section in worker's configuration a
pointer so that it does not show up in the printed worker's
configuration during start up if it was not specified in the actual
configuration file.
Load the GCP credentials file, if provided, during the worker start up to
prevent failure later on while processing a job with GCP upload target.
Pass the loaded GCP credentials as []byte to the OSBuildJobImpl.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Reduce the code related to Compute Node v1 API calls in a similar way as
it is done in the API usage examples.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Modify worker's job implementation to try to share GCP image only if the
provided list of accounts is not empty.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Originally, the internal GCP library in `internal/upload/gcp` was
logging various information and errors. Refactor the code to move all
logging to callers of the library. As a result, some methods now return
additional information to preserve the same amount of information being
logged for GCP.
Refactor methods to have only single purpose and not do any extra work,
such as storage cleanup. Methods which create new resources now don't do
any cleanup at all. The caller is responsible to check for any errors
and perform any cleanup necessary. Necessary methods to perform cleanup
are provided.
Modify worker's job implementation and GCP CLI tool to explicitly do all
necessary cleanup, including in case of errors.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The same test is run in distro/distro_test.go. The redundancy was probably
caused by a bitrot in several commits.
I decided to remove the test from distro implementations to reduce the amount
of duplicated code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit adds NewDefault() method to distroregistry that returns a slice
with all distributions supported by osbuild-composer. This way, there's only
one place where a distribution needs to be defined while its support
is being added to composer.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
My goal is to add a method to distroregistry to return Registry with
all supported distributions. This way, all supported distributions
would be defined only on one place.
To achieve this, the Registry must live outside the distro package
because the distro implementation depends on it and this would create
a circular dependency unsupported by Go.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This replaces Packages() and BuildPackages() by returning a map of
package sets, the semantics of which is up to the distro to define.
They are meant to be depsolved and the result returned back as a
map to Manifest(), with the same keys.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use `generate-test-cases` to generate the qcow2 image types with
customization for fedora 32, 33 and CentOS 8 and RHEL 8 and 8.4.
The previous commit changed the customization blueprint but the
changes to the resulting manifest and test data was done by hand.
Use en_US.UTF-8 as default for LANG, which is what previously was
used and is also needed to properly work on non-us/latin setups[1].
In the customization tests, use a different value than the default
one to check that the customization does in fact work.
[1] http://git.app.eng.bos.redhat.com/git/spin-kickstarts.git/tree/rhel8/rhel-8.2-kvm-x86_64.ks#n4
Co-authored-by: Achilleas Koutsou <achilleas@koutsou.net>
Since kernel upgrading from 'default' to 'rt kernel' has SSH
connection issue, 'install' kernel and 'upgrade' kernel must
have 'kernel-rt' included.
https://github.com/osbuild/osbuild-composer/issues/1222
Only RHEL 8.4 repo has 'rt kernel' repo, 'rt kernel' test will be
RHEL 8.4 only
Note that this doesn't actually test for the ostree fields, I'm not sure
if that's possible with this test framework. But it does make sure that
a test compose won't try to fetch the url.
We have integration tests that boot most of the image types, let
the image tests focus on the actual content of the images, shaving
off considerable CI time.
The only missing one is OpenStack (which should be ported over) and
tar (which we need to define the integration test for).
Signed-off-by: Tom Gundersen <teg@jklm.no>
A few recent changes in image type definitions haven't been reflected in
the test cases yet. This also acts as a check to make sure that the
changes in composer don't affect the old behaviour.
Causes for (some) changes:
- Kernel modules added to package lists:
Prior to PR #1175 image types defined the kernel package in their
package list. Some only included `kernel-core` and not the `kernel`
metapackage. Now images default to having the `kernel` metapackage
included which also adds `kernel-modules` and `alsa-sof-firmware`.
- New package source for rt kernel.
Move the container build to the same phase as the RPM builds. This does not make a huge difference, but should
shave off about two minutes of total CI runtime.
