weldr-client returns different json structures than the original
composer-cli does. It is being replaced in RHEL-9 and these changes make
it possible to run the tests with both old composer-cli and
weldr-client.
This introduces a script to run cloud-cleaner with a schedule. It's
currently working with Azure only and therefore needed a change to
cloud-cleaner code as well. Using azure-cli it gets a list of resources
from $AZURE_RESOURCE_GROUP and then sorts out only resources that
are older than $HOURS_BACK and are not storage accounts.
Then it processes the list further leaving only list with unique
TEST_ID to supply to the cloud-cleaner.
Previously, we just assumed that all test manifests are of version 1, or we
should export the pipeline named assembler. However, this is no longer true
in RHEL 8.5 and 9 - they are only manifest v2 and they don't have a pipeline
named assembler.
This commit introduces a new way to guess the export name - if the manifest
is of version 1, we just export the assembler. In the case v2 manifest, the
last pipeline is exported.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The problem: osbuild-composer used to have a rather uncomplete logic for
selecting client certificates and keys while fetching data from
repositories that use the "subscription model". In this scenario, every
repo requires the user to use a client-side TLS certificate. The problem
is that every repo can use its own CA and require a different pair of
a certificate and a key. This case wasn't handled at all in composer.
Furthermore, osbuild-composer can use remote workers which complicates
things even more.
Assumptions: The problem outlined above is hard to solve in the general
case, but Red Hat Subscription Manager places certain limitations on how
subscriptions might be used. For example, a subscription must be tight to
a host system, so there is no way to use such a repository in osbuild-composer
without it being available on the host system as well.
Also, if a user wishes to use a certain repository in osbuild-composer it
must be available on both hosts: the composer and the worker. It will come
with different pair of a client certificate and a key but otherwise, its
configuration remains the same.
The solution: Expect all the subscriptions to be registered in the
/etc/yum.repos.d/redhat.repo file. Read the mapping of URLs to certificates
and keys from there and use it. Don't change the manifest format and let
osbuild guess the appropriate subscription to use.
The `ec2` and `ec2-ha` images include RHUI client packages, which are
not publicly available. For this reason, building of such images in the
on-premise use case via WeldrAPI would always fail, unless the system
would be inside the Red Hat internal network or VPN.
Mark the `ec2` and `ec2-ha` image types for `rhel-*` distribution as
denied in WeldrAPI by default.
Extend and modify affected unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Change the Image Type denylist in Weldr API from being applied to all
distributions to being distribution-specific. A special name `*`
can be used in the configuration to match any distribution
or any image type.
Modify NEWS entry and unit tests to reflect this change.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend Weldr API to accept a list of denied image types, which should
not be exposed via API for any supported distribution. This
functionality will be needed to not expose image types which can't be
successfully built outside of Red Hat VPN. Example of such images are
the official RHEL EC2 images, which include RHUI client packages not
available publicly.
Image Types are filters when listing available compose types and
creating a new compose using Weldr API.
Extend osbuild-composer configuration to allow specifying the list of
denied Image Types for Weldr API.
Add unit tests for implemented changes.
Add NEWS entry describing the newly introduced functionality.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
To help along with debugging, this commit makes the worker able to print
the status of the different stages with a oneliner for each successfull
stages and a detailed message for failed ones.
Sample output:
Jul 23[..]: Build stages results:
Jul 23[..]: org.osbuild.rpm success
Jul 23[..]: org.osbuild.selinux success
Jul 23[..]: Stages results:
Jul 23[..]: org.osbuild.rpm success
Jul 23[..]: org.osbuild.fix-bls success
Jul 23[..]: org.osbuild.fstab success
Jul 23[..]: org.osbuild.grub2 success
Jul 23[..]: org.osbuild.locale success
Jul 23[..]: org.osbuild.timezone success
Jul 23[..]: org.osbuild.users failure:
Jul 23[..]: [/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
Jul 23[..]: All rules containing unresolvable specifiers will be skipped.
Jul 23[..]: Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
Jul 23[..]: useradd: group 'toto' does not exist
Fixes#1584
The `api.sh` test currently always defaults to "<REGION>-a" zone when
creating instance using the built image. The resources in a zone may get
exhausted and the solution is to use a different zone. Currently even a
CI job retry won't help with mitigation of such error during a CI run.
Modify `api.sh` to pick random GCP zone for a given region when creating
a compute instance. Use only GCP zones which are "UP".
The `cloud-cleaner` relied on the behavior of `api.sh` to always choose
the "<REGION>-a" zone. Guessing the chosen zone in `cloud-cleaner` is
not viable, but thankfully the instance name is by default unique for
the whole GCP project. Modify `cloud-cleaner` to iterate over all
available zones in the used region and try to delete the specific
instance in each of them.
Make `ComputeZonesInRegion` method from the `internal/cloud/gcp` package
exported and use it in `cloud-cleaner` for getting the list of available
zones in a region.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the OSBuildStagesToRPMs function, associated test, and RPM type
from the worker into the rpmmd subpackge. We will use this function in
the cloud API to compile the NEVRAs for the new metadata endpoint.
If a user uses a temporary access key for login, a session token is also
needed.
This commit adds support for it to the internal aws library and also
to the osbuild-upload-aws helper. Note that this doesn't affect the main
osbuild-composer executable nor the worker. Everything here should work
as before and session tokens are not supported. Something for a follow up
if anyone needs it.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add a new CLI option to `osbuild-image-tests` called
`-skip-selinux-ctx-check` to workaround the limitation of `setfiles` on
RHEL-8 [1]. If the option is passed to the binary, then the
'selinux/context-mismatch' part is removed from the "expected" and
"actual" image-info report, before these two reports are compared.
Modify `image_tests.sh` to run `osbuild-image-tests` with
`-skip-selinux-ctx-check` when run on RHEL-8.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1973754
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Koji image request handling now reads the exports defined by each image
type. All APIs now support reading the exports defined by each image
type. The worker still falls back to "assembler" in case the call comes
from an older version of composer.
Uploads an artifact to an S£ bucket and returns a presigned URL to allow
the user to download the file.
Although it uses a lot of common code with the AWS AMI upload target,
it's treated as a completely separate target.
Modify composer to use RepoRegistry, instead of loading the host
repositories, when initializing WeldrAPI.
Modify WeldrAPI to use RepoRegistry, instead of a map of repository
definitions. Make sure that the RepoRegistry method specific to image
type is used in Welder where appropriate. Specifically when depsolving a
Blueprint, which is used to build a specific image type. Update Weldr
API unit tests to reflect the change.
Add a new method to RepoRegistry, allowing to get list of repositories,
which should be used for building an image for a given architecture,
without specifying the exact image type. Add relevant unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Composer does not have 1:1 mapping of what can be the Host Distro name
and the names of supported distributions held in the Distroregistry.
The fact that the host distro `Name()` method as passed to the Weldr API
does not return the same name as what is used as distro name for
repository definitions. This makes it hard to use `distro.Distro` and
`distro.Arch` directly and rely on the values returned by them as their
name.
Add `New*HostDistro()` to all distro definitions, accepting the name
that should be returned by the distro's `Name()` method. This is useful
mainly if the host distro is Beta or Stream variant of the distro.
Change the distroregistry.Registry to contain host distro as a separate
value set when creating it using `New()` function. This value is
returned by `Registry.FromHost()` method. Determining the host distro is
handled by the `NewDefault()` function. Move the distro name mangling to
distroregistry package. Add relevant unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add method to fetch Cloudbuild job log.
Add method to parse Cloudbuild job log for created resources. Parsing is
specific to the Image import Cloudbuild job and its logs format. Add
unit tests for the parsing function.
Add method to clean up all resources (instances, disks, storage objects)
after a Cloudbuild job.
Modify the worker osbuild job implementation and also the GCP upload CLI
tool to use the new cleanup method CloudbuildBuildCleanup().
Keep the StorageImageImportCleanup() method, because it is still used by
the cloud-cleaner tool. There is no way for the cloud-cleaner to figure
out the Cloudbuild job ID to be able to call CloudbuildBuildCleanup()
instead.
Add methods to delete Compute instance and disk.
Add method to get Compute instance information. This is useful for
checking if the instance has been already deleted, or whether it still
exists.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Modify all relevant methods in the internal GCP library to accept
context from the caller.
Modify all places which call the internal GCP library methods to pass
the context.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The previous version constructed multiple temporary variables and then
create job result from them. This was needed because we had multiple
upload targets but now that we have only one, this is only fragile
version of what can be done in a simplified way.
This PR removes the temporary variables and assigns errors and success
states right after the upload or build has finished.
Multiple upload targets are not supported by osbuild-composer any more.
Dropping support for this in worker therefore doesn't change anything
from the user's perspective, but it allows us to simplify the code a
bit.
Replace calls to "continue" with "return nil" because the job finished
correctly even though it failed to perform the task. But the failure was
reported to osbuild-composer for further processing so there is no need
to duplicate and report the same error in worker process
Drop support for LocalTarget, this has not been used in a long time,
and we don't really need to stay compatible across many releases
(just as long as we don't get problems with having to deploy in
lock-step), at least not yet.
Also drop support for KojiTarget, this has been replaced by the
osbuild-koji job type.
The previous implementation exited before reporting back to the worker
API in few branches. This left the compose status in RUNNING state even
though the worker did not work of the job any more. Refactoring the
API call into the `deref` part makes sure it gets called every time.
This commit only moves bits of the code around so that the status gets
back to osbuild-composer, but it still doesn't contain any useful
information in case osbuild fails etc. This will be introduced in
subsequent commits.
When shelling out for a CLI test the error returned from the Start()
command prints the exit code which is not very informative. Capturing
and printing stderr is a lot more useful.
osbuild now supports using the `--export` flag (can be invoked multiple
times) to request the exporting of one or more artefacts. Omitting it
causes the build job to export nothing.
The Koji API doesn't support the new image types (yet) so it simply uses
the "assembler" name, which is the final stage of the old (v1)
Manifests.
Fix a bug in the worker job implementation and GCP CLI upload tool,
which causes the code to report wrong error instance in case the image
import failed for some reason.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
