jobimpl-osbuild
---------------
Add GenericS3Creds to struct
Add method to create AWS with Endpoint for Generic S3 (with its own credentials file)
Move uploading to S3 and result handling to a separate method (along with the special VMDK handling)
adjust the AWS S3 case to the new method
Implement a new case for uploading to a generic S3 service
awscloud
--------
Add wrapper methods for endpoint support
Set the endpoint to the AWS session
Set s3ForcePathStyle to true if endpoint was set
Target
------
Define a new target type for the GenericS3Target and Options
Handle unmarshaling of the target options and result for the Generic S3
Weldr
-----
Add support for only uploading to AWS S3
Define new structures for AWS S3 and Generic S3 (based on AWS S3)
Handle unmarshaling of the providers settings' upload settings
main
----
Add a section in the main config for the Generic S3 service for credentials
If provided pass the credentials file name to the osbuild job implementation
Upload Utility
--------------
Add upload-generic-s3 utility
Makefile
------
Do not fail if the bin directory already exists
Tests
-----
Add test cases for both AWS and a generic S3 server
Add a generic s3_test.sh file for both test cases and add it to the tests RPM spec
Adjust the libvirt test case script to support already created images
GitLabCI - Extend the libvirt test case to include the two new tests
We want to ensure that cloud images connect to Red Hat[1] independently
of how the content was acquired (PAYG, BYOS, or marketplace).
This auto-registration feature is already enabled for AWS and this
patch enables it for Azure with the same recommended settings:
Services:
rhsmcertd: Enabled (already done, so not changed in the patch)
/etc/rhsm/rhsm.conf:
auto_registration: enabled
auto_registration_interval: 60 (the default, so not explicitly set)
manage_repos: false
The latter value `manage_repos` is left enabled (the default) in case
the user explicitly requested to have the system subscribed, i.e. the
`RHSMConfigWithSubscription` code path.
Regenerate the relevant test manifests and image information.
[1] https://cloud.redhat.com
[2] https://docs.google.com/document/d/1VeZFJxNUlyZMQJh6s3NA3RLvadqATsGxVet6uuP87_4
The users anaconda module enables users to create user accounts at
install time if one is not already created in the payload. This is
required for the cloud API (Image Builder service) for the image
installer where user customizations are not supported. Without it, user
creation isn't possible on the installed system.
The module also enables user creation at install time through the
kickstart file for both the image-installer and the edge-installer
(Anaconda only).
Therefore, for the image-installer, the users and groups are no longer
created as part of the payload.
This commit adapts the changes from the following commits (originally
made in the RHEL 8.6 and RHEL 9.0 distros) to the rest of the RHEL
distro definitions:
ebc3330cbd5825294dad
Use single NewGroupsStageOptions() from osbuild1 and osbuild2 instead of
implementing in each distro.
- Followup from 2eef6e6e2d, copied to the
rest of the RHEL distro definitions.
- Added NewGroupsStageOptions() to osbuild1 for rhel8 and rhel84.
NB: The change was not made in the Fedora distro definitions as they are
currently being rewritten.
Followup from, f34380d5b5 and
3a1765a5a8, copied to the rest of the RHEL
distro definitions.
For now, these customizations have no effect on the manifest.
The new `with-users` variants of the edge-installer test cases include
the user customizations in the blueprint, but the manifests are
(currently) the same as the corresponding base cases.
Regression test suite has grown considerably and is taking too long to
run with a single wrapper. Splitting them into individual standalone
tests instead and making them run in parallel.
The VMDK image must be in stream-optimized format in order to be
imported to VSphere. osbuild-composer does not produce VMDK by default
as stream-optimized. Instead, it is converted on the fly when the image
build job has been submitted via Weldr API.
Since we are aiming mainly for the VSphere use case with the VMDK image
in the service, the image should be ready for importing to VSphere.
Implement a temporary workaround for the Cloud API and AWS S3 target to
upload stream-optimized VMDK image.
Adjust the `api.sh` test case to not convert the VMDK image downloaded
form S3, before importing it to VSphere.
Ensure that the content of the database is not printed to the console
when dumped at the end of the test case. The output is still preserved
as a CI run artifact.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Kill and remove the DB container as part of the test case cleanup.
Without this change, running the test case more than once fails.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `api.sh` test to verify the VMDK images uploaded to S3 in
VSphere by booting them and configuring using cloud-init.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Install `cloud-init` by default on the VMDK image on RHEL-85/86/90 and
as a result also CentOS Stream 8/9.
Regenerate image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
test cases fixup
RHEL 9.0 will ship a brand new auxiliary key. Let's use it everywhere in our
RHEL 9 stuff. Taken from current RHEL 9.0's redhat-release package.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We should honour `pkg.CheckGPG` when creating the file inputs for the
rpm stage. This was lost in the transition from v1 to v2 manifests.
Regenerate image test manifests.
Co-authored-by: Tomas Hozza <thozza@redhat.com>
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Remove comment out code
Use three different IP address for different test scenarios
Move /boot/device-credentials file checking into playbook
Some shell script improvements
Enable the user module unconditionally for the image-installer:
- If users are specified for the kickstart file, the module is required
to set up the users.
- If no users are specified, the module can be used at install time to
create users.
Updated relevant test cases (manifests).
Use single NewGroupsStageOptions() from osbuild2 instead of implementing
in each distro.
The new function does not set the Group.Name field anymore. The field
does not exist in the osbuild schema and was silently ignored.
The field in the stage has been marked 'omitempty' and the relevant
manifests have been updated.
This test get stuck randomly on centos-stream-8 and is making the CI
unreliable. Adding hard wait limit and destroying the VM afterwards
helps the test get unstuck and continue as expected. See
https://github.com/osbuild/osbuild-composer/issues/2413 for details.
We have to do a small hack to enable edge-commit on Fedora because its name
is different. We can also change this in the image definition but I want to
iterate quickly on the Fedora Integration MVP and don't want to run in
any conflicts with
https://github.com/osbuild/osbuild-composer/pull/2461
This commit also enables a test for Fedora IoT built through the API.
While enabling the test, I also simplified our decision logic for SSH_USER
and DISTRO.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Use the Extended Bootloader Partition GUID for `/boot`, instead of the
Linux filesystem data GUID. This is useful for autodetection of a
partition purpose based on its GUID without reading the `/etc/fstab`
first.
Ensure that when creating mountpoints, e.g. when converting the
partitions layout to LVM, the `/boot` partition get the proper GUID
assigned.
Regenerate RHEL-90 and centos-9 image test cases.
Related to https://bugzilla.redhat.com/show_bug.cgi?id=2057231
This will allow us to use the service accounts which work against
identity.api.openshift.com. These are much easier to manage, especially
with the new multi-tenancy, as there's a single page to create/expire
them across an account.
They also have the added benefit of not expiring automatically when
they're not used like offline tokens, and immediate expiration when
desired.
Previously, the DB was not dumped in case the compose failed. Ensure
that the DB is dumped before the script exits in any case.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Do not create files directly in `/tmp`, but use `$WORKDIR`, which is a
temporary directory for transient files, which gets cleaned up when the
test case finishes. Without this change, running `api.sh` twice fails
the second time.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Most test scripts don't have any documentation regarding it's purpose,
although it can be guessed by the code. There's value in adding this
small comment.
[skip-ci]
This manifest is intended only for internal use and is currently failing
in nightly pipelines. This will be moved to a different test script in
the future, see COMPOSER-1397.
