jobimpl-osbuild
---------------
Add GenericS3Creds to struct
Add method to create AWS with Endpoint for Generic S3 (with its own credentials file)
Move uploading to S3 and result handling to a separate method (along with the special VMDK handling)
adjust the AWS S3 case to the new method
Implement a new case for uploading to a generic S3 service
awscloud
--------
Add wrapper methods for endpoint support
Set the endpoint to the AWS session
Set s3ForcePathStyle to true if endpoint was set
Target
------
Define a new target type for the GenericS3Target and Options
Handle unmarshaling of the target options and result for the Generic S3
Weldr
-----
Add support for only uploading to AWS S3
Define new structures for AWS S3 and Generic S3 (based on AWS S3)
Handle unmarshaling of the providers settings' upload settings
main
----
Add a section in the main config for the Generic S3 service for credentials
If provided pass the credentials file name to the osbuild job implementation
Upload Utility
--------------
Add upload-generic-s3 utility
Makefile
------
Do not fail if the bin directory already exists
Tests
-----
Add test cases for both AWS and a generic S3 server
Add a generic s3_test.sh file for both test cases and add it to the tests RPM spec
Adjust the libvirt test case script to support already created images
GitLabCI - Extend the libvirt test case to include the two new tests
De-duplicate the `qemuStageOptions()` implementation from many distro
implementations and move it to the `osbuild2` package as
`NewQEMUStageOptions()`.
Adjust all affected pipeline implementations.
The `NewQEMUStageOptions()` function takes an optional format-specific
options argument, instead of a dedicated format-specific options (the
previously used qcow2 compat version). This is a preparation to produce
VMDK images with a different than default subformat.
Add unit test for `NewQEMUStageOptions()`.
Support setting a specific VMDK subformat in the type format options.
The required osbuild version in the SPEC file is not bumped, since the
new functionality is currently not used by any image type.
Related to https://github.com/osbuild/osbuild/pull/999
Define a new aliased type `QEMUFormat` for the format type used by the
osbuild2 QEMU stage and define constants for all allowed values.
Use QEMU format type constants in all relevant places, instead of string
literals. Not using string literals minimizes the room for making a
typo.
We want to ensure that cloud images connect to Red Hat[1] independently
of how the content was acquired (PAYG, BYOS, or marketplace).
This auto-registration feature is already enabled for AWS and this
patch enables it for Azure with the same recommended settings:
Services:
rhsmcertd: Enabled (already done, so not changed in the patch)
/etc/rhsm/rhsm.conf:
auto_registration: enabled
auto_registration_interval: 60 (the default, so not explicitly set)
manage_repos: false
The latter value `manage_repos` is left enabled (the default) in case
the user explicitly requested to have the system subscribed, i.e. the
`RHSMConfigWithSubscription` code path.
Regenerate the relevant test manifests and image information.
[1] https://cloud.redhat.com
[2] https://docs.google.com/document/d/1VeZFJxNUlyZMQJh6s3NA3RLvadqATsGxVet6uuP87_4
The `ManifestJobStatus` and `ManifestByIdJobStatus` both
had identical functionality. The `ManifestByIdJobStatus`
is not being referenced anywhere in the codebase and so
this function has been removed.
Implement an error case for empty manifests in the osbuild jobs.
This is already in place in the koji-osbuild job so this change
introduces the same checks in case a job receives an empty manifest
or a job has no manifest at all.
The image type is only ever known (externally) as image-installer.
Renaming the internal variables and functions to reflect the name makes
the code easier to navigate.
The users anaconda module enables users to create user accounts at
install time if one is not already created in the payload. This is
required for the cloud API (Image Builder service) for the image
installer where user customizations are not supported. Without it, user
creation isn't possible on the installed system.
The module also enables user creation at install time through the
kickstart file for both the image-installer and the edge-installer
(Anaconda only).
Therefore, for the image-installer, the users and groups are no longer
created as part of the payload.
This commit adapts the changes from the following commits (originally
made in the RHEL 8.6 and RHEL 9.0 distros) to the rest of the RHEL
distro definitions:
ebc3330cbd5825294dad
Use single NewAnacondaStageOptions() from osbuild2 instead of
implementing in each distro.
Followup from 974b0a434b, copied to the
rest of the RHEL distro definitions.
Use single NewKickstartStageOptions() and replace image-type-specific
implementation from each distro.
- Followup from cb186df208, copied to the
rest of the RHEL distro definitions.
NB: The change was not made in the Fedora distro definitions as they are
currently being rewritten.
Use single NewGroupsStageOptions() from osbuild1 and osbuild2 instead of
implementing in each distro.
- Followup from 2eef6e6e2d, copied to the
rest of the RHEL distro definitions.
- Added NewGroupsStageOptions() to osbuild1 for rhel8 and rhel84.
NB: The change was not made in the Fedora distro definitions as they are
currently being rewritten.
Use single NewUsersStageOptions() from osbuild1 and osbuild2 instead of
implementing in each distro.
- Followup from ca8b371142, copied to the
rest of the RHEL distro definitions.
- Added NewUsersStageOptions() to osbuild1 for rhel8 and rhel84.
NB: The change was not made in the Fedora distro definitions as they are
currently being rewritten.
Followup from, f34380d5b5 and
3a1765a5a8, copied to the rest of the RHEL
distro definitions.
For now, these customizations have no effect on the manifest.
The new `with-users` variants of the edge-installer test cases include
the user customizations in the blueprint, but the manifests are
(currently) the same as the corresponding base cases.
All dependency errors, whether they are 4xx or 5xx,
are currently being considered as a 5xx error in parent
jobs. This is causing some of the build alerts to fire
off when a depsolve job has failed, for example, when
in reality, this is an expected result. This commit
ensures that dependency errors are being reported as
4xx status in monitoring.
When composer exits, it doesn't wait for the manifest generation goroutines
to finish. This is generally a bad practice so let's introduce a bit of
syncing and a new Shutdown method to prevent this.
This also prevents the manifest generation goroutine from creating weird
states when interrupted on a random line of code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I think it's a natural fit, they access some fields from the server so let's
find a new home for them there.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
update koji init & finalize to use custom leveled logging
This is mainly affects logging, but it also changes functionality slightly
since init & finalize are now using the customCheckRetry, they are able
to retry the "TLS timeout" error.
update koji upload to use custom leveled logging, this only affects
logging
since uploading uses a different connection to send the chunks, it is
done separately in this commit
Add support for promoting certain `Debug` log messages to `Info` so we
can monitor them while the logging level set to `Info`, having it set
to Debug is far too noisy.
This allows us to use t.TempDir() everywhere. This was not possible before
because t.TempDir() fails the test if something was written in the temp dir
after the test case is finished. So by waiting on the depsolving goroutine
to finish, we can be sure that nothing is written in the tempdir anymore.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the goroutine was never stopped because it was all the time
calling RequestJob that returned a TimeoutError.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Deduplicate the `rpmStageOptions()` function implemented in every distro
`stage_options.go` to `osbuild2` package.
RHEL-8.4 and code using osbuild v1 manifests are not touched.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
