When composer exits, it doesn't wait for the manifest generation goroutines
to finish. This is generally a bad practice so let's introduce a bit of
syncing and a new Shutdown method to prevent this.
This also prevents the manifest generation goroutine from creating weird
states when interrupted on a random line of code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I think it's a natural fit, they access some fields from the server so let's
find a new home for them there.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
update koji init & finalize to use custom leveled logging
This is mainly affects logging, but it also changes functionality slightly
since init & finalize are now using the customCheckRetry, they are able
to retry the "TLS timeout" error.
update koji upload to use custom leveled logging, this only affects
logging
since uploading uses a different connection to send the chunks, it is
done separately in this commit
Add support for promoting certain `Debug` log messages to `Info` so we
can monitor them while the logging level set to `Info`, having it set
to Debug is far too noisy.
This allows us to use t.TempDir() everywhere. This was not possible before
because t.TempDir() fails the test if something was written in the temp dir
after the test case is finished. So by waiting on the depsolving goroutine
to finish, we can be sure that nothing is written in the tempdir anymore.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the goroutine was never stopped because it was all the time
calling RequestJob that returned a TimeoutError.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Deduplicate the `rpmStageOptions()` function implemented in every distro
`stage_options.go` to `osbuild2` package.
RHEL-8.4 and code using osbuild v1 manifests are not touched.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Regression test suite has grown considerably and is taking too long to
run with a single wrapper. Splitting them into individual standalone
tests instead and making them run in parallel.
The VMDK image must be in stream-optimized format in order to be
imported to VSphere. osbuild-composer does not produce VMDK by default
as stream-optimized. Instead, it is converted on the fly when the image
build job has been submitted via Weldr API.
Since we are aiming mainly for the VSphere use case with the VMDK image
in the service, the image should be ready for importing to VSphere.
Implement a temporary workaround for the Cloud API and AWS S3 target to
upload stream-optimized VMDK image.
Adjust the `api.sh` test case to not convert the VMDK image downloaded
form S3, before importing it to VSphere.
Ensure that the content of the database is not printed to the console
when dumped at the end of the test case. The output is still preserved
as a CI run artifact.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Kill and remove the DB container as part of the test case cleanup.
Without this change, running the test case more than once fails.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `api.sh` test to verify the VMDK images uploaded to S3 in
VSphere by booting them and configuring using cloud-init.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Install `cloud-init` by default on the VMDK image on RHEL-85/86/90 and
as a result also CentOS Stream 8/9.
Regenerate image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
test cases fixup
RHEL 9.0 will ship a brand new auxiliary key. Let's use it everywhere in our
RHEL 9 stuff. Taken from current RHEL 9.0's redhat-release package.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We should honour `pkg.CheckGPG` when creating the file inputs for the
rpm stage. This was lost in the transition from v1 to v2 manifests.
Regenerate image test manifests.
Co-authored-by: Tomas Hozza <thozza@redhat.com>
Signed-off-by: Tomas Hozza <thozza@redhat.com>
If a kernel is specified we already set the `SavedEntry`. To preserve
that the saved entry setting will survive grub2 config regeneration
we also need to set `Config.Default` to "saved". Do so. All older
pipelines have been prepared to reset this value to preserve the old
behavior. All new pipelines have been prepared to not overwrite this
setting when `ImageConfig.Grub2Config` is applied.
The `grub2.Config.Default` key is owned by `NewGrub2StageOptionsUnified`
and thus needs to be preserved in case there is a `Grub2Config` setting
present in the image configuration (`ImageConfig`).
The `grub2.Config.Default` key is owned by `NewGrub2StageOptions`
and thus needs to be preserved in case there is a `Grub2Config`
setting present in the image configuration (`ImageConfig`).
A future change to `NewGrub2StageOptions` will result in a change of
`Config.Default` to `saved` if a kernel is set. Preserve the current
behavior by manually resetting it to the empty string.
A future change to `NewGrub2StageOptions` will result in a change of
`Config.Default` to `saved` if a kernel is set. Preserve the current
behavior by manually resetting it to the empty string.
Add a new debugging / development tool `osbuild-package-sets` for
printing JSON object with all package sets of a specific distro x arch x
image type combination.
This is useful, since due to the way package sets are implemented in
composer, the actual package set of a vanilla image type is very
difficult to determine just by looking at the code.
Example usage:
`go run cmd/osbuild-package-sets/main.go -distro rhel-90 -arch x86_64
-image qcow2`
Remove comment out code
Use three different IP address for different test scenarios
Move /boot/device-credentials file checking into playbook
Some shell script improvements
We are running into a GitLab CI limitation:
jobs:integration:parallel:matrix config generates too many jobs (maximum is 50)
Let's split these jobs into separate ones.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The nightly compose (currently) only has a 'latest' directory for 8.7.
Switching to the development composes which have 'latest' for 8.4
onwards.
Enable -x in script for easier troubleshooting.
If a home directory has a trailing slash, the `useradd` command fails to
set the correct selinux contexts for the home directory on creation.
This can lead to various issues, but the one that we came across was
that the ~/.ssh directory and authorized_keys file cannot be read by
sshd and we couldn't log in to the system.
This only manifests if the user is created through the kickstart file
because:
1. `useradd` does not set the selinux contexts when creating the
directory
2. Anaconda runs `restorecon` on the home directory and authorized_keys
file when it creates them, but uses the install-time mount path
`/mnt/sysroot/...` for which selinux does not have contexts.
In most cases we get around this bug because we run `setfiles` on the
tree at the end of our pipelines.
For the ostree case, the relabeling in Anaconda is done correctly.
Enable the user module unconditionally for the image-installer:
- If users are specified for the kickstart file, the module is required
to set up the users.
- If no users are specified, the module can be used at install time to
create users.
Updated relevant test cases (manifests).
Use single NewAnacondaStageOptions() from osbuild2 instead of
implementing in each distro.
The new function conditionally adds the user module when there are users
that need to be created at install time (image- and edge-installers).
