Commit graph

3765 commits

Author SHA1 Message Date
Jakub Rusz
e1bb8ab47c tests/ci: split regression tests
Regression test suite has grown considerably and is taking too long to
run with a single wrapper. Splitting them into individual standalone
tests instead and making them run in parallel.
2022-04-04 12:18:53 +02:00
Jakub Rusz
b782b6cced tools/define-compose-url: change url back to nightly
The latest nightly link for 8.6 was missing only by accident, changing
back as it's fixed already.
2022-04-04 12:11:59 +02:00
Tomas Hozza
72019740c2 Cloud API: upload stream-optimized VMDK to S3
The VMDK image must be in stream-optimized format in order to be
imported to VSphere. osbuild-composer does not produce VMDK by default
as stream-optimized. Instead, it is converted on the fly when the image
build job has been submitted via Weldr API.

Since we are aiming mainly for the VSphere use case with the VMDK image
in the service, the image should be ready for importing to VSphere.

Implement a temporary workaround for the Cloud API and AWS S3 target to
upload stream-optimized VMDK image.

Adjust the `api.sh` test case to not convert the VMDK image downloaded
form S3, before importing it to VSphere.
2022-04-02 08:47:02 +01:00
Tomas Hozza
fa1424e724 Worker: explicitly close opened file after uploading it to the server 2022-04-02 08:47:02 +01:00
Tomas Hozza
c98b3afcc4 Add comments to some values in target.go 2022-04-02 08:47:02 +01:00
Tomas Hozza
3bf62ca789 CI: test Cloud API with vsphere image
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-04-02 08:47:02 +01:00
Tomas Hozza
5acaaabc1d test/api.sh: do not print to stdout when dumping the DB
Ensure that the content of the database is not printed to the console
when dumped at the end of the test case. The output is still preserved
as a CI run artifact.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-04-02 08:47:02 +01:00
Tomas Hozza
7a44d00cec test/api.sh: cleanup the DB container
Kill and remove the DB container as part of the test case cleanup.
Without this change, running the test case more than once fails.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-04-02 08:47:02 +01:00
Tomas Hozza
27c260daaf test/api.sh: verify VMDK images from S3 in VSphere
Extend the `api.sh` test to verify the VMDK images uploaded to S3 in
VSphere by booting them and configuring using cloud-init.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-04-02 08:47:02 +01:00
Tomas Hozza
c6507c21b9 distro: add cloud-init to VMDK image on RHEL-85/86/90
Install `cloud-init` by default on the VMDK image on RHEL-85/86/90 and
as a result also CentOS Stream 8/9.

Regenerate image test cases.

Signed-off-by: Tomas Hozza <thozza@redhat.com>

test cases fixup
2022-04-02 08:47:02 +01:00
Ondřej Budai
fd49ed9a07 repositories: update key for RHEL 9.0
RHEL 9.0 will ship a brand new auxiliary key. Let's use it everywhere in our
RHEL 9 stuff. Taken from current RHEL 9.0's redhat-release package.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-04-01 19:22:23 +02:00
Jakub Rusz
b81e3f28fb Schutzfile: add centos-8 pin + update structure
It is now possible to specify mutliple repository files, updating the
structure to enable that + adding centos-stream 8 pin.
2022-03-31 18:10:23 +02:00
Xiaofeng Wang
87679fcab2 test: Add retries on ubi8 image and greenboot package downloading
To fix issue: https://github.com/osbuild/osbuild-composer/issues/2492,
retry 30 times when downloading ubi8 image and greenboot package.
2022-03-30 16:35:11 +02:00
Christian Kellner
45b1fc3cd1 osbuild2: honor GPG key setting for rpm inputs
We should honour `pkg.CheckGPG` when creating the file inputs for the
rpm stage. This was lost in the transition from v1 to v2 manifests.

Regenerate image test manifests.

Co-authored-by: Tomas Hozza <thozza@redhat.com>
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-30 14:32:28 +02:00
Tomas Hozza
947acf74ab Regenerate affected image test cases 2022-03-29 11:43:54 +02:00
Christian Kellner
d29325164d osbuild2: set default boot entry to saved if a kernel is set
If a kernel is specified we already set the `SavedEntry`. To preserve
that the saved entry setting will survive grub2 config regeneration
we also need to set `Config.Default` to "saved". Do so. All older
pipelines have been prepared to reset this value to preserve the old
behavior. All new pipelines have been prepared to not overwrite this
setting when `ImageConfig.Grub2Config` is applied.
2022-03-29 11:43:54 +02:00
Christian Kellner
61c1ccff21 distro/rhel90: ensure grub2.Config.Default is preserved
The `grub2.Config.Default` key is owned by `NewGrub2StageOptionsUnified`
and thus needs to be preserved in case there is a `Grub2Config` setting
present in the image configuration (`ImageConfig`).
2022-03-29 11:43:54 +02:00
Christian Kellner
55b7b82714 distro/rhel86: ensure grub2.Config.Default is preserved
The `grub2.Config.Default` key is owned by `NewGrub2StageOptions`
and thus needs to be preserved in case there is a `Grub2Config`
setting present in the image configuration (`ImageConfig`).
2022-03-29 11:43:54 +02:00
Christian Kellner
96411158e6 distro/rhel90beta: ensure grub default is not set
A future change to `NewGrub2StageOptions` will result in a change of
`Config.Default` to `saved` if a kernel is set. Preserve the current
behavior by manually resetting it to the empty string.
2022-03-29 11:43:54 +02:00
Christian Kellner
111cd8871f distro/rhel85: ensure grub default is not set
A future change to `NewGrub2StageOptions` will result in a change of
`Config.Default` to `saved` if a kernel is set. Preserve the current
behavior by manually resetting it to the empty string.
2022-03-29 11:43:54 +02:00
Christian Kellner
fa8f01543d osbuild2: expose config.default option
This option was added in osbuild 52 and controls the default boot
entry setting via `/etc/default/grub`.
2022-03-29 11:43:54 +02:00
Tomas Hozza
8ab8013535 cmd: add osbuild-package-sets for printing package sets of an image
Add a new debugging / development tool `osbuild-package-sets` for
printing JSON object with all package sets of a specific distro x arch x
image type combination.

This is useful, since due to the way package sets are implemented in
composer, the actual package set of a vanilla image type is very
difficult to determine just by looking at the code.

Example usage:
`go run cmd/osbuild-package-sets/main.go -distro rhel-90 -arch x86_64
-image qcow2`
2022-03-29 11:36:52 +02:00
Alexander Todorov
d117b84dc3 tests: Run SonarQube analysis only on main branch 2022-03-29 11:35:47 +02:00
Xiaofeng Wang
21b6e95136 test: Clean up and improve ostree-simplified-installer.sh
Remove comment out code
Use three different IP address for different test scenarios
Move /boot/device-credentials file checking into playbook
Some shell script improvements
2022-03-29 08:09:37 +02:00
Jakub Rusz
55a3556e9f test/ostree-simplified-installer: destroy VM fixup
There was another virt-install added recently and it was missed in
1d51c0764936902c681283a3489845ef660fef5e
2022-03-29 08:01:08 +02:00
Ondřej Budai
0eca6b4027 gitlab: split integration tests
We are running into a GitLab CI limitation:

jobs:integration:parallel:matrix config generates too many jobs (maximum is 50)

Let's split these jobs into separate ones.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-28 21:22:48 +02:00
Achilleas Koutsou
5261987726 define-compose-url: change to development for 8.x
The nightly compose (currently) only has a 'latest' directory for 8.7.
Switching to the development composes which have 'latest' for 8.4
onwards.

Enable -x in script for easier troubleshooting.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
f2849e2165 blueprint: sanitize user home directories
If a home directory has a trailing slash, the `useradd` command fails to
set the correct selinux contexts for the home directory on creation.
This can lead to various issues, but the one that we came across was
that the ~/.ssh directory and authorized_keys file cannot be read by
sshd and we couldn't log in to the system.

This only manifests if the user is created through the kickstart file
because:
1. `useradd` does not set the selinux contexts when creating the
   directory
2. Anaconda runs `restorecon` on the home directory and authorized_keys
   file when it creates them, but uses the install-time mount path
   `/mnt/sysroot/...` for which selinux does not have contexts.

In most cases we get around this bug because we run `setfiles` on the
tree at the end of our pipelines.
For the ostree case, the relabeling in Anaconda is done correctly.
2022-03-28 14:08:57 +02:00
Xiaofeng Wang
15a135fcbb test: Remove RHEL 8.4 test and not used configurations
CI will not run RHEL 8.4 test any more, remove it.
2022-03-28 14:08:57 +02:00
Xiaofeng Wang
75a3426515 test: Add a new user in installer bp and test new user created 2022-03-28 14:08:57 +02:00
Achilleas Koutsou
14aa3b9f54 test: save installer test manifest and build log
Same as with ostree tests.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
5825294dad distro/rhel86+90: enable the anaconda user modeule for image-installer
Enable the user module unconditionally for the image-installer:
- If users are specified for the kickstart file, the module is required
  to set up the users.
- If no users are specified, the module can be used at install time to
  create users.

Updated relevant test cases (manifests).
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
fe514b9c2e test: create new test cases (manifests)
New test cases for edge- and image-installer with users.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
3ac1f940a4 format-request-map: add installers with users
Add two new test cases:
- image-installer-with-users
- edge-installer-with-users
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
ebc3330cbd distro/rhel86+90: don't create users in the payload for image-installer
Users are created at install time now.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
974b0a434b osbuild2: deduplicate anacondaStageOptions()
Use single NewAnacondaStageOptions() from osbuild2 instead of
implementing in each distro.

The new function conditionally adds the user module when there are users
that need to be created at install time (image- and edge-installers).
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
cb186df208 obsuild2: deduplicate *kickstartStageOptions()
Use single NewKickstartStageOptions() and replace image-type-specific
implementations from each distro.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
2eef6e6e2d osbuild2: deduplicate groupStageOptions()
Use single NewGroupsStageOptions() from osbuild2 instead of implementing
in each distro.

The new function does not set the Group.Name field anymore.  The field
does not exist in the osbuild schema and was silently ignored.
The field in the stage has been marked 'omitempty' and the relevant
manifests have been updated.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
ca8b371142 osbuild2: deduplicate userStageOptions()
Use single NewUsersStageOptions() from osbuild2 instead of implementing
in each distro.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
d022a23b8c osbuild2: add Users and Groups to kickstart stage
Supported by the osbuild stage for a while now.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
db2910fed9 distro/rhel86+90: common error message for unsupported customizations
Use the same error message for both edge-installer and
edge-simplified-installer when an unsupported customization is added.
2022-03-28 14:08:57 +02:00
Achilleas Koutsou
3a1765a5a8 distro/rhel90: allow user and group customizations for edge-installer 2022-03-28 14:08:57 +02:00
Achilleas Koutsou
f34380d5b5 distro/rhel86: allow user and group customizations for edge-installer 2022-03-28 14:08:57 +02:00
Sanne Raymaekers
11890682b7 templates/composer: Drop unused variables 2022-03-28 12:02:37 +02:00
Sanne Raymaekers
eba355bb60 templates/composer: Remove unused acl claims
This leaves fedora and consoledot tenants.
2022-03-28 11:38:48 +02:00
Alexander Todorov
7ea3920cdf Add Xiaofeng to notifications for nightly pipeline 2022-03-28 11:30:44 +02:00
Diaa Sami
68639b4bf9 koji: increment retry counter only when retrying 2022-03-26 09:33:36 +01:00
Diaa Sami
6b08b8ed63 koji: don't decrement retry counter on the first call
After examining the logic of retryablehttp library, the callback does not happen for the first HTTP call, so no need to decrement when counting.
2022-03-26 09:33:36 +01:00
Diaa Sami
3496efe70d koji: initialize retryable client properly
Previously used client has MaxRetries of zero, so was not effectively
retrying
Fixes COMPOSER-1420
2022-03-26 09:33:36 +01:00
Jakub Rusz
f1006c5adb ci/tests: re-enable Installers.sh test
Related bug has been fixed. Updating snapshots and enabling the test.
2022-03-25 16:00:39 +01:00