Commit graph

2922 commits

Author SHA1 Message Date
Achilleas Koutsou
9eff6f1e95 osbuild2: v2 Result writer
Implementing writer for osbuild2 Result type.
Since Go maps don't have stable ordering, sorting by the pipeline name
provides stable output.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-16 09:49:37 +01:00
Achilleas Koutsou
2004c71f89 cloudapi: use osbuild v2 result struct to extract metadata
Reading stage metadata using osbuild's v2 result format.
For RPM stages we only want the core (OS) RPMs (not the build root
RPMs). Skip the build pipeline by name, but this should be handled
better since names are arbitrary.

Using type switch to convert metadata types instead of relying on the
type string of the stage result.

The rpmmd helper function isn't used anymore since that requires two
conversion passes (osbuild.StageMetadata -> rpmmd.RPM ->
cloudapi.PackageMetadata).

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-16 09:49:37 +01:00
Achilleas Koutsou
9dff17e172 rpmmd: convert from v2 result metadata
Function renamed to better fit the argument element type
(StageMetadata).
Argument is a map to fit the pipeline metadata in the result object.
Signature function is made public to be reused in the cloud API
conversion.

Metadata test raw value updated to v2 result format.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-16 09:49:37 +01:00
sanne
6757916c54 worker: Introduce manifest-id-only job
A job intended to run in composer itself, after which a dependant
osbuild job can parse the manifest from it's dynamic arguments.
2021-11-15 16:04:12 +01:00
Ondřej Budai
d3a3dbafed jobqueue: add DequeueByID
We will soon need to dequeue a job using its ID. This commit adds ability
to do that to the Jobqueue interface. As always, the fsjobqueue implementation
is slightly naive but it should fine for the usecases that it's designed for.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-14 10:17:03 +01:00
Ondřej Budai
2ecc48727f fsjobqueue: factor out finished deps check
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-14 10:17:03 +01:00
Ondřej Budai
5f4db72777 fsjobqueue: do not delete empty channels
Previously, we deleted empty channels when a job was dequeued. This is
completely wrong because there still might be some clients waiting for
a job. This commit removes the cleanup and adds a regression test.

Note that this has the potential to leak memory if we ever use a lot of
job types. Currently, we have just handful of them, so this is fine.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-14 10:17:03 +01:00
Martin Sehnoutka
a880c9c019 osbuild2: new stage yum config
This stage was introduced in osbuild 41. Add support into
osbuild-composer and a test using test data from osbuild repo.
2021-11-12 20:27:35 +01:00
Martin Sehnoutka
b159d04af7 osbuild2: new stage pwquality conf
This stage was introduced in osbuild 41. Add support into
osbuild-composer and a test using test data from osbuild repo.
2021-11-12 20:27:35 +01:00
Martin Sehnoutka
59be127daf osbuild2: new stage authconfig
This stage was introduced in osbuild 41. Add support into
osbuild-composer.
2021-11-12 20:27:35 +01:00
Xiaofeng Wang
85df63ce52 test: Use YAML as Ansible output format 2021-11-12 14:43:55 +01:00
Alexander Todorov
87b4a95e09 ci: Send results to Coverity Scan daily
https://scan.coverity.com/projects/osbuild-osbuild-composer
2021-11-12 14:20:15 +01:00
Gianluca Zuccarelli
44017890ca containers: mock oauth container
Add a mock oauth container to simulate
the openshift SSO offline_token
2021-11-12 14:07:13 +01:00
Martin Sehnoutka
af9cca1b50 osbuild2: new stage sshd config
This stage was introduced in osbuild 41. Add support into
osbuild-composer and a test using test data from osbuild repo.
2021-11-12 11:40:13 +01:00
Simon Steinbeiss
f9e2924e00 packit: Use upstream github release description
Setting this option enables packit to use the Github upstream release
description for the specfile instead of just using all commit messages.
2021-11-12 08:48:47 +01:00
Juan Abia
0bc6ea1172 schuzbot: clean aws unused resources
list all unused resources on ec2 and remove them.

each resource originated by test scripts should be tagged with
"gitlab-ci-test"
2021-11-11 15:42:32 +01:00
Juan Abia
4c4e26d996 test/cases: use the same naming pattern on every test
make api.sh and was.sh use the same naming pattern, TEST_ID.
2021-11-11 15:42:32 +01:00
Xiaofeng Wang
ab3712fd92 test: Update ostree test scripts to support different ostree refs 2021-11-11 12:11:01 +01:00
diaasami
bb190f1280 Post release version bump
[skip ci]
2021-11-10 16:37:05 +00:00
Sanne Raymaekers
1fdc18856a Revert "templates: Add prometheus scrape annotations to composer-api"
This reverts commit 7f86dae69b.
2021-11-10 15:24:24 +01:00
sanne
7f86dae69b templates: Add prometheus scrape annotations to composer-api 2021-11-10 15:13:53 +01:00
Achilleas Koutsou
155a887bd1 distroregistry: disable CentOS Stream 9
CS9 is untested and probably doesn't fully work now. Disabling until it
is ready to be fully verified.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
89f0613f3d test/cases: rm ostree-ng-og.sh
Obsolete test script
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
d4f801c48e distro/rhel90: remove all mentions of obsolete firmware packages
Removed packages:
- iwl3945-firmware
- iwl4965-firmware
- iwl6000-firmware
- libertas-sd8686-firmware
- libertas-usb8388-firmware
- libertas-usb8388-olpc-firmware

Source: https://issues.redhat.com/browse/ENGCMP-1287
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
e02911bc45 test/data: update manifests for edge-container images 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
82cf71c5a1 distro/rhel90: make nginx log and lib directories world writable
Previously, we only needed the log directory to be writeable.
In newer versions of nginx, it also needs to create directories in
/var/lib/nginx, so we make that directory writeable and traversable as
well.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
a8eb58bc62 distro/rhel90: disable edge-simplified-installer image type
Critical dependency is not available yet
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
9f4c5b08e3 Schutzfile: remove osbuild version pin for RHEL 9.0 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
5786db9caa tools: update distro-arch-imagetype-map for RHEL 9.0 types
`rhel-90-ga` no longer valid.  Now called `rhel-90`.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
0c185eca7a tools: update RHEL 9.0 repos for test case generators
No more `rhel-90-ga`.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
2cd3c71370 test/data: update RHEL 9.0 and beta manifests
Distro name is now `rhel-90` (was `rhel-90-ga`)
Beta distro name is `rhel-90-beta` (was `rhel-90`)
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
90219720d4 test/ostree-simplified: variable string fixes
- All variants use the same image type names
- OSTREE_REF differs for each case
- Use OS_VARIANT in virt-install command
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
fa18ba0431 distro/rhel90: update unit tests 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
5765d5af9a rhel90: drop -ga suffix and alias from beta
`rhel-90` now refers to the RHEL 9.0 distro (rhel90 package) and the
beta is only referred to by `rhel-90-beta`.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
affe02ceed test/api: ssh key fixes for RHEL 9.0
RHEL 9.0 AWS API test is failing with

  Host key verification failed.

This is probably due to a recent change in openssh deprecating rsa host
keys (or likely rsa keys in general).
- turn off StrictHostKeyChecking when checking groups
- use 'ed25519' type for user ssh keys

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
8ef0cfe618 test/cases: support weldr-client output structure
Using get_build_info() function in all scripts where both composer-cli
and weldr-client need to be supported for consistency.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
4e4f55b058 CI: test new edge types on RHEL 9.0
Edge Raw Image and Edge Simplified Installer are supported on
RHEL 9.0 GA.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
2f30823b41 test/ostree: install python3 instead of specific version
Better cross-distro compatibility

Same as bdc428fbb7 but for all the tests.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
0b41cd8003 mockbuild: change RHEL 9 template to use latest non-beta repos 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
a025601b85 test/cases: add RHEL 9.0 and CentOS 9 cases to test scripts 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
1dd139e758 CI: enable tests for RHEL 9.0 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
ad896fb69e schutzbot: update terraform sha 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
7e9f1f554e test: update test manifests for rhel-90-ga
Changes from previous (RHEL 9.0 Beta):
- Edge container: Switched to nginx from apache. Change in pipeline and
  package sets.
- All bootable x86_64: Packages for hybrid boot added (efibootmgr).
- Edge commit and edge container no longer require explicitly enabling
  greenboot services.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
cba73ee9e3 tools: add centos-9 to distro-arch-imagetype-map 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
994dd1e136 tools: copy rhel-90 distro-arch-imagetype-map to -beta and -ga 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
6b73dc5a92 distro/rhel90: update to match 8.6 and add centos-9 alias
- Copied distro and image definitions from RHEL 8.6
    - New package set handling
    - Distribution-specific strings
    - New image types: edge-raw and edge-simplified-installer
    - Edge container with nginx
    - Removed greenboot services from edge enabled services (enabled
      automatically)
- Adapted to required changes from RHEL 9.0 Beta
- Added CentOS 9 (centos-9) alias

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
ba4a4541d9 distro/rhel86: remove genisoimage
Not required.
Discovered it's unneeded when it was removed from RHEL 9.0.
Updating RHEL 8.6 to keep clean and consistent.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
c7cab92e88 distro/rhel86: single osbuild import
osbuild2 was imported twice, once with its own name and once aliased as
plain osbuild.
Use only the plain osbuild alias.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
c8465e3161 test/data: copy rhel-90 test manifests to -ga and -beta
All three "distros" share the same code for now. Adding manifests to
track changes before updating the RHEL 9.0 GA definitions.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
bbf825ef12 test: add repositories for rhel-90-beta and -ga
- Updated repos to use latest nightly (20211105)
- Added rt repository
- Both -ga and -beta are identical
- rhel-90.json repo file is a symlink to rhel-90-beta.json

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00