Commit graph

2157 commits

Author SHA1 Message Date
Yuxin Sun
6cd6e81e85 Add subscription test case and Azure boot up case
Signed-off-by: Yuxin Sun <yuxisun@redhat.com>
2021-06-30 12:29:33 +02:00
Achilleas Koutsou
1e54cd4493 CI: run ostree tests in RHEL 8.5 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
f557bf9a8b test/cases/ostree: test new image types
ostree.sh: add case for RHEL 8.5 edge-commit
ostree-ng.sh: conditional container and installer type and file names
2021-06-29 17:26:43 +01:00
Achilleas Koutsou
034bdaafab distro-arch-imagetype-map: add new distro and types 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
0bac6806f5 tools/provision: set up nightly repos for RHEL 8.5 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
54a811c3f1 schutzbot/mockbuild.sh: RHEL 8.5 support
Add RHEL 8.5 support to the mockbuild script.
2021-06-29 17:26:43 +01:00
Achilleas Koutsou
c8bf77bad8 news: add entry about RHEL 8.5 Edge images 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
0b6452bb34 test: new manifests for RHEL 8.5 edge types 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
779103cb96 rhel85: add tests 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
4651b8ba22 tools: add 8.5 edge types to format request map
For test case generation. These are the new names used by the 8.5 edge
image types.
2021-06-29 17:26:43 +01:00
Achilleas Koutsou
e388706712 store: add new image type mappings and fix typos 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
139c8eb3c4 Revert "rhel85: nolint package for unused private functions"
Pipeline generation functions are now used by the new image definitions.

This reverts commit cca084d5ce.
2021-06-29 17:26:43 +01:00
Achilleas Koutsou
f789f785a4 rhel85: rhel-edge image definitions
Image definitions added for all three edge types:
edge-commit
edge-container
edge-installer

Manifest pipelines generation methods attached to image types.
2021-06-29 17:26:43 +01:00
Xiaofeng Wang
9a7016f451 Remove the subscription check on mock config for RHEL 8.5 2021-06-29 16:14:45 +02:00
Xiaofeng Wang
e00ba2f075 gitlab-ci: Add two new RHEL 8.5 runners on rpmbuild stage
Now CI is able to build RHEL 8.5 RPM package on both x86_64 and
aarch64
2021-06-29 16:14:45 +02:00
Xiaofeng Wang
bc43eab535 Update schutzbot/terraform to point to latest gitlab-ci-terraform
repo commit

The latest gitlab-ci-terraform repo commit includes new runners:
1. openstack/rhel-8.5-x86_64
2. aws/rhel-8.5-x86_64
3. aws/rhel-8.5-aarch64
2021-06-29 16:14:45 +02:00
Ondřej Budai
419ac4c769 ci: remove EXTRA_REPO_PATH_SEGMENT
EXTRA_REPO_PATH_SEGMENT was needed when both Jenkins and Gitlab CI
were running in parallel (so they don't override their artifacts).

Jenkins is now decommissioned so we can drop the variable.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:58:58 +02:00
Ondřej Budai
cc2c354371 tests: regenerate manifests for rhel 9.0
Previous commits bumped the repo snapshots and changed the manifest generation
for RHEL 9.0.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Ondřej Budai
39f1d4d485 tests: update snapshot for rhel 9.0
We have a newer one so let's use it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Ondřej Budai
9bcd219528 distro/rhel90: make selinux stage the last one
There are currently some context mismatches in the image because of the new
files from the sysconfig stage. Let's move the selinux stage to the end so
we fix them.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Ondřej Budai
4c16b2f77e distro/rhel90: include qemu-guest-agent in qcow2
Fixes COMPOSER-990

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Ondřej Budai
c37705e7d5 distro/rhel90: move all package lists to a separate file
There's currently the Package Deletion Process going on in CentOS Stream 9.
When a maintainer wants to get rid of a package, several steps must be taken
before the package is completely removed from all places. We want to include
osbuild-composer in the process so packages that are used here are not deleted
without us noticing it. Thus, maintainers going through the process should be
able to easily see all packages that osbuild-composer includes in images.

This commit moves all package lists from rhel90/distro to rhel90/packages
to allow CentOS Stream maintainers to easily packages required by composer.

Not a functional change.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Ondřej Budai
42e4a5f6ac ci: remove fedora 34 hack from mockbuild
This is apparently not needed anymore, I checked with 20 jobs here:
https://gitlab.com/osbuild/ci/osbuild-composer/-/pipelines/328779200

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 11:53:56 +02:00
Achilleas Koutsou
0ddfc32957 test: verify selected package appears in metadata
Add postgresql to all builds and verify that it's listed in the metadata
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
cfb244ab34 cloudapi: check if metadata is nil before converting 2021-06-29 09:33:05 +01:00
Achilleas Koutsou
51503cc000 cloudapi: collect RPMs from v2 result stages
Fix for v1 results: collect RPMs for main stages, not build.
Support for v2 results: filter out build stages from stage results,
which include stages from all pipelines (flattened).
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
b46addea5c osbuild1: convert stage metadata from v2
While unmarshalling osbuild output and converting from v1 to v2, convert
the metadata as well.

Added checks to successful parsing and conversion test.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
1dcaa26966 osbuild2: add support for reading stage metadata
Metadata from stages in the output from osbuild in the v2 schema are
stored at the top level of the result structure.

Adding the same Metadata types from v1:
- org.osbuild.rpm stage
- org.osbuild.ostree.commit stage
- Raw metadata (for every other stage type)
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
2212fedae3 docs/news: add news item for new feature 2021-06-29 09:33:05 +01:00
Achilleas Koutsou
7e01a9125b test: verify commit ID hash returned from API 2021-06-29 09:33:05 +01:00
Achilleas Koutsou
dce3e1de9e distro: test manifest version detection 2021-06-29 09:33:05 +01:00
Achilleas Koutsou
39e646ae68 cloudapi: return ostree commit ID with metadata
When the assembler (v1) or any stage (v2) has type ostree.commit,
retrieve the commit ID (hash) from the osbuild stage metadata and return
it with the rest of the metadata (package NEVRAs).
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
e87394fea1 distro: method for detecting manifest version
Small helper method for distro.Manifest that can detect the schema
version of the Manifest.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
72f0f7ffed osbuild1: store all stage result metadata
Currently, metadata from osbuild is discarded for all stages except RPM.
Adding explicit support for the ostree.commit stage/assembler and
storing the metadata in a known type.
For all other stages, store the metadata directly without parsing.

The rawAssemblerResult is removed. Assembler results are treated as
stage results.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
df5921403d cloudapi: add compose/id/metadata endpoint
Returns NEVRAs, compiled from the job manifest.
Will also return the ostree commit ID for edge commit types.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
6b3920783f rpmmd: move RPM metadata tooling to internal pkg
Move the OSBuildStagesToRPMs function, associated test, and RPM type
from the worker into the rpmmd subpackge. We will use this function in
the cloud API to compile the NEVRAs for the new metadata endpoint.
2021-06-29 09:33:05 +01:00
Jakub Rusz
b1e40b5ce7 tests: modify image_tests for Gitlab CI 2021-06-28 13:49:57 +03:00
Ondřej Budai
579a5df698 upload/aws: add support for session tokens
If a user uses a temporary access key for login, a session token is also
needed.

This commit adds support for it to the internal aws library and also
to the osbuild-upload-aws helper. Note that this doesn't affect the main
osbuild-composer executable nor the worker. Everything here should work
as before and session tokens are not supported. Something for a follow up
if anyone needs it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-28 13:14:19 +03:00
Jakub Rusz
bb2f866470 tests: updating README with info about nightly testing in Gitlab CI 2021-06-25 15:10:40 +03:00
Jakub Rusz
b6c579cbe3 tests: Enable use of custom osbuild-composer repo
This enables the user to specify custom REPO_URL for use during
internal RHEL build testing as ENV in scheduled pipelines.
It also gives the option to use osbuild build specified in
Schutzfile or from the nightly build repository.
2021-06-25 15:10:40 +03:00
Jakub Rusz
905c297bc7 tests: enable nightly test execution in gitlab CI
This adds several stages to the gitlab-ci.yaml file which are
triggered by a schedule specified in Gitlab->CI/CD->Schedules
2021-06-25 15:10:40 +03:00
Jakub Rusz
beead86da6 tests: update cloud-cleaner
In 0680214c9b20b76f82c381fb3472d83846ca0c71 the BUILD_ID was changed
in azure.sh test but not in cloud_cleaner causing cloud-cleaner to
not clean up properly. This fixes that.
2021-06-25 14:20:07 +03:00
Yuxin Sun
704a3675b9 Update fedora-33 test repo
Signed-off-by: Yuxin Sun <yuxisun@redhat.com>
2021-06-25 08:40:49 +03:00
Yuxin Sun
23cf70a0d7 Remove Fedora 33 ostree test
Signed-off-by: Yuxin Sun <yuxisun@redhat.com>
2021-06-24 12:12:26 +02:00
sanne
6fac4dcb70 distribution: Add cache dir to composer clouddot deployment 2021-06-23 10:33:22 +02:00
sanne
4f86b4fd45 worker: Use http.PostForm to post data
Avoid having to encode the data ourselves.
2021-06-23 10:33:22 +02:00
sanne
acf88309c2 worker: Trim offline token 2021-06-23 10:33:22 +02:00
sanne
9161553170 worker: Tidy up help message 2021-06-23 10:33:22 +02:00
Ondřej Budai
86baf4879a tools/deploy-qemu: use sys.exit instead of exit
exit is just a helper for the interactive shell, see:

https://stackoverflow.com/questions/6501121/difference-between-exit-and-sys-exit-in-python
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-22 14:45:54 +02:00
Ondřej Budai
99d1d4e5d0 tools/deploy-qemu: use default temp dir location
Previously, the temporary directory was created in the script dir. It just
imo created unnecessary and confusing files in my git checkout. Let's just
use the default temporary directory because we don't really have special
needs for it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-22 14:45:54 +02:00