Commit graph

2800 commits

Author SHA1 Message Date
Achilleas Koutsou
4e5ce288ab 36
Release osbuild-composer 36

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-13 15:52:08 +01:00
Sanne Raymaekers
c65b1e9b26 Revert "cloudapi/v1: Move depsolving to workers"
This reverts commit 0f44250237.
2021-10-13 12:01:51 +02:00
sanne
cb5acbac74 worker: Prefix https always
Also, with oauth, add a root CA if present.
2021-10-12 17:59:59 +01:00
sanne
0f44250237 cloudapi/v1: Move depsolving to workers
We can't enqueue a job without a worker doing the depsolve step now, so
adapt the tests as well.
2021-10-12 14:21:03 +01:00
Achilleas Koutsou
395a22aa21 cloudapi/v2: fix newV2Server() call in test
Bug caused by two consecutive PR merge rebases, one that added a
function call and another that changed the signature of the same
function.
2021-10-11 23:48:26 +02:00
sanne
93dec413f3 templates: Name service ports 2021-10-11 22:41:36 +01:00
Tomas Hozza
46b8a2e977 SPEC: Exclude armv7hl architecture
Exclude the `armv7hl` architecture, since osbuild-composer does not
support building images for it.

Fix #1839

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-11 15:33:53 +02:00
Achilleas Koutsou
cac3fa10db distro/rhel90 -> distro/rhel90beta: rename package
- Rename rhel90 package to rhel90beta.
- Adjust all imports
- Adjust tests
2021-10-11 13:29:57 +02:00
Achilleas Koutsou
972b9ee14e distro/rhel90: rename to rhel-90-beta and alias base name 2021-10-11 13:29:57 +02:00
sanne
b81d58b895 spec: Split dnf-json into a subpackage 2021-10-11 13:16:51 +02:00
Ondřej Budai
e904397fdb cloudapi/v2: Use worker to depsolve
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-11 13:16:51 +02:00
Tom Gundersen
0f90aa9c78 worker: Add a depsolve job type
Allow depsolving to be done in a worker through the job queue rather
than synchronously in composer.

The benefit this might unlock include:
 - no more blocking calls in the cloud/koji APIs
 - only workers accessing repositoires
   - no VPN access from composer
   - composer not needing to be subscribed to CDN, etc
 - no dnf cache managment in composer

Potential problems:
 - the version of composer (so the distro definitions) that
   triggered a depsolve, may not be the same that uses the
   result to generate a manfiset

Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-11 13:16:51 +02:00
Simon Steinbeiss
e2f6dd7965 packit: Push downstream instead of creating PR 2021-10-11 10:23:30 +02:00
sanne
2c92473936 templates: Name services after endpoints 2021-10-11 09:52:21 +02:00
sanne
ce7ac9a756 worker: Make BasePath configurable 2021-10-11 09:52:21 +02:00
sanne
6ce20a9ef6 cloudapi/v2: Listen on /api/image-builder-composer/v2 2021-10-11 09:52:21 +02:00
Achilleas Koutsou
01d920ac01 test/ostree: remove --ip-range from podman network
Bug in podman causes the command to fail with this flag. Not really
needed.
2021-10-11 09:49:15 +02:00
Tomas Hozza
eac6b2c083 RHEL-8.6: add support for official EC2 SAP image
Add support for building the official RHEL EC2 SAP image for RHEL-8.6.
The new image type is available under the `ec2-sap` name. Since the
image type's package set includes RHUI client RPMs, which are not
available publicly, it is by default not exposed through the Weldr API.

Extend the relevant files and definitions used for generating image test
cases. This includes addition of repository definitions needed by the
image type, specifically: `sap`, `saphana` and `ansible`.

Generate image test case for the new image type.

Add News entry for the new image type.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
46fc9b66c5 tools/provision: set up nightly repos for RHEL 8.6
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
be16a6e502 test/cases: add support for RHEL 8.6 to test scripts
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
97f9d02d92 schutzbot: pin osbuild to current main for 8.6
First osbuild mock builds for 8.6

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
ed9e57dc98 schutzbot: update terraform sha
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
3609c1b54e news: add entry about RHEL 8.6
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
1cff6c7dd2 test: add RHEL 8.6 test manifests
Add images to distro-arch-imagetype map and generate all test cases.
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
29ef7efe1b test: add rpmrepo snapshots for RHEL 8.6
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
71c7c3ed87 ci: run all tests on RHEL 8.6
For every test that runs on 8.5, run it on 8.6 as well.
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
0a818641a9 test/data/repositories: add test repos for rhel-86 2021-10-11 09:49:15 +02:00
Achilleas Koutsou
c3077ede4a distro/rhel85: remove rhel86 alias
Remove alias for RHEL 8.6 from RHEL 8.5 and update the registry to point
to the new rhel86 subpackage.
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
513b7f94b9 distro/rhel86: copy all definitions from rhel85
Adjust strings from 85, 8.5, 8-5, to 86, 8.6, 8-6.
2021-10-11 09:49:15 +02:00
sanne
4eca54b6ed cloudapi/v2: Configurable aws bucket
Fixes #1855
2021-10-09 00:55:09 +01:00
sanne
b6c08f3056 cloudapi/v2: Replace upload types with image types
Because there's only a few combinations of upload types and image types
that make sense, enforce correct combinations by eliminating upload
types.

Fixes #1775
2021-10-09 00:55:09 +01:00
Tomas Hozza
510bd1d221 Tests/RHEL-9.0: add EC2 SAP image test
Because the RHUI client RPMs for RHEL-9 don't exist yet, this test
case uses the RHUI client RPMs for RHEL-8 to satisfy the required
package set. These RPMs contain the entitlement certificate and repo
definitions for RHEL-8 AWS RHUI repos, but RHUI will not work without
the image being imported with the proper Billing Code to AWS. So these
installed RHUI RPMs are technically useless on the RHEL-9 image, but
this approach is good enough to allow us to at least build and test
the image from configuration point of view.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
Tomas Hozza
37911599ce Tests/RHEL-9.0: add repos needed for EC2 SAP image tests
Temporarily use the RHEL-8 RHUI client repo, because RHUI for RHEL-9
does not exist yet. Using at least RHEL-8 RHUI repo for EC2 images
allows us to at least successfully build them and inspect them.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
Tomas Hozza
0e90217fcc RHEL-9.0: add EC2 SAP image definition.
The image is not available via Weldr API, because it requires RHUI
client RPMs.

The content and configuration is based on RHEL-8.6 EC2 SAP image, since
there is no definition for the RHEL-9 SAP image yet.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
Tomas Hozza
7c6c67ef98 Tests/RHEL-9.0: add EC2 and EC2 HA image tests
Because the RHUI client RPMs for RHEL-9 don't exist yet, these test
cases use the RHUI client RPMs for RHEL-8 to satisfy the required
package set. These RPMs contain the entitlement certificate and repo
definitions for RHEL-8 AWS RHUI repos, but RHUI will not work without
the image being imported with the proper Billing Code to AWS. So these
installed RHUI RPMs are technically useless on the RHEL-9 image, but
this approach is good enough to allow us to at least build and test
these images from configuration point of view.

Regenerate all image test cases because of the updated repositories.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
Tomas Hozza
d11db7ed83 Tests/RHEL-9.0: add repos needed for EC2 and EC2 HA image tests
Temporarily use the RHEL-8 RHUI client repo, because RHUI for RHEL-9
does not exist yet. Using at least RHEL-8 RHUI repo for EC2 images
allows us to at least successfully build them and inspect them.

Update all RHEL-9 repos to the latest snapshot to ensure the consistency
of package split among repositories.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
Tomas Hozza
9497ef8ca9 Image tests: skip rpm-ostree-1-autovar.conf tmpfiles.d config on Fedora
The content of `/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf` is
dynamically created and its lines are not ordered deterministically.
This causes the rpm-ostree based image test cases to fail often on
Fedora, because the diff of the expected and actual image-info report
always produces some different lines.

Add a new option `-skip-tmpfilesd-path` to `osbuild-image-tests`
accepting a tmpfiles.d configuration path, which should be ignored when
comparing the expected and actual image-info report. The option can be
specified multiple times and all paths will be ignored.

Modify the `image_tests.sh` test case to use the new option and ignore the
`/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf` file when testing Fedora
images.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:21:25 +01:00
sanne
9d5c16f623 composer: Don't dump sensitive fields from config 2021-10-08 13:58:47 +02:00
Jakub Rusz
e9c0f45bf7 tests: Update image_tests 2021-10-07 12:41:53 +02:00
sanne
4b48c194a3 templates: Duplicate value in composer config
[skip ci]
2021-10-07 12:18:35 +02:00
sanne
973c1c4795 templates: Port names should be less than 15 characters
[skip ci]
2021-10-07 12:03:21 +02:00
sanne
14370e3c49 templates: Make sure ports are unquoted
[skip ci]
2021-10-07 11:56:02 +02:00
sanne
cda54ef926 cloudapi/v1: Return status created in compose handler 2021-10-07 10:54:12 +02:00
Diaa Sami
12ca5325d6 worker: Use Recover middleware to handle panics
recover from panics such as out-of-bounds array access & nil
pointer access, print a stack trace and return 5xx error
2021-10-06 17:04:52 +02:00
Diaa Sami
22f151df68 worker: Improve logging
Use logrus library for logging
Use appropriate log-level for different log statements
2021-10-06 17:04:52 +02:00
Tomas Hozza
f9fbf5bd5e Regenerate affected image test cases
Regenerate affected image test cases with the image-info fix (affecting
mainly `tar` image type) and with the latest upstream `osbuild` build
(affecting rhel-85 qcow2 images).

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-06 15:17:06 +02:00
Tomas Hozza
9ec2788ac8 generate-all-test-cases: allow specifying additional DNF repos
Add a new option `--repofrompath` allowing to specify additional DNF
repositories, which will be used on the Runner when installing any
packages (such as osbuild).

Extend the `test/README.md` to mention the new option. In addition,
specify some aspects of the script in more detail, because some of
them were not easy to figure out by users.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-06 15:17:06 +02:00
Tomas Hozza
71cfc35b67 image-info: ensure that directory is analysed as read-only
In some cases, e.g. when analysing an extracted `tar` image, it could
happen that the tools used by `image-info` to analyse the image could
modify its content (e.g. create new files which were originally not on
the image). This is especially an issue with `rpm`, which DB backend
seemed to create files in `/var/lib/rpm/` when run.

Ensure that the analysed directory can not be modified by bind-mounting
it as a read only, before the analysis.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-06 15:17:06 +02:00
Jakub Rusz
1a7858c196 tests: enable koji.sh test on RHEL-9 2021-10-06 15:42:36 +03:00
Jakub Rusz
9e7e16c9cf tests/ci: enable vmware.sh and cross-distro.sh on rhel-9 2021-10-06 15:42:36 +03:00