The previous value for parent was not a valid commit ID. In a regular
compose request, the parent ref is resolved before being added as a
source in the manifest. However, when building test manifests, since
the values aren't real, we don't resolve the ostree refs, meaning that
the test manifests weren't valid — they failed validity checks if run in
osbuild.
Replaced the fake parent ref with a fake parent commit ID
"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff".
The manifests still aren't buildable, since they don't point to a valid
ostree repository or commit, but now they're at least valid (in the
technical sense).
When the test upload a vhd.xz image to koji, it returns `koji.GenericError:
multiple matches for file extension: vhd.xz`. It seems like the default
schema is not valid for vhd.xz images because it contains two archive types
for them which koji cannot handle. I reported this issue as
https://pagure.io/koji/issue/3605
This commit works around that by removing one of the archive types, so koji
isn't confused by two same records.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Add manifests for container embedding in edge-commits and qcow2s.
Limit qcow2 container embedding to RHEL 9.1 for now since it's not
supported on Fedora or RHEL 8 yet.
It needs to be listed in format-request-map.json, and the repo snapshots
required to depsolve the packages need to be added to repos.json
NOTE: The snapshots are from https://github.com/osbuild/rpmrepo/
Podman doesn't work when installed from 9.1 repositories on 9.0 image.
I found that upgrading the whole system helps. Sadly, that requires removing
python-unversioned-command for some reason.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
F37 no longer ships sil-scheherazade-fonts, but
sil-scheherazade-new-fonts instead. Let's change this. The repos for
test manifests must have been updated in order to get the new package.
Co-authored-by: Ondřej Budai <ondrej@budai.cz>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Don't list specific RHEL repos which should be copied over to the repo
overrides directory, but instead copy all of them, similar to how it's
done for Fedora and CentOS Stream. This is less error prone when support
for a new RHEL release is added to composer.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
GCP Bucket to use can be now configured in the worker configuration.
Make the `Bucket` optional in the Cloud API when uploading image to GCP.
Adjust the Cloud API test case to configure GCP Bucket on the worker and
not provide it in the API request.
There was a missing `overrides` property in the `azure-rhui` request
definition, which resulted in:
```
$ ./tools/test-case-generators/generate-test-cases --distro rhel-90
--arch x86_64 --image-types azure-rhui --store /dev/null --output
/home/thozza/devel/osbuild-composer/test/data/manifests
--keep-image-info
Traceback (most recent call last):
File "/home/thozza/devel/osbuild-composer/./tools/test-case-generators/generate-test-cases", line 176, in <module>
main(args.distro, args.arch, args.image_types, args.keep_image_info, args.store, args.output)
File "/home/thozza/devel/osbuild-composer/./tools/test-case-generators/generate-test-cases", line 153, in main
if distro in test_case_request["overrides"]:
KeyError: 'overrides'
```
Instead of keeping the loop device of the base image and then opening
each partition as a loop device, remove the original loop device of the
base image and then create a loop device for each partition from the
file itself using the partition offsets.
The open_image() function is renamed to convert_image() and now only
handles converting qcow2 files to raw files if necessary.
The loop_open() context is done in analyse_image() instead, so that the
base loop device can be closed without removing the converted image.
This fixes the following issue with LVM partitions:
When the same lvm partition UUID is on two devices (e.g., /dev/loop0p4
and /dev/loop1), the 'vgchange -ay' command fails with the following
error:
Cannot activate LVs in VG rootvg while PVs appear on duplicate
devices.
This happens when we open the LVM partition as a separate loop device,
which we do for all partitions that we want to inspect.
NB: It's possible to restrict the vgchange command to a specific device
with --devices, but this isn't available in older versions of lvm2 (it
was introduced in 2.03.11).
make ansible playbooks arch-agnostic
extract embedded bash script into separate file with parameters
update packer template to support aarch64
Convert parts of bash script to python code that can start multi-arch instances to build RPMS
Extend the implementation of mock openid server to take the `grant_type`
into consideration for the `/token` endpoint.
In addition to the previously supported `refresh_topen`, the
implementation now supports also `client_credentials`.
This is necessary to make it possible to use the mock server in
the `koji-osbuild` CI, because the builder plugin uses
`client_credentials` to get access token.
The implementation behaves in the following way:
- For `refresh_token` grant type, it takes the `refresh_token` value
from the request and adds it to the `rh-org-id` field in the custom
claim, which is part of the returned token.
- For `client_credentials` grant type, it takes the `client_secret`
value from the request and adds it to the `rh-org-id` field in the
custom claim, which is part of the returned token.
Requests without the supported `grant_type` set are rejected.
Modify affected test cases to specify `grant_type` when fetching a new
access token.
Do not start local worker (mask the unit) and Weldr API socket when
provisioning the SUT with TLS client cert authentication method. This
method is used only in the Service scenario, therefore starting these
units / sockets was not reflecting the intended deployment.
Modify `api.sh` to not rely on local worker.
Modify `base_tests.sh` to provision SUT with TLS for
`osbuild-auth-tests`, while provisioning SUT with no authentication
method for the rest of test cases.
`tools/provision.sh` is provisioning SUT always in the same way for
both, the Service scenario and the on-premise scenario. While this is
not causing any issues, it does not realistically represent how we
expect osbuild-composer and worker to be used in these scenarios.
The script currently supports the following authentication options:
- `none`
- Intended for the on-premise scenario with Weldr API.
- NO certificates are generated.
- NO osbuild-composer configuration file is created.
- NO osbuild-worker configuration file is created. This means that no
cloud provider credentials are configured directly in the worker.
- Only the local worker is started and used.
- Only the Weldr API socker is started.
- Appropriate repository definitions are copied to
`/etc/osbuild-composer/repositories/`.
- `jwt`
- Intended for the Service scenario with Cloud API.
- Should be the only method supported in the Service scenario in the
future.
- Certificates are generated and copied to `/etc/osbuild-composer`.
- osbuild-composer configuration file is created and configured for
JWT authentication.
- osbuild-worker configuration file is created, configured for JWT
authentication and with appropriate cloud provider credentials.
- Local worker unit is masked. Only the remote worker is used (the
socket is started and one remote-worker instance is created).
- Only the Cloud API socket is started (Weldr API socket is stopped).
- NO repository definitions are copied to
`/etc/osbuild-composer/repositories/`.
- `tls`
- Intended for the Service scenario with Cloud API.
- Should eventually go away.
- Certificates are generated and copied to `/etc/osbuild-composer`.
- osbuild-composer configuration file is created and configured for
TLS client cert authentication.
- osbuild-worker configuration file is created, configured for TLS
authentication and with appropriate cloud provider credentials.
- Services and sockets are started as they used to be originally:
- Both local and remote worker sockets are started.
- Both Weldr and Cloud API sockets are started.
- Only the local worker unit will be started automatically.
- NO repository definitions are copied to
`/etc/osbuild-composer/repositories/`.
Move some code related to using JWT tokens from the `multi-tenancy.sh`
test case to `test/cases/api/common/common.sh`, `tools/provision.sh`
and `tools/run-mock-auth-servers.sh`. Move the composer and worker
configuration from the test to new testing configuration files.
The `tools/provision.sh` now accepts an optional argument specifying the
authentication method to use with the provisioned composer and workers.
Valid values are `tls` and `jwt`. If no argument is specified, the `tls`
option is used and the script defaults to its previous behavior.
The provision tools was calling to the Weldr API using a CLI client to do
a basic verification of the provisioned software. This is however not
practical nor needed. Eventually, we may want to not enable the Weldr
API socket when testing scenarios related to the Service, to make it
more realistic. Another reason to not do it is that test cases which are
using this script to provision the software are doing the actual
verification, so this just duplicates it.
Extend the `tools/koji-compose.py.sh` script to allow also testing the
upload to cloud, in addition to the testing that it supports currently.
If only the `DISTRO` and `ARCH` arguments are passed to the script, it
submits a new Koji compose with two image requests, as it always did.
If a `CLOUD_TARGET` and `IMAGE_TYPE` arguments are provided in addition
to `DISTRO` and `ARCH`, then the script submits a new Koji compose with
a single image request, which has the upload options set to make the
image be uploaded to cloud.
Supported cloud targets are:
- `aws`
- `azure`
- `gcp`
The image types are those that are accepted by the Cloud API. The script
does not check at all if the provided combination of the cloud target
and image type is valid and submits anything that it gets to composer.
Modify the `tools/koji-compose.py` script to print all log messages to
STDERR and to print only the Koji compose ID to STDOUT. This way, the
caller of the script can easily get the ID of the compose created by the
script and use it later.
Add support for reporting the install container images in an image.
NB: this does not use `podman` but reads the overlay storage
directly and therefore does currently not take additional image
locations or different storage drivers into account. For now this
is not a problem since we don't support any of that.
Fedora 34 is EOL, let's remove all traces of it, including:
- distro definition
- repositories (and test one)
- test manifests
- special package set rules
- hacks from the spec file
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
