Commit graph

5881 commits

Author SHA1 Message Date
Brian C. Lane
139bf4dec2 cloudapi: Add ability to skip uploading and save image locally
During development it can be very useful to store the results locally
instead of uploading to a remote system. This implements a development
only option to help with that.

To use it you need to add OSBUILD_LOCALSAVE to the server's environment.
This can be done by editing /usr/lib/systemd/system/osbuild-composer.service
and adding:

Environment="OSBUILD_LOCALSAVE=1"

You can then use an 'upload_options' object to skip trying to upload to
the default service for the type of image, eg:

    "image_requests": [
    {
      "architecture": "x86_64",
      "image_type": "guest-image",
      "upload_options": {
          "local_save": true
      },
      ...
    }]

The results will be saved to /var/lib/osbuild-composer/artifacts/UUID/
using the default filename for the image type.

If local_save is used without OSBUILD_LOCALSAVE being set it will return
an error with id=36 saying 'local_save is not enabled'.
2023-08-09 16:48:46 +02:00
schutzbot
98c7a06545 Post release version bump
[skip ci]
2023-08-09 08:19:58 +00:00
Tomáš Hozza
4ac6a7a11d Koji: expose boot mode in image extra metadata
Also extend the Koji test case to verify that the boot mode information
is in the build extra metadata and that it contains valid value.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-08-08 15:53:07 +02:00
Tomáš Hozza
0a5c82086a Weldr API: set the image boot mode in OSBuildJob options
Set the image boot mode (as a string value) in the OSBuildJob options.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-08-08 15:53:07 +02:00
Tomáš Hozza
8367a4500a Cloud API: set the image boot mode in OSBuildJob options
Set the image boot mode (as a string value) in the OSBuildJob options.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-08-08 15:53:07 +02:00
Tomáš Hozza
6dbc3c200e Worker/OSBuild: copy boot mode information to job result
Copy the image boot mode information from the job options to the
job result, so that the `KojiFinalize` job can later access it
and import as an extra metadata to Koji.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-08-08 15:53:07 +02:00
Tomáš Hozza
5fd5cedd08 OSBuildJob: add image boot mode to options and result
Add the information about the image boot mode to the OSBuildJob options
as well as to the OSBuildJobResult options. The intention is that the
worker will simply copy the value from the job options to job result,
so that the `KojiFinalize` job can then access this information and
upload it to Koji as extra metadata.

This information is required in Koji in order for Red Hat's SP tooling
to know how to import image to the cloud environment in case the boot
mode affects the import parameters.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-08-08 15:53:07 +02:00
Tomáš Hozza
af1c373407 Koji: expose image output metadata in build extra metadata
Expose the extra metadata information for each image output stored in
`ImageExtraInfo` also in the build extra metadata. The extra metadata
for each image is nested under key corresponding to the image filename.

Extend the Koji test case to check information in the buildinfo output
and specifically check for all expected metadata in the build Extra
metadata field.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-08-08 15:53:07 +02:00
Tomáš Hozza
350762497f Koji: refactor and consolidate structures
The original Koji implementation expected that the output of a content
generator is only an image. While in reality, we will eventually upload
other types of files as outputs to Koji, such as logs and osbuild
manifest.

Rename Koji structures and their members to better map to the upstream
Koji documentation and their JSON representation. Add comments to
structures. Define type aliases and constants for string values which
are more like enums, than a free-form values.

These changes have no effect on the actual JSON representation of any of
the structures

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-08-08 15:53:07 +02:00
Sanne Raymaekers
1b04a78120 test/cases/api: add check for subscription-manager facts 2023-08-07 20:47:20 +02:00
Brian C. Lane
50c5147cf1 deps: Update osbuild images module to commit 03212162ff49
This pulls in the Size(0) change for test_distro.

Ran:
go get github.com/osbuild/images@latest
./tools/prepare-source.sh
2023-08-05 11:28:29 +02:00
Brian C. Lane
e96ed30d4b weldr: Adjust test to use image type's Size(0) method
This will use the default image size which will be changed when images
is pulled in with a default of 1GiB in test_distro instead of 0.
2023-08-05 11:28:29 +02:00
Brian C. Lane
5821098c29 cloudapi: Add tests for refactored handler code
Add tests for GetBlueprintWithCustomizations, GetPayloadRepositories,
GetSubscription, and GetOstreeOptions methods.
2023-08-05 11:28:29 +02:00
Brian C. Lane
6a1f12c465 cloudapi: Add a test for the new size option 2023-08-05 11:28:29 +02:00
Brian C. Lane
c1e52aebc3 cloudapi: Refactor handler.go code to make testing easier
This moves some of the code from the PostCompose function in handler.go
into methods on the OpenAPI ComposeRequest and ImageRequest structs.

In compose.go I have added several methods.
GetBlueprintWithCustomizations takes the ComposeRequest customizations
and builds a Blueprint struct.

GetPayloadRepositories returns the custom payload repos.

GetSubscription returns the ImageOptions setup with optional
subscription information from the request.

In imagerequest.go I have added GetTarget which takes the upload
options and returns a Target. This moves the giant switch statement,
which may also benefit from further simplification at some point.

GetOSTreeOptions returns the OSTree ImageOptions if there are ostree
settings in the ImageRequest.

GetImageOptions returns the distro.ImageOptions with the size set.

This commit only moves the code, making PostCompose easier to read. All
tests still pass.

cloudapi: Move the size handling to a method on ImageRequest
2023-08-05 11:28:29 +02:00
Brian C. Lane
818d434303 cloudapi: Add optional image size support
This adds a 'size' parameter to the image_request object. It can be used
to specify the minimum image size in bytes

This behaves in the same way as the size parameter of the weldr API

For raw images the root partition is grown to fill the available space.
For LVM images the PV uses the available space, but the LV does not,
leaving space available for other LVs to be created after boot.

See COMPOSER-1883
2023-08-05 11:28:29 +02:00
schutzbot
8e29448044 schutzfile: Update snapshots to 20230801 2023-08-03 12:27:40 +02:00
Achilleas Koutsou
6a3b0af934 test: update repositories for building Fedora images
Update the rpmrepo snapshots used in tests for building Fedora images.
2023-08-03 10:35:21 +02:00
Achilleas Koutsou
d3ef361dd5 test: update repos and manifests
Summary of changes:
- ova image type uses ustar format
    - https://github.com/osbuild/images/pull/16
- New image type "minimal-raw"
    - https://github.com/osbuild/images/pull/29
- Fedora repositories and test manifests updated
    - https://github.com/osbuild/images/pull/57
- Distro aliases for major version without minor updated
    - https://github.com/osbuild/images/pull/59
2023-08-03 10:35:21 +02:00
Achilleas Koutsou
a4798ea64d deps: update osbuild/images to 157e798fdf8d
Update the osbuild/images dependency from 246b718310ea to 157e798fdf8d.
2023-08-03 10:35:21 +02:00
Brian C. Lane
4c7b3dd25a test: Make log errors easier to find
We were using greenprint for failures, which makes it hard to quickly
find where the tests failed. This switches errors to use redprint, and
adds it to places that were simply using echo before doing an exit 1.
2023-08-02 14:46:35 +02:00
yih
f889be9e05 test edge-vsphere image 2023-08-01 14:18:17 +08:00
Jakub Rusz
8734c35b7a Juan is no longer with Red Hat 2023-07-31 16:05:28 +03:00
Brian C. Lane
8ff4c0c40a tests: Add a check for valid snapshot urls
This pulls the list of snapshots from the rpmrepo API, greps the
codebase for all uses of rpmrepo.osbuild.org that look like a snapshot
name, and then checks to make sure they are still valid.
2023-07-28 11:28:40 +03:00
Xiaofeng Wang
967570a2a6 test: Add workaround for BZ#2213660 and #2225667 2023-07-26 22:23:41 +08:00
Brian C. Lane
2e7afcffe1 test: Fix Fedora 39 snapshot urls in test repo 2023-07-26 12:16:24 +02:00
schutzbot
ee5493b44a Post release version bump
[skip ci]
2023-07-26 08:14:26 +00:00
Sanne Raymaekers
b2451c6247 test: add ubi-wsl test 2023-07-25 10:53:21 +02:00
Ondřej Budai
cac9327b44 update to go 1.19
UBI and the oldest support Fedora (37) now all have go 1.19, so we are
cleared to switch.

gofmt now reformats comments in certain cases, so that explains the formatting
changes in this commit.
See https://go.dev/doc/go1.19#go-doc

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-07-21 19:18:00 +02:00
Simon de Vlieger
89274e538d spec: require osbuild >= 89
The recently merged `live-installer` image type depends on osbuild >=
89, see the following commit in osbuild:

c90b587dcc
2023-07-21 14:20:37 +02:00
Diaa Sami
b6d6ecf874 cloudapi: add validation where missing 2023-07-21 12:52:32 +02:00
Diaa Sami
2bdfa045c1 cloudapi: fix typo in property name 2023-07-21 12:52:32 +02:00
Ondřej Budai
ba417dbf3d packer: use gp3 volumes
GP3 is cheaper than GP2, let's switch to it for storing our images:
https://fedoraproject.org/wiki/Changes/CloudEC2gp3

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-07-21 12:20:47 +02:00
Ondřej Budai
b461e403ef packer: move Fedora to 38
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-07-21 12:20:47 +02:00
Ondřej Budai
3acc0d1d2e ci: fix the gitlab trigger
There were two issues:

- ${{ }} converts the expression to a string and strings are apparently always truthy
  https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idif
- There was a typo in pr_data

Therefore, the check didn't work properly.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-07-21 12:12:44 +02:00
Alexander Todorov
38ed9a39be Start collecting logs from virt-install 2023-07-21 12:11:02 +02:00
Achilleas Koutsou
ee0c7b53dc test/README: update test manifest generation instructions 2023-07-21 12:05:50 +02:00
Achilleas Koutsou
52234c5fd1 test/README: describe vendoring of modified images repo
Add instructions for how to test an updated osbuild/images repo against
osbuild-composer.
2023-07-21 12:05:50 +02:00
Xiaofeng Wang
8ba6487b82 test: update for minimal raw new xz format 2023-07-21 12:05:00 +02:00
Achilleas Koutsou
3bbcc8a7cc Remove test that depends on removed code from images
Manifest tests can no longer run and also aren't needed.
All image type manifests are generated and built in the obsuild/images
CI tests and the test function in images
(distro_test_common.TestDistro_Manifest()) has been removed.
2023-07-21 12:05:00 +02:00
Achilleas Koutsou
5c292c61c6 deps: update osbuild/images to 246b718310ea
Current main.
246b718310
2023-07-21 12:05:00 +02:00
Simon de Vlieger
326f0cfa2f cloudapi: add live-installer
This was missing to enable `live-installer` in the CloudAPI.
2023-07-20 18:53:28 +02:00
Brian C. Lane
d0a5da6866 lint: Enable check for replace in go.mod 2023-07-20 15:07:20 +02:00
Mario Cattamo
5b42eeb8b3 Include test for edge-ami image type 2023-07-20 17:18:35 +08:00
Diaa Sami
cb0734145d remove old prometheus listener 2023-07-19 10:56:36 +02:00
Brian C. Lane
3cfde5860d tools: Make pylint happier 2023-07-18 23:20:19 +02:00
Brian C. Lane
08e5c53d31 tools: Add support for local go package replacement
Local packages do not have a version number. Fake it with 'HEAD' to
indicate it's using whatever the working directory is pointing at.

The Provides line will then look like:

Provides: bundled(golang(../osbuild-images)) = HEAD

which rpmbuild is happy to process into an rpm.

This is ONLY for test builds on local systems and should never be seen
in the wild.
2023-07-18 23:20:19 +02:00
Brian C. Lane
f7dcc79bd8 tools: Set a+x on rpm_spec_vendor2provides 2023-07-18 23:20:19 +02:00
Achilleas Koutsou
f478323e3a tools: replace spec Provides generator
Replace the generator for the spec file "Provides:" list with the one
from the current go-rpm-macros [1].  The new generator can handle
'replace' commands in vendor/modules.txt, which makes it possible to
build RPMs with temporarily modified dependencies.  This enables making
scratch builds against forks or branches of the new osbuild/images
repository for development and testing.

In the future, we can use the packaged version of the script, but this
isn't yet available in all supported distros (EL8).

Made some minor modifications to the original script to prepend each
dependency with "Provides:" and read input files as arguments instead of
lines from stdin.

https://pagure.io/go-rpm-macros/blob/c32fbbd25bbcedee8c0b898d3653255b18a0d30e/f/rpm/go_mod_vendor.prov
2023-07-18 18:08:49 +02:00
schutzbot
1c148c162f Post release version bump
[skip ci]
2023-07-12 08:19:40 +00:00