Commit graph

142 commits

Author SHA1 Message Date
Martin Sehnoutka
ea97adfdf8 spec: bump osbuild version to 41
This will be needed for new stages which are available only in osbuild
41.
2021-11-16 10:13:06 +01:00
diaasami
bb190f1280 Post release version bump
[skip ci]
2021-11-10 16:37:05 +00:00
sanne
117886a75f spec: Only run worker preun if systemd is running
Before, when systemd was not present (for instance in a container), the
preun script would error out.

This checks if systemd is present and running in the same way the
systemd rpm macros do.

Fixes #1915
2021-11-10 01:26:00 +00:00
Simon Steinbeiss
65a3753d2c Post release version bump
This will soon be done by schutzbot, but is done manually this one time.
2021-11-01 20:27:36 +01:00
Simon Steinbeiss
3c9b92b80b Bump version numbers ahead of release
This is necessary for the new simplified release process and is done
ahead of time once for the upcoming release now.
After osbuild-composer 37 this will be done by the GitHub composite action.
2021-10-27 13:03:53 +02:00
Achilleas Koutsou
c1944095d0 spec: dnf-json conflicts with old composer
dnf-json provides files that osbuild-composer used to provide, so they
should be marked as conflicting.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-21 15:07:26 +01:00
Achilleas Koutsou
4e5ce288ab 36
Release osbuild-composer 36

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-13 15:52:08 +01:00
Tomas Hozza
46b8a2e977 SPEC: Exclude armv7hl architecture
Exclude the `armv7hl` architecture, since osbuild-composer does not
support building images for it.

Fix #1839

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-11 15:33:53 +02:00
sanne
b81d58b895 spec: Split dnf-json into a subpackage 2021-10-11 13:16:51 +02:00
Simon Steinbeiss
fed4b97e4e 35
Release osbuild-composer 35

Signed-off-by: Simon Steinbeiss <simon.steinbeiss@redhat.com>
2021-10-02 09:11:07 +02:00
Ondřej Budai
1b59b2973e spec: bump osbuild depedendency to 37
The bootloader option in ostree.config stage is new in osbuild 37.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-27 20:10:04 +02:00
sanne
2f328b0e97 workers: Backwards compatible api.openshift.com spec compliance
The main changes are:
- Kind, Href, Id fields for every object returned
- Attach operationIds to each request, return it for errors
- Errors are predefined and queryable
2021-09-27 13:10:05 +01:00
Ondřej Budai
4c5d1ae45a test: move env variables into set-env-variables helper
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-24 17:03:53 +02:00
Martin Sehnoutka
d7d4e02c8c internal/common: introduce git revision and rpm version
The variables are set to the git revision from which the build is
triggered and rpm version from the spec file, if it is build using RPM.

This can be later used to query exact source version while
running osbuild-composer.

It is necessary to use both, because none of them is available in all
possible scenarios.

Use either git-rev (preferably) or RPM version (NEVRA) instead of the
"devel" build type. It was just a placeholder.
2021-09-24 14:13:23 +02:00
Diaa Sami
aa08e29243 34
Release osbuild-composer 34

Signed-off-by: Diaa Sami <disami@redhat.com>
2021-09-15 21:12:35 +02:00
sanne
5a9d8c792b cloudapi: V2
V2 is compliant with api.openshift.com design guidelines.

Errors are predefined, have codes, and are queryable.

All requests have an operationId set: a unique identifier which is
sortable by time. This is added to the response in case of an error.

All returned objects have the href, id, and kind field set.
2021-09-14 15:32:21 +02:00
Tom Gundersen
8b84d92264 repositories: exclude downstream from upstream
We don't want to give the impression we support building Fedora on RHEL or
CentOS Stream, or CentOS Stream on RHEL, so drop these repositories from
the package.

Also adjust dnf-json to use test repositories, rather than the upstream ones.
These are shipped in the test package, so makes sure we are unaffected by
what is shipped in the RPM shipped in the distros.
2021-09-14 10:43:09 +01:00
Martin Sehnoutka
de41a49f2f spec: stop providing golang-github-osbuild-composer
The golang-github-osbuild-composer package was created by mistake a long
time ago. Stop providing it in Fedora 34 and newer and add a note about
removing the condition when Fedora 33 hits EOL.
2021-09-13 19:20:05 +02:00
Martin Sehnoutka
bb3b69f9c3 spec: bump osbuild dependency to 36
This is needed to include the fix for the RHSM module which is tested by
the newly improved regression test.
2021-09-13 13:14:49 +02:00
yih
645815a11c add test case for httpboot simplified installer 2021-09-07 19:15:22 +02:00
sanne
4a057bf3d5 auth: OpenID/OAUth2 middleware
2 configurations for the listeners are now possible:
- enableJWT=false with client ssl auth
- enableJWT=true with https

Actual verification of the tokens is handled by
https://github.com/openshift-online/ocm-sdk-go.

An authentication handler is run as the top level handler, before any
routing is done. Routes which do not require authentication should be
listed as exceptions.

Authentication can be restricted using an ACL file which allows
filtering based on JWT claims. For more information see the inline
comments in ocm-sdk/authentication.

As an added quirk the `-v` flag for the osbuild-composer executable was
changed to `-verbose` to avoid flag collision with glog which declares
the `-v` flag in the package `init()` function. The ocm-sdk depends on
glog and pulls it in.
2021-09-04 02:48:52 +02:00
Ondřej Budai
8f21f0b873 33
Release osbuild-composer 33

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-08-30 19:03:46 +02:00
Simon Steinbeiss
63fe1c4c83 32
Release osbuild-composer 32

Signed-off-by: Simon Steinbeiss <simon.steinbeiss@redhat.com>
Co-Developed-by: Christian Kellner <christian@kellner.me>
Co-Developed-by: Tom Gundersen <teg@jklm.no>
2021-08-28 16:41:26 +02:00
Antonio Murdaca
4fdf4dbd01 osbuild2: add stages for raw images and grub iso
Add the new stages included in osbuild 30 that can be used to
deploy an OSTree commit as well as the grub iso stage that
is used to configure grub2 (efi) for isos.
Bump the spec file requirement accordingly.

Co-Developed-by: Achilleas Koutsou <achilleas@koutsou.net>
Co-Developed-by: Christian Kellner <christian@kellner.me>
2021-08-28 09:20:19 +02:00
Xiaofeng Wang
bff49c410d test: Add case - running edge-container on OCP as stage repo 2021-08-27 09:29:28 +02:00
Ondřej Budai
fa111f82a4 test: do not install podman-plugins on RHEL 9.0
They are not yet shipped in RHEL 9.0, this makes the tests subpackage
installable.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-08-17 16:54:22 +02:00
Thomas Lavocat
b896ee4d8c 31
Release osbuild-composer 31
2021-08-11 14:04:41 +02:00
Martin Sehnoutka
2d75b25ffe test/cases: new case that verifies composer works behind satellite
Introduce a test case where 2 httpd proxies are used to simulate custom
repos with custom subscriptions. That is, every repo requires
client-side tls certificate with its own CA. Make sure both osbuild and
osbuild-composer can use these repositories by issuing an image build.
2021-08-09 12:40:23 +02:00
Martin Sehnoutka
1ada606ed8 internal/rhsm: introduce package that handles subscriptions
The problem: osbuild-composer used to have a rather uncomplete logic for
selecting client certificates and keys while fetching data from
repositories that use the "subscription model". In this scenario, every
repo requires the user to use a client-side TLS certificate. The problem
is that every repo can use its own CA and require a different pair of
a certificate and a key. This case wasn't handled at all in composer.

Furthermore, osbuild-composer can use remote workers which complicates
things even more.

Assumptions: The problem outlined above is hard to solve in the general
case, but Red Hat Subscription Manager places certain limitations on how
subscriptions might be used. For example, a subscription must be tight to
a host system, so there is no way to use such a repository in osbuild-composer
without it being available on the host system as well.

Also, if a user wishes to use a certain repository in osbuild-composer it
must be available on both hosts: the composer and the worker. It will come
with different pair of a client certificate and a key but otherwise, its
configuration remains the same.

The solution: Expect all the subscriptions to be registered in the
/etc/yum.repos.d/redhat.repo file. Read the mapping of URLs to certificates
and keys from there and use it. Don't change the manifest format and let
osbuild guess the appropriate subscription to use.
2021-08-09 12:40:23 +02:00
Martin Sehnoutka
cc9b74ead6 spec: bump osbuild dependency to >= 29
Version 29 contains the support for the new RHSM lookup alghorithm which
is needed for cooperation with the new algorithm in composer.
2021-08-09 12:40:23 +02:00
Juan Abia
c4fa4e4bb7 test: Replace hard coded ssh keys on some test
Generate a new ssh key each time. Only implemented on bash test scripts,
go tests pending.
2021-07-29 09:38:03 +03:00
Lars Karlitski
9c2c92f729 jobqueue: Introduce jobqueue backed by a postgres database
Co-authored-by: sanne <sanne.raymaekers@gmail.com>
2021-07-28 21:52:31 +01:00
Chloe Kaubisch
4c800f29a7 worker: add metrics
use prometheus to gather metrics
2021-07-23 21:54:28 +02:00
Ondřej Budai
385648223d spec: drop hacks for Fedora 32
There are not needed anymore, yay!

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-07-05 11:16:08 +02:00
Ondřej Budai
b5c65b687a 30
Release osbuild-composer 30.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-10 14:59:08 +02:00
yih
661d59e534 add test case to push/pull edge container images in quay.io 2021-05-26 12:16:42 +03:00
Alexander Todorov
57cebbfaea tests: Define COMPOSE_URL/COMPOSE_ID in a single place. Fix #1397
we don't want this defined multiple times in different places
2021-05-12 09:26:31 +02:00
Ondřej Budai
31a0ca3284 spec: fix rpm build on Fedora 34
Not proud of the fix but it should work for now. See the comment in the spec
file for more information and also the upstream PR for more context:

https://github.com/getkin/kin-openapi/pull/351

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-05-08 20:15:56 +02:00
Jakub Rusz
06361267d5 Add image boot testing in Azure
This uses an image created and uploaded to Azure using composer-cli
and then terraform to spin up a linux vm from that image, check
if the machine works and then cleans up everything.
2021-05-05 12:05:32 +03:00
Ondřej Budai
973b4b2714 distro/rhel84: build qcow2 images with compat=0.10
By default, `qemu-img convert` creates qcow2 images usable in qemu 1.1 and
newer. RHEL 8 guest images are meant to be bootable on RHEL 6 though.
Unfortunately, RHEL 6 has qemu 0.12, therefore these images cannot be used
there.

To fix this, we need to use the new qcow2_compat option in qemu assembler
to override the default compat version and make qcow2 images that can be used
in qemu 0.10 and newer.

For this, we need osbuild 28 that isn't yet available in of any of
downstreams, therefore we need to pin it everywhere.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-04-10 19:18:13 +01:00
Ondřej Budai
54a458af5c test: replace genisoimage with mkisofs
genisoimage might be removed from RHEL 9. The users are advised to switch
to mkisofs tools from the xorriso package. It should be a drop-in replacement.

The same change was recently done by libguestfs:

efb8a766ca
2216ab2e32

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-04-02 14:21:29 +02:00
Martin Sehnoutka
bb235deb62 29
Release osbuild-composer version 29
2021-03-05 15:01:38 +01:00
Martin Sehnoutka
12880950e9 spec: Improve description of the osbuild-composer package
The previous version referred to lorax-composer as the definition of
what osbuild-composer does. This worked fine while osbuild-composer was
considered an alternative for it. Now that osbuild-composer is the
default one, it should describe what it does without references to
lorax. Furthemore, composer is now able to build OSTree commits as well
as VM images, to the previous description was slightly incomplete.

This commit introduces description which is up-to-date and does not
refer to lorax any more.
2021-03-04 12:48:38 +01:00
Tomas Hozza
02ad5a3199 test: Fix installation of osbuild-worker.toml in provision.sh
The tools/provision.sh script is sourced by all test cases and it sets
up the system and software for running test cases. As part of the setup,
it copied over the whole content of test/data/composer/ to
/etc/osbuild-composer. However the source directory contains not only
osbuild-composer's configuration, but also configuration for the worker.
The worker however expects its configuration in /etc/osbuild-worker.

The fact that provision.sh does not copy the worker configuration to the
correct directory didn't affect the CI, because the only test case that
relied on it is koji.sh, which copies the worker configuration
explicitly.

Move osbuild-worker test configuration to a separate 'test/data/worker/'
subdirectory. Also install the osbuild-worker test configuration to its
own subdirectory in the "-test" RPM.

Move the copying of worker configuration to the correct destination
directory from koji.sh to provision.sh, so that all test cases can rely
on the system being set up properly. Do not use wildcard for copying
osbuild-{composer,worker} configuration files, but explicitly copy each
file to its respective destination directory.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-02-25 18:44:21 +00:00
Tomas Hozza
ff95059748 internal/upload: Add support for upload to GCP and CLI tool using it
Add new internal upload target for Google Cloud Platform and
osbuild-upload-gcp CLI tool which uses the API.

Supported features are:
- Authenticate with GCP using explicitly provided JSON credentials
  file or let the authentication be handled automatically by the
  Google cloud client library. The later is useful e.g. when the worker
  is running in GCP VM instance, which has associated permissions with
  it.
- Upload an existing image file into existing Storage bucket.
- Verify MD5 checksum of the uploaded image file against the local
  file's checksum.
- Import the uploaded image file into Compute Node as an Image.
- Delete the uploaded image file after a successful image import.
- Delete all cache files from storage created as part of the image
  import build job.
- Share the imported image with a list of specified accounts.

GCP-specific image type is not yet added, since GCP supports importing
VMDK and VHD images, which the osbuild-composer already supports.

Update go.mod, vendor/ content and SPEC file with new dependencies.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-02-25 18:44:21 +00:00
Ondřej Budai
f2243151be test: remove podman-plugins workaround for RHEL/CentOS
https://bugzilla.redhat.com/show_bug.cgi?id=1877865 is now fixed and
podman-plugins is now shipped in stable RHEL. One less hack in our codebase!

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-02-25 12:14:56 +01:00
Tom Gundersen
e4b9453d40 spec: update osbuild dependency
We now depend on osbuild 26.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2021-02-20 20:01:15 +01:00
Martin Sehnoutka
8ca6b1ea15 28
Release osbuild-composer version 28
2021-02-20 17:56:23 +00:00
Martin Sehnoutka
f98e231f64 cloudapi: extend the API spec with /version and /openapi.json
These endpoints are useful for clients while exploring the API. They are
also required for deploying the service into clouddot.
2021-02-12 14:25:19 +01:00
Jacob Kozol
52823e3c43 spec: update to osbuild version 24
In order to add the newly supported sysconfig stage, the osbuild
dependency needs to be updated to version 24. The version is updated to
24 in the specfile and README. Also, for testing purposes, the Schutzfile
now points to the commit hash for osbuild version 24 for rhel-8.3 tests.
2021-02-09 14:13:25 +01:00