Commit graph

1350 commits

Author SHA1 Message Date
Ondřej Budai
58423c262b ci: rotate secret names
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-21 07:28:46 +02:00
Diaa Sami
751ef84fc1 jobqueue: Better logging
Use appropriate logging levels for different statements
Log important jobqueue events with relevant information
2021-10-19 10:03:39 +01:00
sanne
d25ae71fef worker: Configurable timeout for RequestJob
This is backwards compatible, as long as the timeout is 0 (never
timeout), which is the default.

In case of the dbjobqueue the underlying timeout is due to
context.Canceled, context.DeadlineExceeded, or net.Error with Timeout()
true. For the fsjobqueue only the first two are considered.
2021-10-19 00:12:18 +01:00
sanne
87cc722021 cloudapi/v2: Plural path section when querying a collection 2021-10-14 16:22:24 +02:00
Thomas Lavocat
010a1f5022 worker: Configure AWS credentials in the worker 2021-10-14 02:10:54 +01:00
Ondřej Budai
7760ca1c92 cloudapi/v2: ensure only one image per a compose in the API spec
We've never had the ability to build multiple images per a compose, this
commit thus rips out support for this on the API level:

image_requests is now image_request and it accepts only one ImageRequest
object instead of an array of them.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Ondřej Budai
b2dc90e404 cloudapi/v2: clean up targets
tbh, I don't fully understand why we wrote the code in this way.

Not a functional change.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Ondřej Budai
6b8f758c54 cloudapi/v2: move multi-image compose check to the beginning
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Sanne Raymaekers
c65b1e9b26 Revert "cloudapi/v1: Move depsolving to workers"
This reverts commit 0f44250237.
2021-10-13 12:01:51 +02:00
sanne
0f44250237 cloudapi/v1: Move depsolving to workers
We can't enqueue a job without a worker doing the depsolve step now, so
adapt the tests as well.
2021-10-12 14:21:03 +01:00
Achilleas Koutsou
395a22aa21 cloudapi/v2: fix newV2Server() call in test
Bug caused by two consecutive PR merge rebases, one that added a
function call and another that changed the signature of the same
function.
2021-10-11 23:48:26 +02:00
Achilleas Koutsou
cac3fa10db distro/rhel90 -> distro/rhel90beta: rename package
- Rename rhel90 package to rhel90beta.
- Adjust all imports
- Adjust tests
2021-10-11 13:29:57 +02:00
Achilleas Koutsou
972b9ee14e distro/rhel90: rename to rhel-90-beta and alias base name 2021-10-11 13:29:57 +02:00
Ondřej Budai
e904397fdb cloudapi/v2: Use worker to depsolve
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-11 13:16:51 +02:00
Tom Gundersen
0f90aa9c78 worker: Add a depsolve job type
Allow depsolving to be done in a worker through the job queue rather
than synchronously in composer.

The benefit this might unlock include:
 - no more blocking calls in the cloud/koji APIs
 - only workers accessing repositoires
   - no VPN access from composer
   - composer not needing to be subscribed to CDN, etc
 - no dnf cache managment in composer

Potential problems:
 - the version of composer (so the distro definitions) that
   triggered a depsolve, may not be the same that uses the
   result to generate a manfiset

Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-11 13:16:51 +02:00
sanne
ce7ac9a756 worker: Make BasePath configurable 2021-10-11 09:52:21 +02:00
sanne
6ce20a9ef6 cloudapi/v2: Listen on /api/image-builder-composer/v2 2021-10-11 09:52:21 +02:00
Tomas Hozza
eac6b2c083 RHEL-8.6: add support for official EC2 SAP image
Add support for building the official RHEL EC2 SAP image for RHEL-8.6.
The new image type is available under the `ec2-sap` name. Since the
image type's package set includes RHUI client RPMs, which are not
available publicly, it is by default not exposed through the Weldr API.

Extend the relevant files and definitions used for generating image test
cases. This includes addition of repository definitions needed by the
image type, specifically: `sap`, `saphana` and `ansible`.

Generate image test case for the new image type.

Add News entry for the new image type.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
c3077ede4a distro/rhel85: remove rhel86 alias
Remove alias for RHEL 8.6 from RHEL 8.5 and update the registry to point
to the new rhel86 subpackage.
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
513b7f94b9 distro/rhel86: copy all definitions from rhel85
Adjust strings from 85, 8.5, 8-5, to 86, 8.6, 8-6.
2021-10-11 09:49:15 +02:00
sanne
4eca54b6ed cloudapi/v2: Configurable aws bucket
Fixes #1855
2021-10-09 00:55:09 +01:00
sanne
b6c08f3056 cloudapi/v2: Replace upload types with image types
Because there's only a few combinations of upload types and image types
that make sense, enforce correct combinations by eliminating upload
types.

Fixes #1775
2021-10-09 00:55:09 +01:00
Tomas Hozza
0e90217fcc RHEL-9.0: add EC2 SAP image definition.
The image is not available via Weldr API, because it requires RHUI
client RPMs.

The content and configuration is based on RHEL-8.6 EC2 SAP image, since
there is no definition for the RHEL-9 SAP image yet.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
sanne
cda54ef926 cloudapi/v1: Return status created in compose handler 2021-10-07 10:54:12 +02:00
Diaa Sami
12ca5325d6 worker: Use Recover middleware to handle panics
recover from panics such as out-of-bounds array access & nil
pointer access, print a stack trace and return 5xx error
2021-10-06 17:04:52 +02:00
Diaa Sami
22f151df68 worker: Improve logging
Use logrus library for logging
Use appropriate log-level for different log statements
2021-10-06 17:04:52 +02:00
Martin Sehnoutka
b2d8d092c9 internal/rpmmd: log repository files loaded during composer startup
This is a confusing part of the startup process. We have changed the naming
and we also added the cross distro building feature. It is unclear which
files are loaded and from where (if /etc or /usr).

Log the files that are loaded so every user can clearly see what
configuration osbuild-composer uses. This complements the log of loaded
configuration.
2021-10-05 14:41:05 +02:00
Diaa Sami
27ca55372a cloudapi: Add extra logging & improve existing
Use different logging levels depending on situation
Log HTTP 5xx errors to stdout with internal error details when possible
2021-09-30 18:34:05 +02:00
Diaa Sami
179009fec4 cloudapi: use Logrus as default logger in Echo
And remove log.Logger references
2021-09-30 18:34:05 +02:00
Ondřej Budai
5ed954bcf5 distro/rhel90: use qcow2 compat 1.1 for qcows
There's no reason to use 0.10 since we don't need to support running
this image on RHEL 6 anymore.
See https://bugzilla.redhat.com/show_bug.cgi?id=2008910

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-30 15:36:15 +02:00
Gianluca Zuccarelli
b8c14a5689 blueprints: change minsize from int to uint64
thozza pointed out that `int` is platform dependent which results in
a fs size that is too small for 32-bit machines. This commit changes
the filesystem custimizations to use `uint64` instead of `int`
2021-09-29 15:08:12 +02:00
Christian Kellner
7a4488e5c4 distro/rhel85: set bootloader to none for edge
Use the new bootloader config for OSTree deployments to set the
bootloader backend to `none` since we use grub2 and BLS in all
currentlysupported architectures. In fact, not setting it to
`none`, will use the default `auto`, which will result in the
run of `grub2-mkconfig`, which is not what we want and might
even fail for us.
2021-09-27 20:10:04 +02:00
Christian Kellner
f6d4228b00 osbuild2: add bootloader option to ostree.config
Expose the new (osbuild > 36) `sysroot.bootloader` option to
configure the bootloader backend for OSTree deployments.
Normally, this should be set to `none` for modern bootloader
that use the BLS, since the BLS snippets are always generated.
2021-09-27 20:10:04 +02:00
Christian Kellner
920863df70 osbuild2: small doc fix for ostree.config 2021-09-27 20:10:04 +02:00
sanne
2f328b0e97 workers: Backwards compatible api.openshift.com spec compliance
The main changes are:
- Kind, Href, Id fields for every object returned
- Attach operationIds to each request, return it for errors
- Errors are predefined and queryable
2021-09-27 13:10:05 +01:00
Martin Sehnoutka
d7d4e02c8c internal/common: introduce git revision and rpm version
The variables are set to the git revision from which the build is
triggered and rpm version from the spec file, if it is build using RPM.

This can be later used to query exact source version while
running osbuild-composer.

It is necessary to use both, because none of them is available in all
possible scenarios.

Use either git-rev (preferably) or RPM version (NEVRA) instead of the
"devel" build type. It was just a placeholder.
2021-09-24 14:13:23 +02:00
Achilleas Koutsou
5368ecf7fe distro/rhel90: explicitly enable greenboot services for edge
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-09-24 13:27:21 +02:00
Achilleas Koutsou
23109df500 distro/rhel90: add gnome-kiosk to installer package set
Required for graphical Anaconda installation

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-09-24 13:27:21 +02:00
Achilleas Koutsou
0c5a0d6763 distro/rhel90: remove unavailable packages from edge-installer
Packages that are no longer not available in RHEL 9.0.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-09-24 13:27:21 +02:00
Diaa Sami
60e403e53e cloudapi: use Recover middleware to handle panics
recover from panics such as out-of-bounds array access & nil
pointer access, print a stack trace and return 5xx error
instead of the service crashing and relying on Execution
framework to handle crashes
2021-09-24 12:11:04 +01:00
Christian Kellner
d9fe813fa5 distro/rhel90: re-include nss-altfiles for edge
The `nss-altfiles` package is actually needed for OSTree based
systems, since in those the user/groups database is located in
the /usr/lib directory (in addition to /etc). It was removed
because it was not available in RHEL 9 (rhbz#1916260). This has
been fixed, so we need to re-include the package.
2021-09-23 10:47:11 +02:00
Brian C. Lane
e92424b5e3 weldr: deleting an unknown source should return an error
This adds a check to see if the source is valid and returns a 400 error
if it doesn't exist.
2021-09-22 11:24:31 +02:00
Tomas Hozza
eef9971027 osbuild2: add support for org.osbuild.tuned stage
Add support for a new osbuild stage `org.osbuild.tuned`, for
setting TuneD profile.

Add unit tests for the new stage.

Related to https://github.com/osbuild/osbuild/pull/797.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-09-15 14:46:25 +02:00
Tomas Hozza
8b623d9463 osbuild2: add support for org.osbuild.dnf.config stage
Add support for a new osbuild stage `org.osbuild.dnf.config`, for
configuring DNF.

Add unit tests for the new stage.

Related to https://github.com/osbuild/osbuild/pull/798.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-09-15 14:46:25 +02:00
Tomas Hozza
d9610b97fc osbuild2: add support for org.osbuild.sysctld stage
Add support for a new osbuild stage `org.osbuild.sysctld`, for
creating tmpfiles.d configuration files.

Add unit tests for the new stage.

Related to https://github.com/osbuild/osbuild/pull/804.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-09-15 14:46:25 +02:00
Tomas Hozza
a5097b2141 osbuild2: add support for org.osbuild.pam.limits.conf stage
Add support for a new osbuild stage `org.osbuild.pam.limits.conf`, for
creating pam_limits module configuration files.

Add unit tests for the new stage.

Related to:
- https://github.com/osbuild/osbuild/pull/802
- https://github.com/osbuild/osbuild/pull/807

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-09-15 14:46:25 +02:00
Tomas Hozza
3f52af2adb osbuild2: add support for org.osbuild.tmpfilesd stage
Add support for a new osbuild stage `org.osbuild.tmpfilesd`, for
creating tmpfiles.d configuration files.

Add unit tests for the new stage.

Related to https://github.com/osbuild/osbuild/pull/801.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-09-15 14:46:25 +02:00
Tomas Hozza
fbb70c2d10 osbuild2: add support for org.osbuild.selinux.config stage
Add support for a new osbuid stage `org.osbuild.selinux.config`,
for setting the desired SELinux policy state and type on the system.

Add unit tests for the new stage.

Related to https://github.com/osbuild/osbuild/pull/799.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-09-15 14:46:25 +02:00
Ondřej Budai
39d77f26b6 distro/rhel90: always install docs in the rpm stage
When we firstly introduced the v2 manifests, we excluded docs from all rpm
stages because it doesn't make sense to include docs in edge images.

However, when we ported the other image types to v2, we left the flag on.
The side effect of --excludedocs is that we no longer install man pages into
the image. This the default behaviour of rpm and can be seen here:
b88f43b9a3/macros.in (L555)

This is actually quite a major regression in non-edge images, thus this commit
reverts this setting. It would be great to have the option not to install
docs in some rpm stages but as this is a difficult topic and we're already
past the deadline, we will consider this in the future.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-15 10:47:57 +02:00
Ondřej Budai
d895c336d6 distro/rhel85: always install docs in the rpm stage
When we firstly introduced the v2 manifests, we excluded docs from all rpm
stages because it doesn't make sense to include docs in edge images.

However, when we ported the other image types to v2, we left the flag on.
The side effect of --excludedocs is that we no longer install man pages into
the image. This the default behaviour of rpm and can be seen here:
b88f43b9a3/macros.in (L555)

This is actually quite a major regression in non-edge images, thus this commit
reverts this setting. It would be great to have the option not to install
docs in some rpm stages but as this is a difficult topic and we're already
past the deadline, we will consider this in the future.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-15 10:47:57 +02:00