Commit graph

1371 commits

Author SHA1 Message Date
Achilleas Koutsou
6b73dc5a92 distro/rhel90: update to match 8.6 and add centos-9 alias
- Copied distro and image definitions from RHEL 8.6
    - New package set handling
    - Distribution-specific strings
    - New image types: edge-raw and edge-simplified-installer
    - Edge container with nginx
    - Removed greenboot services from edge enabled services (enabled
      automatically)
- Adapted to required changes from RHEL 9.0 Beta
- Added CentOS 9 (centos-9) alias

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
ba4a4541d9 distro/rhel86: remove genisoimage
Not required.
Discovered it's unneeded when it was removed from RHEL 9.0.
Updating RHEL 8.6 to keep clean and consistent.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
c7cab92e88 distro/rhel86: single osbuild import
osbuild2 was imported twice, once with its own name and once aliased as
plain osbuild.
Use only the plain osbuild alias.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
cad5c3f13f distroregistry: add rhel-90-ga to registry
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
d290ff5923 distro: copy rhel90beta to rhel90
- Start of RHEL 9.0 GA definition
- Initial distro name: rhel-90-ga
- rhel-90 alias remains for rhel-90-beta for now

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Gianluca Zuccarelli
c0ceede084 cloudapi/v2: 5xx error metrics
Currently error metrics are being recorded for
errors with the exact error code of 500. This
commit enables recording metrics for all 5xx errors.
2021-11-04 08:08:32 +01:00
Martin Sehnoutka
708d985194 internal/blueprint: allow filesystem size specified with units
Allow users to specify filesystem size with units such as kB, MB, etc.
2021-11-02 18:32:47 +01:00
Martin Sehnoutka
f84beee04d internal/blueprint: introduce custom fs customization parser
The automatically generated parser can only parse integers into the size
field. Introduce a new one, which will be able to handle strings as
well.
2021-11-02 18:32:47 +01:00
Martin Sehnoutka
f2f078acd6 internal/blueprint: introduce new test for parsing blueprints
Test case like this was completely missing. A similar one is present in
the Weldr package, but this one is specific to testing blueprints and
thus easier to understand and extend.
2021-11-02 18:32:47 +01:00
Martin Sehnoutka
7d6dadb598 internal/common: introduce function to convert data sizes
This function will be used to parse filesystem sizes specified as
string.
2021-11-02 18:32:47 +01:00
Gianluca Zuccarelli
297a60d238 cloudapi: record error metrics
Hook up the compose error metric for
the prometheus alerts and grafana dashboard.
2021-10-29 20:36:18 +01:00
Gianluca Zuccarelli
f8199ec41d prometheus: add middleware function
Add middleware function to track request count
and measure the latency of compose requests.
2021-10-29 20:36:18 +01:00
Gianluca Zuccarelli
dfa6a48f5d prometheus: compose latency metric
Add metric to measure the latency
of requests made to the composer
cloud api.
2021-10-29 20:36:18 +01:00
Chloe Kaubisch
f749078b0d prometheus: update metrics
Change the name of total https requests to be more specific.
Add a new counter for failed compose requests.
2021-10-29 17:09:45 +01:00
Christian Kellner
93e54cd872 distro/rhel86: special case root user for ssh keys
Add a special case for the root user to the work-around for ssh
keys in OSTree commits. As a little refresher: OSTree does not
support having any content in home directories; we therefore
include a first-boot stage in the commit that will create the
ssh keys on first boot. However, until now we did not special
case the root user, which has a separate root directory (/root,
as a symlink to /var/roothome). This patch fixes this.
2021-10-27 00:41:27 +02:00
Tom Gundersen
f44acd0974 Revert "Revert "cloudapi/v1: Move depsolving to workers""
Workers now depsolve in parallel to image builds, so we can
again move depsolivng to the workers. This will help us deal
with increases in traffic as we currently only have one
depsolve handler per pod. It would also avoid any issues with
composer running out of disk space due to dnf metadata caches.

This reverts commit c65b1e9b26.
2021-10-26 13:58:22 +01:00
Achilleas Koutsou
30d1183909 distro/rhel86: skip RHSM config stage for non-RHEL
EC2 RHEL images keep the RHSM DNF plugins enabled, but this shouldn't be
added on CentOS since it doesn't have RHSM.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-21 15:16:04 +01:00
Achilleas Koutsou
93750dd537 distro/rhel86: distro private method isRHEL()
Returns true if the distribution name starts with "rhel".

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-21 15:16:04 +01:00
Achilleas Koutsou
ed0cb5ea24 distro/rhel86: add distro specific package set
A package set that only gets added for some distros.

Now that the rhel86 subpackage defines both RHEL 8.6 and CentOS 8, we
need to be able to separate out packages that are only available on one
of the two distros. Currently, this only includes "insights-client"
which is not available on CentOS.

The packages are added to the OS package sets conditionally based on the
distro name.
2021-10-21 15:16:04 +01:00
Achilleas Koutsou
8f84e46ab8 distro/rhel86: add CentOS Stream 8 as alias to RHEL 8.6
Removed old alias from RHEL 8.4.

To make aliasing distributions simpler, all distro-specific strings were
added to the local distribution type.

These were previously global constants of the distribution package and
alias alternatives were used conditionally. Now the two distributions
are predefined in global map.

CentOS exclusions:
- s390x arch
- ec2 image types: rhui client is not available

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-21 15:16:04 +01:00
Achilleas Koutsou
20036e7944 distro/rhel86: remove redundant rhel-86 alias
Accidentally left over from copy from RHEL 8.5
2021-10-21 15:16:04 +01:00
Ondřej Budai
58423c262b ci: rotate secret names
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-21 07:28:46 +02:00
Diaa Sami
751ef84fc1 jobqueue: Better logging
Use appropriate logging levels for different statements
Log important jobqueue events with relevant information
2021-10-19 10:03:39 +01:00
sanne
d25ae71fef worker: Configurable timeout for RequestJob
This is backwards compatible, as long as the timeout is 0 (never
timeout), which is the default.

In case of the dbjobqueue the underlying timeout is due to
context.Canceled, context.DeadlineExceeded, or net.Error with Timeout()
true. For the fsjobqueue only the first two are considered.
2021-10-19 00:12:18 +01:00
sanne
87cc722021 cloudapi/v2: Plural path section when querying a collection 2021-10-14 16:22:24 +02:00
Thomas Lavocat
010a1f5022 worker: Configure AWS credentials in the worker 2021-10-14 02:10:54 +01:00
Ondřej Budai
7760ca1c92 cloudapi/v2: ensure only one image per a compose in the API spec
We've never had the ability to build multiple images per a compose, this
commit thus rips out support for this on the API level:

image_requests is now image_request and it accepts only one ImageRequest
object instead of an array of them.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Ondřej Budai
b2dc90e404 cloudapi/v2: clean up targets
tbh, I don't fully understand why we wrote the code in this way.

Not a functional change.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Ondřej Budai
6b8f758c54 cloudapi/v2: move multi-image compose check to the beginning
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Sanne Raymaekers
c65b1e9b26 Revert "cloudapi/v1: Move depsolving to workers"
This reverts commit 0f44250237.
2021-10-13 12:01:51 +02:00
sanne
0f44250237 cloudapi/v1: Move depsolving to workers
We can't enqueue a job without a worker doing the depsolve step now, so
adapt the tests as well.
2021-10-12 14:21:03 +01:00
Achilleas Koutsou
395a22aa21 cloudapi/v2: fix newV2Server() call in test
Bug caused by two consecutive PR merge rebases, one that added a
function call and another that changed the signature of the same
function.
2021-10-11 23:48:26 +02:00
Achilleas Koutsou
cac3fa10db distro/rhel90 -> distro/rhel90beta: rename package
- Rename rhel90 package to rhel90beta.
- Adjust all imports
- Adjust tests
2021-10-11 13:29:57 +02:00
Achilleas Koutsou
972b9ee14e distro/rhel90: rename to rhel-90-beta and alias base name 2021-10-11 13:29:57 +02:00
Ondřej Budai
e904397fdb cloudapi/v2: Use worker to depsolve
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-11 13:16:51 +02:00
Tom Gundersen
0f90aa9c78 worker: Add a depsolve job type
Allow depsolving to be done in a worker through the job queue rather
than synchronously in composer.

The benefit this might unlock include:
 - no more blocking calls in the cloud/koji APIs
 - only workers accessing repositoires
   - no VPN access from composer
   - composer not needing to be subscribed to CDN, etc
 - no dnf cache managment in composer

Potential problems:
 - the version of composer (so the distro definitions) that
   triggered a depsolve, may not be the same that uses the
   result to generate a manfiset

Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-11 13:16:51 +02:00
sanne
ce7ac9a756 worker: Make BasePath configurable 2021-10-11 09:52:21 +02:00
sanne
6ce20a9ef6 cloudapi/v2: Listen on /api/image-builder-composer/v2 2021-10-11 09:52:21 +02:00
Tomas Hozza
eac6b2c083 RHEL-8.6: add support for official EC2 SAP image
Add support for building the official RHEL EC2 SAP image for RHEL-8.6.
The new image type is available under the `ec2-sap` name. Since the
image type's package set includes RHUI client RPMs, which are not
available publicly, it is by default not exposed through the Weldr API.

Extend the relevant files and definitions used for generating image test
cases. This includes addition of repository definitions needed by the
image type, specifically: `sap`, `saphana` and `ansible`.

Generate image test case for the new image type.

Add News entry for the new image type.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
c3077ede4a distro/rhel85: remove rhel86 alias
Remove alias for RHEL 8.6 from RHEL 8.5 and update the registry to point
to the new rhel86 subpackage.
2021-10-11 09:49:15 +02:00
Achilleas Koutsou
513b7f94b9 distro/rhel86: copy all definitions from rhel85
Adjust strings from 85, 8.5, 8-5, to 86, 8.6, 8-6.
2021-10-11 09:49:15 +02:00
sanne
4eca54b6ed cloudapi/v2: Configurable aws bucket
Fixes #1855
2021-10-09 00:55:09 +01:00
sanne
b6c08f3056 cloudapi/v2: Replace upload types with image types
Because there's only a few combinations of upload types and image types
that make sense, enforce correct combinations by eliminating upload
types.

Fixes #1775
2021-10-09 00:55:09 +01:00
Tomas Hozza
0e90217fcc RHEL-9.0: add EC2 SAP image definition.
The image is not available via Weldr API, because it requires RHUI
client RPMs.

The content and configuration is based on RHEL-8.6 EC2 SAP image, since
there is no definition for the RHEL-9 SAP image yet.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
sanne
cda54ef926 cloudapi/v1: Return status created in compose handler 2021-10-07 10:54:12 +02:00
Diaa Sami
12ca5325d6 worker: Use Recover middleware to handle panics
recover from panics such as out-of-bounds array access & nil
pointer access, print a stack trace and return 5xx error
2021-10-06 17:04:52 +02:00
Diaa Sami
22f151df68 worker: Improve logging
Use logrus library for logging
Use appropriate log-level for different log statements
2021-10-06 17:04:52 +02:00
Martin Sehnoutka
b2d8d092c9 internal/rpmmd: log repository files loaded during composer startup
This is a confusing part of the startup process. We have changed the naming
and we also added the cross distro building feature. It is unclear which
files are loaded and from where (if /etc or /usr).

Log the files that are loaded so every user can clearly see what
configuration osbuild-composer uses. This complements the log of loaded
configuration.
2021-10-05 14:41:05 +02:00
Diaa Sami
27ca55372a cloudapi: Add extra logging & improve existing
Use different logging levels depending on situation
Log HTTP 5xx errors to stdout with internal error details when possible
2021-09-30 18:34:05 +02:00
Diaa Sami
179009fec4 cloudapi: use Logrus as default logger in Echo
And remove log.Logger references
2021-09-30 18:34:05 +02:00