Commit graph

133 commits

Author SHA1 Message Date
Sanne Raymaekers
b05723a37e templates/composer: Verify against mass sso and rh sso 2022-02-24 09:48:12 +01:00
Tom Gundersen
0b24099751 jwt: support multiple key providers
We may need to use several SSO providers, so extend our
configuration to allow that.

Based on PoC from Sanne:

```
package main

import (
	"net/http"
	"log"

	"github.com/openshift-online/ocm-sdk-go/authentication"
	"github.com/openshift-online/ocm-sdk-go/logging"
)

type H struct{}

func (h *H) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	log.Println("HURRAY")
}

func main() {

	logBuilder := logging.NewGoLoggerBuilder()
	logger, err := logBuilder.Build()
	if err != nil {
		panic(err)
	}

	aH, err := authentication.NewHandler().
		KeysURL("https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/certs").
		KeysURL("https://identity.api.openshift.com/auth/realms/rhoas/protocol/openid-connect/certs").
			Logger(logger).Next(&H{}).Build()
	if err != nil {
		panic(err)
	}

	log.Fatal(http.ListenAndServe(":8080", aH))

}
```
2022-01-31 20:40:22 +00:00
Ondřej Budai
d967790ea5 api/cloud: drop v1 API
It's deprecated and not used anywhere, let's just drop it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:38:50 +01:00
sanne
60d4f5a751 composer: Disable artifacts for the service
When backed by a DB, composer has no need of a queue directory.

This also addresses "Error moving artifacts for job" logging noise.

Signed-off-by: sanne <sanne.raymaekers@gmail.com>
2021-12-16 17:04:08 +00:00
Diaa Sami
8f80dd06d6 composer: use logrus everywhere for consistency 2021-12-16 11:58:41 +00:00
Thomas Lavocat
ca126e9747 dnf-json: Change dnf-json to be a daemon
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
2021-12-15 09:41:32 +01:00
Juan Abia
ceb72975c4 gosec: G402 - TLS MinVersion
stablish minumim TLS version, so G402 from gosec doesn't get triggered
2021-12-13 12:17:30 +02:00
Ondřej Budai
c6fbf589aa dbjobqueue: add settings for max pool connections
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-19 13:17:10 +01:00
Diaa Sami
37bffc9808 logging: include the source of each log statement 2021-11-16 19:16:34 +01:00
sanne
3101046d44 composer: Add metrics endpoint to auth excludes again
Mistakenly removed in 4577ac0717. Composer
itself does the authentication, not the gateway, therefore we do need
the auth exclude.

Added a comment to explain why it's attached to the api socket and not a
separate listener.
2021-11-16 10:05:52 +01:00
sanne
4577ac0717 composer: Add worker openapi spec endpoint to auth excludes
Also remove metrics from the excludes for the composer api. This
endpoint would not be called via the gateway so it's not relevant here.
2021-11-10 10:43:43 +01:00
Diaa Sami
3b6156a240 composer: add json log formatting
Configurable via config file/env
2021-10-25 13:41:36 +02:00
sanne
d25ae71fef worker: Configurable timeout for RequestJob
This is backwards compatible, as long as the timeout is 0 (never
timeout), which is the default.

In case of the dbjobqueue the underlying timeout is due to
context.Canceled, context.DeadlineExceeded, or net.Error with Timeout()
true. For the fsjobqueue only the first two are considered.
2021-10-19 00:12:18 +01:00
sanne
7e6365742f cloudapi/v2: Do not require auth for /openapi or /errors 2021-10-14 16:22:24 +02:00
sanne
ce7ac9a756 worker: Make BasePath configurable 2021-10-11 09:52:21 +02:00
sanne
6ce20a9ef6 cloudapi/v2: Listen on /api/image-builder-composer/v2 2021-10-11 09:52:21 +02:00
sanne
4eca54b6ed cloudapi/v2: Configurable aws bucket
Fixes #1855
2021-10-09 00:55:09 +01:00
sanne
b6c08f3056 cloudapi/v2: Replace upload types with image types
Because there's only a few combinations of upload types and image types
that make sense, enforce correct combinations by eliminating upload
types.

Fixes #1775
2021-10-09 00:55:09 +01:00
Tomas Hozza
0e90217fcc RHEL-9.0: add EC2 SAP image definition.
The image is not available via Weldr API, because it requires RHUI
client RPMs.

The content and configuration is based on RHEL-8.6 EC2 SAP image, since
there is no definition for the RHEL-9 SAP image yet.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
sanne
9d5c16f623 composer: Don't dump sensitive fields from config 2021-10-08 13:58:47 +02:00
Diaa Sami
22f151df68 worker: Improve logging
Use logrus library for logging
Use appropriate log-level for different log statements
2021-10-06 17:04:52 +02:00
Achilleas Koutsou
1a29f084a1 main: IsNotExist() is no longer a valid check 2021-10-04 13:54:53 +02:00
Achilleas Koutsou
c30c4decb7 config: update NonExisting test to check for default 2021-10-04 13:54:53 +02:00
Achilleas Koutsou
8393cfc322 config: don't fail LoadConfig if file doesn't exist
When the config file doesn't exist, don't return because we need to keep
the default and also load from env.
2021-10-04 13:54:53 +02:00
sanne
1e724be8e9 composer: More configuration of how composer is served
Allows you to disable tls
2021-10-02 14:27:02 +02:00
Diaa Sami
cc54c4deee composer: improve logging 2021-09-30 18:34:05 +02:00
Diaa Sami
179009fec4 cloudapi: use Logrus as default logger in Echo
And remove log.Logger references
2021-09-30 18:34:05 +02:00
sanne
5a9d8c792b cloudapi: V2
V2 is compliant with api.openshift.com design guidelines.

Errors are predefined, have codes, and are queryable.

All requests have an operationId set: a unique identifier which is
sortable by time. This is added to the response in case of an error.

All returned objects have the href, id, and kind field set.
2021-09-14 15:32:21 +02:00
sanne
4a057bf3d5 auth: OpenID/OAUth2 middleware
2 configurations for the listeners are now possible:
- enableJWT=false with client ssl auth
- enableJWT=true with https

Actual verification of the tokens is handled by
https://github.com/openshift-online/ocm-sdk-go.

An authentication handler is run as the top level handler, before any
routing is done. Routes which do not require authentication should be
listed as exceptions.

Authentication can be restricted using an ACL file which allows
filtering based on JWT claims. For more information see the inline
comments in ocm-sdk/authentication.

As an added quirk the `-v` flag for the osbuild-composer executable was
changed to `-verbose` to avoid flag collision with glog which declares
the `-v` flag in the package `init()` function. The ocm-sdk depends on
glog and pulls it in.
2021-09-04 02:48:52 +02:00
sanne
58613788bc composer: Move external import to the right place 2021-09-04 02:48:52 +02:00
sanne
5e72636331 Revert "cloudapi: Add x-rh-identity header filter"
This reverts commit 19db3ff1d4.
2021-09-04 02:48:52 +02:00
sanne
7a0ea5b244 worker: Remove identity filter
Partially reverts "0ea31c39d5"
2021-09-04 02:48:52 +02:00
Tomas Hozza
4e92b65721 composer: don't expose ec2 and ec2-ha RHEL images via WeldrAPI
The `ec2` and `ec2-ha` images include RHUI client packages, which are
not publicly available. For this reason, building of such images in the
on-premise use case via WeldrAPI would always fail, unless the system
would be inside the Red Hat internal network or VPN.

Mark the `ec2` and `ec2-ha` image types for `rhel-*` distribution as
denied in WeldrAPI by default.

Extend and modify affected unit tests.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-05 09:34:07 +02:00
Tomas Hozza
b150d57c18 Weldr API: make Image Type denylist distribution-specific
Change the Image Type denylist in Weldr API from being applied to all
distributions to being distribution-specific. A special name `*`
can be used in the configuration to match any distribution
or any image type.

Modify NEWS entry and unit tests to reflect this change.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 18:51:03 +02:00
Tomas Hozza
076bbc5456 Weldr API: introduce Image Type denylist for filtering exposed images
Extend Weldr API to accept a list of denied image types, which should
not be exposed via API for any supported distribution. This
functionality will be needed to not expose image types which can't be
successfully built outside of Red Hat VPN. Example of such images are
the official RHEL EC2 images, which include RHUI client packages not
available publicly.

Image Types are filters when listing available compose types and
creating a new compose using Weldr API.

Extend osbuild-composer configuration to allow specifying the list of
denied Image Types for Weldr API.

Add unit tests for implemented changes.

Add NEWS entry describing the newly introduced functionality.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 18:51:03 +02:00
Lars Karlitski
9c2c92f729 jobqueue: Introduce jobqueue backed by a postgres database
Co-authored-by: sanne <sanne.raymaekers@gmail.com>
2021-07-28 21:52:31 +01:00
Chloe Kaubisch
4c800f29a7 worker: add metrics
use prometheus to gather metrics
2021-07-23 21:54:28 +02:00
sanne
7555ed7556 worker: Merge configs 2021-07-16 12:14:46 +02:00
Brian C. Lane
9818b4b6b1 osbuild-composer: Move InitWeldr code into weldr.New
This will make it easier to support new features related to building for
other distribution releases.
2021-07-12 08:58:42 +02:00
sanne
0ea31c39d5 worker: Add identity filter and client oauth support 2021-06-17 10:08:35 +02:00
sanne
19db3ff1d4 cloudapi: Add x-rh-identity header filter 2021-06-05 14:47:38 +01:00
Tomas Hozza
aa6665ad01 Use RepoRegistry in composer and Weldr API
Modify composer to use RepoRegistry, instead of loading the host
repositories, when initializing WeldrAPI.

Modify WeldrAPI to use RepoRegistry, instead of a map of repository
definitions. Make sure that the RepoRegistry method specific to image
type is used in Welder where appropriate. Specifically when depsolving a
Blueprint, which is used to build a specific image type. Update Weldr
API unit tests to reflect the change.

Add a new method to RepoRegistry, allowing to get list of repositories,
which should be used for building an image for a given architecture,
without specifying the exact image type. Add relevant unit tests.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-05-14 15:43:00 +02:00
Tomas Hozza
fba9fe1072 Make the Distroregistry FromHost() return distro with correct name
Composer does not have 1:1 mapping of what can be the Host Distro name
and the names of supported distributions held in the Distroregistry.

The fact that the host distro `Name()` method as passed to the Weldr API
does not return the same name as what is used as distro name for
repository definitions. This makes it hard to use `distro.Distro` and
`distro.Arch` directly and rely on the values returned by them as their
name.

Add `New*HostDistro()` to all distro definitions, accepting the name
that should be returned by the distro's `Name()` method. This is useful
mainly if the host distro is Beta or Stream variant of the distro.

Change the distroregistry.Registry to contain host distro as a separate
value set when creating it using `New()` function. This value is
returned by `Registry.FromHost()` method. Determining the host distro is
handled by the `NewDefault()` function. Move the distro name mangling to
distroregistry package. Add relevant unit tests.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-05-14 15:43:00 +02:00
Achilleas Koutsou
34b5c5b3f1 weldr: use rhel-8 repos for all RHEL 8.x distros 2021-05-11 12:10:09 +02:00
Ondřej Budai
3c715c7cf8 distroregistry: add a default distroregistry
This commit adds NewDefault() method to distroregistry that returns a slice
with all distributions supported by osbuild-composer. This way, there's only
one place where a distribution needs to be defined while its support
is being added to composer.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-03-12 08:29:30 +01:00
Ondřej Budai
dd4db353e2 distro: move Registry to its own distroregistry package
My goal is to add a method to distroregistry to return Registry with
all supported distributions. This way, all supported distributions
would be defined only on one place.

To achieve this, the Registry must live outside the distro package
because the distro implementation depends on it and this would create
a circular dependency unsupported by Go.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-03-12 08:29:30 +01:00
Ondřej Budai
5eb402415d distro/rhel84: add centos 8 stream support
The image definition is shared with the latest RHEL 8.y one (8.4 currently).
I expect that we the introduction of 8.5 support, we point the centos 8
distro at it.

The test repositories and manifests use the official CentOS composes. From
what I can tell, they are persistent. This is not guaranteed though, so we
might need to switch to RPMRepo at some point.

The "classic" CentOS 8 should also be buildable but due to the chicken and egg
issue (this commit will get into Centos "8.4" but Centos "8.4" isn't a thing
yet), we cannot test it and therefore it might be broken.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-02-14 16:08:08 +00:00
Ondřej Budai
978e309153 worker/server: move it to the style of koji server
The previous code was smelling a bit (e.g. Server.server field) so I decided
to rewrite it in the style of the much nicer koji server.

Not a functional change.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2020-11-19 17:39:24 +00:00
Jacob Kozol
0dd17ae3f7 distro: add rhel 84 support
cockpit-composer can now build rhel 8.4 images. Our distro name for
rhel 8.4 is rhel-84 unlike prior rhel releases which fall
under the umbrella name rhel-8. rhel 8.4 still uses the same
repos as the rest of the rhel 8 releases but points to a different
nightly repo for testing purposes. Test cases are added. The changes
between rhel 8.3 and 8.4 are as follows:

There is now a hybrid boot partition scheme for x86_64. x86_64 images
now use uefi boot and have 3 gpt partitions: a small unformated
partition for mbr compatibility, an efi boot partition of type vfat, and
a root partition of type xfs. The packages grub2-efi-x64 and shim-x64
are added as bootloader packages for all x86_64 images.

For qcow2 images ro is added as a kernel option and the following
packages are added (+) or removed (-):
+ dosfstools
+ efi-filesystem
+ efivar
+ efivar-libs
+ grub2-efi-x64
+ shim-x64
- rhn-client-tools
- rhnlib
- rhnsd
- rhn-setup
2020-11-19 10:36:49 +01:00
Ondřej Budai
117da5aa8a composer: add sanity checks
Running composer without workers or APIs enabled is pretty much a no-op.
Let's forbid that.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2020-11-17 17:01:18 +00:00