Commit graph

3858 commits

Author SHA1 Message Date
Ondřej Budai
80af9cac98 cloudapi: enable edge-commit on Fedora
We have to do a small hack to enable edge-commit on Fedora because its name
is different. We can also change this in the image definition but I want to
iterate quickly on the Fedora Integration MVP and don't want to run in
any conflicts with
https://github.com/osbuild/osbuild-composer/pull/2461

This commit also enables a test for Fedora IoT built through the API.

While enabling the test, I also simplified our decision logic for SSH_USER
and DISTRO.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-25 13:23:32 +01:00
Antonio Murdaca
bf46113251 test/data/repositories: bump cs8
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-24 18:23:39 +01:00
Jakub Rusz
7bf019efcb test/repositories: update rhel-90 repository gpg keys
SHA-1 keys are deprecated, updating to the newly used ones.
2022-03-24 18:23:39 +01:00
yih
cd0f762e58 add fdo sshkey and rootcert test 2022-03-24 18:23:39 +01:00
Antonio Murdaca
66bc1e8b85 rhel8/9: fix path to fdo diun root certificates
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-24 18:23:39 +01:00
Tomas Hozza
2ec55b4c53 RHEL-90: use XBOOTLDR partition GUID for /boot
Use the Extended Bootloader Partition GUID for `/boot`, instead of the
Linux filesystem data GUID. This is useful for autodetection of a
partition purpose based on its GUID without reading the `/etc/fstab`
first.

Ensure that when creating mountpoints, e.g. when converting the
partitions layout to LVM, the `/boot` partition get the proper GUID
assigned.

Regenerate RHEL-90 and centos-9 image test cases.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=2057231
2022-03-24 15:40:10 +01:00
Sanne Raymaekers
9368b60401 templates/composer: Add prod service accounts owner 2022-03-23 16:43:10 +01:00
schutzbot
3571447fc4 Post release version bump
[skip ci]
2022-03-23 14:30:52 +00:00
Simon Steinbeiss
11593b8318 create-tag: Set the release time to 13UTC 2022-03-23 13:57:56 +01:00
Simon Steinbeiss
53cb872aca create-tag: Don't check week for manual triggers
When a release is manually triggered we want to ignore whether it is an
odd or even week but allow the release in any case.
2022-03-23 13:57:56 +01:00
Simon Steinbeiss
64be96017f create-tag: Fix bash typo
This seems to be a bit special in GitHub Actions with bash:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-environment-variable
2022-03-23 13:57:56 +01:00
Tomas Hozza
60db6ad06f image tests: update GPG keys used for RHEL-9.0 repos
SHA-1 key is no longer supported, update it with a new one from
redhat-release.

Related rhbz#2058497

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-23 13:22:13 +01:00
Simon Steinbeiss
86bf942780 create-tag: Fix upstream release schedule
We're counting the even and odd weeks since the Unix epoch, so this
doesn't match the odd/even calendar weeks. Consequently let's switch to
odd weeks for composer.
2022-03-23 11:30:32 +01:00
Tomas Hozza
d11a2f3bc7 api.sh: encrypt the DB dump artifact 2022-03-23 09:45:09 +01:00
Simon Steinbeiss
f521e9a800 Enable scheduled upstream releases
Instead of a human pushing a tag with the release notes let a bot do the
work.
The bot is part of our composite action in osbuild/release-action on the
create-tag branch. It calculates the next subsequent release version and
creates a tag based on pull request titles associated with the changes
since the last release.
Finally the tag is pushed to the repository.

Unfortunately GH Actions don't allow for reliably fortnightly schedules,
so we do an additional check that determines if this is an even or an
odd week. This will help with correctly scheduling alternating osbuild
and osbuild-composer releases.
2022-03-22 17:50:57 +01:00
Ondřej Budai
e9ce9370c6 dbjoqbqueue: actually use the transaction object when a tx is created
Transactions are tied to a connection so this is actually not a functional
change. Nevertheless, I think it's nice to explicitly state that we are
using a transaction.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-22 17:49:22 +01:00
Ondřej Budai
187eb188da dbjoqbqueue: wait for listener to become ready before returning from New
Otherwise, there might be an already waiting dequeuer and if something is
enqueued before `sqlListen` is called, we will lost this notification.

Also, a small log message was added when shutting down the listener.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-22 17:49:22 +01:00
Tom Gundersen
c21596cd99 containers/composer: forward SIGTERM to children
In docker compose and openshift SIGTERM is used to tell containers
to shut down gracefully. Forward the signal to the processes we spawned.
2022-03-22 14:17:37 +01:00
Tom Gundersen
367444635a containers/composer: terminate composer first
Composer may depend on dnf-json and the worker to shut down cleanly.
2022-03-22 14:17:37 +01:00
Tom Gundersen
c3d66b5a33 cmd/composer: gracefully shut down on SIG{INT,TERM}
Call `Shutdown()` on all http servers. This means we will finish processing
any pending requests (including depsolving), but we will not listen to new
ones.

In particular, we will not answer to the readiness probe, so no new traffic
will be routed to this container.

Once all pending requests have been handled composer will shut down
gracefully and the liveness probe will return failure.

Note that in order for this to work correctly no requests should ever take longer
than the shutdown timeout (by default 30s).
2022-03-22 14:17:37 +01:00
Tom Gundersen
d3cd3197c0 container: make liveness probe independent of webserver
Currently liveness and readiness was treated the same. However, their
behaviour at shutdown is meant to be different. When a service is not read
no new connections are made to it, and when a service is not live it can be
cleaned up.

By considering our service live if and only if it listens to HTTP requests we
don't have the opportunity to clean up after we stop listening to new requests.

Leave readiness probes as they are, and instead use a file in the filesystem to
indicate when the service is live. It is created before composer is spawned and
deleted once composer exits.
2022-03-22 14:17:37 +01:00
Jakub Rusz
15c2044b3c tests/upgrade: update gpg key
We need to use a new gpg key after the SHA-1 deprecation. Also don't
fail immediately on compose failure to be able to retrieve logs from the
test VM.
2022-03-22 10:54:30 +01:00
Ondřej Budai
67e55eaea8 gitlab: run containerbuild on RHEL
Otherwise, we're running into
https://bugzilla.redhat.com/show_bug.cgi?id=2065292
and when I tried implementing a workaround, I ran into
https://bugzilla.redhat.com/show_bug.cgi?id=1897579

Gah.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-21 16:45:49 +01:00
Ondřej Budai
99aad294dd deploy: work around a podman bug in CS8
See the comment.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-21 16:45:49 +01:00
Sanne Raymaekers
f0a17d19f0 templates/composer: Add stage service accounts owner 2022-03-21 12:57:32 +01:00
Jakub Rusz
46a79a48da workflows: Fix Gitlab CI trigger + revert debug
Previous implementation added single quotes to the git command which
made it not trigger the Gitlab CI at all. Changing it to clasic bash if
condition.
2022-03-21 10:42:28 +01:00
Sanne Raymaekers
2023f7731d worker: Support client_credentials grant type in client
This will allow us to use the service accounts which work against
identity.api.openshift.com. These are much easier to manage, especially
with the new multi-tenancy, as there's a single page to create/expire
them across an account.

They also have the added benefit of not expiring automatically when
they're not used like offline tokens, and immediate expiration when
desired.
2022-03-21 09:43:43 +01:00
Sanne Raymaekers
8900bcec40 worker: Client lazy token refresh 2022-03-21 09:43:43 +01:00
Sanne Raymaekers
8a6d6ed6cf worker: Clean up worker client config 2022-03-21 09:43:43 +01:00
Jakub Rusz
eb4c9be168 workflows: debug Gitlab CI trigger 2022-03-18 12:59:40 +01:00
Sanne Raymaekers
815d0ad65b osbuild-worker: Log unexpected dnf-json errors
These errors result in a 5xx status for the depsolve job, marked as
internal failure, it's useful to log them.
2022-03-18 10:14:06 +01:00
Ondřej Budai
9ca74694a7 packer: use unique name tag for Fedora workers
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-16 12:58:05 +01:00
Tomas Hozza
e5595667bc test/api.sh: move the DB dump to the cleanup() function
Previously, the DB was not dumped in case the compose failed. Ensure
that the DB is dumped before the script exits in any case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-16 09:03:47 +00:00
Tomas Hozza
e8a347d1e8 test/api.sh: do not use /tmp, but $WORKDIR
Do not create files directly in `/tmp`, but use `$WORKDIR`, which is a
temporary directory for transient files, which gets cleaned up when the
test case finishes. Without this change, running `api.sh` twice fails
the second time.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-16 09:03:47 +00:00
Antonio Murdaca
b2d18166de test/data/manifests: regenerate
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-14 17:31:40 +01:00
Antonio Murdaca
5f2ad326a6 internal/distro/rhel{86,90}: drop console kargs from raw image deployment
Using the simplified installer we were experiencing slow system boots.
Turns out we're incurring into https://bugzilla.redhat.com/show_bug.cgi?id=1839923
This patch just drops the console kargs - to be aligned with the
anaconda installer that doesn't experience this slow down.
The slow down doesn't happen on virtual machines as there's always a
ttyS0 there

Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-14 17:31:40 +01:00
Gianluca Zuccarelli
19e2fb7fb5 template: composer dashboard queries
Tidy up the queries for the composer dashboard
and making them more readable in grafana. Additionally
add some fallback values for when empty query results
are returned from prometheus.
2022-03-14 16:11:05 +01:00
Gianluca Zuccarelli
1f2fd8cb76 templates: worker depsolve error display
Fix the display of the depsolve error rate
panel. The panel had an incorrect min value of
3 (or 300%).
2022-03-14 16:11:05 +01:00
Jakub Rusz
c91131ee0c github workflows: modify Gitlab CI trigger
In 5e639cba6f the context of the Trigger
Gitalb CI workflow changed and the context
"github.event.pull_request.draft" is no longer available so the
condition for SKIP_CI didn't work. This can be fixed by getting the
variable in the previous workflow and passin it as artifact. Docs:
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#using-data-from-the-triggering-workflow
2022-03-14 14:40:23 +02:00
Jakub Rusz
d8ea259f8b ci: run ci_details.sh in before_script
This is a nice script showing potentially useful details about the
runner so let's execute it at the begining of each job.
2022-03-14 14:24:59 +02:00
Ondřej Budai
418ae32cf8 packer: fix the secret ID variable in get_koji_creds.sh
Oops, we should probably start testing this.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-14 10:27:28 +01:00
Ondřej Budai
424a741de6 packer: make subscribing optional
We don't want to subscribe Fedora.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 22:31:40 +01:00
Ondřej Budai
c46376aea2 packer: add support for koji credentials
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2dd5ae7bca packer: skip retrieving of creds if their ARN is not specified
So we can have workers without public cloud creds.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
4c0ba50ea1 packer: remove config tinkering from worker_service.sh
Let's set each cloud section of the config in the respective cloud script.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2813507ac9 packer: split worker_external_creds.sh into one script per cloud
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2e7815bf53 packer: move worker-config creation to ansible
I think it untangles the initialization a bit and allows me to do some more
refactorings.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Tom Gundersen
2a4d4c4d49 dnf-json: use the default connection timeout
By default `timeout` is 30 seconds, but we had it set to 5. Drop
the override and use the default.

This has two effects: it increases the time before we give up on
connecting (as it says on the tin), and it also increases the time
download has to be slow for before we give up.

Internally, we were seing failures in downlaoding metadata from ODCS
and similar issues have occurred in CI too.

The potential downside to this is in case of having several mirrors
this means it takes longer before giving up on a bad one and trying
a better one. But slow is better than broken, so for now rever to
the default behavior.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2022-03-12 09:09:13 +01:00
Tomas Hozza
562225af4c osbuild-pipeline: use repo name from the request if provided
Almost all repo configurations used for generating image test cases
using `osbuild-pipeline` have `name` defined. Make sure that the repo
name provided in the compose request is used when depsolving.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-12 08:36:40 +01:00
Tomas Hozza
13a9022fd8 rpmmd: rename toDNFRepoConfig() argument i -> repoID
Rename the method argument name to make its purpose obvious.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-12 08:36:40 +01:00