Commit graph

3720 commits

Author SHA1 Message Date
Alexander Todorov
7ea3920cdf Add Xiaofeng to notifications for nightly pipeline 2022-03-28 11:30:44 +02:00
Diaa Sami
68639b4bf9 koji: increment retry counter only when retrying 2022-03-26 09:33:36 +01:00
Diaa Sami
6b08b8ed63 koji: don't decrement retry counter on the first call
After examining the logic of retryablehttp library, the callback does not happen for the first HTTP call, so no need to decrement when counting.
2022-03-26 09:33:36 +01:00
Diaa Sami
3496efe70d koji: initialize retryable client properly
Previously used client has MaxRetries of zero, so was not effectively
retrying
Fixes COMPOSER-1420
2022-03-26 09:33:36 +01:00
Jakub Rusz
f1006c5adb ci/tests: re-enable Installers.sh test
Related bug has been fixed. Updating snapshots and enabling the test.
2022-03-25 16:00:39 +01:00
Jakub Rusz
b639a44899 tests/ostree-simplified-installer: destroy VM when installation is stuck
This test get stuck randomly on centos-stream-8 and is making the CI
unreliable. Adding hard wait limit and destroying the VM afterwards
helps the test get unstuck and continue as expected. See
https://github.com/osbuild/osbuild-composer/issues/2413 for details.
2022-03-25 16:00:39 +01:00
Jakub Rusz
fbefd432c1 Schutzfile: Update centos-stream-9 repos 2022-03-25 16:00:39 +01:00
Jakub Rusz
417c381b44 Schutzfile: fix centos-stream-9 repo pin
During runner repo override the runner name is compared with the key in
Schutzfile. There was the word 'stream' missing for centos runners so
the repos were never acutally overwritten before. See
https://github.com/osbuild/gitlab-ci-terraform-executor/blob/main/prepare#L69
for more context
2022-03-25 16:00:39 +01:00
Jakub Rusz
8390709614 ci/details: list system repositories and run in after_script too
It is good to know what system repositories are used exactly on the
runner. Also running the ci_details.sh in after_script can help during
any debugging as packages that got updated or installed during the run
can be easily identified.
2022-03-25 16:00:39 +01:00
Ondřej Budai
fc86ffd968 container: fix liveness probe
We don't have permissions to write to /run when running on OpenShift so let's
just use /tmp and change the filename to prevent any conflicts.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-25 14:02:12 +01:00
Ondřej Budai
7a194bfcb5 cloudapi: require gpgkey if check_gpg is set
If a user requires that packages from a certain repository are checked using
a GPG key, they should specify it. Now, this is enforced to catch this issue
earlier than in osbuild.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-25 13:23:32 +01:00
Ondřej Budai
af44202b1c cloudapi: rename gpg_key field to gpgkey
Oh no, we made a mistake here: Both our json repositories and repo files in
/etc/yum.repos.d have the GPG key in a field named `gpgkey`. Unfortunately,
cloudapi uses a field named `gpg_key`. One consequence of this issue is that
our api.sh test is meant to pass GPG keys in the compose request but since
it's using a bad field name (`gpgkey`), the key is actually not used.

I've decided to fix this in cloudapi: The `gpg_key` field is now renamed to
`gpgkey`. This is a breaking change but no one is using this API anyway so
we think it's better to do this now than introducing weird backward
compatible hacks.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-25 13:23:32 +01:00
Ondřej Budai
80af9cac98 cloudapi: enable edge-commit on Fedora
We have to do a small hack to enable edge-commit on Fedora because its name
is different. We can also change this in the image definition but I want to
iterate quickly on the Fedora Integration MVP and don't want to run in
any conflicts with
https://github.com/osbuild/osbuild-composer/pull/2461

This commit also enables a test for Fedora IoT built through the API.

While enabling the test, I also simplified our decision logic for SSH_USER
and DISTRO.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-25 13:23:32 +01:00
Antonio Murdaca
bf46113251 test/data/repositories: bump cs8
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-24 18:23:39 +01:00
Jakub Rusz
7bf019efcb test/repositories: update rhel-90 repository gpg keys
SHA-1 keys are deprecated, updating to the newly used ones.
2022-03-24 18:23:39 +01:00
yih
cd0f762e58 add fdo sshkey and rootcert test 2022-03-24 18:23:39 +01:00
Antonio Murdaca
66bc1e8b85 rhel8/9: fix path to fdo diun root certificates
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-24 18:23:39 +01:00
Tomas Hozza
2ec55b4c53 RHEL-90: use XBOOTLDR partition GUID for /boot
Use the Extended Bootloader Partition GUID for `/boot`, instead of the
Linux filesystem data GUID. This is useful for autodetection of a
partition purpose based on its GUID without reading the `/etc/fstab`
first.

Ensure that when creating mountpoints, e.g. when converting the
partitions layout to LVM, the `/boot` partition get the proper GUID
assigned.

Regenerate RHEL-90 and centos-9 image test cases.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=2057231
2022-03-24 15:40:10 +01:00
Sanne Raymaekers
9368b60401 templates/composer: Add prod service accounts owner 2022-03-23 16:43:10 +01:00
schutzbot
3571447fc4 Post release version bump
[skip ci]
2022-03-23 14:30:52 +00:00
Simon Steinbeiss
11593b8318 create-tag: Set the release time to 13UTC 2022-03-23 13:57:56 +01:00
Simon Steinbeiss
53cb872aca create-tag: Don't check week for manual triggers
When a release is manually triggered we want to ignore whether it is an
odd or even week but allow the release in any case.
2022-03-23 13:57:56 +01:00
Simon Steinbeiss
64be96017f create-tag: Fix bash typo
This seems to be a bit special in GitHub Actions with bash:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-environment-variable
2022-03-23 13:57:56 +01:00
Tomas Hozza
60db6ad06f image tests: update GPG keys used for RHEL-9.0 repos
SHA-1 key is no longer supported, update it with a new one from
redhat-release.

Related rhbz#2058497

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-23 13:22:13 +01:00
Simon Steinbeiss
86bf942780 create-tag: Fix upstream release schedule
We're counting the even and odd weeks since the Unix epoch, so this
doesn't match the odd/even calendar weeks. Consequently let's switch to
odd weeks for composer.
2022-03-23 11:30:32 +01:00
Tomas Hozza
d11a2f3bc7 api.sh: encrypt the DB dump artifact 2022-03-23 09:45:09 +01:00
Simon Steinbeiss
f521e9a800 Enable scheduled upstream releases
Instead of a human pushing a tag with the release notes let a bot do the
work.
The bot is part of our composite action in osbuild/release-action on the
create-tag branch. It calculates the next subsequent release version and
creates a tag based on pull request titles associated with the changes
since the last release.
Finally the tag is pushed to the repository.

Unfortunately GH Actions don't allow for reliably fortnightly schedules,
so we do an additional check that determines if this is an even or an
odd week. This will help with correctly scheduling alternating osbuild
and osbuild-composer releases.
2022-03-22 17:50:57 +01:00
Ondřej Budai
e9ce9370c6 dbjoqbqueue: actually use the transaction object when a tx is created
Transactions are tied to a connection so this is actually not a functional
change. Nevertheless, I think it's nice to explicitly state that we are
using a transaction.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-22 17:49:22 +01:00
Ondřej Budai
187eb188da dbjoqbqueue: wait for listener to become ready before returning from New
Otherwise, there might be an already waiting dequeuer and if something is
enqueued before `sqlListen` is called, we will lost this notification.

Also, a small log message was added when shutting down the listener.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-22 17:49:22 +01:00
Tom Gundersen
c21596cd99 containers/composer: forward SIGTERM to children
In docker compose and openshift SIGTERM is used to tell containers
to shut down gracefully. Forward the signal to the processes we spawned.
2022-03-22 14:17:37 +01:00
Tom Gundersen
367444635a containers/composer: terminate composer first
Composer may depend on dnf-json and the worker to shut down cleanly.
2022-03-22 14:17:37 +01:00
Tom Gundersen
c3d66b5a33 cmd/composer: gracefully shut down on SIG{INT,TERM}
Call `Shutdown()` on all http servers. This means we will finish processing
any pending requests (including depsolving), but we will not listen to new
ones.

In particular, we will not answer to the readiness probe, so no new traffic
will be routed to this container.

Once all pending requests have been handled composer will shut down
gracefully and the liveness probe will return failure.

Note that in order for this to work correctly no requests should ever take longer
than the shutdown timeout (by default 30s).
2022-03-22 14:17:37 +01:00
Tom Gundersen
d3cd3197c0 container: make liveness probe independent of webserver
Currently liveness and readiness was treated the same. However, their
behaviour at shutdown is meant to be different. When a service is not read
no new connections are made to it, and when a service is not live it can be
cleaned up.

By considering our service live if and only if it listens to HTTP requests we
don't have the opportunity to clean up after we stop listening to new requests.

Leave readiness probes as they are, and instead use a file in the filesystem to
indicate when the service is live. It is created before composer is spawned and
deleted once composer exits.
2022-03-22 14:17:37 +01:00
Jakub Rusz
15c2044b3c tests/upgrade: update gpg key
We need to use a new gpg key after the SHA-1 deprecation. Also don't
fail immediately on compose failure to be able to retrieve logs from the
test VM.
2022-03-22 10:54:30 +01:00
Ondřej Budai
67e55eaea8 gitlab: run containerbuild on RHEL
Otherwise, we're running into
https://bugzilla.redhat.com/show_bug.cgi?id=2065292
and when I tried implementing a workaround, I ran into
https://bugzilla.redhat.com/show_bug.cgi?id=1897579

Gah.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-21 16:45:49 +01:00
Ondřej Budai
99aad294dd deploy: work around a podman bug in CS8
See the comment.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-21 16:45:49 +01:00
Sanne Raymaekers
f0a17d19f0 templates/composer: Add stage service accounts owner 2022-03-21 12:57:32 +01:00
Jakub Rusz
46a79a48da workflows: Fix Gitlab CI trigger + revert debug
Previous implementation added single quotes to the git command which
made it not trigger the Gitlab CI at all. Changing it to clasic bash if
condition.
2022-03-21 10:42:28 +01:00
Sanne Raymaekers
2023f7731d worker: Support client_credentials grant type in client
This will allow us to use the service accounts which work against
identity.api.openshift.com. These are much easier to manage, especially
with the new multi-tenancy, as there's a single page to create/expire
them across an account.

They also have the added benefit of not expiring automatically when
they're not used like offline tokens, and immediate expiration when
desired.
2022-03-21 09:43:43 +01:00
Sanne Raymaekers
8900bcec40 worker: Client lazy token refresh 2022-03-21 09:43:43 +01:00
Sanne Raymaekers
8a6d6ed6cf worker: Clean up worker client config 2022-03-21 09:43:43 +01:00
Jakub Rusz
eb4c9be168 workflows: debug Gitlab CI trigger 2022-03-18 12:59:40 +01:00
Sanne Raymaekers
815d0ad65b osbuild-worker: Log unexpected dnf-json errors
These errors result in a 5xx status for the depsolve job, marked as
internal failure, it's useful to log them.
2022-03-18 10:14:06 +01:00
Ondřej Budai
9ca74694a7 packer: use unique name tag for Fedora workers
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-16 12:58:05 +01:00
Tomas Hozza
e5595667bc test/api.sh: move the DB dump to the cleanup() function
Previously, the DB was not dumped in case the compose failed. Ensure
that the DB is dumped before the script exits in any case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-16 09:03:47 +00:00
Tomas Hozza
e8a347d1e8 test/api.sh: do not use /tmp, but $WORKDIR
Do not create files directly in `/tmp`, but use `$WORKDIR`, which is a
temporary directory for transient files, which gets cleaned up when the
test case finishes. Without this change, running `api.sh` twice fails
the second time.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-16 09:03:47 +00:00
Antonio Murdaca
b2d18166de test/data/manifests: regenerate
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-14 17:31:40 +01:00
Antonio Murdaca
5f2ad326a6 internal/distro/rhel{86,90}: drop console kargs from raw image deployment
Using the simplified installer we were experiencing slow system boots.
Turns out we're incurring into https://bugzilla.redhat.com/show_bug.cgi?id=1839923
This patch just drops the console kargs - to be aligned with the
anaconda installer that doesn't experience this slow down.
The slow down doesn't happen on virtual machines as there's always a
ttyS0 there

Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-14 17:31:40 +01:00
Gianluca Zuccarelli
19e2fb7fb5 template: composer dashboard queries
Tidy up the queries for the composer dashboard
and making them more readable in grafana. Additionally
add some fallback values for when empty query results
are returned from prometheus.
2022-03-14 16:11:05 +01:00
Gianluca Zuccarelli
1f2fd8cb76 templates: worker depsolve error display
Fix the display of the depsolve error rate
panel. The panel had an incorrect min value of
3 (or 300%).
2022-03-14 16:11:05 +01:00