Commit graph

3674 commits

Author SHA1 Message Date
Antonio Murdaca
b2d18166de test/data/manifests: regenerate
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-14 17:31:40 +01:00
Antonio Murdaca
5f2ad326a6 internal/distro/rhel{86,90}: drop console kargs from raw image deployment
Using the simplified installer we were experiencing slow system boots.
Turns out we're incurring into https://bugzilla.redhat.com/show_bug.cgi?id=1839923
This patch just drops the console kargs - to be aligned with the
anaconda installer that doesn't experience this slow down.
The slow down doesn't happen on virtual machines as there's always a
ttyS0 there

Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-03-14 17:31:40 +01:00
Gianluca Zuccarelli
19e2fb7fb5 template: composer dashboard queries
Tidy up the queries for the composer dashboard
and making them more readable in grafana. Additionally
add some fallback values for when empty query results
are returned from prometheus.
2022-03-14 16:11:05 +01:00
Gianluca Zuccarelli
1f2fd8cb76 templates: worker depsolve error display
Fix the display of the depsolve error rate
panel. The panel had an incorrect min value of
3 (or 300%).
2022-03-14 16:11:05 +01:00
Jakub Rusz
c91131ee0c github workflows: modify Gitlab CI trigger
In 5e639cba6f the context of the Trigger
Gitalb CI workflow changed and the context
"github.event.pull_request.draft" is no longer available so the
condition for SKIP_CI didn't work. This can be fixed by getting the
variable in the previous workflow and passin it as artifact. Docs:
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#using-data-from-the-triggering-workflow
2022-03-14 14:40:23 +02:00
Jakub Rusz
d8ea259f8b ci: run ci_details.sh in before_script
This is a nice script showing potentially useful details about the
runner so let's execute it at the begining of each job.
2022-03-14 14:24:59 +02:00
Ondřej Budai
418ae32cf8 packer: fix the secret ID variable in get_koji_creds.sh
Oops, we should probably start testing this.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-14 10:27:28 +01:00
Ondřej Budai
424a741de6 packer: make subscribing optional
We don't want to subscribe Fedora.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 22:31:40 +01:00
Ondřej Budai
c46376aea2 packer: add support for koji credentials
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2dd5ae7bca packer: skip retrieving of creds if their ARN is not specified
So we can have workers without public cloud creds.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
4c0ba50ea1 packer: remove config tinkering from worker_service.sh
Let's set each cloud section of the config in the respective cloud script.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2813507ac9 packer: split worker_external_creds.sh into one script per cloud
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2e7815bf53 packer: move worker-config creation to ansible
I think it untangles the initialization a bit and allows me to do some more
refactorings.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Tom Gundersen
2a4d4c4d49 dnf-json: use the default connection timeout
By default `timeout` is 30 seconds, but we had it set to 5. Drop
the override and use the default.

This has two effects: it increases the time before we give up on
connecting (as it says on the tin), and it also increases the time
download has to be slow for before we give up.

Internally, we were seing failures in downlaoding metadata from ODCS
and similar issues have occurred in CI too.

The potential downside to this is in case of having several mirrors
this means it takes longer before giving up on a bad one and trying
a better one. But slow is better than broken, so for now rever to
the default behavior.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2022-03-12 09:09:13 +01:00
Tomas Hozza
562225af4c osbuild-pipeline: use repo name from the request if provided
Almost all repo configurations used for generating image test cases
using `osbuild-pipeline` have `name` defined. Make sure that the repo
name provided in the compose request is used when depsolving.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-12 08:36:40 +01:00
Tomas Hozza
13a9022fd8 rpmmd: rename toDNFRepoConfig() argument i -> repoID
Rename the method argument name to make its purpose obvious.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-12 08:36:40 +01:00
Tomas Hozza
180290d016 dnf-json: use repository name from the request if provided
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-12 08:36:40 +01:00
Tomas Hozza
43dafe87fb rpmmd: pass repo name to dnf-json
The repo name is already part of the `rpmmd.RepoConfig` structure. Do
not ignore when calling `dnf-json` and and pass it the value.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-12 08:36:40 +01:00
Tomas Hozza
f9d0412316 dnf-json: do not use reponame as repoid.
Repo name defaults to the repo ID if the name is not set. `dnf-json`
should not rely on the `reponame` being set to the ID and intsead
return the actual `repoid`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-12 08:36:40 +01:00
Ondřej Budai
c4c7f44fcb dbjobqueue: reimplement the jobqueue to use only one listening connection
Previously, all dequeuers (goroutines waiting for a job to be dequeued) were
listening for new messages on postgres channel jobs (LISTEN jobs). This didn't
scale well as each dequeuer required to have its own DB connection and the
number of DB connections is hard-limited in the pool's config.

I changed the logic to work somewhat differently: dbjobqueue.New() now spawns
a goroutine that listens on the postgres channel. If there's a new message,
the goroutine just wakes up all dequeuers using a standard go channel.
Go channels are cheap so this should scale much better.

A test was added that confirms that 100 dequeuers are not a big deal now. This
test failed when I tried to run on it on the previous commit. I tried even 1000
locally and it was still fine.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 16:04:52 +01:00
Ondřej Budai
c8dbe0de74 dbjobqueue: remove unused variables from Dequeue
Removing queued_at and started_at is pretty straightforward, it wasn't needed.
Removing token might seem concerning but basically we were just pulling
the same value from DB as we were pushing there. I think there's no value in
doing that.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 16:04:52 +01:00
Sanne Raymaekers
318a4525c6 cmd/osbuild-worker: dnf-json returns MarkingErrors (plural) 2022-03-11 10:13:27 +01:00
Feng Huang
c64eb98011 use app-sre packer image
Signed-off-by: Feng Huang <fehuang@redhat.com>
2022-03-11 09:24:26 +01:00
Ondřej Budai
72de1b3bbe packer: don't save the AMIs on PRs
This should save us a ton of resources as we don't use AMIs from PRs.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
ad15179faf packer: build Fedora images
The decision logic which jobs to run is quite confusing but that's how we
roll for now:

Jenkins builds RHEL images only on main
Schutzbot builds RHEL images only in PRs
Schutzbot builds Fedora images on both PRs and on main

To achieve this, the commit re-enables running Packer on main on Schutzbot.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
ec070612ff packer: remove RHEL and x86_64-specific bits
Arch was easy.

For passing the repository distribution and osbuild_commit (it can be
different for each distro), I decided to go in the way of ansible
inventory directories. It adds a bit of structure but I think it's
the most clean solution.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
cd394bf67d packer: add default to aws auth variables
So you don't have to pass these if packer is supposed to find them
on its own (instance profile, local profile).

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
4ae71d3f3d packer: move all RHEL-specific options to a source block
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
22ec89f956 packer: add more tags identifying the image
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
7301ea6b9d packer: use newer (=faster) instances
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
8664c1449a packer: reuse the build user for the ansible provisioner
We want to build multiple images at once and some of them use a different user.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
e45578d3b0 packer: remove the ami_id variable
We want to build multiple images at once so they have to be defined elsewhere.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
5ecbfbad9e packer: rename composer.pkr.hcl to worker.pkr.hcl
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Achilleas Koutsou
e5675efc4a github: fix job names and IDs for the tests workflow
Flip the incorrect flip that happened in
e4baddfad1
2022-03-10 10:54:20 +01:00
Achilleas Koutsou
93c6336e24 reporegistry: remove unused ReposByArch() function
Since all calls to `ReposByArch()` have been replaced by
`ReposByArchName()`, remove the function and updates tests.
2022-03-09 23:22:06 +00:00
Achilleas Koutsou
baa2339de4 weldr: remove the (host) architecture member from the API struct
The `API.arch` member was (mostly) used to read the name of the
architecture.
The only non-name use was for the purposes of reading RPM repositories
from the configuration, in `reporegistry.ReposByArch()`, a thin wrapper
around `reporegistry.ReposByArchName()`.
Removing the `arch` member from the API and using the new `archName`
that is set up in the API constructor lets us control the arch name that
is set without relying on a valid `distro.Arch` object being available
(which would depend on having a valid `distro.Distro` object).

Replaced all calls to `ReposByArch()` with `ReposByArchName()` which
depends on the arch and distro name strings instead of a full
`distro.Arch`.
2022-03-09 23:22:06 +00:00
Achilleas Koutsou
f9eb98b14d weldr: allow running on unsupported distros
When the host distribution is not known or supported, instead of failing
with an error, print a warning to the log and initialise the API with
the architecture name and distro name.
This enables running the weldr API on unsupported distros for
cross-distro building.

Guards against a nil arch member when initialising the store.
2022-03-09 23:22:06 +00:00
Diaa Sami
3ab2725042 koji: Reduce excessive logging by retryablehttp
Use LeveledLogger
Fixes COMPOSER-1394
2022-03-09 23:18:25 +00:00
Juan Abia
4c5de045ff test: add test purpose
Most test scripts don't have any documentation regarding it's purpose,
although it can be guessed by the code. There's value in adding this
small comment.

[skip-ci]
2022-03-09 22:46:25 +01:00
Laura Barcziova
a51f52f041 Packit: build SRPMs in Copr
Add srpm_build_deps key to the Packit configuration to specify the needed dependencies for SRPM build
and indicate to build SRPM in Copr.
2022-03-09 22:25:54 +01:00
Jakub Rusz
67c5f91ea4 mockbuild: use public EPEL-9 2022-03-09 22:03:39 +01:00
Sanne Raymaekers
663f1dcbee cmd/osbuild-worker: Pass bucket config to job implementation 2022-03-09 13:28:48 +01:00
kingsleyzissou
0c762d4ca0 Post release version bump
[skip ci]
2022-03-09 10:10:24 +00:00
Gianluca Zuccarelli
761aab6cac cloudapi/v2: add error object to ImageStatus
Add an error object to the ComposeStatus.ImageStatus.
The error object contains a human-readable error reason
and optional details in the case of an error.
2022-03-09 08:49:37 +00:00
Sanne Raymaekers
63a0bbc1f2 cmd/osbuild-worker: Configure s3 bucket on the worker itself
Parameterize aws section of worker config. If credentials is empty,
the iam role will be used.
2022-03-08 21:58:11 +01:00
Tomas Hozza
3a2002f557 CI: temporarily disable Installer test on el9/c9s due to rhbz#2059565
Temporarily disable Installer test case in the CI on RHEL-9 and CentOS
Stream 9 until https://bugzilla.redhat.com/show_bug.cgi?id=2059565 is
resolved. This test case is now consistently failing due to the
mentioned bug and makes it impossible for the CI to pass cleanly.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-03-08 13:53:16 +01:00
Jakub Rusz
05be7c4096 test/image-tests: temporarily skip azure_rhui image testing
This manifest is intended only for internal use and is currently failing
in nightly pipelines. This will be moved to a different test script in
the future, see COMPOSER-1397.
2022-03-08 13:19:34 +01:00
Ondřej Budai
2ea2e9be09 templates/composer: give access to Fedora org
We will be using both offline tokens (account_id) and service accounts
(rh-org-id) for now.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-08 13:06:35 +01:00
Achilleas Koutsou
f449cd45e8 github/workflows: check dnf-json with pylint
Use a fedora container for dnf dependency.
Add a pylintrc file to the project with 120 max line length and
docstrings ignored.
2022-03-08 12:42:12 +01:00
Achilleas Koutsou
e4baddfad1 github/workflows: fix job names for the tests workflow
lint and unit-tests were accidentally swapped.
2022-03-08 12:42:12 +01:00