Commit graph

3107 commits

Author SHA1 Message Date
sanne
f284e2a932 .github: Bump fedora container version for rpmlint 2021-12-17 13:13:05 +01:00
sanne
8406ada6f5 worker: Treat a non echo.HTTPError like a regular error 2021-12-17 13:13:05 +01:00
Tomas Hozza
30f64d190d osbuild2: ensure that empty sysconfig options members are omitted
The `Kernel` and `Network` members of the sysconfig stage options
structure were previously not declared as pointers. As a result, they
always appeared in the resulting JSON object, even though they were
empty. Use pointers to ensure that the members are omitted from the
resulting JSON object, if they were not defined.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:47:28 +01:00
Tomas Hozza
96000173db osbuild2: do not use reflect in pam.limits.conf stage
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Tomas Hozza
09cd5b3576 osbuild2: support org.osbuild.dnf-automatic.config stage
Add support for the new `org.osbuild.dnf-automatic.config` stage for
configuring DNF Automatic.

Add appropriate new unit tests for the stage implementation and modify
necessary existing unit tests.

Related to https://github.com/osbuild/osbuild/pull/936

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Tomas Hozza
97ef7fbf28 osbuild2: support org.osbuild.yum.repos stage
Add support for the new `org.osbuild.yum.repos` stage for creating DNF /
YUM repository configuration files.

Add appropriate new unit tests for the stage implementation and modify
necessary existing unit tests.

Related to https://github.com/osbuild/osbuild/pull/932

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Tomas Hozza
37a39743bc osbuild2: support PermitRootLogin in sshd.config stage
Add support for the `PermitRootLogin` option in the `sshd.config` stage.
Valid values can be of type `bool` or `string`. Due to this reason, a
custom interface type is defined and a custom `UnmarshalJSON()` method is
defined for the `SshdConfigConfig` structure.

Modify unit tests to test the newly added option and test
(un)marhsalling of valid values of both types.

Related to https://github.com/osbuild/osbuild/pull/917

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Tomas Hozza
33c7da9dc3 tools: use image_type_tags in repos used for image test cases
Use image_type_tags in repo definitions used for generating image test
cases. Enhance the `generate-test-cases` script to take the
image_type_tags into account when creating a compose request for a
specific image test case.

The motivation for the change is to keep the list of repositories in the
compose request of a image test case as small as possible. Previously,
all of the defined repositories were part of each image test cases for a
specific architecture, even those that were not needed.

Regenerate affected image test cases.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 18:07:50 +01:00
sanne
60d4f5a751 composer: Disable artifacts for the service
When backed by a DB, composer has no need of a queue directory.

This also addresses "Error moving artifacts for job" logging noise.

Signed-off-by: sanne <sanne.raymaekers@gmail.com>
2021-12-16 17:04:08 +00:00
Diaa Sami
487e2d0669 internal/cloud: use logrus for logging
and log upload & sharing failures
2021-12-16 11:58:41 +00:00
Diaa Sami
8f80dd06d6 composer: use logrus everywhere for consistency 2021-12-16 11:58:41 +00:00
Diaa Sami
b599245284 internal/worker: Use logrus for logging 2021-12-16 11:58:41 +00:00
Diaa Sami
510d2ccac0 worker/server: pass more error details to handler 2021-12-16 11:58:41 +00:00
Diaa Sami
c1aeeeaf0e internal/worker: log internal details when available 2021-12-16 11:58:41 +00:00
Thomas Lavocat
d26a93319c Tracing: measure IO during each job
Using iostat from package sysstat
2021-12-16 10:32:37 +01:00
Thomas Lavocat
8d67b90a2e Tests: trim dependencies for test generation 2021-12-16 10:21:25 +01:00
Gianluca Zuccarelli
e165db63ea metrics: add additional buckets
The change between the 32s bucket and the 64s bucket is too drastic
for measuring the duration of depsolve jobs. At present, 90% of the
depsolve jobs have a duration inbetween 32s and 64s, making the 32s
bucket too sensitive and the 64s bucket not sensitive enough.
2021-12-15 19:53:11 +00:00
sanne
3686e31e49 test/api: Integration test for 3rd-party repositories 2021-12-15 20:12:49 +01:00
Djebran Lezzoum
c93ea748a2 distro/depsolve/cloudapi: Add 3rd-party repository support.
Allow 3rd-party repositories to be supported and custom packages installed.
Fixes #COMPOSER-1273
2021-12-15 20:12:49 +01:00
Gianluca Zuccarelli
10f34de88b templates: add worker dashboard
Add an initial dashboard for the job metrics.
For now, the dashboard includes graphs and
burn rates for osbuild job duration and depsolve
job duration
2021-12-15 08:52:52 +00:00
Thomas Lavocat
fec9a6d3e5 Cosmetics: adjust the vertical alignment 2021-12-15 09:41:32 +01:00
Thomas Lavocat
53cb1b952e Tests: generation works with dnf-json.socket down
Test that the test cases generation will properly start the
dnf-json.socket when needed.
2021-12-15 09:41:32 +01:00
Thomas Lavocat
e6a6ed01ea Tests: start dnf-json-socket when needed 2021-12-15 09:41:32 +01:00
sanne
83e16afda4 dnf-json: Can be started without systemd
Instead of starting the socket in the entrypoint, make dnf-json able to
bind on the unixsocket by itself.
2021-12-15 09:41:32 +01:00
Thomas Lavocat
0877ae3ac0 dnf-json: Avoid leaking memory on the Cpp side
To avoid dnf leaking memory, dnf-json as a service calls fork() on each
request. This allow memory to be freed automatically when the process
handling the request is destroyed.
2021-12-15 09:41:32 +01:00
Thomas Lavocat
f8281eee54 dnf-json: refactor
Prepare the multi-cache architecture by doing some refactoring.
Mainly this commit adds a solver class that embeds all the logic around
dnf. Responsibilities of communicating on the socket and depsolving are
separated.
2021-12-15 09:41:32 +01:00
Thomas Lavocat
ca126e9747 dnf-json: Change dnf-json to be a daemon
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
2021-12-15 09:41:32 +01:00
Juan Abia
4875998cbe schutzbot: remove tagged gcp resources
extend scheduled cloud cleaner functionality to clean old enough and
tagged gcp resources
2021-12-14 12:20:31 +01:00
Juan Abia
9debf97f37 test/cases: tag gcp resources
gcp test images are not tagged. Now we create the "gitlab-ci-test"
labbel so it's easier to remove them later.
2021-12-14 12:20:31 +01:00
Thomas Lavocat
4da7ec68a0 openstack: use rhos-01 2021-12-13 19:11:13 +00:00
Tomas Hozza
d115baec03 Regenerate image test cases
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
66cd704198 RHEL-9.0: use the default partitioning scheme for all EC2 images
This change is part of unifying the default partitioning scheme used by
all RHEL-9.0 images [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
661cd2772d RHEL-9.0: add / fix BIOS boot partition size comment
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
52115716d7 RHEL-9.0: increase the size of /boot/efi partition to 200 MB
Increase the size of /boot/efi partition in the default partition table
used for x86_64 and aarch64 architectures. The size is the same as what
is being used by RHEL EC2 aarch64 image as well as what ie being
suggested by RHEL-8 documentation [1]. There is currently no
documentation equivalent for RHEL-9 yet.

This change is part of unifying the default partitioning scheme used by
all RHEL-9.0 images.

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_installation/partitioning-reference_installing-rhel-as-an-experienced-user
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
4366acc8f4 RHEL-9.0: add separate /boot partition to default partition table
Add a separate /boot partition to the default partition table used on
RHEL-9.0. The size is set to 500 MB, which is the value used by RHEL EC2
images. This change is needed to unify the default partitioning scheme
used by all RHEL-9.0 images [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Alexander Todorov
af18116574 ci: CC QE in notification message 2021-12-13 12:12:41 +01:00
Juan Abia
610db6563a gosec: G601 - Implicit memory aliasing in for loop
G601 warning doen't mean there's a vulnerabilty. But this code could
have unintended bugs. Disabling warnings locally.
2021-12-13 12:17:30 +02:00
Juan Abia
8136209d17 gosec: G404 - Use of weak rng
math/rand is good enough for uuids. disabling rule locally.
2021-12-13 12:17:30 +02:00
Juan Abia
4f91a2cc81 gosec: G306-WriteFile permissions higher than 0600
lower WriteFile permissions to 0600 (could break something)
2021-12-13 12:17:30 +02:00
Juan Abia
b9f1ccbdf1 gosec: G505, G401 - crypto/sha1 module
Disabling G505 and G401 gosec rules on specific lines on
internal/storage/storage.go to be able to use sha1 algorithm.
2021-12-13 12:17:30 +02:00
Juan Abia
eb3fa3e5d4 gosec: G204 - Subproccess launched as function arg
G204 doesn't necessarily indicate a bad behaviour. But could help
discover potential command injection vector.
2021-12-13 12:17:30 +02:00
Juan Abia
0b9372fe0a gosec: G110 - Potential DoS vulnerability
this vulnerability is already tested. Disabling issue in line
2021-12-13 12:17:30 +02:00
Juan Abia
ceb72975c4 gosec: G402 - TLS MinVersion
stablish minumim TLS version, so G402 from gosec doesn't get triggered
2021-12-13 12:17:30 +02:00
Juan Abia
c8cf835db3 gosec: G401, G501 - Weak cryptographic primitive
azure, koji and gcp use md5 hashes. Gosec is not happy with it, so we
create exceptions for them (G401, G501).
2021-12-13 12:17:30 +02:00
Juan Abia
5a1460a6d1 golangci: enable gosec in golangci
gosec is a security tools for golang. This change enables this tools
inside golangci-lint
2021-12-13 12:17:30 +02:00
Simon Steinbeiss
d41ae15220 release-action: Send notification to our Slack channel
Passing the webhook URL is necessary because GH composite actions don't
support handling secrets.
See also https://github.com/osbuild/release-action/pull/3

[skip ci]
2021-12-13 09:48:34 +01:00
ondrejbudai
acafddbcd7 Post release version bump
[skip ci]
2021-12-09 17:29:14 +00:00
Ondřej Budai
df5074f509 test/regression: fix journalctl for workers
Our workers are not named osbuild-worker but usually something like
osbuild-worker@1. Let's steal the code that determines the unit name
from other tests.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-09 12:02:24 +01:00
Ondřej Budai
1ac1409753 test/regression: remove the composes after the build is done
Otherwise, we are leaking disk space.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-09 12:02:24 +01:00
Ondřej Budai
47551ffc67 test/regression: fix if formatting
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-09 12:02:24 +01:00