Commit graph

1479 commits

Author SHA1 Message Date
Tomas Hozza
b200fa8fcd RHEL-90: introduce default image config data structure
Introduce a new data structure `ImageConfig` holding the default OS
configuration applied when building an image. The structure can be used
to hold the default image configuration on the distribution level with
possible overrides defined on the image-type level.

As a starting point, move hard-coded default values and configuration
common for `osPipeline`, `ec2BaseTreePipeline` and `ostreeTreePipeline`
to the distribution and image-type default image configuration. This is
preparing the ground for merging all of these three pipeline functions
into `osPipeline`, which will produce the appropriate OS pipeline based
on the image-type configuration and the fact if it is rpmOstree or not.

Regenerate affected EC2 and AMI manifests. There is however no change in
the resulting image configuration and image-info report.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Ondřej Budai
d967790ea5 api/cloud: drop v1 API
It's deprecated and not used anywhere, let's just drop it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:38:50 +01:00
Ondřej Budai
e668ed0ac5 weldr: return an error if host distro wasn't found in distro registry
Prevents a nil panic, see rhbz#2035956

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-03 18:05:15 +01:00
Tomas Hozza
885fe3d2b4 osbuild2: fix typo in tar stage option value
Related to https://github.com/osbuild/osbuild/pull/940

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-22 10:41:22 +01:00
Tomas Hozza
35ef067396 osbuild2: support 'format' and 'root-node' Tar stage options
Bring the Tar stage implementation on par with the current osbuild
schema. Specifically add the 'format' and 'root-node' options to the
stage options structure.

Add stage options validation along with appropriate unit tests.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-21 20:40:12 +01:00
Ondřej Budai
ab3990b90a dbjobqueue: fix FinishJob not returning an error if already finished
Reported by covscan

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-18 00:14:07 +00:00
sanne
2543459a7f osbuild2: Expand dnf_config stage 2021-12-17 20:07:56 +01:00
sanne
8406ada6f5 worker: Treat a non echo.HTTPError like a regular error 2021-12-17 13:13:05 +01:00
Tomas Hozza
30f64d190d osbuild2: ensure that empty sysconfig options members are omitted
The `Kernel` and `Network` members of the sysconfig stage options
structure were previously not declared as pointers. As a result, they
always appeared in the resulting JSON object, even though they were
empty. Use pointers to ensure that the members are omitted from the
resulting JSON object, if they were not defined.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:47:28 +01:00
Tomas Hozza
96000173db osbuild2: do not use reflect in pam.limits.conf stage
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Tomas Hozza
09cd5b3576 osbuild2: support org.osbuild.dnf-automatic.config stage
Add support for the new `org.osbuild.dnf-automatic.config` stage for
configuring DNF Automatic.

Add appropriate new unit tests for the stage implementation and modify
necessary existing unit tests.

Related to https://github.com/osbuild/osbuild/pull/936

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Tomas Hozza
97ef7fbf28 osbuild2: support org.osbuild.yum.repos stage
Add support for the new `org.osbuild.yum.repos` stage for creating DNF /
YUM repository configuration files.

Add appropriate new unit tests for the stage implementation and modify
necessary existing unit tests.

Related to https://github.com/osbuild/osbuild/pull/932

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Tomas Hozza
37a39743bc osbuild2: support PermitRootLogin in sshd.config stage
Add support for the `PermitRootLogin` option in the `sshd.config` stage.
Valid values can be of type `bool` or `string`. Due to this reason, a
custom interface type is defined and a custom `UnmarshalJSON()` method is
defined for the `SshdConfigConfig` structure.

Modify unit tests to test the newly added option and test
(un)marhsalling of valid values of both types.

Related to https://github.com/osbuild/osbuild/pull/917

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-16 20:01:42 +01:00
Diaa Sami
487e2d0669 internal/cloud: use logrus for logging
and log upload & sharing failures
2021-12-16 11:58:41 +00:00
Diaa Sami
510d2ccac0 worker/server: pass more error details to handler 2021-12-16 11:58:41 +00:00
Diaa Sami
c1aeeeaf0e internal/worker: log internal details when available 2021-12-16 11:58:41 +00:00
Gianluca Zuccarelli
e165db63ea metrics: add additional buckets
The change between the 32s bucket and the 64s bucket is too drastic
for measuring the duration of depsolve jobs. At present, 90% of the
depsolve jobs have a duration inbetween 32s and 64s, making the 32s
bucket too sensitive and the 64s bucket not sensitive enough.
2021-12-15 19:53:11 +00:00
Djebran Lezzoum
c93ea748a2 distro/depsolve/cloudapi: Add 3rd-party repository support.
Allow 3rd-party repositories to be supported and custom packages installed.
Fixes #COMPOSER-1273
2021-12-15 20:12:49 +01:00
Thomas Lavocat
ca126e9747 dnf-json: Change dnf-json to be a daemon
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
2021-12-15 09:41:32 +01:00
Tomas Hozza
66cd704198 RHEL-9.0: use the default partitioning scheme for all EC2 images
This change is part of unifying the default partitioning scheme used by
all RHEL-9.0 images [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
661cd2772d RHEL-9.0: add / fix BIOS boot partition size comment
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
52115716d7 RHEL-9.0: increase the size of /boot/efi partition to 200 MB
Increase the size of /boot/efi partition in the default partition table
used for x86_64 and aarch64 architectures. The size is the same as what
is being used by RHEL EC2 aarch64 image as well as what ie being
suggested by RHEL-8 documentation [1]. There is currently no
documentation equivalent for RHEL-9 yet.

This change is part of unifying the default partitioning scheme used by
all RHEL-9.0 images.

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_installation/partitioning-reference_installing-rhel-as-an-experienced-user
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
4366acc8f4 RHEL-9.0: add separate /boot partition to default partition table
Add a separate /boot partition to the default partition table used on
RHEL-9.0. The size is set to 500 MB, which is the value used by RHEL EC2
images. This change is needed to unify the default partitioning scheme
used by all RHEL-9.0 images [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Juan Abia
610db6563a gosec: G601 - Implicit memory aliasing in for loop
G601 warning doen't mean there's a vulnerabilty. But this code could
have unintended bugs. Disabling warnings locally.
2021-12-13 12:17:30 +02:00
Juan Abia
8136209d17 gosec: G404 - Use of weak rng
math/rand is good enough for uuids. disabling rule locally.
2021-12-13 12:17:30 +02:00
Juan Abia
4f91a2cc81 gosec: G306-WriteFile permissions higher than 0600
lower WriteFile permissions to 0600 (could break something)
2021-12-13 12:17:30 +02:00
Juan Abia
b9f1ccbdf1 gosec: G505, G401 - crypto/sha1 module
Disabling G505 and G401 gosec rules on specific lines on
internal/storage/storage.go to be able to use sha1 algorithm.
2021-12-13 12:17:30 +02:00
Juan Abia
eb3fa3e5d4 gosec: G204 - Subproccess launched as function arg
G204 doesn't necessarily indicate a bad behaviour. But could help
discover potential command injection vector.
2021-12-13 12:17:30 +02:00
Juan Abia
0b9372fe0a gosec: G110 - Potential DoS vulnerability
this vulnerability is already tested. Disabling issue in line
2021-12-13 12:17:30 +02:00
Juan Abia
c8cf835db3 gosec: G401, G501 - Weak cryptographic primitive
azure, koji and gcp use md5 hashes. Gosec is not happy with it, so we
create exceptions for them (G401, G501).
2021-12-13 12:17:30 +02:00
Achilleas Koutsou
2e1f6ead4e store: set RHSM when initialising SourceConfig with a repo
Fixes #2055
2021-12-09 00:28:30 +01:00
Gianluca Zuccarelli
1a709eda5c metrics: add initial job metrics
Add job metrics to track the number of
pending/running jobs, the duration of
the jobs and how long the jobs spent in
the job queue.
2021-12-08 21:49:43 +00:00
Achilleas Koutsou
500e484799 cloudapi: add new image types to tests
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-12-03 16:49:09 +00:00
Achilleas Koutsou
b840f452ea cloudapi: extend supported image types
Add support for:
- qcow2: guest iamge
- vmdk: vmware image
- image-installer: installer ISO

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-12-03 16:49:09 +00:00
sanne
c43ad2b22a osbuild-service-maintenance: Clean up expired images 2021-12-03 00:14:09 +00:00
sanne
742e0e6616 cloudapi/v2: No ObjectReference in request bodies
Because of the way json encoding works in golang we haven't run into any
issues. But if we add automated validation based on the spec it will
break, the image-builder-crc client for instance doesn't specify these
fields.
2021-12-02 21:01:58 +01:00
Achilleas Koutsou
01184f43a0 distro/rhel90: assign random volid for efiboot.img 2021-12-02 12:12:05 +01:00
Achilleas Koutsou
05cc6b98c2 disk: random volume ID generator for FAT
Generate a random 32-bit hex string
2021-12-02 12:12:05 +01:00
Achilleas Koutsou
2b34e4003c distro/rhel90: remove rdma-core for simplified-installer
No longer necessary.
See fed8edc51b
2021-12-02 12:12:05 +01:00
Achilleas Koutsou
13bdde91f7 distro/rhel90: use coreos-installer-dracut
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-12-02 12:12:05 +01:00
Achilleas Koutsou
5d4103af74 distro/rhel90: enable edge-simplified-installer image type
This reverts commit 226ada67f93cdfda73b3cae69d45eaf4fb281d6f.
2021-12-02 12:12:05 +01:00
Tomas Hozza
4f4341d774 RHEL-9.0: clean up package sets of commented lines
Remove commented out lines and some comments, kept only as a reference
when we moved away from using the `@core` group.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Tomas Hozza
bab13f3c90 RHEL-9.0: replace the @core package group with a static list of packages
Don't use the `@core` package group in image definitions, because it is
not intended as the minimal package set for virtual / cloud images. In
addition, its content is changing without us knowing, which has
consequences such as the recent discovery of the fact that TuneD is no
longer installed by default on RHEL images, while it definitely should be.

Replace the `@core` package group with the `coreOsCommonPackageSet`
package set. The content of it is based on the latest `@core` group
definition with a few modifications, so that image package sets
never end up having the same package listed in the `Include` and `Exclude`
package set at the same time. All additions have been accompanied with a
comment and all removals have been kept commented out with a comment.

The fact that the change does not have any effect on image package sets
was verified by regenerating all RHEL-9.0 image test cases. There is
however one change in the VMDK image. Specifically the
`python3-libselinux` package have been added. The reason is that the
latest `@core` group definition was used when defining the content of
`coreOsCommonPackageSet`, however the `@core` group definition in the
RPMRepo snapshot used for the image test case didn't include the package
yet.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Tomas Hozza
aee86c1199 RHEL-9.0: list one package per line in package set definitions
Listing a single package per line in the package set definitions makes
it much more easier to review diffs in code changes and spot potential
issues.

Align EC2 package set functions to use the structure's `.Append()`
method as it is used by all the other package set functions.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Tomas Hozza
9e757fbf11 RHEL-9.0: install tuned by default on all images using @core group
The `@core` package group used to include TuneD package by default on
RHEL-8. It has been removed from the group in Fedora as part of [1] and
inherited into RHEL-9. As a result, TuneD is no longer installed by
default on RHEL images.

After a discussion on rhel-devel there seems to be an agreement, that
TuneD should be installed by default on all RHEL virtual images. At
least we should keep the consistency in this regard with RHEL-8.

Regenerate all RHEL-9.0 image test cases.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=2026709

[1] https://pagure.io/fork/adelton/fedora-comps/c/a5d4f1b6c9fcbe20cb0c38eac5048d7d45d1dd17

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Martin Sehnoutka
d1029fae69 osbuild2: update cloud-init stage with new options
Related to: https://github.com/osbuild/osbuild/pull/866/

Introduce new fields and move structure validation into the constructor.
This will fail faster and hopefully provide less space for programming
errors. Another advantage is simplified code with less type aliases and
lines.
2021-12-01 12:03:13 +01:00
Ondřej Budai
c55ec88505 api/koji: fix /compose/log route
We have been actually unmarshalling into a wrong datatype for a year, by
fixing this, we should get much more logging in Brew.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-30 18:14:10 +01:00
Ondřej Budai
14b29ae98a dbjobqueue: don't log when context's deadline was exceeded
This happens rather often as we limit the request job timeout to 20s on the
service.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-25 08:20:22 +01:00
Tomas Hozza
8420f34b7a RHEL-9.0: install and enable TuneD by default on all EC2 images
Add TuneD package to the base package set for all EC2 image types,
including the `ami` image type. In addition to installing the package,
also enable the service by default. TuneD will by default auto-detect
the environment in which the image is running and set the most
appropriate TuneD profile, with exception of the `ec2-sap` image, which
explicitly sets a specific TuneD profile.

This change affects the `ami`, `ec2`, and `ec2-ha` image types on all
supported architectures.

Regenerate affected image test cases.

Related to RHELPLAN-102615
Fix #1972

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-24 08:51:52 +01:00
Martin Sehnoutka
04f69db2c0 osbuild2: selinux stage - introduce force_autorelabel option
This was added in osbuild: https://github.com/osbuild/osbuild/pull/875

Introduce the same option in composer and make it optional by specifying
it as a pointer to bool value. It would work the same even if it was
there every time, but as it should be an edge case, don't use it
everywhere. Also osbuild doesn't require it to be present, so it seems
like the right thing to do.
2021-11-22 15:28:19 +01:00