Commit graph

1463 commits

Author SHA1 Message Date
Gianluca Zuccarelli
e165db63ea metrics: add additional buckets
The change between the 32s bucket and the 64s bucket is too drastic
for measuring the duration of depsolve jobs. At present, 90% of the
depsolve jobs have a duration inbetween 32s and 64s, making the 32s
bucket too sensitive and the 64s bucket not sensitive enough.
2021-12-15 19:53:11 +00:00
Djebran Lezzoum
c93ea748a2 distro/depsolve/cloudapi: Add 3rd-party repository support.
Allow 3rd-party repositories to be supported and custom packages installed.
Fixes #COMPOSER-1273
2021-12-15 20:12:49 +01:00
Thomas Lavocat
ca126e9747 dnf-json: Change dnf-json to be a daemon
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
2021-12-15 09:41:32 +01:00
Tomas Hozza
66cd704198 RHEL-9.0: use the default partitioning scheme for all EC2 images
This change is part of unifying the default partitioning scheme used by
all RHEL-9.0 images [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
661cd2772d RHEL-9.0: add / fix BIOS boot partition size comment
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
52115716d7 RHEL-9.0: increase the size of /boot/efi partition to 200 MB
Increase the size of /boot/efi partition in the default partition table
used for x86_64 and aarch64 architectures. The size is the same as what
is being used by RHEL EC2 aarch64 image as well as what ie being
suggested by RHEL-8 documentation [1]. There is currently no
documentation equivalent for RHEL-9 yet.

This change is part of unifying the default partitioning scheme used by
all RHEL-9.0 images.

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_installation/partitioning-reference_installing-rhel-as-an-experienced-user
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Tomas Hozza
4366acc8f4 RHEL-9.0: add separate /boot partition to default partition table
Add a separate /boot partition to the default partition table used on
RHEL-9.0. The size is set to 500 MB, which is the value used by RHEL EC2
images. This change is needed to unify the default partitioning scheme
used by all RHEL-9.0 images [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2022805

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-13 12:33:16 +01:00
Juan Abia
610db6563a gosec: G601 - Implicit memory aliasing in for loop
G601 warning doen't mean there's a vulnerabilty. But this code could
have unintended bugs. Disabling warnings locally.
2021-12-13 12:17:30 +02:00
Juan Abia
8136209d17 gosec: G404 - Use of weak rng
math/rand is good enough for uuids. disabling rule locally.
2021-12-13 12:17:30 +02:00
Juan Abia
4f91a2cc81 gosec: G306-WriteFile permissions higher than 0600
lower WriteFile permissions to 0600 (could break something)
2021-12-13 12:17:30 +02:00
Juan Abia
b9f1ccbdf1 gosec: G505, G401 - crypto/sha1 module
Disabling G505 and G401 gosec rules on specific lines on
internal/storage/storage.go to be able to use sha1 algorithm.
2021-12-13 12:17:30 +02:00
Juan Abia
eb3fa3e5d4 gosec: G204 - Subproccess launched as function arg
G204 doesn't necessarily indicate a bad behaviour. But could help
discover potential command injection vector.
2021-12-13 12:17:30 +02:00
Juan Abia
0b9372fe0a gosec: G110 - Potential DoS vulnerability
this vulnerability is already tested. Disabling issue in line
2021-12-13 12:17:30 +02:00
Juan Abia
c8cf835db3 gosec: G401, G501 - Weak cryptographic primitive
azure, koji and gcp use md5 hashes. Gosec is not happy with it, so we
create exceptions for them (G401, G501).
2021-12-13 12:17:30 +02:00
Achilleas Koutsou
2e1f6ead4e store: set RHSM when initialising SourceConfig with a repo
Fixes #2055
2021-12-09 00:28:30 +01:00
Gianluca Zuccarelli
1a709eda5c metrics: add initial job metrics
Add job metrics to track the number of
pending/running jobs, the duration of
the jobs and how long the jobs spent in
the job queue.
2021-12-08 21:49:43 +00:00
Achilleas Koutsou
500e484799 cloudapi: add new image types to tests
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-12-03 16:49:09 +00:00
Achilleas Koutsou
b840f452ea cloudapi: extend supported image types
Add support for:
- qcow2: guest iamge
- vmdk: vmware image
- image-installer: installer ISO

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-12-03 16:49:09 +00:00
sanne
c43ad2b22a osbuild-service-maintenance: Clean up expired images 2021-12-03 00:14:09 +00:00
sanne
742e0e6616 cloudapi/v2: No ObjectReference in request bodies
Because of the way json encoding works in golang we haven't run into any
issues. But if we add automated validation based on the spec it will
break, the image-builder-crc client for instance doesn't specify these
fields.
2021-12-02 21:01:58 +01:00
Achilleas Koutsou
01184f43a0 distro/rhel90: assign random volid for efiboot.img 2021-12-02 12:12:05 +01:00
Achilleas Koutsou
05cc6b98c2 disk: random volume ID generator for FAT
Generate a random 32-bit hex string
2021-12-02 12:12:05 +01:00
Achilleas Koutsou
2b34e4003c distro/rhel90: remove rdma-core for simplified-installer
No longer necessary.
See fed8edc51b
2021-12-02 12:12:05 +01:00
Achilleas Koutsou
13bdde91f7 distro/rhel90: use coreos-installer-dracut
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-12-02 12:12:05 +01:00
Achilleas Koutsou
5d4103af74 distro/rhel90: enable edge-simplified-installer image type
This reverts commit 226ada67f93cdfda73b3cae69d45eaf4fb281d6f.
2021-12-02 12:12:05 +01:00
Tomas Hozza
4f4341d774 RHEL-9.0: clean up package sets of commented lines
Remove commented out lines and some comments, kept only as a reference
when we moved away from using the `@core` group.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Tomas Hozza
bab13f3c90 RHEL-9.0: replace the @core package group with a static list of packages
Don't use the `@core` package group in image definitions, because it is
not intended as the minimal package set for virtual / cloud images. In
addition, its content is changing without us knowing, which has
consequences such as the recent discovery of the fact that TuneD is no
longer installed by default on RHEL images, while it definitely should be.

Replace the `@core` package group with the `coreOsCommonPackageSet`
package set. The content of it is based on the latest `@core` group
definition with a few modifications, so that image package sets
never end up having the same package listed in the `Include` and `Exclude`
package set at the same time. All additions have been accompanied with a
comment and all removals have been kept commented out with a comment.

The fact that the change does not have any effect on image package sets
was verified by regenerating all RHEL-9.0 image test cases. There is
however one change in the VMDK image. Specifically the
`python3-libselinux` package have been added. The reason is that the
latest `@core` group definition was used when defining the content of
`coreOsCommonPackageSet`, however the `@core` group definition in the
RPMRepo snapshot used for the image test case didn't include the package
yet.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Tomas Hozza
aee86c1199 RHEL-9.0: list one package per line in package set definitions
Listing a single package per line in the package set definitions makes
it much more easier to review diffs in code changes and spot potential
issues.

Align EC2 package set functions to use the structure's `.Append()`
method as it is used by all the other package set functions.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Tomas Hozza
9e757fbf11 RHEL-9.0: install tuned by default on all images using @core group
The `@core` package group used to include TuneD package by default on
RHEL-8. It has been removed from the group in Fedora as part of [1] and
inherited into RHEL-9. As a result, TuneD is no longer installed by
default on RHEL images.

After a discussion on rhel-devel there seems to be an agreement, that
TuneD should be installed by default on all RHEL virtual images. At
least we should keep the consistency in this regard with RHEL-8.

Regenerate all RHEL-9.0 image test cases.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=2026709

[1] https://pagure.io/fork/adelton/fedora-comps/c/a5d4f1b6c9fcbe20cb0c38eac5048d7d45d1dd17

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-12-01 19:45:36 +01:00
Martin Sehnoutka
d1029fae69 osbuild2: update cloud-init stage with new options
Related to: https://github.com/osbuild/osbuild/pull/866/

Introduce new fields and move structure validation into the constructor.
This will fail faster and hopefully provide less space for programming
errors. Another advantage is simplified code with less type aliases and
lines.
2021-12-01 12:03:13 +01:00
Ondřej Budai
c55ec88505 api/koji: fix /compose/log route
We have been actually unmarshalling into a wrong datatype for a year, by
fixing this, we should get much more logging in Brew.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-30 18:14:10 +01:00
Ondřej Budai
14b29ae98a dbjobqueue: don't log when context's deadline was exceeded
This happens rather often as we limit the request job timeout to 20s on the
service.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-25 08:20:22 +01:00
Tomas Hozza
8420f34b7a RHEL-9.0: install and enable TuneD by default on all EC2 images
Add TuneD package to the base package set for all EC2 image types,
including the `ami` image type. In addition to installing the package,
also enable the service by default. TuneD will by default auto-detect
the environment in which the image is running and set the most
appropriate TuneD profile, with exception of the `ec2-sap` image, which
explicitly sets a specific TuneD profile.

This change affects the `ami`, `ec2`, and `ec2-ha` image types on all
supported architectures.

Regenerate affected image test cases.

Related to RHELPLAN-102615
Fix #1972

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-24 08:51:52 +01:00
Martin Sehnoutka
04f69db2c0 osbuild2: selinux stage - introduce force_autorelabel option
This was added in osbuild: https://github.com/osbuild/osbuild/pull/875

Introduce the same option in composer and make it optional by specifying
it as a pointer to bool value. It would work the same even if it was
there every time, but as it should be an edge case, don't use it
everywhere. Also osbuild doesn't require it to be present, so it seems
like the right thing to do.
2021-11-22 15:28:19 +01:00
Antonio Murdaca
ea5f153f3c internal: drop not needed kargs
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-20 14:31:44 +01:00
Antonio Murdaca
fed8edc51b internal/rhel86: prune default dracut modules
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-20 14:31:44 +01:00
Gianluca Zuccarelli
91f2457363 metrics: add prometheus namespaces
Make use of the prometheus namespace and subsystem
to give the metrics a consistent namespaces in openshift.
2021-11-19 22:48:25 +01:00
sanne
c8f198166d cloudapi/v2: Add support for edge-container 2021-11-19 21:45:19 +01:00
Tomas Hozza
57d60df24a RHEL-9.0: remove compat-sap-c++-9 from EC2-SAP image package set
The `compat-sap-c++-9` package no longer exists on RHEL-9. It has been
removed from the RHEL-9.0 Beta EC2-SAP image's package set, but then got
readded as part of adding the RHEL-9.0 distro and merging package sets
code with RHEL-8.6.

This issue have been found by the test case which tests manifests from
mage test cases, including the package set depsolving.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-19 19:50:25 +01:00
Tomas Hozza
8eb71484d9 RHEL-8.6: fix package set of the image-installer image type
The commit adding distro-specific package set to RHEL-8.6 [1] introduced
a bug in the `image-installer` image type due to a typo. The
`insights-client` package have been removed from the
`bareMetalPackageSet` and moved to `distroSpecificPackageSet`. However
the `distroBuildPackageSet` got appended to `bareMetalPackageSet`,
instead of `distroSpecificPackageSet`, which caused `insights-client` to
be removed from the image's package set.

This issue have been found by the test case which tests manifests from
image test cases, including the package set depsolving.

[1] ed0cb5ea24#

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-19 19:50:25 +01:00
Tomas Hozza
c5a4946135 Test all manifests with depsolved package sets
Generated image test case manifests for all supported distros, arches and
image-types are being tested as part of distro unit tests. However due
to time constrains, the unit test does not depsolve the image's default
package sets and thus does not check if they changed in the internal
osbuild-composer's representation, compared to the generated image test
case.

Extend the `TestDistro_Manifest()` function used by the unit test to
allow depsolving image's package sets.

Introduce a new test case binary `osbuild-composer-manifest-tests`
allowing to check the manifests generated by composer for all supported
combinations of images against generated manifests, including depsolving
image's default package sets.

Introduce a new CI test case `manifest_tests.sh` executing the
`osbuild-composer-manifest-tests` binary and testing all existing image
test cases. Run it in CI on RHEL-9 runner.

Modify SPEC file to ship the newly added test case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-19 19:50:25 +01:00
Achilleas Koutsou
0712ed9700 cloudapi: add pipeline names to osbuild job 2021-11-19 19:37:08 +01:00
Martin Sehnoutka
f4412ff07f osbuild2: update rhsm stage
The stage now allows for customizations specific to YUM or DNF. So far
it is just an alias to the same definition, meaning that composer can
use exactly the same structures for both.

Ref: https://github.com/osbuild/osbuild/pull/876
2021-11-19 10:43:58 +01:00
Antonio Murdaca
5005ff6184 internal: mandate installation device for the simplified installer
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-18 18:10:09 +01:00
Antonio Murdaca
26b96483a3 edge-simplified-installer: use coreos-installer-dracut
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-18 18:07:52 +01:00
sanne
0a49dbe560 cloudapi/v2: Hard timeout for manifest job 2021-11-18 10:26:17 +01:00
sanne
028eca1b26 cloudapi/v2: Use manifest-id-only job
job dependencies:
depsolve -> manifest -> osbuild

This allows the compose handler to return the osbuild job id
immediately.
2021-11-18 10:26:17 +01:00
sanne
e5aba6bce9 cloudapi/v1: Adapt metadata handler to osbuild2 results 2021-11-17 18:59:13 +01:00
Achilleas Koutsou
e32f1cd697 osbuild: test Write() on nil Result receivers
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-17 15:21:40 +01:00
Achilleas Koutsou
7f8a1b314a osbuild: check if result objects are nil in Write()
Before dereferencing the method receiver in Write(), check if the object
is nil and return early.

Fixes #2002

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-17 15:21:40 +01:00