Commit graph

3007 commits

Author SHA1 Message Date
Ondřej Budai
1b289cc27e packer: import image-builder-packer repository
/templates/packer now contains a copy of image-builder-packer repository
as of b8a4b45f93890090de24e3d043e2d958948fc3c5

Changes:
- LICENSE file was dropped (it was redundant)
- README file was dropped (no longer needed)
- GitHub workflows were removed (will be replaced by schutzbot)
- RPMs were removed (they were huge, will be installed in a different way)

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Martin Sehnoutka
d1029fae69 osbuild2: update cloud-init stage with new options
Related to: https://github.com/osbuild/osbuild/pull/866/

Introduce new fields and move structure validation into the constructor.
This will fail faster and hopefully provide less space for programming
errors. Another advantage is simplified code with less type aliases and
lines.
2021-12-01 12:03:13 +01:00
Alexander Todorov
fbf220707a tests: Small updates to docs 2021-12-01 10:12:04 +01:00
Ondřej Budai
c55ec88505 api/koji: fix /compose/log route
We have been actually unmarshalling into a wrong datatype for a year, by
fixing this, we should get much more logging in Brew.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-30 18:14:10 +01:00
Jakub Rusz
8136e0f069 ci: make some jobs interruptible
Jobs that don't interact with clouds can be canceled at any time without
the risk of leaving unused resources in the cloud. This enables the use
of "automatic cancellation of redundant pipelines" which means that if
and update is pushed to an open PR the current running pipeline is
canceled. This is done by adding an "interruptible" flag to the jobs.
Default value is false so only jobs that have it explicitly set to true
will be canceled.
2021-11-30 10:00:22 +01:00
Jakub Rusz
c38ebb1643 tests/regression: skip satellite regression test on subscribed RHEl
This test does not bring any value when running on subscribed RHEL
machines.
2021-11-26 12:20:54 +01:00
Jakub Rusz
35f8707d59 tests: re-enable satellite regression test on nightlies
Test now also works with nightly repositories. Re-enabling so that we
can run it during nightly compose testing.
2021-11-26 12:20:54 +01:00
Jakub Rusz
dbd9285275 schutzbot/prepare-rhel-internal: add trailing slash to repo url
Regression tests were failing without this trailing slash at the end.
2021-11-26 12:20:54 +01:00
Ondřej Budai
fa73586932 nightly: use boston mirror instead of the redirecter
It often takes a lot of time to synchronize the compose across all mirrors,
so it's safer to use just one, otherwise we can get hit by 404s if the
compose isn't available yet by a mirror.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-26 12:20:54 +01:00
Juan Abia
42daf0c84d schuzbot: update scheduled cloud cleaner runner
fedora 33 will be EOL, change the runner to centos stream 8
2021-11-26 10:00:42 +01:00
Juan Abia
ca36cf155e schutzbot: remove aws s3 resources
remove old enough and tagged s3 objects
2021-11-26 10:00:42 +01:00
Juan Abia
eed20916b0 test/cases: tag aws s3 resources
all resources on aws originated from a test should be tagged with
"gitlab-ci-test". this commit ensures s3 resources get tagged.
2021-11-26 10:00:42 +01:00
sanne
c6c311cc3d osbuild-worker: Recreate rpmmd for each depsolve
The remote workers are long-running, and the subscription certificates
might expire. Before each depsolve refresh the subscriptions.
2021-11-25 17:55:46 +01:00
Ondřej Budai
14b29ae98a dbjobqueue: don't log when context's deadline was exceeded
This happens rather often as we limit the request job timeout to 20s on the
service.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-25 08:20:22 +01:00
Ondřej Budai
76d8a75184 worker: normalize job logging
The format is now always 'JOB_ID' (JOB_TYPE). This means that we also know
the job type when a job is finished or when it failed.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-25 08:20:22 +01:00
Ondřej Budai
dc7968d5c7 terraform: update to use the new instance type
The latest commit switched all our AWS x86_64 workloads to the newly announced
c6i.large instance type. I did some measurements and the test suite runs about
10% faster. Nice!

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-24 17:13:41 +01:00
Ondřej Budai
d61e1c009b spec: add epoch to nevra only if it's set
This change was tested by running:
```
$ rpm --eval '
    %global noepoch 42
    %{?epoch:%epoch:}
  '
$ rpm --eval '
    %global epoch 42
    %{?epoch:%epoch:}
  '
42:
```

Fixes #2054

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-24 16:12:54 +00:00
chloenayon
b0ef1ac256 Post release version bump
[skip ci]
2021-11-24 16:03:02 +00:00
Tomas Hozza
47b1b5b93b Doc: remove unused doc/news directory
The doc/news directory has been re-added by some recent PRs, but it is
not used any more since [1]. Clean up the repository of the unused
files.

[1] https://github.com/osbuild/osbuild-composer/pull/1933

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-24 14:55:47 +01:00
Tomas Hozza
8420f34b7a RHEL-9.0: install and enable TuneD by default on all EC2 images
Add TuneD package to the base package set for all EC2 image types,
including the `ami` image type. In addition to installing the package,
also enable the service by default. TuneD will by default auto-detect
the environment in which the image is running and set the most
appropriate TuneD profile, with exception of the `ec2-sap` image, which
explicitly sets a specific TuneD profile.

This change affects the `ami`, `ec2`, and `ec2-ha` image types on all
supported architectures.

Regenerate affected image test cases.

Related to RHELPLAN-102615
Fix #1972

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-24 08:51:52 +01:00
Antonio Murdaca
c37b688dd2 test/rhel86-manifest: re-generate
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-23 18:23:45 +01:00
Diaa Sami
e4d535ea5f Improve contributing.md
use auto-answer -y for all commands
mention the script tools/prepare-source
mention the go generate step
2021-11-23 08:25:07 +01:00
Martin Sehnoutka
04f69db2c0 osbuild2: selinux stage - introduce force_autorelabel option
This was added in osbuild: https://github.com/osbuild/osbuild/pull/875

Introduce the same option in composer and make it optional by specifying
it as a pointer to bool value. It would work the same even if it was
there every time, but as it should be an edge case, don't use it
everywhere. Also osbuild doesn't require it to be present, so it seems
like the right thing to do.
2021-11-22 15:28:19 +01:00
Achilleas Koutsou
841dd233c7 test/data: update image-installer test manifest 2021-11-20 14:31:44 +01:00
Antonio Murdaca
ea5f153f3c internal: drop not needed kargs
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-20 14:31:44 +01:00
Antonio Murdaca
fed8edc51b internal/rhel86: prune default dracut modules
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-20 14:31:44 +01:00
sanne
58424b39bc test: Adapt api test metrics to namespacing 2021-11-19 22:48:25 +01:00
Gianluca Zuccarelli
3443fb8771 templates: update dashboard metrics
Update the composer dashboard to make use of the
namespaced metrics.
2021-11-19 22:48:25 +01:00
Gianluca Zuccarelli
91f2457363 metrics: add prometheus namespaces
Make use of the prometheus namespace and subsystem
to give the metrics a consistent namespaces in openshift.
2021-11-19 22:48:25 +01:00
sanne
c8f198166d cloudapi/v2: Add support for edge-container 2021-11-19 21:45:19 +01:00
Tomas Hozza
57d60df24a RHEL-9.0: remove compat-sap-c++-9 from EC2-SAP image package set
The `compat-sap-c++-9` package no longer exists on RHEL-9. It has been
removed from the RHEL-9.0 Beta EC2-SAP image's package set, but then got
readded as part of adding the RHEL-9.0 distro and merging package sets
code with RHEL-8.6.

This issue have been found by the test case which tests manifests from
mage test cases, including the package set depsolving.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-19 19:50:25 +01:00
Tomas Hozza
8eb71484d9 RHEL-8.6: fix package set of the image-installer image type
The commit adding distro-specific package set to RHEL-8.6 [1] introduced
a bug in the `image-installer` image type due to a typo. The
`insights-client` package have been removed from the
`bareMetalPackageSet` and moved to `distroSpecificPackageSet`. However
the `distroBuildPackageSet` got appended to `bareMetalPackageSet`,
instead of `distroSpecificPackageSet`, which caused `insights-client` to
be removed from the image's package set.

This issue have been found by the test case which tests manifests from
image test cases, including the package set depsolving.

[1] ed0cb5ea24#

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-19 19:50:25 +01:00
Tomas Hozza
c5a4946135 Test all manifests with depsolved package sets
Generated image test case manifests for all supported distros, arches and
image-types are being tested as part of distro unit tests. However due
to time constrains, the unit test does not depsolve the image's default
package sets and thus does not check if they changed in the internal
osbuild-composer's representation, compared to the generated image test
case.

Extend the `TestDistro_Manifest()` function used by the unit test to
allow depsolving image's package sets.

Introduce a new test case binary `osbuild-composer-manifest-tests`
allowing to check the manifests generated by composer for all supported
combinations of images against generated manifests, including depsolving
image's default package sets.

Introduce a new CI test case `manifest_tests.sh` executing the
`osbuild-composer-manifest-tests` binary and testing all existing image
test cases. Run it in CI on RHEL-9 runner.

Modify SPEC file to ship the newly added test case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-11-19 19:50:25 +01:00
Achilleas Koutsou
0712ed9700 cloudapi: add pipeline names to osbuild job 2021-11-19 19:37:08 +01:00
sanne
08c94da2b6 Revert ".gitlabci.yml: Reduce api test runners"
We need rhel8.5+ testing.

This reverts commit ac39dacfae.
2021-11-19 19:37:08 +01:00
Ondřej Budai
8f0d685b70 template: bump postgres max conns to 20
We actually need 2 * 16 connections at minimum (one worker waits for two
jobs). Let's bump the maximum connection count even moar.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-19 13:25:51 +01:00
Ondřej Budai
c3a8fc19a2 templates: bump max postgres connections to 10
By default, pgxpool.Pool has 4 connections (or number of cpus if higher).
Currently, we have 3 replicas, that means max 3*4=12 DB connections.

The dequeue operation is actually blocking - when a worker is waiting for
a job, one connection is blocked. My theory is that with 16 workers, we just
don't have enough connections that causes all sorts of weird slowdowns.

This commit bumps the number of connection from one replica to 10, therefore
we should be at 30 connections in total.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-19 13:17:10 +01:00
Ondřej Budai
c6fbf589aa dbjobqueue: add settings for max pool connections
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-11-19 13:17:10 +01:00
Martin Sehnoutka
f4412ff07f osbuild2: update rhsm stage
The stage now allows for customizations specific to YUM or DNF. So far
it is just an alias to the same definition, meaning that composer can
use exactly the same structures for both.

Ref: https://github.com/osbuild/osbuild/pull/876
2021-11-19 10:43:58 +01:00
yih
0b4261b4da test mandate installation device
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-18 18:10:09 +01:00
Antonio Murdaca
5005ff6184 internal: mandate installation device for the simplified installer
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-18 18:10:09 +01:00
Antonio Murdaca
26b96483a3 edge-simplified-installer: use coreos-installer-dracut
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2021-11-18 18:07:52 +01:00
Jakub Rusz
a0569c784e osbuild-image-tests: skip Openstack boot testing on aarch64
We don't have access to aarch64 machines in Openstack or with kvm
support so skip the boot test part to be able to run the image tests on
aarch64.
2021-11-18 16:24:36 +01:00
Alexander Todorov
efcd959d2e ci: Update Coverity Scan name
curl will URL encode this properly, no need to be encoded first
2021-11-18 10:26:52 +01:00
sanne
0a49dbe560 cloudapi/v2: Hard timeout for manifest job 2021-11-18 10:26:17 +01:00
sanne
028eca1b26 cloudapi/v2: Use manifest-id-only job
job dependencies:
depsolve -> manifest -> osbuild

This allows the compose handler to return the osbuild job id
immediately.
2021-11-18 10:26:17 +01:00
sanne
ac39dacfae .gitlabci.yml: Reduce api test runners
Only rhel ga and centos-8 are being built by the service.
2021-11-18 10:26:17 +01:00
sanne
ea2812f48f test: Switch api tests to v2 2021-11-18 10:26:17 +01:00
sanne
e5aba6bce9 cloudapi/v1: Adapt metadata handler to osbuild2 results 2021-11-17 18:59:13 +01:00
Achilleas Koutsou
e32f1cd697 osbuild: test Write() on nil Result receivers
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-17 15:21:40 +01:00