Commit graph

4006 commits

Author SHA1 Message Date
Sanne Raymaekers
8219dcdee8 osbuild-service-maintenance: Explicitly enable maintenance parts
Stage and production share the GCP account. To avoid trying to delete
each GCP image twice, the maintenance script needs the ability to
selectively disable certain parts based on the config.
2022-05-14 16:21:21 +02:00
Achilleas Koutsou
e2fe4b8de2 spec: require osbuild v55
The new osbuild input schema, for which we added support in
https://github.com/osbuild/osbuild-composer/pull/2578, requires osbuild
v55 or newer.
2022-05-13 20:22:23 +02:00
Juan Abia
99649ee142 generate-all-test-cases: generate all manifests
regenerate all manifests without image-info and add new ones
2022-05-13 21:01:37 +03:00
Tomas Hozza
287e63735c RHEL-84: panic error on tar image on s390x
Building `tar` image for `s390x` on RHEL-84 ends with panic:
"s390x image must have a partition table, this is a programming error"

A tar image should not need a partition table, so this error does not
make sense.
2022-05-13 21:01:37 +03:00
Juan Abia
967c8734a1 test-case-generators: update repos and add img types overrides
overrides where needed for `qcow2` and `simplified-image-installer` images on specific distros. Also some repos needed to be updated to newer versions.
2022-05-13 21:01:37 +03:00
Juan Abia
a87193369d update repos.json
fix typos and add new repos
2022-05-13 21:01:37 +03:00
Tomas Hozza
1aabc1870d Image tests: don't use customizations for rhel-85/90beta edge-installer
Do not apply the user customizations on edge-installer on RHEL-85/90beta,
since they are not supported there yet. The way we generate image test
cases from the `format-request-map.json` makes even the customized image
types being generated for all distributions automatically.
2022-05-13 21:01:37 +03:00
Tomas Hozza
04f612d758 Manifests test: ensure that every image type has test coverage
Extend the manifests test to ensure that each an every image type of
each architecture and each distribution is covered by at least one
image test case.

Since now we have the ability to generate image test cases for more
complicated image types, which consists only of the manifest, we should
have test coverage for each and every image type.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-05-13 21:01:37 +03:00
Tomas Hozza
5be81326eb generate-all-test-cases: capture manifests output
Capture stdout and stderr output when running generate-test-casesin the manifests command.
This is helpful for debugging test case generation failures.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-05-13 21:01:37 +03:00
Tomas Hozza
40c095a850 Tools: fetch image test case generation matrix from composer
Add a simple tool `osbuild-composer-image-definitions` which dumps the
matrix of all distributions, architectures and image types names
supported by composer as a JSON to the stdout.

Default to fetching the image test case generation matrix directly from
composer. This eliminates the need to update a JSON source file with
this information every time a new distro or image type are added to
composer.

Delete the previously used JSON source file with the image test case
generation matrix.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-05-13 21:01:37 +03:00
Tomas Hozza
035b616308 test-case-generators: add request definitions for edge-raw-image
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-05-13 21:01:37 +03:00
Tomas Hozza
97cb091ef7 test-case-generators: add RHEL-87/91 and F36 repos
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-05-13 21:01:37 +03:00
Achilleas Koutsou
1ff36bce9a tools/format-request-map: add 'core' group to qcow2 customize
Removed in 2beb707def, possibly
accidentally.
The affected manifests were not regenerated based on this change so
they all already contain the core group.
2022-05-13 13:25:34 +02:00
Ondřej Budai
056e095419 github: pin fedora:35 for the pylint check
Fedora 36 ships pylint 2.13 that newly reports:

dnf-json:436:20: E0601: Using variable 'cache_state' before assignment (used-before-assignment)

As dnf-json is pending a big rewrite
(https://github.com/osbuild/osbuild-composer/pull/2537),
I decided to pin fedora to 35 and let the other PR decide how to proceed in
order to prevent any conflicts.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-05-13 12:03:26 +02:00
Thomas Lavocat
c00aae0a4a worker: provide the region for the ASG
Before, the autoscaling group discovery is failing with error:
Error getting the Autoscaling instances: MissingRegion MissingRegion:
could not find region configuration
2022-05-13 11:52:30 +02:00
Diaa Sami
5a4488c829 templates/composer: fix access to private repos
update secret name to the correct one
2022-05-12 14:49:22 +02:00
Diaa Sami
941fe3513f templates/composer: add missing fluentd-config volume 2022-05-12 14:02:00 +02:00
Sanne Raymaekers
809afbd0ad templates/composer: Specify registry for fluentd-hec image 2022-05-12 11:03:17 +02:00
Diaa Sami
33711d7d51 composer: add support for logrus syslog hook
Which will be used on crc in the log forwarding setup
NeededBy: COMPOSER-1285
2022-05-12 11:02:27 +02:00
Diaa Sami
631133eabb templates/composer: give access to private quay repos 2022-05-12 10:30:54 +02:00
Diaa Sami
ca83eccc47 templates/composer: add fluentd sidecar
The sidecar receives logs from the service and forwards them to Splunk
HEC
2022-05-12 10:30:54 +02:00
Ondřej Budai
069d08fa64 Schutzfile: update Fedora snapshots to 20220504
We need the latest golang-github-getkin-kin-openapi in order to unblock
https://github.com/osbuild/osbuild-composer/pull/1953

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-05-12 09:56:09 +02:00
Ondřej Budai
de46e85712 cloudapi: make Repository.rhsm optional
I think that we can spare the users of clouadpi of writing "rhsm": "false"
into the requests so I decided to make this property optional and default
to false.

This is nice because it matches the behaviour of Weldr repositories and
sources so we can also use test/data/repositories without any changes after
openapi validation is enabled.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-05-11 13:46:47 +02:00
Jordi Gil
616258ee25 distro: housekeeping with cpu arch and arch.Name() 2022-05-10 19:53:41 +02:00
Sanne Raymaekers
02debc0cda templates/composer: Parametrize tenants in acl
This will allow us to specify tenants in the acl per namespace.
2022-05-10 15:40:38 +02:00
Jakub Rusz
ef87c917d5 Nightly testing: remove edge testing from nightly pipelines
All of the edge tests are being ran on every nightly compose inside
virt-qe Jenkins so no need to run all of them here as well. Keep just
ostree-raw-image.sh to cover edge testing for sign-off.
2022-05-10 14:13:24 +02:00
dependabot[bot]
60e6d936f0 build(deps): bump github.com/vmware/govmomi from 0.27.4 to 0.28.0
Bumps [github.com/vmware/govmomi](https://github.com/vmware/govmomi) from 0.27.4 to 0.28.0.
- [Release notes](https://github.com/vmware/govmomi/releases)
- [Changelog](https://github.com/vmware/govmomi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/vmware/govmomi/compare/v0.27.4...v0.28.0)

---
updated-dependencies:
- dependency-name: github.com/vmware/govmomi
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-05-10 13:31:39 +02:00
Simon Steinbeiss
4825f549e3 packit: Enable Koji build integration
Also drop the create_pr option, which was dropped by Packit.
2022-05-10 13:29:32 +02:00
Jakub Rusz
d388d30ef2 rpmplint: trust osbuild-composer repository
Git v2.35.2 brought some security fixes which prevents interaction with
repository if it's not owned by the user who cloned it. More info here:
https://github.blog/2022-04-12-git-security-vulnerability-announced/
2022-05-10 13:25:01 +02:00
Thomas Lavocat
ab7fe6558a worker: protect the instance from upgrading
Before the instance was vulnerable to an OTA update while processing a
request. Because there is no way of retriggering a job in Composer, it
is better to avoid this situation.
The way we are doing it is by setting the `protected` flag onto the
instance when a job is being processed. This way the AWS scheduler
does hopefully not shutdown the machine at the wrong time.

Main caveats of this solution:
* Starvation: If a worker keeps accepting new jobs, then it might not be
  updated.
* Inconsistency: There exist a window between the job acceptation and the
  protection where the worker can be shutdown without having the time to
  protect itself.
2022-05-10 11:45:29 +02:00
Jordi Gil
b6f61d26ea distro/fedora: extend kernel options for openstack, qcow2, vhd and vmdk types
* Use the same value from AMI to these image types:

"ro no_timer_check console=ttyS0,115200n8 biosdevname=0 net.ifnames=0"

* Regenerate manifests
2022-05-09 12:25:21 +02:00
Jordi Gil
69095f5034 distro/fedora: produce VMDK as stream-optimized
Added code in fedora/pipelines.go to add the subformat field in the
manifests

Added manifests for f34 and f35 for x86_64 only (image type not
available in aarch64)
2022-05-09 12:25:21 +02:00
Jordi Gil
3a079eec26 distro/fedora: Regenerated manifests
* IoT image types now correctly point to the fedora-identity-iot package
* QCOW2, VMDK and OCI types use Fedora Cloud as identity package
* Changed default target for AMI from graphical.target to multi-user.target. This matches the behaviour with the RHEL types, which all target the multi-user.
* Readded the image-info field for some manifests which was missing due to issues regenerating the manifests.
2022-05-09 12:25:21 +02:00
Jordi Gil
0c3d17983b distro/fedora: Added package 'fedora-release-iot' for iot-commit image type 2022-05-09 12:25:21 +02:00
Jordi Gil
f256303854 distro/fedora: Changed kernelOptions value for ec2/ami image to be ro no_timer_check console=ttyS0,115200n8 biosdevname=0 net.ifnames=0 2022-05-09 12:25:21 +02:00
Jordi Gil
b19b824e38 distro/fedora: Regenerated manifests for f34-aarch and f35-x86_64 that include the image-info field 2022-05-09 12:25:21 +02:00
Jordi Gil
7cf937d668 distro/fedora: removed specific locale and timezone for ec2/ami since they are the same as the default one for all image types 2022-05-09 12:25:21 +02:00
Jordi Gil
6593e30d63 distro/fedora: Regenerated test manifests using V2 API 2022-05-09 12:25:21 +02:00
Jordi Gil
6d89a0f333 distro/fedora: change default target for qcow2,oci,vhd image types to multi-user instead of graphical 2022-05-09 12:25:21 +02:00
Jordi Gil
dbd163d49c distro/fedora: use GetTestingImagePackageSpecSets() and remove references to "centos" 2022-05-09 12:25:21 +02:00
Jordi Gil
5936abf85e distro/fedora: refactor osBuild pipeline 2022-05-09 12:25:21 +02:00
Jordi Gil
9a4125a37d distro/fedora: Added packageSetChain references
* Removed specific function that packaged the fedora cloud package group to avoid collision between fedora-identity-cloud and fedora-identity-basic packages. With the introduction of the PackageSetChains() it is no longer necessary to filter the packages
2022-05-09 12:25:21 +02:00
Jordi Gil
426085c060 distro/fedora: added smartmontools-selinux for iot commit 2022-05-09 12:25:21 +02:00
Jordi Gil
40d5950f1e distro/fedora: Set default target for ami/ec2 image type to multi-user and remove extra console entry in kernel opts 2022-05-09 12:25:21 +02:00
Jordi Gil
1d0e5c3410 distro/fedora: changed x86_64 bootType to hybrid 2022-05-09 12:25:21 +02:00
Jordi Gil
96d9e63b77 distro/fedora: capitalized IoT in fedora 2022-05-09 12:25:21 +02:00
Xiaofeng Wang
0af53e4cad test: Add fedora-iot-container and fedora-iot-installer tests 2022-05-09 12:25:21 +02:00
Jordi Gil
00a84d1fdc distro: allow user and group customizations for iot-installer 2022-05-09 12:25:21 +02:00
Jordi Gil
6c11c27c48 distro/fedora: removed old fedora code 2022-05-09 12:25:21 +02:00
Jordi Gil
2beb707def distro/fedora: regenerated test manifests 2022-05-09 12:25:21 +02:00