Commit graph

3187 commits

Author SHA1 Message Date
Roy Golan
d9051c23e6 Add build requirement in README.md
koji depends on uccr/kerbi go module which depends on kerberos C header
files. Make sure new commers know about what to install in order to
compile the project.

Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-01-28 15:16:47 +01:00
Roy Golan
da17cfe872 Add rpm build requierments
Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-01-28 15:16:47 +01:00
Roy Golan
932e0269f5 OCI: vendor dependencies
Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-01-28 15:16:47 +01:00
Christian Kellner
c5feb93279 distro/rhel90: no uuids in dos partition table
The unification of the partition table also introduced uuids and
types in uuid form for partition tables in dos layout, sill used
on PPC64LE and s390x. The org.osbuild.sfdisk stage did work with
that but produced a `/boot` partition with the wrong type, which
grub2 refused to read from and thus prevented boot. Fix this by
removing uuids from the dos partition tables.

Reported-by: Jakub Rusz <jrusz@redhat.com>
2022-01-28 12:16:39 +01:00
Gianluca Zuccarelli
88b5529cc4 osbuild-worker: test error backwards compatability
Since the workers will use structured error messages
going forward, it is necessary to maintain backwards
compatability for there errors in composer. Tests have
been added to the various apis to ensure that each api
checks for both kinds of errors, old and new.
2022-01-27 16:45:14 +01:00
Gianluca Zuccarelli
cc981b887a osbuild-worker: implement structured errors
Implement the structured errors as defined by the worker client.
Every error for each of the job types now returns a structured
error with a reason and a specific error code.  This will make
it possible to differentiate between 4xx errors and 5xx errors.

This commit refactors the way errors are implemented in the workers,
but maintains backwards compatability in composer by checking for
both kinds of errors.
2022-01-27 16:45:14 +01:00
Gianluca Zuccarelli
daf24f8db3 worker: define worker errors
Define worker errors to give more structured
error messages. The error api is:
id: VALIDATION_ERROR_NUMBER, reason: STRING, details: { issues: [{...}, {...}] }

The api was agreed upon with osbuild so that,
in future, osbuild errors will share the same
structure
2022-01-27 16:45:14 +01:00
sanne
1dbbc37cac ostree: Use stable installer image for PRs 2022-01-27 14:07:23 +01:00
sanne
8f8e871e3b schutzfile: Define nightly repos in schutzfile 2022-01-27 14:07:23 +01:00
Ondřej Budai
32651a610e test/koji: remove the koji-cli patch
The fix is already in Koji 1.27.1 which should be available in all downstreams
we are testing against.

See https://src.fedoraproject.org/rpms/koji

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-26 14:44:27 +01:00
Alexander Todorov
864a1a2330 tests: Don't execute koji.sh inside nightly CI pipelines 2022-01-26 11:39:43 +01:00
lavocatt
3899acbf84 Post release version bump
[skip ci]
2022-01-26 09:33:32 +00:00
Alexander Todorov
1bd852ceef tests: Add missing /var/log/audit to explicit checks
- the /var/log/audit mountpoint has been added previously to the
  blueprint but wasn't explicitly checked
- reordered the list of mountpoints to match the blueprint
  (alphabetically)
- added logging of successfull results we that QE can see what has been
  tested in the logs and use it for verification purposes.

Related: rhbz#2002727, rhbz#2001891
2022-01-25 11:59:20 +01:00
Alexander Todorov
e6212cced6 Add Image Tests to nightly pipelines 2022-01-25 11:53:16 +01:00
Jakub Rusz
1c2c97c900 tests/api: Make cleanup best effort
We should not fail on missing variables or errors during cleanup. The
test can fail in any stage and the cleanup should be run full each time.
2022-01-19 19:56:28 +01:00
sanne
6b716a4bbc tools/appsre-build-worker-packer: Fix branch parsing on jenkins 2022-01-19 12:48:14 +01:00
sanne
8a8ed14319 templates/dashboards: Fixed grafana uids
This way we get a nice URL `.../d/image-builder-(composer|worker)`.
2022-01-19 12:27:33 +01:00
Jakub Rusz
98f5623914 test/cases: properly kill journal for installers.sh
CI gets stuck if we don't kill the journal watcher on exit.
See bc1a7dc30c for more info.
2022-01-19 09:59:09 +01:00
Jakub Rusz
35fd20c0de ci: don't run iostats temporarily
CI job execution gets stuck if it fails and waits for timeout because
isostat are running in background preventing the main scrip from
exiting. Removing temporarily until a better solution is found.
2022-01-19 09:59:09 +01:00
Jakub Rusz
18ee1607a1 ci: change openstack runners to rhos-01
We can use both Openstack clouds now so switch to the new rhos-01
2022-01-19 09:59:09 +01:00
Jakub Rusz
1b951c5377 ci: update terraform SHA 2022-01-19 09:59:09 +01:00
sanne
ef6c5df9fa templates/packer: Make cdn host check less sensitive 2022-01-18 17:00:17 +01:00
sanne
68e98244b9 templates/packer: Correct priority for worker rpms
Lower priority means higher, currently the images built through AppSRE's
infra install the worker from epel.
2022-01-17 14:30:11 +01:00
sanne
a6f40f807d tools/appsre-build-worker-packer: Run entire workflow in container
Avoids the issue with artifacts that jenkins can't clean up.
2022-01-13 15:05:52 +01:00
sanne
2d0bfedc9d tools/appsre-ansible: Correct rpm destination when pulling rpms 2022-01-12 19:47:45 +01:00
sanne
e8b790e8d4 tools/appsre-build-worker-packer: Disable ansible debug output 2022-01-12 19:47:45 +01:00
sanne
6b43594840 tools/appsre-build-worker-packer: Ansible rpmbuild debug output 2022-01-12 17:33:50 +01:00
sanne
e6fd492372 tools/appsre-build-worker-packer: Prevent ssh from timing out 2022-01-12 17:33:50 +01:00
sanne
a21061fc5c tools/appsre-ansible: Don't use /tmp for rpmbuilds 2022-01-12 15:27:27 +01:00
lavocatt
b303e95827 Post release version bump
[skip ci]
2022-01-12 13:46:04 +00:00
sanne
93ec151c60 tools/appsre-build-worker-packer: Verbose ssh check 2022-01-12 12:33:04 +01:00
sanne
8c514308e7 tools/appsre-build-worker-packer: Correct user in ssh check 2022-01-12 12:13:41 +01:00
sanne
01163a1384 tools/appsre-build-worker-packer: Use ssh instead of ssh-keyscan
`ssh-keyscan` on appsre's infra has a bug where it doesn't exit 1 in
case of failure.
2022-01-12 12:02:58 +01:00
sanne
a83cf95d5b go.mod: Update oapi-codegen and kin-openapi 2022-01-12 11:35:06 +01:00
Ondřej Budai
add17bba45 spec: drop Fedora 33 hacks
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Ondřej Budai
b667df4443 bump the minimum Go version to 1.16
Fedora 34 ships 1.16
RHEL 8.5 also ships 1.16

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Ondřej Budai
f605fe9feb ci: drop Fedora 33
It's EOL.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Ondřej Budai
5da89ccce0 ci/container: switch to CS8
Fedora 33 is EOL, let's use something that will last longer than a single
Fedora release.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Tomas Hozza
ec6099f7f6 RHEL-86: port over the RHEL-90 pipeline refactoring
Port all of the pipeline refactoring done to RHEL-90 to RHEL-86. Both
distros now use the same approach.

Regenerate all RHEL-8.6 and CentOS 8 image test cases.

[1] https://git.centos.org/centos/kickstarts/tree/master

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
960aad0b0a RHEL-86: apply subscriptions and RHSM configuration only on RHEL
RHSM configuration is now applied conditionally only on RHEL. The same
applies to the customization to subscribe the system on first boot.

The reason is that the CentOS `@core` package group does not contain
`subscription-manager`. Thus it is not installed on CentOS Stream by
default and also CentOS 8 image definitions don't apply any changes
to the RHSM configuration [1].

In addition, make sure to not install any subscription-manager
packages on CentOS Stream images.

Regenerate all CentOS 8 image test cases.

[1] https://git.centos.org/centos/kickstarts/tree/master

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
e6c55efe08 RHEL-90: replace ostreeTreePipeline by osPipeline
Enhance the `osPipeline` to add necessary stages to the returned
pipeline, in case the image is RPM OSTree based. As a result, delete the
`ostreeTreePipeline` and replace its uses by `osPipeline`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
90e00c48fa RHEL-90: make the osPipeline self-contained
Make the `osPipeline` self-contained in the sense, that no stages are
added to the returned pipeline outside of the function and the returned
pipeline is usable as returned.

Modify the `osPipeline` to add Kernel Cmdline, FSTab and bootloader
config stages to the pipeline if a valid partition table was passed to
the function. As the last one, the SELinux stage is appended to the
returned pipeline.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
36e81bfdac RHEL-90: replace rhelEc2SapPipelines by rhelEc2Pipelines`
Move the EC2 SAP image specific configuration from `ec2SapPipelines`
to the EC2 SAP default image configurations data structure. As a
result, remove the `ec2SapPipelines` and `rhelEc2SapPipelines` entirely
and use `rhelEc2Pipelines` for all RHEL EC2 images.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
43ea54f8de RHEL-90: replace ec2X86_64BaseTreePipeline by osPipeline
Move the x86_64 specific configuration from `ec2X86_64BaseTreePipeline`
to x86_64-specific image configurations for EC2 / AMI images. As a
result, remove the `ec2X86_64BaseTreePipeline` entirely and replace it
with `osPipeline`.

Regenerate image test cases. While there are changed in the manifests,
the actual image configuration didn't change at all and thus the
`image-info` report was not changed.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
776de86e6d RHEL-90: replace ec2BaseTreePipeline by osPipeline
Move all hard-coded image configuration from the `ec2BaseTreePipeline`
function to the `ImageConfig` structure and update the respective EC2
images default configuration structure.

Update `osPipeline` and `ostreeTreePipeline` to handle all of the new
configuration values from `ImageConfig`.

Completely remove the `ec2BaseTreePipeline` and replace it with
`osPipeline`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
755154179d RHEL-90: do not install subscription-manager on non-RHEL
Do not install any subscription-manager packages on non-RHEL distro
variant of RHEL-90 (meaning CentOS Stream). Subscription-manager is not
needed on CentOS Stream and it does not add value to install it by
default. It is also not included on images produced by CentOS Stream.

Relevant links:
- https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge_requests/151
- https://bugzilla.redhat.com/show_bug.cgi?id=1962385
- https://gitlab.com/redhat/centos-stream/release-engineering/kickstarts/-/blob/main/CentOS-Stream-9-kvm.ks#L115

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
8130c892f0 RHEL-90: move RHSM configuration to ImageConfig structure
Move the RHSM configuration settings to `ImageConfig` structure and use
when handling subscriptions in `osPipeline`, `ec2BaseTreePipeline` and
`ostreeTreePipeline` functions.

Regenerate image test cases. While there are changed in the manifests,
the actual image configuration didn't change at all and thus the
`image-info` report was not changed.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
b200fa8fcd RHEL-90: introduce default image config data structure
Introduce a new data structure `ImageConfig` holding the default OS
configuration applied when building an image. The structure can be used
to hold the default image configuration on the distribution level with
possible overrides defined on the image-type level.

As a starting point, move hard-coded default values and configuration
common for `osPipeline`, `ec2BaseTreePipeline` and `ostreeTreePipeline`
to the distribution and image-type default image configuration. This is
preparing the ground for merging all of these three pipeline functions
into `osPipeline`, which will produce the appropriate OS pipeline based
on the image-type configuration and the fact if it is rpmOstree or not.

Regenerate affected EC2 and AMI manifests. There is however no change in
the resulting image configuration and image-info report.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
sanne
3c729be3c5 tools/appsre-build-worker-packer: Add image_users variable
packer will share the ami with those users.
2022-01-11 14:30:19 +01:00
sanne
2420504e1d tools/appsre-build-worker-packer: Use WORKSPACE variable
Docker doesn't accept "." as a mountpoint, use the WORKSPACE variable on
jenkins and default to PWD.
2022-01-11 14:30:19 +01:00