Commit graph

3192 commits

Author SHA1 Message Date
Christian Kellner
5e2617ed49 kojiapi: successful osbuild jobs must have output
When we compute the overall status of a koji compose, the individual
build jobs are checked. Currently, a job is considered a failure, if
a build job has output (`OSBuildOutput`) and the output's `Success`
field is `false`. But `OSBuildOutput` will be `nil` when osbuild
crashed or refused the manifest input. Therefore the job status is
a failure if `OSBuildOutput` is `nil`, since if osbuild was run,
and the run was successful we must have a non-`OSBuildOutput` field.
2022-01-28 20:37:23 +00:00
Christian Kellner
987f4c4795 worker: fix job error check for koji-osbuild job
This check got inverted during the work on "Worker errors backwards
compatibility". As a consequence, osbuild was never run and the
result structure `buildResult.OSBuildOutput` was `nil` Since the
overall status reporting is not complete, and does not take this,
i.e. `buildResult.OSBuildOutput`, being `nil` as an error case,
the overall status was reported as "success". See the function
`composeStatusFromJobStatus` in `internal/kojiapi/server.go`.
2022-01-28 20:37:23 +00:00
Achilleas Koutsou
edfdcf0be6 osbuild-worker: change error handling for OCI upload
Issues caused by rebase-merge:
- Error handling was changed in one PR
- Errors using the old method were introduced in another
2022-01-28 17:33:48 +01:00
Achilleas Koutsou
022a52d56b distro/rhel90: special case root user for ssh keys
Add a special cases for the root user to the work-around for ssh keys in
OSTree commits.

See 93e54cd872 for the original,
equivalent change in RHEL 8.6.
2022-01-28 15:16:56 +01:00
Roy Golan
bee932e222 Add support for OCI upload provider
Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-01-28 15:16:47 +01:00
Roy Golan
d9051c23e6 Add build requirement in README.md
koji depends on uccr/kerbi go module which depends on kerberos C header
files. Make sure new commers know about what to install in order to
compile the project.

Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-01-28 15:16:47 +01:00
Roy Golan
da17cfe872 Add rpm build requierments
Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-01-28 15:16:47 +01:00
Roy Golan
932e0269f5 OCI: vendor dependencies
Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-01-28 15:16:47 +01:00
Christian Kellner
c5feb93279 distro/rhel90: no uuids in dos partition table
The unification of the partition table also introduced uuids and
types in uuid form for partition tables in dos layout, sill used
on PPC64LE and s390x. The org.osbuild.sfdisk stage did work with
that but produced a `/boot` partition with the wrong type, which
grub2 refused to read from and thus prevented boot. Fix this by
removing uuids from the dos partition tables.

Reported-by: Jakub Rusz <jrusz@redhat.com>
2022-01-28 12:16:39 +01:00
Gianluca Zuccarelli
88b5529cc4 osbuild-worker: test error backwards compatability
Since the workers will use structured error messages
going forward, it is necessary to maintain backwards
compatability for there errors in composer. Tests have
been added to the various apis to ensure that each api
checks for both kinds of errors, old and new.
2022-01-27 16:45:14 +01:00
Gianluca Zuccarelli
cc981b887a osbuild-worker: implement structured errors
Implement the structured errors as defined by the worker client.
Every error for each of the job types now returns a structured
error with a reason and a specific error code.  This will make
it possible to differentiate between 4xx errors and 5xx errors.

This commit refactors the way errors are implemented in the workers,
but maintains backwards compatability in composer by checking for
both kinds of errors.
2022-01-27 16:45:14 +01:00
Gianluca Zuccarelli
daf24f8db3 worker: define worker errors
Define worker errors to give more structured
error messages. The error api is:
id: VALIDATION_ERROR_NUMBER, reason: STRING, details: { issues: [{...}, {...}] }

The api was agreed upon with osbuild so that,
in future, osbuild errors will share the same
structure
2022-01-27 16:45:14 +01:00
sanne
1dbbc37cac ostree: Use stable installer image for PRs 2022-01-27 14:07:23 +01:00
sanne
8f8e871e3b schutzfile: Define nightly repos in schutzfile 2022-01-27 14:07:23 +01:00
Ondřej Budai
32651a610e test/koji: remove the koji-cli patch
The fix is already in Koji 1.27.1 which should be available in all downstreams
we are testing against.

See https://src.fedoraproject.org/rpms/koji

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-26 14:44:27 +01:00
Alexander Todorov
864a1a2330 tests: Don't execute koji.sh inside nightly CI pipelines 2022-01-26 11:39:43 +01:00
lavocatt
3899acbf84 Post release version bump
[skip ci]
2022-01-26 09:33:32 +00:00
Alexander Todorov
1bd852ceef tests: Add missing /var/log/audit to explicit checks
- the /var/log/audit mountpoint has been added previously to the
  blueprint but wasn't explicitly checked
- reordered the list of mountpoints to match the blueprint
  (alphabetically)
- added logging of successfull results we that QE can see what has been
  tested in the logs and use it for verification purposes.

Related: rhbz#2002727, rhbz#2001891
2022-01-25 11:59:20 +01:00
Alexander Todorov
e6212cced6 Add Image Tests to nightly pipelines 2022-01-25 11:53:16 +01:00
Jakub Rusz
1c2c97c900 tests/api: Make cleanup best effort
We should not fail on missing variables or errors during cleanup. The
test can fail in any stage and the cleanup should be run full each time.
2022-01-19 19:56:28 +01:00
sanne
6b716a4bbc tools/appsre-build-worker-packer: Fix branch parsing on jenkins 2022-01-19 12:48:14 +01:00
sanne
8a8ed14319 templates/dashboards: Fixed grafana uids
This way we get a nice URL `.../d/image-builder-(composer|worker)`.
2022-01-19 12:27:33 +01:00
Jakub Rusz
98f5623914 test/cases: properly kill journal for installers.sh
CI gets stuck if we don't kill the journal watcher on exit.
See bc1a7dc30c for more info.
2022-01-19 09:59:09 +01:00
Jakub Rusz
35fd20c0de ci: don't run iostats temporarily
CI job execution gets stuck if it fails and waits for timeout because
isostat are running in background preventing the main scrip from
exiting. Removing temporarily until a better solution is found.
2022-01-19 09:59:09 +01:00
Jakub Rusz
18ee1607a1 ci: change openstack runners to rhos-01
We can use both Openstack clouds now so switch to the new rhos-01
2022-01-19 09:59:09 +01:00
Jakub Rusz
1b951c5377 ci: update terraform SHA 2022-01-19 09:59:09 +01:00
sanne
ef6c5df9fa templates/packer: Make cdn host check less sensitive 2022-01-18 17:00:17 +01:00
sanne
68e98244b9 templates/packer: Correct priority for worker rpms
Lower priority means higher, currently the images built through AppSRE's
infra install the worker from epel.
2022-01-17 14:30:11 +01:00
sanne
a6f40f807d tools/appsre-build-worker-packer: Run entire workflow in container
Avoids the issue with artifacts that jenkins can't clean up.
2022-01-13 15:05:52 +01:00
sanne
2d0bfedc9d tools/appsre-ansible: Correct rpm destination when pulling rpms 2022-01-12 19:47:45 +01:00
sanne
e8b790e8d4 tools/appsre-build-worker-packer: Disable ansible debug output 2022-01-12 19:47:45 +01:00
sanne
6b43594840 tools/appsre-build-worker-packer: Ansible rpmbuild debug output 2022-01-12 17:33:50 +01:00
sanne
e6fd492372 tools/appsre-build-worker-packer: Prevent ssh from timing out 2022-01-12 17:33:50 +01:00
sanne
a21061fc5c tools/appsre-ansible: Don't use /tmp for rpmbuilds 2022-01-12 15:27:27 +01:00
lavocatt
b303e95827 Post release version bump
[skip ci]
2022-01-12 13:46:04 +00:00
sanne
93ec151c60 tools/appsre-build-worker-packer: Verbose ssh check 2022-01-12 12:33:04 +01:00
sanne
8c514308e7 tools/appsre-build-worker-packer: Correct user in ssh check 2022-01-12 12:13:41 +01:00
sanne
01163a1384 tools/appsre-build-worker-packer: Use ssh instead of ssh-keyscan
`ssh-keyscan` on appsre's infra has a bug where it doesn't exit 1 in
case of failure.
2022-01-12 12:02:58 +01:00
sanne
a83cf95d5b go.mod: Update oapi-codegen and kin-openapi 2022-01-12 11:35:06 +01:00
Ondřej Budai
add17bba45 spec: drop Fedora 33 hacks
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Ondřej Budai
b667df4443 bump the minimum Go version to 1.16
Fedora 34 ships 1.16
RHEL 8.5 also ships 1.16

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Ondřej Budai
f605fe9feb ci: drop Fedora 33
It's EOL.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Ondřej Budai
5da89ccce0 ci/container: switch to CS8
Fedora 33 is EOL, let's use something that will last longer than a single
Fedora release.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-12 11:35:06 +01:00
Tomas Hozza
ec6099f7f6 RHEL-86: port over the RHEL-90 pipeline refactoring
Port all of the pipeline refactoring done to RHEL-90 to RHEL-86. Both
distros now use the same approach.

Regenerate all RHEL-8.6 and CentOS 8 image test cases.

[1] https://git.centos.org/centos/kickstarts/tree/master

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
960aad0b0a RHEL-86: apply subscriptions and RHSM configuration only on RHEL
RHSM configuration is now applied conditionally only on RHEL. The same
applies to the customization to subscribe the system on first boot.

The reason is that the CentOS `@core` package group does not contain
`subscription-manager`. Thus it is not installed on CentOS Stream by
default and also CentOS 8 image definitions don't apply any changes
to the RHSM configuration [1].

In addition, make sure to not install any subscription-manager
packages on CentOS Stream images.

Regenerate all CentOS 8 image test cases.

[1] https://git.centos.org/centos/kickstarts/tree/master

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
e6c55efe08 RHEL-90: replace ostreeTreePipeline by osPipeline
Enhance the `osPipeline` to add necessary stages to the returned
pipeline, in case the image is RPM OSTree based. As a result, delete the
`ostreeTreePipeline` and replace its uses by `osPipeline`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
90e00c48fa RHEL-90: make the osPipeline self-contained
Make the `osPipeline` self-contained in the sense, that no stages are
added to the returned pipeline outside of the function and the returned
pipeline is usable as returned.

Modify the `osPipeline` to add Kernel Cmdline, FSTab and bootloader
config stages to the pipeline if a valid partition table was passed to
the function. As the last one, the SELinux stage is appended to the
returned pipeline.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
36e81bfdac RHEL-90: replace rhelEc2SapPipelines by rhelEc2Pipelines`
Move the EC2 SAP image specific configuration from `ec2SapPipelines`
to the EC2 SAP default image configurations data structure. As a
result, remove the `ec2SapPipelines` and `rhelEc2SapPipelines` entirely
and use `rhelEc2Pipelines` for all RHEL EC2 images.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
43ea54f8de RHEL-90: replace ec2X86_64BaseTreePipeline by osPipeline
Move the x86_64 specific configuration from `ec2X86_64BaseTreePipeline`
to x86_64-specific image configurations for EC2 / AMI images. As a
result, remove the `ec2X86_64BaseTreePipeline` entirely and replace it
with `osPipeline`.

Regenerate image test cases. While there are changed in the manifests,
the actual image configuration didn't change at all and thus the
`image-info` report was not changed.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00
Tomas Hozza
776de86e6d RHEL-90: replace ec2BaseTreePipeline by osPipeline
Move all hard-coded image configuration from the `ec2BaseTreePipeline`
function to the `ImageConfig` structure and update the respective EC2
images default configuration structure.

Update `osPipeline` and `ostreeTreePipeline` to handle all of the new
configuration values from `ImageConfig`.

Completely remove the `ec2BaseTreePipeline` and replace it with
`osPipeline`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-01-11 17:08:19 +01:00