Commit graph

4279 commits

Author SHA1 Message Date
Tomas Hozza
20cb2e1b2c worker/osbuild: set error in target result
Ensure that a target result with a proper error is added to the Job
result, in case the there was any error encountered. This error is not
used at all for now. Keep setting the `JobError` to the same error set
in the target result for now.

This is a step towards job results containing multiple target results
with each or them having potentially an error set as well.
2022-07-01 18:55:01 +01:00
Tomas Hozza
ff669b8f2c worker/osbuild: do not set job results in uploadToS3()
Do not pass the `worker.OSBuildJobResult` to `uploadToS3()`, but instead
return target errors from the function. This will make the error
handling of all upload targets consistent and easier to modify to
support multiple targets.
2022-07-01 18:55:01 +01:00
Tomas Hozza
93fd5e3821 target: ensure that each used target has NewXXXTargetResult defined
Ensure that `UnmarshalTargetResultOptions()` is called only when there
are any options to unmarshal in the JSON object.

Since results of some Targets don't have any options defined, mark
`TargetResult.Options` as optional in the JSON tag.
2022-07-01 18:55:01 +01:00
Tomas Hozza
6f13db5b92 target: extend TargetResult with TargetError
This will enable reporting of target-specific errors from jobs, once
they'll support multiple targets.

Target errors are currently reported via `JobResult.JobError`.
2022-07-01 18:55:01 +01:00
Tomas Hozza
59ded68457 worker: delete TargetErrors from OSBuildJobResult
The `TargetErrors` is not used any more since PR#2192 [1] and there is
no need to keep the backward compatibility any more, because there are
no composer / worker instances in production, which are not running the
modified code.

In addition, delete unit tests covering this legacy error handling.

[1] https://github.com/osbuild/osbuild-composer/pull/2192
2022-07-01 18:55:01 +01:00
Tomas Hozza
c63bfe6d83 target: use constants for target names, instead of string literals 2022-07-01 18:55:01 +01:00
Juan Abia
f6fa5ccca1 remove cloud cleaner
scheduled cloud cleaner now uses a new method to remove azure resources.
So cloud cleaner code is not used
2022-07-01 17:47:44 +02:00
Jakub Rusz
6d15c03d2f tools/generic_s3: remove unused condition
This is a copypaste from other tests. Test can be ran on nightlies and
is passing.
2022-07-01 13:00:52 +03:00
Jakub Rusz
3547764687 CI: Don't run ContainerUpload job on scheduled pipelines 2022-07-01 13:00:52 +03:00
Jakub Rusz
8d6b998b0a tests/libvirt: boot with uefi on rhel and centos
Updating condition to run on all rhel and centos runners. Before this
eufi was not tested on all supported runners.
2022-07-01 13:00:52 +03:00
Achilleas Koutsou
444b8e33c9 distro: don't enable fdo-client service on RHEL 8.5
RHEL 8.5 doesn't have FDO and it's not installed, but we were enabling
the service for it in edge.
2022-06-30 18:34:45 +02:00
Christian Kellner
54ae069b94 distro: remove rhel 9 beta
This was a stop-gap until the actual rhel 9 distro was created. It
is in a sad state, quite broken and shout not be used by anybody.
Put it out of its misery.
2022-06-30 14:15:33 +02:00
Juan Abia
8f9bb1bc01 tag azure resources with gitlab-ci-test
cloud cleaner removes old enough resources with 'gitlab-ci-test' tag.
2022-06-30 13:46:15 +02:00
Ygal Blum
08811dd632 Composer API - add support for services enable/disable
Add service sections to the customization unittests
2022-06-30 13:32:12 +02:00
Achilleas Koutsou
fc1d754999 distro: remove rhel85
Remove RHEL 8.5 distro source files.
RHEL 8.5 is now defined in the rhel86 package.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
436d8f9b43 distro/rhel86: add RHEL 8.5 variant
Define the distribution strings for RHEL 8.5 in distro/rhel86 and add
constructors.  Remove the old 8.5 from the distro registry and use the
new constructors.

Composer can now build RHEL 8.5 image-installer on aarch64, which wasn't
supported before.

RHEL 8.5 manifests have changed to minimise the differences from 8.6.
Some changes are fixes made in 8.6 but never backported to 8.5 because
of our (older) policy of not changing definitions after the release of a
distro.
Other changes are non-functional (e.g., stage or package order).

See the list below for the source of each change.

Manifest changes:
- Stage order changed for org.osbuild.systemd-logind and
  org.osbuild.rhsm.
- org.osbuild.grub2 options: config.default = "saved"
  Reverted 111cd8871f
- Partition sizes: RHEL 8.5 had extra arbitrarily sized padding for the
  header.  Now all partitions are sized to fit headers exactly.
  Original change at b7abef54e8.
- SELinux set to permissive in Anaconda.  This was changed in RHEL 8.6
  and 9.0 but never backported to 8.5.
  See a7fbe916b7.
- Installer isolevel set to 3.  Like above, this was changed in
  8.6 and 9.0.
  Original change at d8d161480e.
- Specify a remote for edge deployments.
  Original change at b18b4e80a0.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
56e6208fcf distro/rhel86: imagetypes and packages per RHEL version
Added utility function for comparing RHEL version strings.

Conditions added:
- greenboot subpackages were changed between RHEL 8.5 and RHEL 8.6.
- fido client packages aren't available in RHEL prior to 8.6.
- the ec2 SAP image type is not supported in RHEL prior to 8.6.
- the edge-simplified-installer and edge-raw-image image types are not
  supported in RHEL prior to 8.6.
    - They were previously supported in 8.5 without FDO support, but now
      it's dropped from 8.5 completely.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
e2e77a1190 distro/rhel86: reformat package sets
- One package per line for easier reading and diffing.
- Sorted alphabetically.  Exceptions made for sub-grouping.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
0a82fe4561 distro/rhel86: reorganise addImageTypes() call
One image type per line.
Sorted alphabetically.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
062e38da69 osbuild-package-sets: check for required cl args
Print usage when one or more required arguments are omitted.
Fix the error message for incorrect distro: print distroName, not distro
(nil).
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
5e14c34462 gen-manifests: add descriptions for command line flags 2022-06-30 11:14:21 +02:00
Achilleas Koutsou
8adb502aa0 gen-manifests: add flag for setting cache location 2022-06-30 11:14:21 +02:00
Achilleas Koutsou
5ac0ef2f77 test: update RHEL 8.5 manifests
Package changes from RHUI repository updates.
2022-06-30 11:14:21 +02:00
Ondřej Budai
3fcb737243 worker/koji-finalize: fix result options casting
It's actually a pointer, this caused a panic before, let's fix it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-29 16:44:10 +02:00
Ondřej Budai
0693274ffe worker/server: set a job error when heartbeat gets missing
Previously, we just used an empty struct when heartbeat failed. This is fine
for the osbuild job because it's treated as a failed one when
result.OSBuildResult == false which is the default value.

koji-finalize works differently though: It's in a failed state if there's
an job error of kojiError != "". So when failed heartbeat set the struct to
be empty, this was treated as success because there's no error.

Let's fix this by introducing a new error for the situation where we don't get
a heartbeat in time for a specific job.
2022-06-29 16:44:10 +02:00
Jakub Rusz
358e58f3d3 image-tests: skip azure-rhui test on rhel-86 2022-06-29 15:21:59 +02:00
schutzbot
d1b6a6455c Post release version bump
[skip ci]
2022-06-29 08:30:35 +00:00
Achilleas Koutsou
321295e34d test/diff-manifests: don't fail on diffs
The script isn't supposed to fail when the manifests differ.
Initialise err to 0 and assign it the exit code of the diff call if it
returns with an error.
2022-06-29 10:02:46 +02:00
Christian Kellner
dbd2ed6cc8 ci: add new container uploading test
Add a new integration test that will create a fedora container and
upload it to the gitlab container registry.
2022-06-29 10:02:46 +02:00
Christian Kellner
c5a5da5f56 distro/fedora: add container image type
Add a container image type that is based on the existing fedora
container image. There is a delta in terms of the configuration
because osbuild does not yet provide all the neccessary means,
but the package set is already very close.
2022-06-29 10:02:46 +02:00
Christian Kellner
59edcc4a46 distro/fedora: only pass kernelName to NewOSPipeline for bootable images
If the image is not bootable it means that we do not need the kernel
version and thus we should not pass it along to `NewOSPipeline`,
because we might not even have a kernel in the package set.
2022-06-29 10:02:46 +02:00
Christian Kellner
49559d53d7 pipeline/os: only infer kernel version if name is not empty
Do not try to infer the kernel verison if `kernelName` is the
empty string.
2022-06-29 10:02:46 +02:00
Christian Kellner
776dfd2da6 distro: option to exclude docs during installation
We don't ship documentation in container images, add an option to
ImageConfig to disable the installation of docs during installation.
2022-06-29 10:02:46 +02:00
Christian Kellner
f37a791bd2 distro: new image config to disable selinux
We don't need to label container artifacts, so include an option
in the image config to disable it.
NB: This is a boolean that with the default false in order to
not have to change every existing image configuration.
2022-06-29 10:02:46 +02:00
Christian Kellner
4b7dc3bbec pipelines/os: new option to exclude documentation
Add a new `ExcludeDocs` to control the if documentation is included
during the package installation.
2022-06-29 10:02:46 +02:00
Christian Kellner
d88bde8ab7 pipelines/os: new option to control selinux labeling
Make the SELinux labeling of the tree optional via a `SELinux`
variable.
2022-06-29 10:02:46 +02:00
Christian Kellner
516f30de17 distro: new blueprint helper where kernel package is optional
Add a new extended helper `GetPackagesEx` that includes a flag
to specify if the target is bootlable or not and only include
the kernel package in case it is set to true. Delegate to this
new helper from the existing `GetPackages`.
2022-06-29 10:02:46 +02:00
Christian Kellner
986f076276 container: add support for uploading to registries
Add a new generic container registry client via a new `container`
package. Use this to create a command line utility as well as a
new upload target for container registries.

The code uses the github.com/containers/* project and packages to
interact with container registires that is also used by skopeo,
podman et al. One if the dependencies is `proglottis/gpgme` that
is using cgo to bind libgpgme, so we have to add the corresponding
devel package to the BuildRequires as well as installing it on CI.

Checks will follow later via an integration test.
2022-06-29 10:02:46 +02:00
Jakub Rusz
d136a075bc manifests: regenerate RHEL-8.6 qcow2 test cases 2022-06-29 10:15:50 +03:00
Ondřej Budai
c6c401a322 worker/config: add tests for parseConfig
Tests are always nice, aren't they?

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
9af6d1d43b worker: make default config values more idiomatic
The only functional change is that
base_path = ""

will be now parsed as:

config.BasePath == ""

which wasn't possible before.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
dc78b05a19 worker: flip error handling when parsing the config
It's more idiomatic this way.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
c7716c2072 worker/config: extract all subtypes
...to make this testable.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
04f053689d worker: factor out config parsing
The struct is factored out 1:1. The only functional change in this commit is
worker now logging in case of a missing config (which means just loading the
defaults).

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
e4a66b0373 worker: clean up the RelaxTimeoutFactor
This commit moves the field to the koji struct where it actually belongs.
Also, it renames it to relax_timeout_factor for the sake of consistency.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
3b32480d45 worker: factor out kojiServer struct
In order to match the config structure.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
8666abc4ba worker: add a forgotten toml tag
The TOML library translates the field names 1:1, so now you have to use:

[Composer]
proxy: "abcd"

This is not idiomatic though so let's add the toml tag to make it [composer].

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Juan Abia
fa38246575 remove test skip
COMPOSER-1397 temporarily skipped azure-rhui image. Now that we test on
rhel 8.7, this skip can be removed.
2022-06-28 15:23:52 +02:00
dependabot[bot]
2c6e28b01d build(deps): bump github.com/aws/aws-sdk-go from 1.44.4 to 1.44.43
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.4 to 1.44.43.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.4...v1.44.43)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 14:25:12 +02:00
Chloe Kaubisch
765e93bfc8 cloudapi: get specific error from openapi
Error handling is structured in such a way that typically, a ServiceCodeError is passed
through the echo HTTP error, in reference to internally defined errors. We want to be able
to obtain and return specific external errors, for example during validation from openapi3.
Add a 'details' field to the serviceError struct, to contain extra / externally defined
information. Modify HTTPErrorHandler to anticipate either a string or a ServiceErrorCode
from echo, and respond accordingly. Edit the affected tests to expect the appropriate response.
2022-06-28 10:02:55 +02:00