Commit graph

4267 commits

Author SHA1 Message Date
Juan Abia
8f9bb1bc01 tag azure resources with gitlab-ci-test
cloud cleaner removes old enough resources with 'gitlab-ci-test' tag.
2022-06-30 13:46:15 +02:00
Ygal Blum
08811dd632 Composer API - add support for services enable/disable
Add service sections to the customization unittests
2022-06-30 13:32:12 +02:00
Achilleas Koutsou
fc1d754999 distro: remove rhel85
Remove RHEL 8.5 distro source files.
RHEL 8.5 is now defined in the rhel86 package.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
436d8f9b43 distro/rhel86: add RHEL 8.5 variant
Define the distribution strings for RHEL 8.5 in distro/rhel86 and add
constructors.  Remove the old 8.5 from the distro registry and use the
new constructors.

Composer can now build RHEL 8.5 image-installer on aarch64, which wasn't
supported before.

RHEL 8.5 manifests have changed to minimise the differences from 8.6.
Some changes are fixes made in 8.6 but never backported to 8.5 because
of our (older) policy of not changing definitions after the release of a
distro.
Other changes are non-functional (e.g., stage or package order).

See the list below for the source of each change.

Manifest changes:
- Stage order changed for org.osbuild.systemd-logind and
  org.osbuild.rhsm.
- org.osbuild.grub2 options: config.default = "saved"
  Reverted 111cd8871f
- Partition sizes: RHEL 8.5 had extra arbitrarily sized padding for the
  header.  Now all partitions are sized to fit headers exactly.
  Original change at b7abef54e8.
- SELinux set to permissive in Anaconda.  This was changed in RHEL 8.6
  and 9.0 but never backported to 8.5.
  See a7fbe916b7.
- Installer isolevel set to 3.  Like above, this was changed in
  8.6 and 9.0.
  Original change at d8d161480e.
- Specify a remote for edge deployments.
  Original change at b18b4e80a0.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
56e6208fcf distro/rhel86: imagetypes and packages per RHEL version
Added utility function for comparing RHEL version strings.

Conditions added:
- greenboot subpackages were changed between RHEL 8.5 and RHEL 8.6.
- fido client packages aren't available in RHEL prior to 8.6.
- the ec2 SAP image type is not supported in RHEL prior to 8.6.
- the edge-simplified-installer and edge-raw-image image types are not
  supported in RHEL prior to 8.6.
    - They were previously supported in 8.5 without FDO support, but now
      it's dropped from 8.5 completely.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
e2e77a1190 distro/rhel86: reformat package sets
- One package per line for easier reading and diffing.
- Sorted alphabetically.  Exceptions made for sub-grouping.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
0a82fe4561 distro/rhel86: reorganise addImageTypes() call
One image type per line.
Sorted alphabetically.
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
062e38da69 osbuild-package-sets: check for required cl args
Print usage when one or more required arguments are omitted.
Fix the error message for incorrect distro: print distroName, not distro
(nil).
2022-06-30 11:14:21 +02:00
Achilleas Koutsou
5e14c34462 gen-manifests: add descriptions for command line flags 2022-06-30 11:14:21 +02:00
Achilleas Koutsou
8adb502aa0 gen-manifests: add flag for setting cache location 2022-06-30 11:14:21 +02:00
Achilleas Koutsou
5ac0ef2f77 test: update RHEL 8.5 manifests
Package changes from RHUI repository updates.
2022-06-30 11:14:21 +02:00
Ondřej Budai
3fcb737243 worker/koji-finalize: fix result options casting
It's actually a pointer, this caused a panic before, let's fix it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-29 16:44:10 +02:00
Ondřej Budai
0693274ffe worker/server: set a job error when heartbeat gets missing
Previously, we just used an empty struct when heartbeat failed. This is fine
for the osbuild job because it's treated as a failed one when
result.OSBuildResult == false which is the default value.

koji-finalize works differently though: It's in a failed state if there's
an job error of kojiError != "". So when failed heartbeat set the struct to
be empty, this was treated as success because there's no error.

Let's fix this by introducing a new error for the situation where we don't get
a heartbeat in time for a specific job.
2022-06-29 16:44:10 +02:00
Jakub Rusz
358e58f3d3 image-tests: skip azure-rhui test on rhel-86 2022-06-29 15:21:59 +02:00
schutzbot
d1b6a6455c Post release version bump
[skip ci]
2022-06-29 08:30:35 +00:00
Achilleas Koutsou
321295e34d test/diff-manifests: don't fail on diffs
The script isn't supposed to fail when the manifests differ.
Initialise err to 0 and assign it the exit code of the diff call if it
returns with an error.
2022-06-29 10:02:46 +02:00
Christian Kellner
dbd2ed6cc8 ci: add new container uploading test
Add a new integration test that will create a fedora container and
upload it to the gitlab container registry.
2022-06-29 10:02:46 +02:00
Christian Kellner
c5a5da5f56 distro/fedora: add container image type
Add a container image type that is based on the existing fedora
container image. There is a delta in terms of the configuration
because osbuild does not yet provide all the neccessary means,
but the package set is already very close.
2022-06-29 10:02:46 +02:00
Christian Kellner
59edcc4a46 distro/fedora: only pass kernelName to NewOSPipeline for bootable images
If the image is not bootable it means that we do not need the kernel
version and thus we should not pass it along to `NewOSPipeline`,
because we might not even have a kernel in the package set.
2022-06-29 10:02:46 +02:00
Christian Kellner
49559d53d7 pipeline/os: only infer kernel version if name is not empty
Do not try to infer the kernel verison if `kernelName` is the
empty string.
2022-06-29 10:02:46 +02:00
Christian Kellner
776dfd2da6 distro: option to exclude docs during installation
We don't ship documentation in container images, add an option to
ImageConfig to disable the installation of docs during installation.
2022-06-29 10:02:46 +02:00
Christian Kellner
f37a791bd2 distro: new image config to disable selinux
We don't need to label container artifacts, so include an option
in the image config to disable it.
NB: This is a boolean that with the default false in order to
not have to change every existing image configuration.
2022-06-29 10:02:46 +02:00
Christian Kellner
4b7dc3bbec pipelines/os: new option to exclude documentation
Add a new `ExcludeDocs` to control the if documentation is included
during the package installation.
2022-06-29 10:02:46 +02:00
Christian Kellner
d88bde8ab7 pipelines/os: new option to control selinux labeling
Make the SELinux labeling of the tree optional via a `SELinux`
variable.
2022-06-29 10:02:46 +02:00
Christian Kellner
516f30de17 distro: new blueprint helper where kernel package is optional
Add a new extended helper `GetPackagesEx` that includes a flag
to specify if the target is bootlable or not and only include
the kernel package in case it is set to true. Delegate to this
new helper from the existing `GetPackages`.
2022-06-29 10:02:46 +02:00
Christian Kellner
986f076276 container: add support for uploading to registries
Add a new generic container registry client via a new `container`
package. Use this to create a command line utility as well as a
new upload target for container registries.

The code uses the github.com/containers/* project and packages to
interact with container registires that is also used by skopeo,
podman et al. One if the dependencies is `proglottis/gpgme` that
is using cgo to bind libgpgme, so we have to add the corresponding
devel package to the BuildRequires as well as installing it on CI.

Checks will follow later via an integration test.
2022-06-29 10:02:46 +02:00
Jakub Rusz
d136a075bc manifests: regenerate RHEL-8.6 qcow2 test cases 2022-06-29 10:15:50 +03:00
Ondřej Budai
c6c401a322 worker/config: add tests for parseConfig
Tests are always nice, aren't they?

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
9af6d1d43b worker: make default config values more idiomatic
The only functional change is that
base_path = ""

will be now parsed as:

config.BasePath == ""

which wasn't possible before.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
dc78b05a19 worker: flip error handling when parsing the config
It's more idiomatic this way.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
c7716c2072 worker/config: extract all subtypes
...to make this testable.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
04f053689d worker: factor out config parsing
The struct is factored out 1:1. The only functional change in this commit is
worker now logging in case of a missing config (which means just loading the
defaults).

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
e4a66b0373 worker: clean up the RelaxTimeoutFactor
This commit moves the field to the koji struct where it actually belongs.
Also, it renames it to relax_timeout_factor for the sake of consistency.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
3b32480d45 worker: factor out kojiServer struct
In order to match the config structure.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
8666abc4ba worker: add a forgotten toml tag
The TOML library translates the field names 1:1, so now you have to use:

[Composer]
proxy: "abcd"

This is not idiomatic though so let's add the toml tag to make it [composer].

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Juan Abia
fa38246575 remove test skip
COMPOSER-1397 temporarily skipped azure-rhui image. Now that we test on
rhel 8.7, this skip can be removed.
2022-06-28 15:23:52 +02:00
dependabot[bot]
2c6e28b01d build(deps): bump github.com/aws/aws-sdk-go from 1.44.4 to 1.44.43
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.4 to 1.44.43.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.4...v1.44.43)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 14:25:12 +02:00
Chloe Kaubisch
765e93bfc8 cloudapi: get specific error from openapi
Error handling is structured in such a way that typically, a ServiceCodeError is passed
through the echo HTTP error, in reference to internally defined errors. We want to be able
to obtain and return specific external errors, for example during validation from openapi3.
Add a 'details' field to the serviceError struct, to contain extra / externally defined
information. Modify HTTPErrorHandler to anticipate either a string or a ServiceErrorCode
from echo, and respond accordingly. Edit the affected tests to expect the appropriate response.
2022-06-28 10:02:55 +02:00
Achilleas Koutsou
d59d870574 dnf-json: fix depsolve error handling
When a DepsolveError exception occurs, the error message would print the
packages in the request.  When the request arguments changed, the error
message handling wasn't updated and would fail to produce the correct
error message.

Compile a list of packages from all transactions and print them in the
error message as a comma-separated list.
2022-06-27 20:41:34 +02:00
Tom Gundersen
be2195b235 pipeline: add a bit more documentation
Make sure that each of the types have at least a bit of documentation.
2022-06-27 19:11:26 +01:00
Tom Gundersen
4556312d22 pipeline: split qcow2 into format specific pipelines
Stages are procedural and named after the tool they wrap, but pipelines are declarative and should
be named after the kind of artefact they produce.

This splits the qemu (the tool) pipeline into qcow2, vmdk, and vpc (the formats) pipelines. In theory
we may have wanted to implemented through some shared helpers, but for now it seems trivial
enough that it is not worth it.
2022-06-27 19:11:26 +01:00
Tom Gundersen
1fa92f9091 pipeline: distinguish between optional and mandatory properties
The ideal is that the constructor takes mandatory properties as arguments, and fields in the struct
are all optional.

This clarifies that across the pipelines (or leaves TODOs where work remains), and where possible
makes fields optional by providing a valid default value.
2022-06-27 19:11:26 +01:00
Tom Gundersen
c001af63ec pipeline: further cleanups
This adds more documentation and makes more properties implicitly inherited rather than
repeated. This makes for less boilerplate, and gives us fewer things to keep in sync.
2022-06-27 19:11:26 +01:00
Tom Gundersen
e5ef7933ac pipeline: drop stage_options.go
This was barely used. Making it explicit seemed more readable.
2022-06-27 19:11:26 +01:00
Tom Gundersen
f60092033b pipelines: make OSPipeline own the PartitionTable
The OSPipeline might need to know what disk layout it will be put onto, enforce this by making
the PartitionTable a property of the OSPipeline, and require child pipelines to query it when needed.
2022-06-27 19:11:26 +01:00
Tom Gundersen
ae34513d18 pipeline: move package to top level
The pipeline package is exists conceptually between the distro and the osbuild packages, so move
it to the top level rather than as a child of distro.

No functional change.
2022-06-27 19:11:26 +01:00
Tom Gundersen
e024ceb138 pipelines: cleanups
This is a collection of minor cleanups:
 - Start documenting the API
 - Enforce dependent pipelines have the correct type where necessary
 - Use data from dependent pipelines where possible
 - Start enforcing required fields
 - Move logic into the pipeline implementation where we can
2022-06-27 19:11:26 +01:00
Tom Gundersen
040a4ef6a1 pipelines: add Fedora IoT installer pipelines 2022-06-27 19:11:26 +01:00
Tom Gundersen
284eb9af59 distro/fedora: drop dead code 2022-06-27 19:11:26 +01:00
Tom Gundersen
0f015801d7 pipelines: add Fedora IoT commit and container pipelines 2022-06-27 19:11:26 +01:00