Commit graph

4242 commits

Author SHA1 Message Date
Christian Kellner
986f076276 container: add support for uploading to registries
Add a new generic container registry client via a new `container`
package. Use this to create a command line utility as well as a
new upload target for container registries.

The code uses the github.com/containers/* project and packages to
interact with container registires that is also used by skopeo,
podman et al. One if the dependencies is `proglottis/gpgme` that
is using cgo to bind libgpgme, so we have to add the corresponding
devel package to the BuildRequires as well as installing it on CI.

Checks will follow later via an integration test.
2022-06-29 10:02:46 +02:00
Jakub Rusz
d136a075bc manifests: regenerate RHEL-8.6 qcow2 test cases 2022-06-29 10:15:50 +03:00
Ondřej Budai
c6c401a322 worker/config: add tests for parseConfig
Tests are always nice, aren't they?

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
9af6d1d43b worker: make default config values more idiomatic
The only functional change is that
base_path = ""

will be now parsed as:

config.BasePath == ""

which wasn't possible before.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
dc78b05a19 worker: flip error handling when parsing the config
It's more idiomatic this way.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
c7716c2072 worker/config: extract all subtypes
...to make this testable.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
04f053689d worker: factor out config parsing
The struct is factored out 1:1. The only functional change in this commit is
worker now logging in case of a missing config (which means just loading the
defaults).

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
e4a66b0373 worker: clean up the RelaxTimeoutFactor
This commit moves the field to the koji struct where it actually belongs.
Also, it renames it to relax_timeout_factor for the sake of consistency.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
3b32480d45 worker: factor out kojiServer struct
In order to match the config structure.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Ondřej Budai
8666abc4ba worker: add a forgotten toml tag
The TOML library translates the field names 1:1, so now you have to use:

[Composer]
proxy: "abcd"

This is not idiomatic though so let's add the toml tag to make it [composer].

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-28 15:55:31 +01:00
Juan Abia
fa38246575 remove test skip
COMPOSER-1397 temporarily skipped azure-rhui image. Now that we test on
rhel 8.7, this skip can be removed.
2022-06-28 15:23:52 +02:00
dependabot[bot]
2c6e28b01d build(deps): bump github.com/aws/aws-sdk-go from 1.44.4 to 1.44.43
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.4 to 1.44.43.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.4...v1.44.43)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 14:25:12 +02:00
Chloe Kaubisch
765e93bfc8 cloudapi: get specific error from openapi
Error handling is structured in such a way that typically, a ServiceCodeError is passed
through the echo HTTP error, in reference to internally defined errors. We want to be able
to obtain and return specific external errors, for example during validation from openapi3.
Add a 'details' field to the serviceError struct, to contain extra / externally defined
information. Modify HTTPErrorHandler to anticipate either a string or a ServiceErrorCode
from echo, and respond accordingly. Edit the affected tests to expect the appropriate response.
2022-06-28 10:02:55 +02:00
Achilleas Koutsou
d59d870574 dnf-json: fix depsolve error handling
When a DepsolveError exception occurs, the error message would print the
packages in the request.  When the request arguments changed, the error
message handling wasn't updated and would fail to produce the correct
error message.

Compile a list of packages from all transactions and print them in the
error message as a comma-separated list.
2022-06-27 20:41:34 +02:00
Tom Gundersen
be2195b235 pipeline: add a bit more documentation
Make sure that each of the types have at least a bit of documentation.
2022-06-27 19:11:26 +01:00
Tom Gundersen
4556312d22 pipeline: split qcow2 into format specific pipelines
Stages are procedural and named after the tool they wrap, but pipelines are declarative and should
be named after the kind of artefact they produce.

This splits the qemu (the tool) pipeline into qcow2, vmdk, and vpc (the formats) pipelines. In theory
we may have wanted to implemented through some shared helpers, but for now it seems trivial
enough that it is not worth it.
2022-06-27 19:11:26 +01:00
Tom Gundersen
1fa92f9091 pipeline: distinguish between optional and mandatory properties
The ideal is that the constructor takes mandatory properties as arguments, and fields in the struct
are all optional.

This clarifies that across the pipelines (or leaves TODOs where work remains), and where possible
makes fields optional by providing a valid default value.
2022-06-27 19:11:26 +01:00
Tom Gundersen
c001af63ec pipeline: further cleanups
This adds more documentation and makes more properties implicitly inherited rather than
repeated. This makes for less boilerplate, and gives us fewer things to keep in sync.
2022-06-27 19:11:26 +01:00
Tom Gundersen
e5ef7933ac pipeline: drop stage_options.go
This was barely used. Making it explicit seemed more readable.
2022-06-27 19:11:26 +01:00
Tom Gundersen
f60092033b pipelines: make OSPipeline own the PartitionTable
The OSPipeline might need to know what disk layout it will be put onto, enforce this by making
the PartitionTable a property of the OSPipeline, and require child pipelines to query it when needed.
2022-06-27 19:11:26 +01:00
Tom Gundersen
ae34513d18 pipeline: move package to top level
The pipeline package is exists conceptually between the distro and the osbuild packages, so move
it to the top level rather than as a child of distro.

No functional change.
2022-06-27 19:11:26 +01:00
Tom Gundersen
e024ceb138 pipelines: cleanups
This is a collection of minor cleanups:
 - Start documenting the API
 - Enforce dependent pipelines have the correct type where necessary
 - Use data from dependent pipelines where possible
 - Start enforcing required fields
 - Move logic into the pipeline implementation where we can
2022-06-27 19:11:26 +01:00
Tom Gundersen
040a4ef6a1 pipelines: add Fedora IoT installer pipelines 2022-06-27 19:11:26 +01:00
Tom Gundersen
284eb9af59 distro/fedora: drop dead code 2022-06-27 19:11:26 +01:00
Tom Gundersen
0f015801d7 pipelines: add Fedora IoT commit and container pipelines 2022-06-27 19:11:26 +01:00
Tom Gundersen
1243f84cb0 pipelines: add fedora liveimg and qemu pipelines 2022-06-27 19:11:26 +01:00
Tom Gundersen
562a5b1127 pipelines: add fedora OS pipeline
This pulls out the OS pipeline, without changing the parameters. The dependency
between the OS pipeline and build pipeline, is now explicit, rather than by name.
2022-06-27 19:11:26 +01:00
Tom Gundersen
b6d6626a5d pipelines: introduce pipeline abstractions
Create a new package to encapsulate pipelines. This introduces only the build
pipeline and uses it in fedora.

No functional change.
2022-06-27 19:11:26 +01:00
Tom Gundersen
b8815aab4c distro/fedora: avoid double import
Fix typo. No functional chnage.
2022-06-27 19:11:26 +01:00
Ygal Blum
534625fa38 Fedora - Use vendor instead of rpm dependencies
RPM Spec
--------
Remove all Go dependecies
Add Start and End marker comments for bundling information
Add '-k' to goprep to preserve the vendor directory

tools
-----
Add script to update the RPM spec file to generate the indication lines
based on vendor/modules.txt

Packit
------
Run the new script as a post-upstream-clone hook

Makefile
--------
Run the new script on the generated spec file before generating the RPM

mockbuild.sh
------------
Run the new script before creating the RPM
2022-06-23 18:02:09 +02:00
Jakub Rusz
8fef39c4cf rpmbuild: add fedora-36 2022-06-23 12:30:20 +02:00
Achilleas Koutsou
365e1d109c distro/rhel7: remove vhd image type
1. Broken: specifies packages that aren't available in the repositories
   (langpacks-en and dhcp-client).
2. Not needed.
2022-06-23 10:12:35 +02:00
Christian Kellner
292ffd59a0 cmd/osbuild-upload-gcp: fix typo in skip-import's help string 2022-06-22 12:37:34 +02:00
Jakub Rusz
0201b0bff9 tests: Update compose version in Upgrade test 2022-06-21 10:40:58 +02:00
Jakub Rusz
2eee0632d6 manifests: regenerate rhel-91-gce-boot
We don't want to do gpg check for the google repositories.
2022-06-21 10:40:58 +02:00
Jakub Rusz
70688173bb tests: update RHEL version in unsupported tests 2022-06-21 10:40:58 +02:00
Jakub Rusz
81c66b5128 CI: move centos out of RHEL runners
We have this lits of separate RHEL runners only to build test rpms
during nightly pipelines.
2022-06-21 10:40:58 +02:00
Jakub Rusz
eb78fcd85e tests/koji: build latest RHEL when running on RHEL
Based on
aff680542d
this test is supposed to always build the latest available compose.
2022-06-21 10:40:58 +02:00
Jakub Rusz
5fcf0b1d53 tests/ostree: Update to support new RHEL nightlies 2022-06-21 10:40:58 +02:00
Jakub Rusz
1829ff24ea Update terraform SHA 2022-06-21 10:40:58 +02:00
Jakub Rusz
d1c6b46ac3 Schutzfile: update snapshots and osbuild commits
New osbuild commmit SHA and newer rpmrepo snapshots
osbuild commit: stages/mkfs.fat: pass -I command line option
2022-06-21 10:40:58 +02:00
Jakub Rusz
d87c60857c mockbuild: update mockbuild script after RHEL-9 GA
Removing rhel-9 templates as they are shipped upstream already and
making necessary modifications to mockbuild.sh
2022-06-21 10:40:58 +02:00
Alexander Todorov
0fd77f4594 Update osbuild commit 2022-06-21 10:40:58 +02:00
Alexander Todorov
0d3ef8817a Update repositories in repo-*.json files 2022-06-21 10:40:58 +02:00
Alexander Todorov
b82895b52a Define 8.7 & 9.1 nightly repos in Schutzfile 2022-06-21 10:40:58 +02:00
Alexander Todorov
8c03162867 COMPOSER-1576: Switch to latest RHEL versions for testing 2022-06-21 10:40:58 +02:00
Achilleas Koutsou
e8d6dee9ac CI: run the new test script on gitlab
We only need one runner and it should use the internal network for
access to all repositories.

Set a rule so it doesn't run on 'main' (makes no sense).

Set git depth to 500:
We need a long history in order to find the merge-base between the PR
and 'main'.  It's unclear whether there's a straightforward way to find
the depth of the PR to limit the clone depth accurately.  500 should be
enough for any PR (I'd hate to see a PR that makes this statement
false).
2022-06-21 09:50:45 +02:00
Achilleas Koutsou
7076f9b8dc test: new script that detects changes in manifests in a PR
The script runs the gen-manifests command first on the PR head and then
on the merge-base with the PR's base branch (typically 'main') and
checks for any differences.  It creates a review comment on the PR on
GitHub if any changes are detected.

The message is posted as a simple COMMENT type review to inform the
author and reviewers that changes exist.

The script doesn't fail if there's a diff.  CI shouldn't fail if changes
are detected since they can be intentional.  The job fails if something
goes wrong with the script execution (manifest generation, comment
posting, etc).

The script exits immediately if not run from a PR.

The gen-manifests run is silenced with `> /dev/null`.  In the future,
this should be handled by flags to the command itself to control the
output format noisiness.

The gen-manifests command is run 50 workers.  Testing with 100 seemed to
make the execution stall, likely because of the resources on the worker.
We can experiment with this value more in the future.
2022-06-21 09:50:45 +02:00
Christian Kellner
c38fcb128c distro/rhel7: add support for azure images
Add support for building Azure images, including RHUI based ones for
the private market place.
2022-06-21 00:06:26 +02:00
Christian Kellner
06e05df620 distro: add support for building a rhel 7 (qcow2)
Based on the RHEL 8.6 pipelines, needs a special buildroot with two
extra packages: python3-iniparse and python3-PyYAML.
Only x86_64 support for now.
2022-06-21 00:06:26 +02:00