Commit graph

3266 commits

Author SHA1 Message Date
Jakub Rusz
a4c19eee85 Schutzfile: add RPMREPO to repo titles
This will make it easier to distinguish nightly rpmrepo snapshots from
actual nightly repositories in the test logs.
2022-02-08 14:28:06 +02:00
Jakub Rusz
29a8810d6e tests/ci: fix sonarqube analysis for main branch
Only use the branch options if it's actually running in a branch and
analyse main without them, sonarqube otherwise doesn't recognize the
main branch.
2022-02-08 14:28:06 +02:00
Tomas Hozza
1401a7a659 test: add image test cases for Fedora 34 and 35
Delete Fedora 33 image test cases.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Tomas Hozza
d9d5a6c1a6 Schutzfile: don't pin osbuild on Fedora and drop F33
Drop unnecessary pinning of osbuild release on Fedora. The latest version
available in all supported Fedora releases should be good enough for
osbuild-composer testing.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Tomas Hozza
617572ad6c test/api.sh: remove parts related to Fedora 33
We no longer test Cloud API on Fedora and Fedora 33 is EOL anyway.
Remove all Fedora 33 related lines from the test case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Tomas Hozza
2c04329645 distro: drop support for Fedora 33
Fedora 33 is already EOL, therefore there is no point in supporting
image builds for it. Drop F33 from the distroregistry list and remove
F33 repositories definition.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Ondřej Budai
f66740fdbd github: fix gitlab trigger
With the quoting, we are passing an empty variable instead of no argument.
Also, if the skip is required, we are passing "-o ci.skip" instead of
"-o" "ci.skip".

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-08 10:03:59 +01:00
Roy Golan
e8dd3654d5 Allow skipping CI for draft PR or WIP label
Save resources and needless CI runs by skipping the trigger-gitlab job if a PR is 'draft' (can be set in the pull request page at any time) or by setting a label "WIP".

Asside from saving resources, WIP PRs that don't trigger CI are a good message to a contributers to keep testing it locally and to only get the project resources after serious testing been made on their side.
2022-02-08 08:24:23 +01:00
Sanne Raymaekers
4956e48a0b service-maintenance: Skip db cleanup
Let's enable the cloud cleanup first, and then move on to the db.
2022-02-07 20:42:45 +01:00
Gianluca Zuccarelli
dbf396db2b templates/dashboards: worker error metrics
Update the grafana dashboard for the workers
to show information on the success rate for
osbuild and depsolve jobs.
2022-02-07 20:40:37 +01:00
Tomas Hozza
91097fa007 test: regenerate RHEL 8.5 and 8.6 EC2 image test cases
Fix missing missing `image-info` report updates after PR#2235.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-07 20:33:54 +01:00
dependabot[bot]
c7cba177d9 build(deps): bump cloud.google.com/go/cloudbuild from 0.2.0 to 1.0.0
Bumps [cloud.google.com/go/cloudbuild](https://github.com/googleapis/google-cloud-go) from 0.2.0 to 1.0.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/v0.2.0...dlp/v1.0.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/cloudbuild
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 19:08:06 +01:00
Juan Abia
cf72a8963e test: update README with cloud cleaner info
update testing documentation with the purpose of 'cloud cleaner' and
'scheduled cloud cleaner'
2022-02-07 10:51:47 +01:00
Jakub Rusz
0d64331ffe tests/ci: run all integration tests on fedora and centos 2022-02-07 11:37:13 +02:00
Jakub Rusz
409eb3f33b tests/cross-distro: filter out beta and stream repos
Currently these repositories are not recognized by composer which is a
bug described in issue #2257. This commit will be reverted when related
issue is fixed.
2022-02-07 11:37:13 +02:00
Jakub Rusz
17f84ad6cc tests: expand cross-distro tests
This test now checks if all the installed repositories are recognized,
there are no extra or missing and that blueprint push with unsupported
distro fails gracefuly.
2022-02-07 11:37:13 +02:00
Tomas Hozza
eb36b86161 RHEL-90: Remove deprecated crashkernel=auto option
Setting of the `crashkernel` option to the appropriate value is now done
by the `kexec-tools` package when installed and when any new kernel is
installed.

Regenerate relevant image test cases.

Fix #1819
Fix rhbz#2006692

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-07 09:29:07 +01:00
Sanne Raymaekers
8cb3900dd6 templates: Add dnf-json template 2022-02-06 14:48:32 +00:00
Gianluca Zuccarelli
a8f20811f6 worker: add koji build worker validation
Validate the results for a koji build job
and check for empty osbuild output. This commit
ensures that there aren't any gaps in the error
handling. Additionally this commit adds a few
extra error cases rather than a generic error.
2022-02-05 00:13:59 +00:00
Gianluca Zuccarelli
3f8bd5da67 worker: check for empty osbuild job output
Add an additional check to ensure empty
osbuild output is marked as an error.
Add an error case for failed parsing of
dynamic args.
2022-02-05 00:13:59 +00:00
Alexander Todorov
0fb96345f2 tests: Disable SonarQube for nightly pipelines 2022-02-04 23:43:43 +01:00
Alexander Todorov
966692be11 tests: Conditionally enable osbuild-dnf-json-tests
based on package version
2022-02-04 23:43:43 +01:00
Alexander Todorov
8b3a1367be tests: Add nvr checking into a shared library
can be used for conditionally enabling parts of the test scripts:

source /usr/libexec/tests/osbuild-composer/shared_lib.sh

if nvrGreaterOrEqual "osbuild-composer" "41"; then
    .... do some more testing ...
fi
2022-02-04 23:43:43 +01:00
Alexander Todorov
c5d62909e8 tests: Don't run cloud API tests for nightly trees
https://coreos.slack.com/archives/C0235DZB0DT/p1643969527395979?thread_ts=1643968867.786629&cid=C0235DZB0DT

Note: koji.sh already checks the $NIGHTLY variable and just exits
2022-02-04 23:43:43 +01:00
Alexander Todorov
b31f61649a ci: Overwrite REPO_PATH based on the value of $NIGHTLY
Override only for osbuild-composer, not other projects

will produce different repositories/rpms for the same commit when
executed from a PR and from the scheduled pipelines. We need the
distinction b/c osbuild-composer-tests RPMs end up slightly different in
these two cases. For more info see:
https://github.com/osbuild/osbuild-composer/pull/2093#pullrequestreview-854328838
https://github.com/osbuild/osbuild-composer/pull/2093#issuecomment-1014469915
2022-02-04 23:43:43 +01:00
Alexander Todorov
43738d6caa tests: Fix for unbound variable 2022-02-04 23:43:43 +01:00
Alexander Todorov
8a46796cf7 ci: Don't account for SCHUTZ_OSBUILD variable
if you want to use a custom osbuild.rpm you can supply that in REPO_URL,
no?

Note: this removal of .repo files breaks building the -tests RPM from
source in downstream branches/nightlies b/c the repo is defined in
osbuild-composer.repo
2022-02-04 23:43:43 +01:00
Alexander Todorov
e07bd4bebf Don't override osbuild-composer-tests-multi-arch repo if not needed
drop the discovery mechanism from Brew b/c we're building the tests RPM
from main but overwrite the repository if REPO_URL is configured in ENV.
2022-02-04 23:43:43 +01:00
Alexander Todorov
63bca237cf ci: Check that installed osbuild-composer RPM matches SUT 2022-02-04 23:43:43 +01:00
Alexander Todorov
570e5f595f Relax the requirements between osbuild-composer-tests and osbuild-composer
this is only useful for our RHEL nightly pipeline testing b/c we intend
to be building the -tests RPM from the main branch.
2022-02-04 23:43:43 +01:00
Alexander Todorov
3d86af796a Prepare CI to build -tests RPM for downstream testing from source 2022-02-04 23:43:43 +01:00
Gianluca Zuccarelli
1e443cf0fa worker: fix error status codes
The DNFDepsolveError and DNFMarking error should have
a `4xx` code instead of a `5xx` error code.
2022-02-04 19:30:25 +01:00
Gianluca Zuccarelli
290472dfdf metrics: add worker error metrics
This commit introduces the collection of error
metrics since it is now possible to differentiate
between internal errors and user input errors.
Additionally, the error status is reported for
job duration metrics.
2022-02-03 23:40:42 +00:00
Gianluca Zuccarelli
6c4caec022 metrics: move metrics to worker server
For simplicity, the collection of the job metrics
was carried out in the job queue. This was only
being done in the dbqueue and not in the fsqueue.
This pr refactors the metric collection and moves
the job metrics to the worker server, by adding a
wrapper function to enqueueing jobs so that the
metrics only have to be recorded in one place when
queueing a job.
2022-02-03 23:40:42 +00:00
Gianluca Zuccarelli
bce12b7bea metrics: extract metric collection
Refactor the current metric collection to make use
of re-usable functions, since some of the same queries
are repeated. This will also make it easier to move
the collection of metrics from the job queue.
2022-02-03 23:40:42 +00:00
Juan Abia
2c8b12febd gitlab-ci: make every stage interruptible
scheduled cloud cleaner is now fully funcitoning. This means we can
auto-cancel whole pipelines on new commits.
2022-02-03 22:58:27 +01:00
Tomas Hozza
07a5745875 internal/cloud/gcp: use pkg.go.dev/cloud.google.com/go for Compute Engine
The internal GCP package used `pkg.go.dev/google.golang.org/api` [1] to
interact with Compute Engine API. Modify the package to use the new and
idiomatic `pkg.go.dev/cloud.google.com/go` [2] library for interacting
with the Compute Engine API. The new library have been already used to
interact with the Cloudbuild and Storage APIs. The new library was not
used for Compute Engine since the beginning, because at that time, it
didn't support Compute Engine.

Update go.mod and vendored packages.

[1] https://github.com/googleapis/google-api-go-client
[2] https://github.com/googleapis/google-cloud-go

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-03 15:35:28 +01:00
Christian Kellner
9302befa67 distro/rhel90: disable pw auth for EC2 images
Disable loging in via password authentication since this is an
official Amazon marketplace requirement

  Linux-based AMIs must not allow SSH password authentication.
  Disable password authentication via your sshd_config file by
  setting PasswordAuthentication to NO.

  Section "Security policies" from
  https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
2022-02-03 14:57:16 +01:00
Christian Kellner
90cebf0337 distro/rhel86: disable pw auth for EC2 images
Disable loging in via password authentication since this is an
official Amazon marketplace requirement

  Linux-based AMIs must not allow SSH password authentication.
  Disable password authentication via your sshd_config file by
  setting PasswordAuthentication to NO.

  Section "Security policies" from
  https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
2022-02-03 14:57:16 +01:00
Christian Kellner
8aede24057 distro/rhel85: disable pw auth for EC2 images
Disable loging in via password authentication since this is an
official Amazon marketplace requirement

  Linux-based AMIs must not allow SSH password authentication.
  Disable password authentication via your sshd_config file by
  setting PasswordAuthentication to NO.

  Section "Security policies" from
  https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
2022-02-03 14:57:16 +01:00
Christian Kellner
6f132181d9 distro: add sshd config to ImageConfig
Ability to specify the sshd configuration via the ImageConfig
data structure.
2022-02-03 14:57:16 +01:00
Tomas Hozza
b9efe82bd7 distro/fedora: implementation cleanups for newer releases
Clean up some implementation aspects of the Fedora distro definition:
 - Do not have default Fedora distro version and use `fedora` as the
   package name in all places that use it, instead of `fedora33`.
 - Fix bugs when wrong (Fedora 33) values were returned by `OSTreeRef()`
   and `Releasever()` for newer Fedora releases.
 - Test Fedora 35 in package unit tests.
 - Add unit test for `OSTreeRef()` method.
 - Use architecture name constants from `distro` package, instead of
   string literals.

Fix #1802

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-03 14:51:07 +01:00
Tomas Hozza
8b8c7bbbbe Fix FS label of the ESP in Fedora distro definition
The QEMU assembler in Fedora distro definition for UEFI systems used
longer than allowed label for the VFAT filesystem of the EFI System
Partition. The maximum allowed label length is 11 characters.

This worked before with dosfstools, but in 2018, they added a label
validation [1]. This change got into the v4.2 release of dosfstools,
released in Jan 2021. And subsequently since F34, this new version of
dosfstools is present in Fedora repositories.

[1] ca54953476

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-03 14:51:07 +01:00
sanne
08f668200a .gitlabci.yml: Add fedora-35 testing
Enable F34 testing on AWS as there is nothing blocking it. F34 is not
yet supported on `rhos-01` as there is no runner definition.

Remove F33 repositories for testing and add repo definitions for F34 and
F35.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-03 14:51:07 +01:00
Pavel Raiskup
00eda75691 packit: re-enable builds for s390x architecture
Fedora Copr has native s390x builders now:
https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/message/AR3ZDKET3EXZHV3MSU3UHMO7EIKBGAN2/

Closes: #2246
Signed-off-by: Pavel Raiskup <praiskup@redhat.com>
2022-02-03 12:10:48 +01:00
Sanne Raymaekers
b53ee4135c schutzbot/terraform: New 86 runners 2022-02-03 00:24:59 +01:00
Sanne Raymaekers
1a6f6fdc4b test: Drop ANSIBLE_STDOUT_CALLBACK
There's conflicting ansible versions in the 86 nightlies and epel. There
should be a correct combination of plugins which fixes the callback on
86. But let's drop it to unblock for now.
2022-02-03 00:24:59 +01:00
Achilleas Koutsou
93c0485531 tools/provision: remove json_query usage from ansible playbooks
`json_query` requires python3-jmespath which, while available in the
repos, it can sometimes cause issues when the ansible interpreter is
different from the system interpreter.
The `json_query` is only used in a handful of locations that can easily
be served by `jq`, which we use in other places already.
2022-02-03 00:24:59 +01:00
Sanne Raymaekers
5617a0744a schutzfile: Snapshot updates
For rhel9 the CRBs snapshots ran successfully, appstream is still
broken.

Also update osbuild to mitigated el9 selinux issues.
2022-02-03 00:24:59 +01:00
sanne
fe00e1efd3 containers/osbuild-composer: Allow dnf-json to accept http connections
Revert 83e16afda4: With dnf-json running
in a container it's easy to run it standalone.
2022-02-02 11:15:46 +01:00