Commit graph

3275 commits

Author SHA1 Message Date
Roy Golan
439699fcec Add oracle-oci.md under image-types/rhel8 2022-02-10 10:58:46 +01:00
Sanne Raymaekers
b3796a7319 tools/appsre-ansible: Explicitly specific mock config 2022-02-09 18:33:14 +01:00
Sanne Raymaekers
a173a3513d tools/appsre-build-worker-packer: Run on subscribed 8.5 machine 2022-02-09 16:54:22 +01:00
Juan Abia
3699c52b59 test: update govc env variables documentation
govc cli no longer uses GOVMOMI_* enviroment variables. As the govc
documentation indicates, this variables should be GOVC_*.

https://github.com/vmware/govmomi/blob/master/govc/README.md#usage

[skip-ci]
2022-02-09 16:00:37 +01:00
Tomas Hozza
f43d9f4744 Add missing F34 and F35 image test cases
Add F34 and F35 image test cases for all remaining image types, which
were previously not tested. With this PR, image test cases are now
generated for all image types on all architectures as supported by the
Fedora distro definition.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-09 15:29:33 +01:00
Gianluca Zuccarelli
e8d7519c7d templates/dashboard: worker metric queries
The prometheus queries have been updated with
the correct namepsace for the job metrics
Additionally, this commit fixes some of the
queries to add fallback values when the
query results are returned empty.
2022-02-09 14:09:50 +01:00
Ondřej Budai
b1a40c1040 distro: add an alias for RHEL 8.7
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-08 17:37:36 +01:00
Gianluca Zuccarelli
80f24dbd61 metrics: change job metrics namespace
Currently the job metrics are namespaced with the composer
subsystem, i.e. `composer_worker`. Since we plan to split
the components to their own namespaces in app interface,
the worker subsystem should be split too.
2022-02-08 15:57:12 +01:00
Sanne Raymaekers
a739151c71 Revert "templates: Add dnf-json template"
This reverts commit 8cb3900dd6.
2022-02-08 14:05:48 +01:00
Jakub Rusz
a4c19eee85 Schutzfile: add RPMREPO to repo titles
This will make it easier to distinguish nightly rpmrepo snapshots from
actual nightly repositories in the test logs.
2022-02-08 14:28:06 +02:00
Jakub Rusz
29a8810d6e tests/ci: fix sonarqube analysis for main branch
Only use the branch options if it's actually running in a branch and
analyse main without them, sonarqube otherwise doesn't recognize the
main branch.
2022-02-08 14:28:06 +02:00
Tomas Hozza
1401a7a659 test: add image test cases for Fedora 34 and 35
Delete Fedora 33 image test cases.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Tomas Hozza
d9d5a6c1a6 Schutzfile: don't pin osbuild on Fedora and drop F33
Drop unnecessary pinning of osbuild release on Fedora. The latest version
available in all supported Fedora releases should be good enough for
osbuild-composer testing.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Tomas Hozza
617572ad6c test/api.sh: remove parts related to Fedora 33
We no longer test Cloud API on Fedora and Fedora 33 is EOL anyway.
Remove all Fedora 33 related lines from the test case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Tomas Hozza
2c04329645 distro: drop support for Fedora 33
Fedora 33 is already EOL, therefore there is no point in supporting
image builds for it. Drop F33 from the distroregistry list and remove
F33 repositories definition.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-08 10:19:52 +01:00
Ondřej Budai
f66740fdbd github: fix gitlab trigger
With the quoting, we are passing an empty variable instead of no argument.
Also, if the skip is required, we are passing "-o ci.skip" instead of
"-o" "ci.skip".

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-08 10:03:59 +01:00
Roy Golan
e8dd3654d5 Allow skipping CI for draft PR or WIP label
Save resources and needless CI runs by skipping the trigger-gitlab job if a PR is 'draft' (can be set in the pull request page at any time) or by setting a label "WIP".

Asside from saving resources, WIP PRs that don't trigger CI are a good message to a contributers to keep testing it locally and to only get the project resources after serious testing been made on their side.
2022-02-08 08:24:23 +01:00
Sanne Raymaekers
4956e48a0b service-maintenance: Skip db cleanup
Let's enable the cloud cleanup first, and then move on to the db.
2022-02-07 20:42:45 +01:00
Gianluca Zuccarelli
dbf396db2b templates/dashboards: worker error metrics
Update the grafana dashboard for the workers
to show information on the success rate for
osbuild and depsolve jobs.
2022-02-07 20:40:37 +01:00
Tomas Hozza
91097fa007 test: regenerate RHEL 8.5 and 8.6 EC2 image test cases
Fix missing missing `image-info` report updates after PR#2235.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-07 20:33:54 +01:00
dependabot[bot]
c7cba177d9 build(deps): bump cloud.google.com/go/cloudbuild from 0.2.0 to 1.0.0
Bumps [cloud.google.com/go/cloudbuild](https://github.com/googleapis/google-cloud-go) from 0.2.0 to 1.0.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/v0.2.0...dlp/v1.0.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/cloudbuild
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 19:08:06 +01:00
Juan Abia
cf72a8963e test: update README with cloud cleaner info
update testing documentation with the purpose of 'cloud cleaner' and
'scheduled cloud cleaner'
2022-02-07 10:51:47 +01:00
Jakub Rusz
0d64331ffe tests/ci: run all integration tests on fedora and centos 2022-02-07 11:37:13 +02:00
Jakub Rusz
409eb3f33b tests/cross-distro: filter out beta and stream repos
Currently these repositories are not recognized by composer which is a
bug described in issue #2257. This commit will be reverted when related
issue is fixed.
2022-02-07 11:37:13 +02:00
Jakub Rusz
17f84ad6cc tests: expand cross-distro tests
This test now checks if all the installed repositories are recognized,
there are no extra or missing and that blueprint push with unsupported
distro fails gracefuly.
2022-02-07 11:37:13 +02:00
Tomas Hozza
eb36b86161 RHEL-90: Remove deprecated crashkernel=auto option
Setting of the `crashkernel` option to the appropriate value is now done
by the `kexec-tools` package when installed and when any new kernel is
installed.

Regenerate relevant image test cases.

Fix #1819
Fix rhbz#2006692

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-07 09:29:07 +01:00
Sanne Raymaekers
8cb3900dd6 templates: Add dnf-json template 2022-02-06 14:48:32 +00:00
Gianluca Zuccarelli
a8f20811f6 worker: add koji build worker validation
Validate the results for a koji build job
and check for empty osbuild output. This commit
ensures that there aren't any gaps in the error
handling. Additionally this commit adds a few
extra error cases rather than a generic error.
2022-02-05 00:13:59 +00:00
Gianluca Zuccarelli
3f8bd5da67 worker: check for empty osbuild job output
Add an additional check to ensure empty
osbuild output is marked as an error.
Add an error case for failed parsing of
dynamic args.
2022-02-05 00:13:59 +00:00
Alexander Todorov
0fb96345f2 tests: Disable SonarQube for nightly pipelines 2022-02-04 23:43:43 +01:00
Alexander Todorov
966692be11 tests: Conditionally enable osbuild-dnf-json-tests
based on package version
2022-02-04 23:43:43 +01:00
Alexander Todorov
8b3a1367be tests: Add nvr checking into a shared library
can be used for conditionally enabling parts of the test scripts:

source /usr/libexec/tests/osbuild-composer/shared_lib.sh

if nvrGreaterOrEqual "osbuild-composer" "41"; then
    .... do some more testing ...
fi
2022-02-04 23:43:43 +01:00
Alexander Todorov
c5d62909e8 tests: Don't run cloud API tests for nightly trees
https://coreos.slack.com/archives/C0235DZB0DT/p1643969527395979?thread_ts=1643968867.786629&cid=C0235DZB0DT

Note: koji.sh already checks the $NIGHTLY variable and just exits
2022-02-04 23:43:43 +01:00
Alexander Todorov
b31f61649a ci: Overwrite REPO_PATH based on the value of $NIGHTLY
Override only for osbuild-composer, not other projects

will produce different repositories/rpms for the same commit when
executed from a PR and from the scheduled pipelines. We need the
distinction b/c osbuild-composer-tests RPMs end up slightly different in
these two cases. For more info see:
https://github.com/osbuild/osbuild-composer/pull/2093#pullrequestreview-854328838
https://github.com/osbuild/osbuild-composer/pull/2093#issuecomment-1014469915
2022-02-04 23:43:43 +01:00
Alexander Todorov
43738d6caa tests: Fix for unbound variable 2022-02-04 23:43:43 +01:00
Alexander Todorov
8a46796cf7 ci: Don't account for SCHUTZ_OSBUILD variable
if you want to use a custom osbuild.rpm you can supply that in REPO_URL,
no?

Note: this removal of .repo files breaks building the -tests RPM from
source in downstream branches/nightlies b/c the repo is defined in
osbuild-composer.repo
2022-02-04 23:43:43 +01:00
Alexander Todorov
e07bd4bebf Don't override osbuild-composer-tests-multi-arch repo if not needed
drop the discovery mechanism from Brew b/c we're building the tests RPM
from main but overwrite the repository if REPO_URL is configured in ENV.
2022-02-04 23:43:43 +01:00
Alexander Todorov
63bca237cf ci: Check that installed osbuild-composer RPM matches SUT 2022-02-04 23:43:43 +01:00
Alexander Todorov
570e5f595f Relax the requirements between osbuild-composer-tests and osbuild-composer
this is only useful for our RHEL nightly pipeline testing b/c we intend
to be building the -tests RPM from the main branch.
2022-02-04 23:43:43 +01:00
Alexander Todorov
3d86af796a Prepare CI to build -tests RPM for downstream testing from source 2022-02-04 23:43:43 +01:00
Gianluca Zuccarelli
1e443cf0fa worker: fix error status codes
The DNFDepsolveError and DNFMarking error should have
a `4xx` code instead of a `5xx` error code.
2022-02-04 19:30:25 +01:00
Gianluca Zuccarelli
290472dfdf metrics: add worker error metrics
This commit introduces the collection of error
metrics since it is now possible to differentiate
between internal errors and user input errors.
Additionally, the error status is reported for
job duration metrics.
2022-02-03 23:40:42 +00:00
Gianluca Zuccarelli
6c4caec022 metrics: move metrics to worker server
For simplicity, the collection of the job metrics
was carried out in the job queue. This was only
being done in the dbqueue and not in the fsqueue.
This pr refactors the metric collection and moves
the job metrics to the worker server, by adding a
wrapper function to enqueueing jobs so that the
metrics only have to be recorded in one place when
queueing a job.
2022-02-03 23:40:42 +00:00
Gianluca Zuccarelli
bce12b7bea metrics: extract metric collection
Refactor the current metric collection to make use
of re-usable functions, since some of the same queries
are repeated. This will also make it easier to move
the collection of metrics from the job queue.
2022-02-03 23:40:42 +00:00
Juan Abia
2c8b12febd gitlab-ci: make every stage interruptible
scheduled cloud cleaner is now fully funcitoning. This means we can
auto-cancel whole pipelines on new commits.
2022-02-03 22:58:27 +01:00
Tomas Hozza
07a5745875 internal/cloud/gcp: use pkg.go.dev/cloud.google.com/go for Compute Engine
The internal GCP package used `pkg.go.dev/google.golang.org/api` [1] to
interact with Compute Engine API. Modify the package to use the new and
idiomatic `pkg.go.dev/cloud.google.com/go` [2] library for interacting
with the Compute Engine API. The new library have been already used to
interact with the Cloudbuild and Storage APIs. The new library was not
used for Compute Engine since the beginning, because at that time, it
didn't support Compute Engine.

Update go.mod and vendored packages.

[1] https://github.com/googleapis/google-api-go-client
[2] https://github.com/googleapis/google-cloud-go

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-03 15:35:28 +01:00
Christian Kellner
9302befa67 distro/rhel90: disable pw auth for EC2 images
Disable loging in via password authentication since this is an
official Amazon marketplace requirement

  Linux-based AMIs must not allow SSH password authentication.
  Disable password authentication via your sshd_config file by
  setting PasswordAuthentication to NO.

  Section "Security policies" from
  https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
2022-02-03 14:57:16 +01:00
Christian Kellner
90cebf0337 distro/rhel86: disable pw auth for EC2 images
Disable loging in via password authentication since this is an
official Amazon marketplace requirement

  Linux-based AMIs must not allow SSH password authentication.
  Disable password authentication via your sshd_config file by
  setting PasswordAuthentication to NO.

  Section "Security policies" from
  https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
2022-02-03 14:57:16 +01:00
Christian Kellner
8aede24057 distro/rhel85: disable pw auth for EC2 images
Disable loging in via password authentication since this is an
official Amazon marketplace requirement

  Linux-based AMIs must not allow SSH password authentication.
  Disable password authentication via your sshd_config file by
  setting PasswordAuthentication to NO.

  Section "Security policies" from
  https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
2022-02-03 14:57:16 +01:00
Christian Kellner
6f132181d9 distro: add sshd config to ImageConfig
Ability to specify the sshd configuration via the ImageConfig
data structure.
2022-02-03 14:57:16 +01:00