Commit graph

2885 commits

Author SHA1 Message Date
Achilleas Koutsou
2f30823b41 test/ostree: install python3 instead of specific version
Better cross-distro compatibility

Same as bdc428fbb7 but for all the tests.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
0b41cd8003 mockbuild: change RHEL 9 template to use latest non-beta repos 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
a025601b85 test/cases: add RHEL 9.0 and CentOS 9 cases to test scripts 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
1dd139e758 CI: enable tests for RHEL 9.0 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
ad896fb69e schutzbot: update terraform sha 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
7e9f1f554e test: update test manifests for rhel-90-ga
Changes from previous (RHEL 9.0 Beta):
- Edge container: Switched to nginx from apache. Change in pipeline and
  package sets.
- All bootable x86_64: Packages for hybrid boot added (efibootmgr).
- Edge commit and edge container no longer require explicitly enabling
  greenboot services.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
cba73ee9e3 tools: add centos-9 to distro-arch-imagetype-map 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
994dd1e136 tools: copy rhel-90 distro-arch-imagetype-map to -beta and -ga 2021-11-10 14:54:31 +01:00
Achilleas Koutsou
6b73dc5a92 distro/rhel90: update to match 8.6 and add centos-9 alias
- Copied distro and image definitions from RHEL 8.6
    - New package set handling
    - Distribution-specific strings
    - New image types: edge-raw and edge-simplified-installer
    - Edge container with nginx
    - Removed greenboot services from edge enabled services (enabled
      automatically)
- Adapted to required changes from RHEL 9.0 Beta
- Added CentOS 9 (centos-9) alias

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
ba4a4541d9 distro/rhel86: remove genisoimage
Not required.
Discovered it's unneeded when it was removed from RHEL 9.0.
Updating RHEL 8.6 to keep clean and consistent.
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
c7cab92e88 distro/rhel86: single osbuild import
osbuild2 was imported twice, once with its own name and once aliased as
plain osbuild.
Use only the plain osbuild alias.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
c8465e3161 test/data: copy rhel-90 test manifests to -ga and -beta
All three "distros" share the same code for now. Adding manifests to
track changes before updating the RHEL 9.0 GA definitions.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
bbf825ef12 test: add repositories for rhel-90-beta and -ga
- Updated repos to use latest nightly (20211105)
- Added rt repository
- Both -ga and -beta are identical
- rhel-90.json repo file is a symlink to rhel-90-beta.json

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
cad5c3f13f distroregistry: add rhel-90-ga to registry
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
Achilleas Koutsou
d290ff5923 distro: copy rhel90beta to rhel90
- Start of RHEL 9.0 GA definition
- Initial distro name: rhel-90-ga
- rhel-90 alias remains for rhel-90-beta for now

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-11-10 14:54:31 +01:00
sanne
4577ac0717 composer: Add worker openapi spec endpoint to auth excludes
Also remove metrics from the excludes for the composer api. This
endpoint would not be called via the gateway so it's not relevant here.
2021-11-10 10:43:43 +01:00
Thomas Lavocat
82ee9661a9 CI: Journal-log is accessible and encrypted 2021-11-10 10:31:33 +01:00
sanne
117886a75f spec: Only run worker preun if systemd is running
Before, when systemd was not present (for instance in a container), the
preun script would error out.

This checks if systemd is present and running in the same way the
systemd rpm macros do.

Fixes #1915
2021-11-10 01:26:00 +00:00
Martin Sehnoutka
03074303cc gitlab-ci: Remove RHEL9.0-beta runners
The RHEL9.0 Beta repos are gone. It will soon be replaced by RHEL9 GA,
so there is no point in running the CI for this distro any more.
2021-11-09 15:32:29 +01:00
Martin Sehnoutka
01acebda64 schutzbot: double quote jq argument to prevent shellcheck failures
The argument contains a variable $i. Shellcheck complains that it must
be quoted.
2021-11-09 15:32:29 +01:00
Gianluca Zuccarelli
cfa8d7a636 distribution: worker dnf-json & cache dir
Since the depsolving has been moved to the worker,
the Dockerfile for the worker needed to have the
dnf-json executable. Additionally there was a missing
cache directory.
2021-11-05 13:34:43 +01:00
Gianluca Zuccarelli
c0ceede084 cloudapi/v2: 5xx error metrics
Currently error metrics are being recorded for
errors with the exact error code of 500. This
commit enables recording metrics for all 5xx errors.
2021-11-04 08:08:32 +01:00
Martin Sehnoutka
708d985194 internal/blueprint: allow filesystem size specified with units
Allow users to specify filesystem size with units such as kB, MB, etc.
2021-11-02 18:32:47 +01:00
Martin Sehnoutka
f84beee04d internal/blueprint: introduce custom fs customization parser
The automatically generated parser can only parse integers into the size
field. Introduce a new one, which will be able to handle strings as
well.
2021-11-02 18:32:47 +01:00
Martin Sehnoutka
f2f078acd6 internal/blueprint: introduce new test for parsing blueprints
Test case like this was completely missing. A similar one is present in
the Weldr package, but this one is specific to testing blueprints and
thus easier to understand and extend.
2021-11-02 18:32:47 +01:00
Martin Sehnoutka
7d6dadb598 internal/common: introduce function to convert data sizes
This function will be used to parse filesystem sizes specified as
string.
2021-11-02 18:32:47 +01:00
Jakub Rusz
2d13fa3a87 prepare_rhel_internal: configure s3cmd explicitly
A similar change was added in 65e429fc4a
so adding it here as well.
2021-11-02 08:43:40 +01:00
Gianluca Zuccarelli
47c41a0b8d templates: add latency metrics to dashboard
Update the grafana dashboard to with metrics
for latency requests, including error budget
burn for compose latency.
2021-11-02 00:23:57 +00:00
Simon Steinbeiss
bb15007f35 Let schutzbot do the post-release version bump
We need a privileged / admin user doing the post-release version bump as
this is a direct commit to main (i.e. without a PR) so switch to using
schutzbot with a scoped personal access token (only public_repo).
2021-11-01 20:27:36 +01:00
Simon Steinbeiss
65a3753d2c Post release version bump
This will soon be done by schutzbot, but is done manually this one time.
2021-11-01 20:27:36 +01:00
Gianluca Zuccarelli
297a60d238 cloudapi: record error metrics
Hook up the compose error metric for
the prometheus alerts and grafana dashboard.
2021-10-29 20:36:18 +01:00
Gianluca Zuccarelli
f8199ec41d prometheus: add middleware function
Add middleware function to track request count
and measure the latency of compose requests.
2021-10-29 20:36:18 +01:00
Gianluca Zuccarelli
dfa6a48f5d prometheus: compose latency metric
Add metric to measure the latency
of requests made to the composer
cloud api.
2021-10-29 20:36:18 +01:00
Chloe Kaubisch
f749078b0d prometheus: update metrics
Change the name of total https requests to be more specific.
Add a new counter for failed compose requests.
2021-10-29 17:09:45 +01:00
Ondřej Budai
01445cfdfb templates: fix liveness/readiness check url
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-29 13:36:16 +02:00
Ondřej Budai
7cf02091d1 templates: add s3 bucket name
Composer API v2 requires a bucket name to be set in composer configuration.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-29 11:50:37 +01:00
Gianluca Zuccarelli
57250f5496 templates: update dashboard config map
Minor fix for a capitalisation of `image-builder`
in the grafana configmap
2021-10-28 22:17:45 +01:00
Gianluca Zuccarelli
22aed692f1 templates: add grafana dashboard
Add initial grafana dashboard with
reporting on compose success rate,
error budget and the number of
total composes.
2021-10-28 21:17:55 +01:00
dependabot[bot]
27b00ed61a build(deps): bump github.com/openshift-online/ocm-sdk-go
Bumps [github.com/openshift-online/ocm-sdk-go](https://github.com/openshift-online/ocm-sdk-go) from 0.1.208 to 0.1.214.
- [Release notes](https://github.com/openshift-online/ocm-sdk-go/releases)
- [Changelog](https://github.com/openshift-online/ocm-sdk-go/blob/master/CHANGES.adoc)
- [Commits](https://github.com/openshift-online/ocm-sdk-go/compare/v0.1.208...v0.1.214)

---
updated-dependencies:
- dependency-name: github.com/openshift-online/ocm-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-28 19:15:16 +01:00
Tom Gundersen
6a671112f0 templates: hook up simple probes and default limits
Use fetching the OpenAPI spec as a simple readiness/liveness, as
there is not much else we can/need to verify.

Set the default CPU and memory limits in accordance with AppSRE
requirements.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2021-10-27 22:51:35 +01:00
Tom Gundersen
b0f36fccd3 templates: add service account
Avoid using the default account, but use a dedicated one.

This follows the guidelines from AppSRE and is what was done for
image-builder.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2021-10-27 22:50:40 +01:00
Simon Steinbeiss
3c9b92b80b Bump version numbers ahead of release
This is necessary for the new simplified release process and is done
ahead of time once for the upcoming release now.
After osbuild-composer 37 this will be done by the GitHub composite action.
2021-10-27 13:03:53 +02:00
Simon Steinbeiss
dcb5220329 Switch to simple upstream releases
This commit changes our release process from the model of having a
release commit (and pull request) which also updated the NEWS.md file
and bumped the versions in the osbuild.spec and setup.py files to simply
pushing a tag.

After the tag (containing the release notes) is pushed, a GitHub
composite action is triggered that creates a GitHub release with the
contents of the git release tag. Furthermore the bumping of the version
number now always has to happen directly after a release to avoid having
to push a(n untested) commit to main for the release and this is also
handled by the GitHub composite action.

Finally packit pushes directly to dist-git now on pushing the release
tag, so no pull-request needs to be reviewed and merged anymore.

Consequently, we also drop the docs/news folder and its content and
adjust the PR template.
2021-10-27 13:03:53 +02:00
Christian Kellner
93e54cd872 distro/rhel86: special case root user for ssh keys
Add a special case for the root user to the work-around for ssh
keys in OSTree commits. As a little refresher: OSTree does not
support having any content in home directories; we therefore
include a first-boot stage in the commit that will create the
ssh keys on first boot. However, until now we did not special
case the root user, which has a separate root directory (/root,
as a symlink to /var/roothome). This patch fixes this.
2021-10-27 00:41:27 +02:00
Tom Gundersen
f44acd0974 Revert "Revert "cloudapi/v1: Move depsolving to workers""
Workers now depsolve in parallel to image builds, so we can
again move depsolivng to the workers. This will help us deal
with increases in traffic as we currently only have one
depsolve handler per pod. It would also avoid any issues with
composer running out of disk space due to dnf metadata caches.

This reverts commit c65b1e9b26.
2021-10-26 13:58:22 +01:00
Tom Gundersen
cfe9f7a87f templates: image-builder-ci access to composer
This should all move to app-interface, as it is configuration, and
we should distinguish between staging and production.

But for now, enable this where it is.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2021-10-26 10:39:50 +02:00
Ondřej Budai
83a35fb9b8 ci: remove 8.5 nightly testing
It no longer makes sense because:

- we don't make any changes to 8.5
- we don't regenerate test manifests for 8.5
- osbuild-composer for 8.5 is in the rhel-8.5.0 branch

Also, the latest-8.5.0 symlink was removed, which broke the CI.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-26 09:30:21 +01:00
Diaa Sami
3b6156a240 composer: add json log formatting
Configurable via config file/env
2021-10-25 13:41:36 +02:00
Achilleas Koutsou
aea77ed3de test: timestamp messages in test scripts
The commonly used 'greenprint' function now adds a date + timestamp to
each message for debugging and tracking the duration of segments of each
scripts.
2021-10-24 16:13:53 +01:00
Achilleas Koutsou
03fe1337a5 Add news item for updated CentOS Stream 8 definitions 2021-10-21 15:16:04 +01:00