Commit graph

203 commits

Author SHA1 Message Date
Sanne Raymaekers
aa156028b5 templates/packer: make worker_config script executable 2024-03-06 20:38:36 +01:00
Tomáš Hozza
f02fe86366 Packer: make osbuild-executor worker config more configurable
We need to use custom IAM policy name used by the worker for
osbuild-executor on Fedora workers (in prod vs. stage). And we have the
same requirement for the CloudWatch log group used by the
osbuild-executor.

Modify the Ansible playbook used by Packer to use the values from
/tmp/cloud_init_vars if set and defaulting to the current values if not
set.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-03-06 12:02:56 +01:00
Sanne Raymaekers
9004ca6310 Revert "packer: pin the vector version"
This reverts commit 5315264f2e.

This got fixed in 0.22.1.
2024-03-05 19:10:28 +01:00
Sanne Raymaekers
cad29f93a7 templates/packer: migrate vector to new repos
See https://vector.dev/highlights/2023-11-07-new-linux-repos/ for more
information.
2024-03-05 19:10:28 +01:00
Tomáš Hozza
e561ba0854 templates/composer: set DISTRO_ALIASES for composer
Set the RHEL release names without the minor version to point to the
latest GA release. Set the 'rhel-10.0' to the latest RHEL-9 minor
release in development, so that one can start building RHEL-10 images
without referencing RHEL-9.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-02-21 12:06:33 +01:00
Sanne Raymaekers
1abd887c23 templates/packer: rename executor log group
In app-interface the output resource names need to be unique, and the
log group name is already shared with the role.
2024-02-16 11:46:03 +01:00
Sanne Raymaekers
3400c5c20d templates/packer: setup vector in osbuild-executor 2024-02-15 12:05:17 +01:00
Sanne Raymaekers
a9010479e3 templates/packer: set -builder-path to /var/cache/osbuild-builder
The builder uses `/run/osbuild` as a default path for this argument. Yet
this directory doesn't exist when the builder writes the manifest. But
osbuild should own this directory, not the builder.

Furthermore `/run` is a tmpfs, so the executor might run into memory
issues if we use `/run` as the store and output directory (on the "host"
workers these are in `/var/cache`).

While `/tmp` might seem like a good candidate on RHEL, it's a tmpfs on
Fedora, so it's also to be avoided.
2024-02-15 01:02:10 +01:00
Sanne Raymaekers
04ce056b6a templates/packer: let the executor listen on all interfaces 2024-02-14 22:55:54 +01:00
Sanne Raymaekers
f03dac5834 templates/packer: don't subscribe fedora executor 2024-02-14 13:45:52 +01:00
Sanne Raymaekers
d5fd1bbbf0 templates/packer: add worker-executor service to image 2024-02-14 13:45:52 +01:00
Sanne Raymaekers
a9066ee793 templates/packer: remove worker_builder script from init service
This script doesn't actually exist.
2024-02-14 13:45:52 +01:00
Sanne Raymaekers
d0caac9d69 templates/packer: use aws.ec2 osbuild executor 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
4111ca4e0e templates/packer: add worker-executor service 2024-02-14 09:54:11 +01:00
Diaa Sami
c9c51613a4 composer: glitchtip integration 2024-02-13 14:57:57 +01:00
Sanne Raymaekers
44426bb48f templates/dashboards: add community stage service to orgs 2024-02-05 11:38:53 +01:00
Jakub Rusz
db0e6c9643 Packer: change fedora-38 aarch64 ami
This ami is currently broken, switch to a slightly older one.
2024-01-31 10:11:50 +01:00
Sanne Raymaekers
e289b763e7 templates/packer: deal with unbound variables
Don't allow unbound variables, but for the variables that are used to
determine whether or not that part of the setup should continue, default
to empty/undefined.
2024-01-30 21:41:31 +01:00
Diaa Sami
6cfa26399f composer: use logrus hook instead of k8s sidecar
for splunk log forwarding
Fixes COMPOSER-2051
2023-11-28 12:42:00 +01:00
Gianluca Zuccarelli
3fe36d0012 templates/packer: configure pulp creds on startup 2023-11-07 10:48:00 +01:00
Sanne Raymaekers
bf3ff40a65 dashboards: drop interval from composer dashboard and fix slo
The latency budget remaining used $__range instead of the 28d constant.
2023-10-03 11:48:37 +02:00
Sanne Raymaekers
f05a5b59f3 dashboards: drop API section from worker job stats dashboard
Renames the worker dashboard to worker job stats dashboard.

Drops the interval variable and relies solely on $__range and
$__rate_interval.
2023-10-03 11:48:37 +02:00
Sanne Raymaekers
1475e216d2 dashboards: add worker api dashboard
Also this one is made without a separate interval variable, instead
relying on $__rate_interval and $__interval.
2023-10-03 11:48:37 +02:00
Sanne Raymaekers
33f9a6726e dashboards: fix composer dash request rate errors 2023-10-02 18:50:37 +02:00
Sanne Raymaekers
9d7159dab3 templates/packer: retry subscribtion 2023-09-25 11:56:42 +02:00
Sanne Raymaekers
0dc1a01077 templates/packer: configure oracle cloud credentials on startup 2023-09-22 09:55:48 +02:00
Sanne Raymaekers
3a9bcded32 templates/composer: fix cpu request/limits
The fluentd sidecar had the same request/limit as the service container,
and the migrate init-container had the fluentd request/limit. It should
be the other way round.
2023-09-21 12:41:06 +02:00
Sanne Raymaekers
5bb9d414a2 templates/compose: add startingDeadlineSeconds to maintenance job
The job won't run if it doesn't get scheduled within 30 minutes. This
prevents the job running multiple times in a row if it didn't get
scheduled, for instance due to resource limits.
2023-09-21 12:41:06 +02:00
Sanne Raymaekers
e0b2455acf templates/composer: parameterise maintenance job cpu req/limit 2023-09-21 11:11:35 +02:00
Sanne Raymaekers
715bdba1bf dashboards/worker: default to showing the past 6 hours
The worker dashboards contains slow queries, running these on 28 days of
data take a very long time (and they often time out).
2023-08-24 17:01:23 +02:00
Ondřej Budai
ba417dbf3d packer: use gp3 volumes
GP3 is cheaper than GP2, let's switch to it for storing our images:
https://fedoraproject.org/wiki/Changes/CloudEC2gp3

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-07-21 12:20:47 +02:00
Ondřej Budai
b461e403ef packer: move Fedora to 38
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-07-21 12:20:47 +02:00
Sanne Raymaekers
a2c07ea83a templates/dashboards: rework composer dashboard
splits the board into 3 sections:
- SLO
- API throughput
- API latency

It's also possible to filter by tenant. And some colours were adjusted
to improve readability.
2023-06-30 11:06:51 +02:00
Sanne Raymaekers
170feba87b templates/packer: use RH SSO for the default token endpoint
MAS SSO (identity.api.openshift.com) was deprecated, RH SSO should be
the default.
2023-06-29 11:32:44 +02:00
Sanne Raymaekers
38093100e3 templates/composer: No longer accept MAS SSO 2023-06-29 11:32:44 +02:00
Diaa Sami
8398f27742 internal/cloudapi: additional prometheus listener
Listening on another port, while keeping the existing endpoint until
transition is complete
2023-06-07 17:05:32 +02:00
Sanne Raymaekers
53198bed6e templates/composer: fix fluentd requests/limits
No separate request for memory was defined in #3472, only cpu
request/limit.
2023-06-05 16:16:18 +02:00
Sanne Raymaekers
3faab2f102 templates/composer: add separate CPU request/limit for sidecar 2023-06-05 11:51:36 +02:00
Sanne Raymaekers
0ddbee11cd templates/composer: parametrise replicas 2023-06-05 11:51:36 +02:00
Ondřej Budai
dce2ced50b packer: bump the amazon plugin to 1.2.3
Since the previous commit removed the associate_public_ip_address, we should
not be hitting the new behaviour introduced in 1.2.3, thus everything will
hopefully work as before.
2023-05-05 11:07:05 +02:00
Ondřej Budai
a2a5618149 packer: remove associate_public_ip_address
The documentation for this option says the following:

> If using a non-default VPC, public IP addresses are not provided by default.
> If this is true, your new instance will get a Public IP. default: unset

We don't specify a VPC in the packer build, thus we are using the default
one. Therefore, I don't think we actually need this option as it's useful
only for non-default VPCs.

See
https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs#run-configuration

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-05-05 11:07:05 +02:00
Ondřej Budai
edf4f7e879 packer: pin the version of the amazon plugin to 1.2.2
Version 1.2.3 made changes to how the plugin handles auto-selection of a
subnet when it's not specified, see

f1ec287c77

Sadly, the new algorithm selects us-east-1e for us that doesn't support
the machine types we use (c6*.large) which causes the build to fail.
I reported it here:
https://github.com/hashicorp/packer-plugin-amazon/issues/368

One workaround might be to pin a working subnet, but that's apparently also
broken in 1.2.3, see
https://github.com/hashicorp/packer-plugin-amazon/issues/367

Therefore, I decided to pin the plugin to 1.2.2 for now, and see what's
the recommended approach from terraform guys.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-20 13:02:34 +02:00
Sanne Raymaekers
a2a3a2602c templates/dashboards/worker: add arch label to job wait duration
Display the wait duration of jobs per architecture.
2023-03-21 12:34:09 +01:00
Sanne Raymaekers
b13865d361 templates/dashboards/worker: edit thresholds
95th percentile duration is now a fixed colour, as it's tricky to get
dynamic thresholds based on the job type.

Budget remaining thresholds are now only green at infinity, turn yellow
below 4 weeks, and turn red when budget consumption would only last 3
weeks (out of 4).
2023-03-21 12:34:09 +01:00
Sanne Raymaekers
63d5132aa6 templates/dashboards/worker: change panel alignment
This aligns vertical dividers between panels across rows.
2023-03-21 12:34:09 +01:00
Sanne Raymaekers
865bb98034 templates/dashboards/worker: bump version 2023-03-21 12:34:09 +01:00
Sanne Raymaekers
5a9f8d3457 templates/dashboards/worker: show request throughput per path 2023-03-21 12:34:09 +01:00
Sanne Raymaekers
26a521f54d templates/dashboards/worker: use jobtype variable for job stats
This removes the rows of panels per job type, and uses the jobtype
variable.
2023-03-21 12:34:09 +01:00
Sanne Raymaekers
5d2f84cb9e templates/dashboards/worker: add target duration 2023-03-21 12:34:09 +01:00
Sanne Raymaekers
0b7e94b097 templates/dashboard/worker: add job type variable 2023-03-21 12:34:09 +01:00