This should save us some money. 10 GB is the size of the underlying
RHEL 8.5 AMI so this should be the minimum.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The prometheus queries have been updated with
the correct namepsace for the job metrics
Additionally, this commit fixes some of the
queries to add fallback values when the
query results are returned empty.
Because of the way the gcp secrets are stored for the workers, and how
the mapping from vault to openshift works (unable to map a multiple key
secret into a single json file), there's a bit of juggling required to
get the gcp credentials in the right format.
This directory is not used on worker instances. It was a left-over from the
times when this AMI was also used for running composer.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The worker needs quite a lot of configuration involving secrets. Baking them
in the AMI is just awful so we need to fetch them during the instance startup.
Previously, this was all done using cloud-init. This makes the cloud-init
config huge and it is also very hard to test.
This commit moves all the configuration scripts into the image itself.
Cloud-init still needs to be used to push the secret variables into the
instance. The configuration scripts are run after cloud-init. They pick up
yhe secrets and initialize the worker correctly.
These scripts were adopted from
75b752a1c0
(private repository).
During the adoption, some changes has to be applied to make shellcheck happy.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When backed by a DB, composer has no need of a queue directory.
This also addresses "Error moving artifacts for job" logging noise.
Signed-off-by: sanne <sanne.raymaekers@gmail.com>
Previously, monit and vector RPMs were embedded directly in the
image-builder-packer repository. This was not ideal because hosting big
binary files in git is always ugly.
This commit brings back monit and vector:
- monit is installed from EPEL
- vector is installed from the upstream RPM repository
Ansible was dropped because we don't need it in the image.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We want an absolute path, otherwise packer doesn't know where to find the
playbook if called from a wrong directory.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
osbuild and composer commit SHAs now must be passed into packer using
variables, no defaults are defined. Also, packer is no longer responsible
for naming the AMIs, the name is also passed as a variable.
imagebuilder_packer_sha was dropped entirely as the packer configuration
now lives directly in osbuild-composer repository.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Console support in AWS EC2 is very basic. We now use vector that works much
better than console so we can just drop the forwarding and rely on vector
dumping the logs into cloudwatch.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This is currently not working because workers in aoc no longer use mTLS.
Definitely something we want to fix in the future I think.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
/templates/packer now contains a copy of image-builder-packer repository
as of b8a4b45f93890090de24e3d043e2d958948fc3c5
Changes:
- LICENSE file was dropped (it was redundant)
- README file was dropped (no longer needed)
- GitHub workflows were removed (will be replaced by schutzbot)
- RPMs were removed (they were huge, will be installed in a different way)
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We actually need 2 * 16 connections at minimum (one worker waits for two
jobs). Let's bump the maximum connection count even moar.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
By default, pgxpool.Pool has 4 connections (or number of cpus if higher).
Currently, we have 3 replicas, that means max 3*4=12 DB connections.
The dequeue operation is actually blocking - when a worker is waiting for
a job, one connection is blocked. My theory is that with 16 workers, we just
don't have enough connections that causes all sorts of weird slowdowns.
This commit bumps the number of connection from one replica to 10, therefore
we should be at 30 connections in total.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
