The worker needs quite a lot of configuration involving secrets. Baking them
in the AMI is just awful so we need to fetch them during the instance startup.
Previously, this was all done using cloud-init. This makes the cloud-init
config huge and it is also very hard to test.
This commit moves all the configuration scripts into the image itself.
Cloud-init still needs to be used to push the secret variables into the
instance. The configuration scripts are run after cloud-init. They pick up
yhe secrets and initialize the worker correctly.
These scripts were adopted from
75b752a1c0
(private repository).
During the adoption, some changes has to be applied to make shellcheck happy.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When backed by a DB, composer has no need of a queue directory.
This also addresses "Error moving artifacts for job" logging noise.
Signed-off-by: sanne <sanne.raymaekers@gmail.com>
Previously, monit and vector RPMs were embedded directly in the
image-builder-packer repository. This was not ideal because hosting big
binary files in git is always ugly.
This commit brings back monit and vector:
- monit is installed from EPEL
- vector is installed from the upstream RPM repository
Ansible was dropped because we don't need it in the image.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We want an absolute path, otherwise packer doesn't know where to find the
playbook if called from a wrong directory.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
osbuild and composer commit SHAs now must be passed into packer using
variables, no defaults are defined. Also, packer is no longer responsible
for naming the AMIs, the name is also passed as a variable.
imagebuilder_packer_sha was dropped entirely as the packer configuration
now lives directly in osbuild-composer repository.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Console support in AWS EC2 is very basic. We now use vector that works much
better than console so we can just drop the forwarding and rely on vector
dumping the logs into cloudwatch.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This is currently not working because workers in aoc no longer use mTLS.
Definitely something we want to fix in the future I think.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
/templates/packer now contains a copy of image-builder-packer repository
as of b8a4b45f93890090de24e3d043e2d958948fc3c5
Changes:
- LICENSE file was dropped (it was redundant)
- README file was dropped (no longer needed)
- GitHub workflows were removed (will be replaced by schutzbot)
- RPMs were removed (they were huge, will be installed in a different way)
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We actually need 2 * 16 connections at minimum (one worker waits for two
jobs). Let's bump the maximum connection count even moar.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
By default, pgxpool.Pool has 4 connections (or number of cpus if higher).
Currently, we have 3 replicas, that means max 3*4=12 DB connections.
The dequeue operation is actually blocking - when a worker is waiting for
a job, one connection is blocked. My theory is that with 16 workers, we just
don't have enough connections that causes all sorts of weird slowdowns.
This commit bumps the number of connection from one replica to 10, therefore
we should be at 30 connections in total.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Use fetching the OpenAPI spec as a simple readiness/liveness, as
there is not much else we can/need to verify.
Set the default CPU and memory limits in accordance with AppSRE
requirements.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Avoid using the default account, but use a dedicated one.
This follows the guidelines from AppSRE and is what was done for
image-builder.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This should all move to app-interface, as it is configuration, and
we should distinguish between staging and production.
But for now, enable this where it is.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is backwards compatible, as long as the timeout is 0 (never
timeout), which is the default.
In case of the dbjobqueue the underlying timeout is due to
context.Canceled, context.DeadlineExceeded, or net.Error with Timeout()
true. For the fsjobqueue only the first two are considered.
