Commit graph

104 commits

Author SHA1 Message Date
Diaa Sami
5ffb9e693e tools/appsre: remove monit setup code & scripts
Since it doesn't not work since we moved workers to app-sre
2022-10-04 16:26:08 +02:00
Ondřej Budai
f25dca793d packer: remove Fedora 35
Our workers already run on Fedora 36 so there's no need to build F35 anymore.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-09-30 14:52:24 +02:00
Diaa Sami
98eda72499 templates/packer: update amazon plugin 2022-09-27 10:47:32 +02:00
Diaa Sami
06fbd926ae app-sre: Update AMIs to rhel-9.0 2022-09-27 10:47:32 +02:00
Sanne Raymaekers
5c12076b4f templates/packer: Allow token url to be set by cloud-init vars
Hardcoding the token url renders the image useless if it ever needs to
be changed.
2022-09-22 14:15:26 +02:00
Ondřej Budai
8f97c4788c packer: add fedora 36
F35 is going EOL soon, so let's update. I want to ditch F35 as soon as possible
after this is merged, but I want to have some overlap just to be sure.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-09-22 11:22:46 +02:00
Sanne Raymaekers
183e10e466 templates/packer: append distro and arch to the ami name
Because the rhel-8 images share the same name, and `force_deregister` is
true, packer will always deregister one of them.
2022-09-15 20:27:59 +02:00
Diaa Sami
819a63e50e templates/packer: reasonable aws_polling limits for rhel AWS builds 2022-09-09 12:08:29 +02:00
Diaa Sami
46d36a0e73 Revert "appsre: disable aarch64 AMI creation until issue is resolved"
This reverts commit 84f46eebdb.
2022-09-09 12:08:29 +02:00
Diaa Sami
84f46eebdb appsre: disable aarch64 AMI creation until issue is resolved
after merging of PR #2718, generation of AMIs has been failing with 'ResourceNotReady: exceeded wait attempts'.
issue tracked in #2961
2022-09-07 12:28:40 +02:00
Sanne Raymaekers
ab3bd7d94f templates/packer: Increase aws timeouts for rhel-8-aarch64
This job is failing with "ResourceNotReady: exceeded wait attempts".

https://www.packer.io/plugins/builders/amazon#resourcenotready-error
2022-09-05 14:39:12 +02:00
Diaa Sami
ec0a1944b4 appsre-ansible: support aarch64
make ansible playbooks arch-agnostic
extract embedded bash script into separate file with parameters
update packer template to support aarch64
Convert parts of bash script to python code that can start multi-arch instances to build RPMS
2022-09-05 12:08:57 +02:00
Ondřej Budai
767283b2d9 packer: use 8.6 as a base for RHEL images
Let's stay updated!

Also, let's remove 8.4 and 8.5 from Schutzfile, I strongly believe that it's
not used anywhere.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-05 11:54:12 +02:00
Ondřej Budai
5315264f2e packer: pin the vector version
See the comment inline.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-06-07 09:08:22 +02:00
Sanne Raymaekers
1ded72b4dc templates/packer: Set region in vector config
Vector 0.21 needs region set otherwise the healthcheck will
fail.
2022-04-19 13:24:33 +02:00
Sanne Raymaekers
2023f7731d worker: Support client_credentials grant type in client
This will allow us to use the service accounts which work against
identity.api.openshift.com. These are much easier to manage, especially
with the new multi-tenancy, as there's a single page to create/expire
them across an account.

They also have the added benefit of not expiring automatically when
they're not used like offline tokens, and immediate expiration when
desired.
2022-03-21 09:43:43 +01:00
Ondřej Budai
9ca74694a7 packer: use unique name tag for Fedora workers
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-16 12:58:05 +01:00
Ondřej Budai
418ae32cf8 packer: fix the secret ID variable in get_koji_creds.sh
Oops, we should probably start testing this.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-14 10:27:28 +01:00
Ondřej Budai
424a741de6 packer: make subscribing optional
We don't want to subscribe Fedora.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 22:31:40 +01:00
Ondřej Budai
c46376aea2 packer: add support for koji credentials
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2dd5ae7bca packer: skip retrieving of creds if their ARN is not specified
So we can have workers without public cloud creds.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
4c0ba50ea1 packer: remove config tinkering from worker_service.sh
Let's set each cloud section of the config in the respective cloud script.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2813507ac9 packer: split worker_external_creds.sh into one script per cloud
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2e7815bf53 packer: move worker-config creation to ansible
I think it untangles the initialization a bit and allows me to do some more
refactorings.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
72de1b3bbe packer: don't save the AMIs on PRs
This should save us a ton of resources as we don't use AMIs from PRs.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
ad15179faf packer: build Fedora images
The decision logic which jobs to run is quite confusing but that's how we
roll for now:

Jenkins builds RHEL images only on main
Schutzbot builds RHEL images only in PRs
Schutzbot builds Fedora images on both PRs and on main

To achieve this, the commit re-enables running Packer on main on Schutzbot.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
ec070612ff packer: remove RHEL and x86_64-specific bits
Arch was easy.

For passing the repository distribution and osbuild_commit (it can be
different for each distro), I decided to go in the way of ansible
inventory directories. It adds a bit of structure but I think it's
the most clean solution.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
cd394bf67d packer: add default to aws auth variables
So you don't have to pass these if packer is supposed to find them
on its own (instance profile, local profile).

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
4ae71d3f3d packer: move all RHEL-specific options to a source block
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
22ec89f956 packer: add more tags identifying the image
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
7301ea6b9d packer: use newer (=faster) instances
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
8664c1449a packer: reuse the build user for the ansible provisioner
We want to build multiple images at once and some of them use a different user.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
e45578d3b0 packer: remove the ami_id variable
We want to build multiple images at once so they have to be defined elsewhere.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
5ecbfbad9e packer: rename composer.pkr.hcl to worker.pkr.hcl
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Sanne Raymaekers
63a0bbc1f2 cmd/osbuild-worker: Configure s3 bucket on the worker itself
Parameterize aws section of worker config. If credentials is empty,
the iam role will be used.
2022-03-08 21:58:11 +01:00
Sanne Raymaekers
55b78bbd36 templates/packer: Remove -u flag from creds mapping script
We test if specific variables are set, and -u interferes with that.
2022-03-07 17:11:54 +01:00
Ondřej Budai
5d304d2957 packer: make the worker image smaller
This should save us some money. 10 GB is the size of the underlying
RHEL 8.5 AMI so this should be the minimum.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-18 09:24:07 +01:00
Sanne Raymaekers
a173a3513d tools/appsre-build-worker-packer: Run on subscribed 8.5 machine 2022-02-09 16:54:22 +01:00
sanne
ef6c5df9fa templates/packer: Make cdn host check less sensitive 2022-01-18 17:00:17 +01:00
sanne
68e98244b9 templates/packer: Correct priority for worker rpms
Lower priority means higher, currently the images built through AppSRE's
infra install the worker from epel.
2022-01-17 14:30:11 +01:00
sanne
3c729be3c5 tools/appsre-build-worker-packer: Add image_users variable
packer will share the ami with those users.
2022-01-11 14:30:19 +01:00
sanne
71da979c81 tools: AppSRE packer build 2022-01-05 22:13:55 +01:00
Ondřej Budai
8d81da7d7b packer: remove /var/lib/osbuild-composer check
This directory is not used on worker instances. It was a left-over from the
times when this AMI was also used for running composer.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:17:59 +01:00
Ondřej Budai
9d0ae3bc1f packer: add initialization scripts
The worker needs quite a lot of configuration involving secrets. Baking them
in the AMI is just awful so we need to fetch them during the instance startup.

Previously, this was all done using cloud-init. This makes the cloud-init
config huge and it is also very hard to test.

This commit moves all the configuration scripts into the image itself.
Cloud-init still needs to be used to push the secret variables into the
instance. The configuration scripts are run after cloud-init. They pick up
yhe secrets and initialize the worker correctly.

These scripts were adopted from
75b752a1c0
(private repository).

During the adoption, some changes has to be applied to make shellcheck happy.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:17:59 +01:00
Ondřej Budai
5697b43ad6 packer: update to RHEL 8.5
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:17:59 +01:00
Ondřej Budai
8bf2dd55a2 packer: remove osbuild-composer.service override
We no longer use this AMI for composer, so we don't need this override.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
2bd2e3d1bc packer: install just osbuild-composer-worker
We don't actually need a composer in these images, so let's just install
the worker.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
b799605f51 packer: install monit and vector
Previously, monit and vector RPMs were embedded directly in the
image-builder-packer repository. This was not ideal because hosting big
binary files in git is always ugly.

This commit brings back monit and vector:

- monit is installed from EPEL
- vector is installed from the upstream RPM repository

Ansible was dropped because we don't need it in the image.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
fbebe4c2cf packer: adjust ansible playbook filepath
We want an absolute path, otherwise packer doesn't know where to find the
playbook if called from a wrong directory.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
b619e4875e packer: rework variables
osbuild and composer commit SHAs now must be passed into packer using
variables, no defaults are defined. Also, packer is no longer responsible
for naming the AMIs, the name is also passed as a variable.

imagebuilder_packer_sha was dropped entirely as the packer configuration
now lives directly in osbuild-composer repository.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00