Commit graph

40 commits

Author SHA1 Message Date
Sanne Raymaekers
1ded72b4dc templates/packer: Set region in vector config
Vector 0.21 needs region set otherwise the healthcheck will
fail.
2022-04-19 13:24:33 +02:00
Sanne Raymaekers
2023f7731d worker: Support client_credentials grant type in client
This will allow us to use the service accounts which work against
identity.api.openshift.com. These are much easier to manage, especially
with the new multi-tenancy, as there's a single page to create/expire
them across an account.

They also have the added benefit of not expiring automatically when
they're not used like offline tokens, and immediate expiration when
desired.
2022-03-21 09:43:43 +01:00
Ondřej Budai
9ca74694a7 packer: use unique name tag for Fedora workers
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-16 12:58:05 +01:00
Ondřej Budai
418ae32cf8 packer: fix the secret ID variable in get_koji_creds.sh
Oops, we should probably start testing this.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-14 10:27:28 +01:00
Ondřej Budai
424a741de6 packer: make subscribing optional
We don't want to subscribe Fedora.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 22:31:40 +01:00
Ondřej Budai
c46376aea2 packer: add support for koji credentials
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2dd5ae7bca packer: skip retrieving of creds if their ARN is not specified
So we can have workers without public cloud creds.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
4c0ba50ea1 packer: remove config tinkering from worker_service.sh
Let's set each cloud section of the config in the respective cloud script.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2813507ac9 packer: split worker_external_creds.sh into one script per cloud
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
2e7815bf53 packer: move worker-config creation to ansible
I think it untangles the initialization a bit and allows me to do some more
refactorings.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-13 09:08:11 +01:00
Ondřej Budai
72de1b3bbe packer: don't save the AMIs on PRs
This should save us a ton of resources as we don't use AMIs from PRs.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
ad15179faf packer: build Fedora images
The decision logic which jobs to run is quite confusing but that's how we
roll for now:

Jenkins builds RHEL images only on main
Schutzbot builds RHEL images only in PRs
Schutzbot builds Fedora images on both PRs and on main

To achieve this, the commit re-enables running Packer on main on Schutzbot.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
ec070612ff packer: remove RHEL and x86_64-specific bits
Arch was easy.

For passing the repository distribution and osbuild_commit (it can be
different for each distro), I decided to go in the way of ansible
inventory directories. It adds a bit of structure but I think it's
the most clean solution.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
cd394bf67d packer: add default to aws auth variables
So you don't have to pass these if packer is supposed to find them
on its own (instance profile, local profile).

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
4ae71d3f3d packer: move all RHEL-specific options to a source block
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
22ec89f956 packer: add more tags identifying the image
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
7301ea6b9d packer: use newer (=faster) instances
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
8664c1449a packer: reuse the build user for the ansible provisioner
We want to build multiple images at once and some of them use a different user.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
e45578d3b0 packer: remove the ami_id variable
We want to build multiple images at once so they have to be defined elsewhere.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Ondřej Budai
5ecbfbad9e packer: rename composer.pkr.hcl to worker.pkr.hcl
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-03-11 09:06:43 +01:00
Sanne Raymaekers
63a0bbc1f2 cmd/osbuild-worker: Configure s3 bucket on the worker itself
Parameterize aws section of worker config. If credentials is empty,
the iam role will be used.
2022-03-08 21:58:11 +01:00
Sanne Raymaekers
55b78bbd36 templates/packer: Remove -u flag from creds mapping script
We test if specific variables are set, and -u interferes with that.
2022-03-07 17:11:54 +01:00
Ondřej Budai
5d304d2957 packer: make the worker image smaller
This should save us some money. 10 GB is the size of the underlying
RHEL 8.5 AMI so this should be the minimum.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-18 09:24:07 +01:00
Sanne Raymaekers
a173a3513d tools/appsre-build-worker-packer: Run on subscribed 8.5 machine 2022-02-09 16:54:22 +01:00
sanne
ef6c5df9fa templates/packer: Make cdn host check less sensitive 2022-01-18 17:00:17 +01:00
sanne
68e98244b9 templates/packer: Correct priority for worker rpms
Lower priority means higher, currently the images built through AppSRE's
infra install the worker from epel.
2022-01-17 14:30:11 +01:00
sanne
3c729be3c5 tools/appsre-build-worker-packer: Add image_users variable
packer will share the ami with those users.
2022-01-11 14:30:19 +01:00
sanne
71da979c81 tools: AppSRE packer build 2022-01-05 22:13:55 +01:00
Ondřej Budai
8d81da7d7b packer: remove /var/lib/osbuild-composer check
This directory is not used on worker instances. It was a left-over from the
times when this AMI was also used for running composer.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:17:59 +01:00
Ondřej Budai
9d0ae3bc1f packer: add initialization scripts
The worker needs quite a lot of configuration involving secrets. Baking them
in the AMI is just awful so we need to fetch them during the instance startup.

Previously, this was all done using cloud-init. This makes the cloud-init
config huge and it is also very hard to test.

This commit moves all the configuration scripts into the image itself.
Cloud-init still needs to be used to push the secret variables into the
instance. The configuration scripts are run after cloud-init. They pick up
yhe secrets and initialize the worker correctly.

These scripts were adopted from
75b752a1c0
(private repository).

During the adoption, some changes has to be applied to make shellcheck happy.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:17:59 +01:00
Ondřej Budai
5697b43ad6 packer: update to RHEL 8.5
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-01-04 16:17:59 +01:00
Ondřej Budai
8bf2dd55a2 packer: remove osbuild-composer.service override
We no longer use this AMI for composer, so we don't need this override.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
2bd2e3d1bc packer: install just osbuild-composer-worker
We don't actually need a composer in these images, so let's just install
the worker.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
b799605f51 packer: install monit and vector
Previously, monit and vector RPMs were embedded directly in the
image-builder-packer repository. This was not ideal because hosting big
binary files in git is always ugly.

This commit brings back monit and vector:

- monit is installed from EPEL
- vector is installed from the upstream RPM repository

Ansible was dropped because we don't need it in the image.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
fbebe4c2cf packer: adjust ansible playbook filepath
We want an absolute path, otherwise packer doesn't know where to find the
playbook if called from a wrong directory.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
b619e4875e packer: rework variables
osbuild and composer commit SHAs now must be passed into packer using
variables, no defaults are defined. Also, packer is no longer responsible
for naming the AMIs, the name is also passed as a variable.

imagebuilder_packer_sha was dropped entirely as the packer configuration
now lives directly in osbuild-composer repository.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
0fb3634c2c packer: remove forwarding to console
Console support in AWS EC2 is very basic. We now use vector that works much
better than console so we can just drop the forwarding and rely on vector
dumping the logs into cloudwatch.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
15c46544b6 packer/monit: remove verify_worker_connection
This is currently not working because workers in aoc no longer use mTLS.
Definitely something we want to fix in the future I think.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
cc81e919ca packer: drop RH IT certificate
I think it was needed for internal workers - not needed anymore.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00
Ondřej Budai
1b289cc27e packer: import image-builder-packer repository
/templates/packer now contains a copy of image-builder-packer repository
as of b8a4b45f93890090de24e3d043e2d958948fc3c5

Changes:
- LICENSE file was dropped (it was redundant)
- README file was dropped (no longer needed)
- GitHub workflows were removed (will be replaced by schutzbot)
- RPMs were removed (they were huge, will be installed in a different way)

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-12-01 16:08:11 +00:00