Commit graph

6270 commits

Author SHA1 Message Date
Ondřej Budai
504a51089c cloudapi: split GetImageRequests off PostCompose
This allows us to do some testing of the ComposeRequest => []imageRequest
conversion without actually enqueueing any jobs (which requires us to
do quite a lot of setup in the testsuite).

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2024-02-21 12:09:01 +01:00
Ondřej Budai
28ef0bc855 cloudapi: move manifest seed into an image request
The goal of this commit is primarily to simplify the API of the enqueue
methods. This way, basically everything needed to generate manifests
is in the imageRequest structure, which simplifies the amount of structures
that we need to think about.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2024-02-21 12:09:01 +01:00
Ondřej Budai
7385cab165 cloudapi: move blueprint into an image request
The goal of this commit is primarily to simplify the API of the enqueue
methods. This way, basically everything needed to generate manifests
is in the imageRequest structure, which simplifies the amount of structures
that we need to think about.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2024-02-21 12:09:01 +01:00
Ondřej Budai
ab8c1ae4f7 clouadpi: remove distribution from enqueue methods
They can just find it from the image type.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2024-02-21 12:09:01 +01:00
Ondřej Budai
c48bf0a143 cloudapi: remove arch from imageRequest
You can absolutely just get it from imageType, let's not duplicate
information, that's error-prone.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2024-02-21 12:09:01 +01:00
Ondřej Budai
08aabe1bef cloudapi: add minimal-raw 2024-02-21 12:09:01 +01:00
Ondřej Budai
eb3bd9dc43 cloudapi: sort image types in getDefaultTarget
Not a functional change.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2024-02-21 12:09:01 +01:00
Tomáš Hozza
e561ba0854 templates/composer: set DISTRO_ALIASES for composer
Set the RHEL release names without the minor version to point to the
latest GA release. Set the 'rhel-10.0' to the latest RHEL-9 minor
release in development, so that one can start building RHEL-10 images
without referencing RHEL-9.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-02-21 12:06:33 +01:00
Tomáš Hozza
f57e88d034 osbuild-composer: support setting distro_aliases via ENV variable
This will allow us to configure the distro aliases in the service by
setting ENV variable.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-02-21 12:06:33 +01:00
Tomáš Hozza
218a784778 Test/base_tests.sh: workaround missing osbuild-depsolve-dnf in nightly
The RHEL nightly composes still contain old osbuild-composer version,
which does not have an explicit dependency on osbuild-depsolve-dnf
package.

As a result, running the latest upstream base_tests.sh on RHEL nightly
composes fails due to osbuild-depsolve-dnf not being installed.

Workaround this in the test case by explicitly installing the
osbuild-depsolve-dnf package in case the osbuild-composer version is
lower than the one which has an explicit dependency on
osbuild-depsolve-dnf.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-02-20 15:55:47 +01:00
Diaa Sami
82add84c31 Test/old-worker-new-composer: use latest composer for verification
To avoid incompatibility between older image-info and latest osbuild
2024-02-20 15:55:47 +01:00
Diaa Sami
76e686df10 switch to images/pkg/dnfjson and remove internal copy
COMPOSER-2068
2024-02-20 15:55:47 +01:00
Diaa Sami
98e3dab9c7 Schutzfile: bump version of osbuild to 619a64f0 2024-02-20 15:55:47 +01:00
Sanne Raymaekers
5025ec31d3 cloud/awscloud: describe security groups using filters
Using the group names option only works for the default VPC, the workers
are not running in the default VPC. For non-default VPCs filters should
be used.
2024-02-20 15:23:52 +01:00
Sanne Raymaekers
bb43f330b8 cmd/osbuild-jobsite-builder: shutdown http server gracefully 2024-02-16 15:21:20 +01:00
Sanne Raymaekers
7fce482baa cloud/awscloud: create secure instance in the same subnet
This reduces network costs as transferring data between AZs is not free.
2024-02-16 15:21:20 +01:00
Sanne Raymaekers
1abd887c23 templates/packer: rename executor log group
In app-interface the output resource names need to be unique, and the
log group name is already shared with the role.
2024-02-16 11:46:03 +01:00
Simon de Vlieger
7533f58fb2 jobsite/manager: turn off compression
We're seeing some unexpected EOFs in staging deployment; going on a
hunch I've seen these before when gzip gets involved in transfering
large files so let's disable that.
2024-02-16 11:45:51 +01:00
Sanne Raymaekers
ab6fd9e53e cmd/osbuild-jobsite-builder: actually assign the stdout buffer
The buffer needs to be assigned to the process' stdout for the buffer to
fill up.
2024-02-15 18:28:07 +01:00
Simon de Vlieger
b9584099ab jobsite/manager: create export directory
Be a bit more friendly and actually create the export directory instead
of assuming it exists.
2024-02-15 16:23:55 +01:00
Sanne Raymaekers
1150f0f27e cmd/osbuild-jobsite: capture osbuild's stdout
Write osbuild's stdout in the progress step. The manager can just copy
it to stdout and the executor will be able to parse the output into an
osbuild result.
2024-02-15 14:31:24 +01:00
Sanne Raymaekers
ee6b198b0a cloud/awscloud: remove restricting egress rule from SG
The machine still needs to be able to fetch sources, so just keep the
default 0.0.0.0/0 rule.
2024-02-15 14:23:18 +01:00
Sanne Raymaekers
3400c5c20d templates/packer: setup vector in osbuild-executor 2024-02-15 12:05:17 +01:00
Sanne Raymaekers
a9010479e3 templates/packer: set -builder-path to /var/cache/osbuild-builder
The builder uses `/run/osbuild` as a default path for this argument. Yet
this directory doesn't exist when the builder writes the manifest. But
osbuild should own this directory, not the builder.

Furthermore `/run` is a tmpfs, so the executor might run into memory
issues if we use `/run` as the store and output directory (on the "host"
workers these are in `/var/cache`).

While `/tmp` might seem like a good candidate on RHEL, it's a tmpfs on
Fedora, so it's also to be avoided.
2024-02-15 01:02:10 +01:00
Sanne Raymaekers
04ce056b6a templates/packer: let the executor listen on all interfaces 2024-02-14 22:55:54 +01:00
Sanne Raymaekers
bfd1978ad8 osbuildexecutor/aws.ec2: pass the manifest to the job manager 2024-02-14 20:28:22 +01:00
Sanne Raymaekers
8e6717fa1b cloud/awscloud: take instance type from host
InstanceRequirements is very flakey, the create fleet request fails
almost consistently with the same error.

To continue with testing use a fixed instance type for now. As a
followup we can expand the instance type selection logic or figure out
what was wrong with the InstanceRequirements.
2024-02-14 18:15:25 +01:00
Sanne Raymaekers
8a1d66a0bd cloud/awscloud: max 4 overrides are allowed when creating a fleet
```
InvalidParameterValue: Your request contains more than the maximum allowed number of InstanceRequirements (4)
```
2024-02-14 15:24:42 +01:00
Sanne Raymaekers
f03dac5834 templates/packer: don't subscribe fedora executor 2024-02-14 13:45:52 +01:00
Sanne Raymaekers
7fd150b938 cloud/awscloud: specify subnets when creating secure instance
For non-default VPCs, AWS needs the subnets it can launch the instance
in, otherwise it will try to launch the instance in the default VPC,
even if the supplied security groups are attached to a non-default VPC.

Furthermore there can only be 1 subnet specified per availability zone,
so query the subnets in the VPC of the host (as the instance needs to be
launched in the same network), and pick 1 of the VPC's subnets per AZ.
2024-02-14 13:45:52 +01:00
Sanne Raymaekers
d5fd1bbbf0 templates/packer: add worker-executor service to image 2024-02-14 13:45:52 +01:00
Sanne Raymaekers
a9066ee793 templates/packer: remove worker_builder script from init service
This script doesn't actually exist.
2024-02-14 13:45:52 +01:00
Diaa Sami
f08d1f6068 composer: send error log messages to sentry 2024-02-14 10:38:42 +01:00
Sanne Raymaekers
d0caac9d69 templates/packer: use aws.ec2 osbuild executor 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
4111ca4e0e templates/packer: add worker-executor service 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
e7cadb16af osbuildexecutor: add aws.ec2 executor
This executor spins up an instance which can only contact the host, and
uses the osbuild-jobsite manager & builder to invoke osbuild.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
a2fb1bfc61 cloud/awscloud: add userdata to secure instance
This way the `worker-initialization.service` knows to spin up the
builder instead of the worker.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
3db88960c2 cloud/awscloud: add ability to run a secure instance to awscloud
This instance can only contact the host, and requires this host to be
running on AWS itself with the appropriate IAM role.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
05a45ed233 cloud/awscloud: add ec2metadata client 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
e10424de2f osbuildexecutor: introduce osbuildexecutor.Executor interface
Wrap the current osbuildexecutor.Executor in an interface so it's easier
to add different executors, which for instance can run osbuild in a VM.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
9e85050633 osbuild-worker: add osbuild_executor config option 2024-02-14 09:54:11 +01:00
Diaa Sami
c9c51613a4 composer: glitchtip integration 2024-02-13 14:57:57 +01:00
Simon de Vlieger
95b4979d88 jobsite/builder: move process into builder
Moves the background process into the builder struct in preparation for
using `sync.Once`.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
3cf2531ae2 jobsite: go-staticcheck appeasement
Log and errors should not end with punctuation.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
7b7de50e23 jobsite/builder: enable --json
Use `--json`; this causes `osbuild` to buffer all output and then print
a huge blob of JSON at the end of the command.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
748db5410e jobsite/manager: always retry on some errors
On network layer errors we will retry the request until the timeout
expires.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
46ace22007 jobsite/manager: rename pipeline and export
A better descriptive name externally is to use `export` and
`export-file`.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
42892754fd jobsite/builder: no caching for osbuild.
As builders are not re-used caching has no use and can likely only
lead to disk-size issues. This can be re-enabled when we have an
outside-of-the-builder cache.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
b9b476c430 osbuild/builder: check deferred serve.
Make sure to keep track of the deferred error for serve.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
323d3a1f9c jobsite/builder: rename.
Consistency, the agent is now called builder.
2024-02-12 12:03:38 +01:00