Commit graph

3170 commits

Author SHA1 Message Date
Florian Schüller
a4068b328d splunk_logger: move environment hook to splunk_logger pt2
for reusability also in image-builder
2024-08-28 16:41:07 +02:00
Michael Vogt
43c6bc7abc worker: simplify the POST in workerHeartbeat()
There is no actual payload when posting the worker status so let's
just post `nil` instead of an empty `bytes.Buffer` to make this
explicit.
2024-08-26 16:39:12 +02:00
Michael Vogt
802fbce596 worker: rename server -> serverURL
The current name `Client.server` feels a bit misleading as it is
unclear if this is an abstraction for a "server" object or an
URL. This rename makes this unambiguous.
2024-08-26 16:38:37 +02:00
Tomáš Hozza
f8c5595413 Delete EOL F37 and F38 repos
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
e53d03fe16 compute/gcp: add guest OS features for el10/c10s
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
a0cad4f9a6 client/compose_test: fix error message checked by the test
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Sanne Raymaekers
2624516f1a osbuild-worker: use aws sdk v2 for asg scale-in protection 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
c90b92f666 cloud/awscloud: test failures when running a secure instance 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
acc415a676 cloud/awscloud: test terminating a secure instance 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
e30fba38fc cloud/awscloud: handle nil describe output when creating LTs/SGs 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
fa3b203178 internal/boot: adapt to aws sdk v2 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
b10bbc0fb0 cloudapi/v2: adapt to aws sdk v2 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
8b0e09ed20 internal/weldr: adapt to aws sdk v2 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
16c9a7be88 cloud/awscloud: add tests for ec2 operations 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
810e9133e8 cloud/awscloud: switch ec2 to v2 sdk 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
8d158f6031 cloud/awscloud: switch s3 to v2 sdk 2024-08-20 15:32:40 +02:00
Tomáš Hozza
768537dba3 Add support for RHSM customizations
Add support for RHSM customizations, which currently allow various
aspects of RHSM. Specifically enabling / disabling DNF plugins shipped
by subscription-manager package and setting a subset of options in the
rhsm.conf.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-13 15:52:44 +02:00
Tomáš Hozza
26dd54a4d7 Add support for RPM customizations
Add support for RPM customizations, which currently allow to import RPM
GPG key from a file installed in the image. This is e.g. done for the
Azure RHUI image type.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-13 15:52:44 +02:00
Tomáš Hozza
c3680ca0a7 Update osbuild/images to v0.75.0
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-13 15:52:44 +02:00
Florian Schüller
9006836afc logrus: add deployment channel as field to the logs 2024-08-07 12:32:57 +02:00
Gianluca Zuccarelli
2da3a73308 cloudapi: json tailoring options
Add support for json tailoring files in the cloudapi. Expand the tests
to check that the options returned are okay and that tailoring options
and json tailoring options can't be provided at the same time.
2024-08-07 12:06:28 +02:00
Gianluca Zuccarelli
16180e9c58 blueprint/customizations: json tailoring
Add support to the blueprints for json tailoring files
2024-08-07 12:06:28 +02:00
Gianluca Zuccarelli
4c225f7d66 cloudapi: openscap tailoring tests
Add an initial test for OpenSCAP tailoring customizations since we will
be expanding the tailoring options to support json tailoring.
2024-08-07 12:06:28 +02:00
Michael Vogt
86b1143923 osbuildexecutor: show full osbuild exector on json decode errors
This is a short term workaround to get better visibility why
the osbuild executor sometimes sends non-json data. When reading
the result from the executor the entire output is now read and
if the json parsing goes wrong it will use the entire body
in the error message for better debug visibility.
2024-08-05 14:51:40 +02:00
Sanne Raymaekers
dc389eaa71 cloud/awscloud: fix nil pointer dereference
When the cleanup function gets called, there's a chance the Instnace
field isn't populated yet, so store the instance ID separately and wait
for it to be terminated in case it's present.

The error would produce the following trace:
```
goroutine 1 [running]:
...
main.(*OSBuildJobImpl).Run.func1()
    osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:404 +0xc5
panic({0x55e2a76a1e40?, 0x55e2a906d2f0?})
    /usr/lib/golang/src/runtime/panic.go:920 +0x270
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).deleteFleetIfExists(0xc000faa840, 0xc0012718c0)
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:441 +0x175
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).TerminateSecureInstance(0x55e2a90825e0?, 0x2?)
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:192 +0x1d
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).RunSecureInstance.func1()
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:75 +0x69
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).RunSecureInstance(0xc000faa840, {0xc000afeade, 0x10}, {0x0, 0x0}, {0x0, 0x0}, {0xc001120f30, 0x24})
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:169 +0x12a7
...
```
2024-08-01 10:58:08 +02:00
Michael Vogt
573b349f16 clienterrors: rename WorkerClientError to clienterrors.New
The usual convention to create new object is to prefix `New*` so
this commit renames the `WorkerClientError`. Initially I thought
it would be `NewWorkerClientError()` but looking at the package
prefix it seems unneeded, i.e. `clienterrors.New()` already
provides enough context it seems and it's the only error we
construct.

We could consider renaming it to `clienterror` (singular) too
but that could be a followup.

I would also like to make `clienterror.Error` implement the
`error` interface but that should be a followup to make this
(mechanical) rename trivial to review.
2024-07-31 17:04:58 +02:00
Andrea Waltlova
7c830e90b0 Add password to User schema
Make sure password is encrypted
Update tests

Signed-off-by: Andrea Waltlova <awaltlov@redhat.com>
2024-07-22 09:11:36 +02:00
Achilleas Koutsou
dab896dd64 weldr: update expected error messages
Some depsolve-related error messages in osbuild/images changed.  Update
the expected error messages in our tests.
2024-07-04 19:01:07 +02:00
Sanne Raymaekers
3b3ffe0d08 internal/prometheus: more human-readable time buckets
Let's make them slightly easier to query and reason about. From 100 ms
to 24 hours.
2024-07-04 11:19:25 +02:00
Sanne Raymaekers
547f74e7db awscloud: try to terminate previous secure instance
In case the previous executor SI belonging to the worker did not get
shut down properly, attempt to do it again when starting a new one,
otherwise replacing the SG or LT will not work.
2024-06-28 15:33:08 +02:00
Sanne Raymaekers
791ec07bc2 internal/awscloud: fix cloud-init userdata for secure instance
The conditional only checked if the cloudwatch group was set, and if it
wasn't, the hostname variable wouldn't be set either. So the executor
would try to look for a hostname but not find any.
2024-06-26 10:56:57 +02:00
Sanne Raymaekers
ca19a5b5ed cloudapi/v2: add test for errors 2024-06-25 17:27:28 +02:00
Sanne Raymaekers
86fd957c9d cloudapi/v2: add details for non-service errors
In case Echo throws an error, or in case even echo doesn't manage to
catch an underlying error and wrap it, return as many details as
possible.
2024-06-25 17:27:28 +02:00
Sanne Raymaekers
39d2068d31 cloudapi/v2: detach HTTPErrorHandler from Server struct 2024-06-25 17:27:28 +02:00
Sanne Raymaekers
135dd94de5 cloudapi/v2: include details in case internal error is set
In case details aren't explicitly given, and the internal error is set,
include the internal error message in the details.
2024-06-25 17:27:28 +02:00
Lukas Zapletal
5ce8f65a58 cloudapi: propagate operation/external id
Signed-off-by: Lukas Zapletal <lzap+git@redhat.com>
2024-06-25 13:58:53 +02:00
Lukas Zapletal
f3c0daebbf cmd/osbuild-composer: journald support 2024-06-25 13:58:53 +02:00
Sanne Raymaekers
2a621521a8 osbuildexecutor/aws.ec2: set hostname of executor via cloud-init
This way much more of the journal will be captured under the new
hostname.
2024-06-25 10:58:10 +02:00
Sanne Raymaekers
14052e25db Revert "osbuildexecutor: write the job-id intro control.json"
This reverts commit c13d37c3c0.
2024-06-25 10:58:10 +02:00
Sanne Raymaekers
ae4467ab0d internal/awscloud: retry CreateFleet
When receiving the "UnfillableCapacity" error from CreateFleet, retry
the request with an OnDemand instance.
2024-06-24 12:50:37 +02:00
Michael Vogt
c13d37c3c0 osbuildexecutor: write the job-id intro control.json
Set the jobID into control.json so that the osbuild-worker-executor
can set the hostname based on the jobID.
2024-06-14 15:02:08 +02:00
Michael Vogt
aa3d70a429 osbuildexecutor: tweak RunOSBuild() signature and use opts
Introduce a new OsbuildOpts struct to make the API a bit easier
to extend and use in the packages.

Also add a new `JobID` field in the `OsbuildOpts`.
2024-06-14 15:02:08 +02:00
Sanne Raymaekers
2e31ea50aa cloud/awscloud: use instance requirements when creating secure instance 2024-06-14 10:59:58 +02:00
Sanne Raymaekers
4bb61da37e Revert "prometheus: active worker gauge"
This reverts commit 68bc8e0c88.
2024-06-12 17:20:01 +02:00
Michael Vogt
fedbd72d57 osbuildexecutor: allow file type tar.TypeGNUSparse too
We need to allow files of type `tar.TypeGNUSparse` in the result
that we get from the osbuild-worker-executor too.
2024-06-12 16:31:44 +02:00
Sanne Raymaekers
971e1df148 internal/osbuildexecutor: verify expected input from handleBuild 2024-06-12 13:10:52 +02:00
Sanne Raymaekers
8a5f486715 osbuildexecutor: wait until worker-executor closes the connection
Otherwise the client will try to fetch the output archive before the
build output is archived on the worker-executor side.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
db43ec8e60 osbuildexecutor: move tests to separate _test package 2024-06-12 11:36:30 +02:00
Michael Vogt
984f51feb8 osbuildexecutor: add validateOutputArchive() and run before extract
The tar file from the `osbuild-worker-executor` is potentially
tainted. Ensure we validate and only extract if it harmless.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
1f52150ff1 internal/osbulid-executor/aws-ec2: return more verbose errors
In RunOsbuild return more verbose errors where the error doesn't
originate from one of the local helper functions.
2024-06-12 11:36:30 +02:00