Commit graph

898 commits

Author SHA1 Message Date
Sanne Raymaekers
f18293c2f1 osbuild-worker-executor: adapt worker-executor to osbuild-composer 2024-06-12 11:36:30 +02:00
Michael Vogt
2704b18663 obuild-worker: extract workerClientErrorFrom() helper and add tests
Tiny commit to extract a helper from DepsolveJobImpl.Run() that
can then be unit tested.

This should help with https://github.com/osbuild/images/issues/727
2024-06-11 10:42:00 +02:00
Michael Vogt
a691df2353 osbuild-worker-executor: fix order of assert.Equal() in tests
The `assert.Equal()` expects that the "expected" value is put
first. Which is not what I'm used to. It's also slightly inconsistent
because `assert.EqualError()` expects the "actual" err first and
then the expected string. But this commit is not about ranting :)

This commit fixes the order in the tests assert.Equal() so that
mismatches actually are displayed correctly.
2024-06-07 08:17:32 +02:00
Michael Vogt
61bf0c3235 osbuild-worker: do not use error in clienterror.Error.Details
This is an alternative/complementary fix for PR#4137. It is very
simple so should be uncontroverisal.

It fixes an issue that @schuellerf discovered, i.e. that when an error
interface is passed into clienterrors.Error.Details the details get
lost because the json.Marshaler will not know how to handler an
error interface.

To find the problematic uses of `error` a custom vet checker was
build in https://github.com/mvo5/osbuild-cvet. With that the
result is:
```
$ go run github.com/mvo5/osbuild-cvet@latest ./...
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-depsolve.go:93:26: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:404:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:519:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:556:31: do not pass '[]error' to WorkerClientError() details, use []string instead
```
and once this commit is in no more errors.

Just like PR#4137 this is not perfect because it will not do a
recursive check for the passed argument.
2024-06-07 01:19:11 +02:00
Michael Vogt
8ebefbdbc9 main: rework the way the mock logger is passed
Pass the mock logger directly to `run()` instead of mocking
`logrus.New`. Doing the later leads to a data race when multiple
parallel tests modify the (global) `var logrusNew logrus.New`.

Thanks to Tomas Hozza for reporting.
2024-06-06 21:14:31 +02:00
Michael Vogt
95b4a9e250 osbuild-worker-executor: make test output silent again
Do not use the global logger but pass instead the locally created
logger. This means the test output is silent again.

Sadly using the global logger is difficult because it is a global
resource so replacing it in tests means all tests (that are
potentially run in parallel) will write to it which makes testing
specific log output hard.
2024-06-06 16:16:33 +02:00
Michael Vogt
e34728b466 osbuild-worker-executor: appease errcheck 2024-06-05 18:26:08 +02:00
Michael Vogt
138bc73e37 osbuild-worker-executor: appease gosec
Note that gosec IMHO is a bit silly here, the heuristics used are
note very good, i.e. the code is already validating the external
inputs and it's not clear to me that "filepath.Clean()" will help
but it seems to supress the error. I hope gosec provides value
in other places, here it seems to be adding work :/

I also excluded "gosec" from any _test.go files, I do not see
why we should gosec tests?
2024-06-05 18:26:08 +02:00
Sanne Raymaekers
b0543e89f4 osbuild-worker-executor: fix lint warnings/errors
The osbuild-composer linting found a bunch of issues that this
commit fixes.
2024-06-05 18:26:08 +02:00
Michael Vogt
cbb8d79baf c/osbuild-worker-executor: update to match new name
Update the imports/names to match the new name
"osbuild-worker-executor".
2024-06-05 18:26:08 +02:00
Michael Vogt
372d9f07dd cmd/osbuild-worker-executor: import verbatim from mvo5/oaas
This commit imports the repository https://github.com/mvo5/oaas
without keeping the history. The history is largely unimportant
and can be looked up in https://github.com/mvo5/oaas/commits/main
if needed.
2024-06-05 18:26:08 +02:00
Tomáš Hozza
b01b080565 Update 'rhel-9' distro alias to 'rhel-9.5'
New releases are landing in 9.5 already, so the on-prem version should
reflect that. The service can and does override this using a
configuration.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-04 13:03:37 +02:00
Tomáš Hozza
695febf39c Define distro alias for 'rhel-10'
Set it to RHEL-10.0.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-04 13:03:37 +02:00
Sanne Raymaekers
5a776c5b79 templates/openshift: split worker from composer maintenance 2024-04-25 17:32:21 +02:00
Sanne Raymaekers
0b277937dd worker/server: make worker timeout configurable 2024-04-19 19:56:25 +02:00
Sanne Raymaekers
c9eb60d674 osbuild-worker: fix nil pointer in depsolve job 2024-04-18 12:21:07 +02:00
Sanne Raymaekers
7684dbeedd osbuild-worker: fix mtls credentials injection in depsolve job 2024-04-17 20:27:06 +02:00
Sanne Raymaekers
05a1e8f054 worker: support a proxy for repository mtls configuration
Allows setting a proxy for the mtls settings for a specific
repository. This is useful when consuming content from the content
service in consoledot.
2024-04-17 16:17:57 +02:00
Sanne Raymaekers
d6cfd04704 cmd/osbuild-dnf-json-tests: check repo configs 2024-04-17 10:27:08 +02:00
Sanne Raymaekers
98e3892e23 cmd/osbuild-store-dump: pass repoconfigs when serializing manifests 2024-04-17 10:27:08 +02:00
Sanne Raymaekers
3094eb474d cmd/gen-manifests: pass repoconfigs when serializing manifests 2024-04-17 10:27:08 +02:00
Sanne Raymaekers
e0759e01af worker: save repoconfigs in depsolve result 2024-04-17 10:27:08 +02:00
Achilleas Koutsou
809f2544ad deps: update images to v0.54.0
Update images dependency and adjust import paths for distro package
changes.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2024-04-11 16:43:40 +02:00
Sanne Raymaekers
53f77368fd osbuild-worker: add support for mtls dnf repo secrets 2024-03-29 14:46:54 +01:00
Sanne Raymaekers
f311adf35d osbuild-jobsite-builder: disable http.Server timeouts
Let's just rely on the timeouts surrounding the whole step.
2024-03-21 17:08:07 +01:00
Sanne Raymaekers
d3074fc265 osbuild-jobsite-manager: clean up store archive after use 2024-03-21 15:20:52 +01:00
Sanne Raymaekers
bc17204001 osbuild-jobsite-manager: reseek when necessary 2024-03-21 15:20:52 +01:00
Sanne Raymaekers
c19e9fb3b9 osbuild-jobsite-manager: close writer before sending the store 2024-03-21 15:20:52 +01:00
Sanne Raymaekers
dabe63cb78 osbuild-jobsite: increase populate timeout
30 seconds can be a bit too little for the entire store.
2024-03-20 21:58:17 +01:00
Sanne Raymaekers
b4e361e093 osbuild-jobsite-builder: unpack source tarball in populate 2024-03-19 17:07:30 +01:00
Sanne Raymaekers
07598ea65c osbuild-jobsite-manager: pass store to builder 2024-03-19 17:07:30 +01:00
Brian C. Lane
01ba674cac cloudapi: Pass the RepoRegistry to the cloudapi Server 2024-03-11 03:04:54 -07:00
Brian C. Lane
b8967d53bc Move RepoRegistry setup into Composer
This is so that both the weldr and cloud api's can use it as the source
of their repositories.
2024-03-11 03:04:54 -07:00
Tomáš Hozza
e7743f17ec Worker: allow configuring executor CloudWatch group
We need the ability to use different CloudWatch group for the
osbuild-executor on Fedora workers in staging and production
environment.

Extend the worker confguration to allow configuring the CloudWatch group
name used by the osbuild-executor. Extend the secure instance code to
instruct cloud-init via user data to create /tmp/cloud_init_vars file
with the CloudWatch group name in the osbuild-executor instance, to make
it possible for the executor to configure its logging differently based
on the value.

Cover new changes by unit tests.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-03-08 13:13:44 +01:00
Brian C. Lane
ceddabc395 osbuild-worker: Convert osbuild raw json error to a string for logging
Without this you get an array of bytes in the logs. Convert the raw json
output from osbuild to a string so there is some hope that it is
readable.
2024-03-08 01:17:00 +01:00
Sanne Raymaekers
040eec4089 osbuild-worker: allow adding key to aws.ec2 executor
This is useful during testing to set up the executor machine.
2024-03-01 19:20:51 +01:00
Gianluca Zuccarelli
f6b76cce31 Update osbuild/images to v0.41.0
Multiple blueprint fixes:

- Extend the blueprint service customizations to accept services to be
  masked.

- The `storage-path` and `container-transport` fields were removed in
  imagees 41.0 in order to simplify the way local storage containers are
  handled.
2024-02-29 20:57:39 +01:00
Tomáš Hozza
f57e88d034 osbuild-composer: support setting distro_aliases via ENV variable
This will allow us to configure the distro aliases in the service by
setting ENV variable.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-02-21 12:06:33 +01:00
Diaa Sami
76e686df10 switch to images/pkg/dnfjson and remove internal copy
COMPOSER-2068
2024-02-20 15:55:47 +01:00
Sanne Raymaekers
bb43f330b8 cmd/osbuild-jobsite-builder: shutdown http server gracefully 2024-02-16 15:21:20 +01:00
Simon de Vlieger
7533f58fb2 jobsite/manager: turn off compression
We're seeing some unexpected EOFs in staging deployment; going on a
hunch I've seen these before when gzip gets involved in transfering
large files so let's disable that.
2024-02-16 11:45:51 +01:00
Sanne Raymaekers
ab6fd9e53e cmd/osbuild-jobsite-builder: actually assign the stdout buffer
The buffer needs to be assigned to the process' stdout for the buffer to
fill up.
2024-02-15 18:28:07 +01:00
Simon de Vlieger
b9584099ab jobsite/manager: create export directory
Be a bit more friendly and actually create the export directory instead
of assuming it exists.
2024-02-15 16:23:55 +01:00
Sanne Raymaekers
1150f0f27e cmd/osbuild-jobsite: capture osbuild's stdout
Write osbuild's stdout in the progress step. The manager can just copy
it to stdout and the executor will be able to parse the output into an
osbuild result.
2024-02-15 14:31:24 +01:00
Diaa Sami
f08d1f6068 composer: send error log messages to sentry 2024-02-14 10:38:42 +01:00
Sanne Raymaekers
e7cadb16af osbuildexecutor: add aws.ec2 executor
This executor spins up an instance which can only contact the host, and
uses the osbuild-jobsite manager & builder to invoke osbuild.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
3db88960c2 cloud/awscloud: add ability to run a secure instance to awscloud
This instance can only contact the host, and requires this host to be
running on AWS itself with the appropriate IAM role.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
05a45ed233 cloud/awscloud: add ec2metadata client 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
e10424de2f osbuildexecutor: introduce osbuildexecutor.Executor interface
Wrap the current osbuildexecutor.Executor in an interface so it's easier
to add different executors, which for instance can run osbuild in a VM.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
9e85050633 osbuild-worker: add osbuild_executor config option 2024-02-14 09:54:11 +01:00