Commit graph

914 commits

Author SHA1 Message Date
Michael Vogt
573b349f16 clienterrors: rename WorkerClientError to clienterrors.New
The usual convention to create new object is to prefix `New*` so
this commit renames the `WorkerClientError`. Initially I thought
it would be `NewWorkerClientError()` but looking at the package
prefix it seems unneeded, i.e. `clienterrors.New()` already
provides enough context it seems and it's the only error we
construct.

We could consider renaming it to `clienterror` (singular) too
but that could be a followup.

I would also like to make `clienterror.Error` implement the
`error` interface but that should be a followup to make this
(mechanical) rename trivial to review.
2024-07-31 17:04:58 +02:00
Tomáš Hozza
286236b698 Config: don't override undefined keys when loading from ENV
Composer can load configuration values defined as map from ENV.
Previously, when loading the configuration from ENV, the whole map would
get overridden, not just values defined in the ENV. This is however not
intended and not consistent with how loading configuration from file
works.

Adjust the configuration loading from ENV and adjust the unit test
accordingly.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-17 11:02:41 +02:00
Michael Vogt
919b423953 osbuild-worker: tweak error to not include a \n for a failed stage
Small followup for
https://github.com/osbuild/osbuild-composer/pull/4113#discussion_r1670063775

Given that the failed stage is a relatively short string the `\n`
seems unneccessary and quotes are enough.
2024-07-11 09:33:40 +02:00
Florian Schüller
7cd5abd17c cmd/osbuild-worker/jobimpl-depsolve: show error.Reason only once
as now the .Reason is properly passed over - it was printed twice
2024-07-09 12:12:36 +02:00
Florian Schüller
b0a737421a osbuild-worker: improve error "reason" in case of stage failures 2024-07-09 12:12:36 +02:00
Lukas Zapletal
5ce8f65a58 cloudapi: propagate operation/external id
Signed-off-by: Lukas Zapletal <lzap+git@redhat.com>
2024-06-25 13:58:53 +02:00
Lukas Zapletal
f3c0daebbf cmd/osbuild-composer: journald support 2024-06-25 13:58:53 +02:00
Sanne Raymaekers
2a621521a8 osbuildexecutor/aws.ec2: set hostname of executor via cloud-init
This way much more of the journal will be captured under the new
hostname.
2024-06-25 10:58:10 +02:00
Sanne Raymaekers
4853bf3ec0 Revert "osbuild-worker-executor: job-id in control.json as hostname"
This reverts commit fc1d1c3b8f.
2024-06-25 10:58:10 +02:00
Florian Schüller
55c5602f91 osbuild-worker-executor/main_test: use random port for tests
this for sure is racy but better than colliding with other tests
with a fixed port for sure
2024-06-24 09:18:44 +02:00
Michael Vogt
aa3d70a429 osbuildexecutor: tweak RunOSBuild() signature and use opts
Introduce a new OsbuildOpts struct to make the API a bit easier
to extend and use in the packages.

Also add a new `JobID` field in the `OsbuildOpts`.
2024-06-14 15:02:08 +02:00
Michael Vogt
fc1d1c3b8f osbuild-worker-executor: job-id in control.json as hostname
This commit adds support to set the hostname to the job-id that
is part of the control.json.
2024-06-14 15:02:08 +02:00
Michael Vogt
22769305d8 osbuild-worker-executor: fix tar warning and log unexpected output
This commit fixes a warning from tar that the archive cannot contain
itself. It also makes any tar output a warning (maybe even an error?)
as we do not expect anything from the tar command. The test is updated
to also check this.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
840f46b07f osbuild-worker-executor: expect osbuild-store as store directory 2024-06-12 11:36:30 +02:00
Sanne Raymaekers
a769d73745 osbuild-worker: print stacktrace in case osbuild job panics 2024-06-12 11:36:30 +02:00
Sanne Raymaekers
4df04643ab internal/osbuildexecutor/aws-ec2: use osbuild-worker-executor
Adds some unit tests as well.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
f18293c2f1 osbuild-worker-executor: adapt worker-executor to osbuild-composer 2024-06-12 11:36:30 +02:00
Michael Vogt
2704b18663 obuild-worker: extract workerClientErrorFrom() helper and add tests
Tiny commit to extract a helper from DepsolveJobImpl.Run() that
can then be unit tested.

This should help with https://github.com/osbuild/images/issues/727
2024-06-11 10:42:00 +02:00
Michael Vogt
a691df2353 osbuild-worker-executor: fix order of assert.Equal() in tests
The `assert.Equal()` expects that the "expected" value is put
first. Which is not what I'm used to. It's also slightly inconsistent
because `assert.EqualError()` expects the "actual" err first and
then the expected string. But this commit is not about ranting :)

This commit fixes the order in the tests assert.Equal() so that
mismatches actually are displayed correctly.
2024-06-07 08:17:32 +02:00
Michael Vogt
61bf0c3235 osbuild-worker: do not use error in clienterror.Error.Details
This is an alternative/complementary fix for PR#4137. It is very
simple so should be uncontroverisal.

It fixes an issue that @schuellerf discovered, i.e. that when an error
interface is passed into clienterrors.Error.Details the details get
lost because the json.Marshaler will not know how to handler an
error interface.

To find the problematic uses of `error` a custom vet checker was
build in https://github.com/mvo5/osbuild-cvet. With that the
result is:
```
$ go run github.com/mvo5/osbuild-cvet@latest ./...
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-depsolve.go:93:26: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:404:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:519:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:556:31: do not pass '[]error' to WorkerClientError() details, use []string instead
```
and once this commit is in no more errors.

Just like PR#4137 this is not perfect because it will not do a
recursive check for the passed argument.
2024-06-07 01:19:11 +02:00
Michael Vogt
8ebefbdbc9 main: rework the way the mock logger is passed
Pass the mock logger directly to `run()` instead of mocking
`logrus.New`. Doing the later leads to a data race when multiple
parallel tests modify the (global) `var logrusNew logrus.New`.

Thanks to Tomas Hozza for reporting.
2024-06-06 21:14:31 +02:00
Michael Vogt
95b4a9e250 osbuild-worker-executor: make test output silent again
Do not use the global logger but pass instead the locally created
logger. This means the test output is silent again.

Sadly using the global logger is difficult because it is a global
resource so replacing it in tests means all tests (that are
potentially run in parallel) will write to it which makes testing
specific log output hard.
2024-06-06 16:16:33 +02:00
Michael Vogt
e34728b466 osbuild-worker-executor: appease errcheck 2024-06-05 18:26:08 +02:00
Michael Vogt
138bc73e37 osbuild-worker-executor: appease gosec
Note that gosec IMHO is a bit silly here, the heuristics used are
note very good, i.e. the code is already validating the external
inputs and it's not clear to me that "filepath.Clean()" will help
but it seems to supress the error. I hope gosec provides value
in other places, here it seems to be adding work :/

I also excluded "gosec" from any _test.go files, I do not see
why we should gosec tests?
2024-06-05 18:26:08 +02:00
Sanne Raymaekers
b0543e89f4 osbuild-worker-executor: fix lint warnings/errors
The osbuild-composer linting found a bunch of issues that this
commit fixes.
2024-06-05 18:26:08 +02:00
Michael Vogt
cbb8d79baf c/osbuild-worker-executor: update to match new name
Update the imports/names to match the new name
"osbuild-worker-executor".
2024-06-05 18:26:08 +02:00
Michael Vogt
372d9f07dd cmd/osbuild-worker-executor: import verbatim from mvo5/oaas
This commit imports the repository https://github.com/mvo5/oaas
without keeping the history. The history is largely unimportant
and can be looked up in https://github.com/mvo5/oaas/commits/main
if needed.
2024-06-05 18:26:08 +02:00
Tomáš Hozza
b01b080565 Update 'rhel-9' distro alias to 'rhel-9.5'
New releases are landing in 9.5 already, so the on-prem version should
reflect that. The service can and does override this using a
configuration.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-04 13:03:37 +02:00
Tomáš Hozza
695febf39c Define distro alias for 'rhel-10'
Set it to RHEL-10.0.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-04 13:03:37 +02:00
Sanne Raymaekers
5a776c5b79 templates/openshift: split worker from composer maintenance 2024-04-25 17:32:21 +02:00
Sanne Raymaekers
0b277937dd worker/server: make worker timeout configurable 2024-04-19 19:56:25 +02:00
Sanne Raymaekers
c9eb60d674 osbuild-worker: fix nil pointer in depsolve job 2024-04-18 12:21:07 +02:00
Sanne Raymaekers
7684dbeedd osbuild-worker: fix mtls credentials injection in depsolve job 2024-04-17 20:27:06 +02:00
Sanne Raymaekers
05a1e8f054 worker: support a proxy for repository mtls configuration
Allows setting a proxy for the mtls settings for a specific
repository. This is useful when consuming content from the content
service in consoledot.
2024-04-17 16:17:57 +02:00
Sanne Raymaekers
d6cfd04704 cmd/osbuild-dnf-json-tests: check repo configs 2024-04-17 10:27:08 +02:00
Sanne Raymaekers
98e3892e23 cmd/osbuild-store-dump: pass repoconfigs when serializing manifests 2024-04-17 10:27:08 +02:00
Sanne Raymaekers
3094eb474d cmd/gen-manifests: pass repoconfigs when serializing manifests 2024-04-17 10:27:08 +02:00
Sanne Raymaekers
e0759e01af worker: save repoconfigs in depsolve result 2024-04-17 10:27:08 +02:00
Achilleas Koutsou
809f2544ad deps: update images to v0.54.0
Update images dependency and adjust import paths for distro package
changes.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2024-04-11 16:43:40 +02:00
Sanne Raymaekers
53f77368fd osbuild-worker: add support for mtls dnf repo secrets 2024-03-29 14:46:54 +01:00
Sanne Raymaekers
f311adf35d osbuild-jobsite-builder: disable http.Server timeouts
Let's just rely on the timeouts surrounding the whole step.
2024-03-21 17:08:07 +01:00
Sanne Raymaekers
d3074fc265 osbuild-jobsite-manager: clean up store archive after use 2024-03-21 15:20:52 +01:00
Sanne Raymaekers
bc17204001 osbuild-jobsite-manager: reseek when necessary 2024-03-21 15:20:52 +01:00
Sanne Raymaekers
c19e9fb3b9 osbuild-jobsite-manager: close writer before sending the store 2024-03-21 15:20:52 +01:00
Sanne Raymaekers
dabe63cb78 osbuild-jobsite: increase populate timeout
30 seconds can be a bit too little for the entire store.
2024-03-20 21:58:17 +01:00
Sanne Raymaekers
b4e361e093 osbuild-jobsite-builder: unpack source tarball in populate 2024-03-19 17:07:30 +01:00
Sanne Raymaekers
07598ea65c osbuild-jobsite-manager: pass store to builder 2024-03-19 17:07:30 +01:00
Brian C. Lane
01ba674cac cloudapi: Pass the RepoRegistry to the cloudapi Server 2024-03-11 03:04:54 -07:00
Brian C. Lane
b8967d53bc Move RepoRegistry setup into Composer
This is so that both the weldr and cloud api's can use it as the source
of their repositories.
2024-03-11 03:04:54 -07:00
Tomáš Hozza
e7743f17ec Worker: allow configuring executor CloudWatch group
We need the ability to use different CloudWatch group for the
osbuild-executor on Fedora workers in staging and production
environment.

Extend the worker confguration to allow configuring the CloudWatch group
name used by the osbuild-executor. Extend the secure instance code to
instruct cloud-init via user data to create /tmp/cloud_init_vars file
with the CloudWatch group name in the osbuild-executor instance, to make
it possible for the executor to configure its logging differently based
on the value.

Cover new changes by unit tests.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-03-08 13:13:44 +01:00