Commit graph

6583 commits

Author SHA1 Message Date
Michael Vogt
fc1d1c3b8f osbuild-worker-executor: job-id in control.json as hostname
This commit adds support to set the hostname to the job-id that
is part of the control.json.
2024-06-14 15:02:08 +02:00
Sanne Raymaekers
2e31ea50aa cloud/awscloud: use instance requirements when creating secure instance 2024-06-14 10:59:58 +02:00
dependabot[bot]
7ce03144da build(deps): bump golangci/golangci-lint-action from 5 to 6
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 5 to 6.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-13 15:44:16 +02:00
Sanne Raymaekers
4bb61da37e Revert "prometheus: active worker gauge"
This reverts commit 68bc8e0c88.
2024-06-12 17:20:01 +02:00
Sanne Raymaekers
55439fc6d3 templates/dashboards: remove active worker count
It's misleading since it counts the amount of workers that have
registered to the current composer pods, it doesn't actually keep track
of the active workers.

Remove it and keep the worker-api stats as a proxy for active workers.
2024-06-12 17:20:01 +02:00
Michael Vogt
fedbd72d57 osbuildexecutor: allow file type tar.TypeGNUSparse too
We need to allow files of type `tar.TypeGNUSparse` in the result
that we get from the osbuild-worker-executor too.
2024-06-12 16:31:44 +02:00
Sanne Raymaekers
971e1df148 internal/osbuildexecutor: verify expected input from handleBuild 2024-06-12 13:10:52 +02:00
Sanne Raymaekers
8a5f486715 osbuildexecutor: wait until worker-executor closes the connection
Otherwise the client will try to fetch the output archive before the
build output is archived on the worker-executor side.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
db43ec8e60 osbuildexecutor: move tests to separate _test package 2024-06-12 11:36:30 +02:00
Michael Vogt
984f51feb8 osbuildexecutor: add validateOutputArchive() and run before extract
The tar file from the `osbuild-worker-executor` is potentially
tainted. Ensure we validate and only extract if it harmless.
2024-06-12 11:36:30 +02:00
Michael Vogt
22769305d8 osbuild-worker-executor: fix tar warning and log unexpected output
This commit fixes a warning from tar that the archive cannot contain
itself. It also makes any tar output a warning (maybe even an error?)
as we do not expect anything from the tar command. The test is updated
to also check this.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
1f52150ff1 internal/osbulid-executor/aws-ec2: return more verbose errors
In RunOsbuild return more verbose errors where the error doesn't
originate from one of the local helper functions.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
840f46b07f osbuild-worker-executor: expect osbuild-store as store directory 2024-06-12 11:36:30 +02:00
Sanne Raymaekers
7d7bce76c0 templates/packer: use osbuild-worker-executor 2024-06-12 11:36:30 +02:00
Sanne Raymaekers
a769d73745 osbuild-worker: print stacktrace in case osbuild job panics 2024-06-12 11:36:30 +02:00
Sanne Raymaekers
4df04643ab internal/osbuildexecutor/aws-ec2: use osbuild-worker-executor
Adds some unit tests as well.
2024-06-12 11:36:30 +02:00
Sanne Raymaekers
9d62f01c7d test/worker-executor: switch to oaas or osbuild-worker-executor 2024-06-12 11:36:30 +02:00
Sanne Raymaekers
f18293c2f1 osbuild-worker-executor: adapt worker-executor to osbuild-composer 2024-06-12 11:36:30 +02:00
Tomáš Hozza
591ff8c46c Ship el8 repos without aux key on el10
The RHEL-8 auxiliary key uses SHA-1 in its signature. To enable RHEL-8
cross-builds on RHEL-10, we can't include the RHEL-8 auxiliary key in
the repo definitions.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-12 11:29:56 +02:00
Tomáš Hozza
8c54226793 Test manifests: update RH release key and regenerate manifests
Regenerate all manifests.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-12 11:29:56 +02:00
Tomáš Hozza
ab4dd4995c Remove F37 and F38 test manifest, because they are EOL
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-12 11:29:56 +02:00
Tomáš Hozza
ff433563c0 Remove c8s test manifest, because it is EOL
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-12 11:29:56 +02:00
Tomáš Hozza
30f221401a Repositories: use RH GPG release key signed using SHA256
The RPM GPG release key used by Red Hat to sign its content used to be
signed using SHA-1. SHA-1 is no longer accepted on RHEL-10 / c10s and as
a result, such key can't be imported during image build. The RH GPG
release key has been resigned using SHA256 some time ago. Let's use this
version of the key for all RHEL repositories.

The key is taken from:
https://access.redhat.com/security/team/key

Specifically:
https://access.redhat.com/security/data/fd431d51.txt

The second key (auxiliary key 3) was not changed.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-12 11:29:56 +02:00
schutzbot
08fe4ef4fb Post release version bump
[skip ci]
2024-06-12 08:15:23 +00:00
Sanne Raymaekers
7e89085808 templates/openshift/composer: remove maintenance cronjob
This is now deployed from a separate tempate.
2024-06-12 09:42:27 +02:00
Michael Vogt
2704b18663 obuild-worker: extract workerClientErrorFrom() helper and add tests
Tiny commit to extract a helper from DepsolveJobImpl.Run() that
can then be unit tested.

This should help with https://github.com/osbuild/images/issues/727
2024-06-11 10:42:00 +02:00
Achilleas Koutsou
7abcd279eb go.mod: bump osbuild/images to v0.65.0 2024-06-07 17:11:16 +02:00
Alexander Todorov
c80ca0a64d Temporary disable 2 regression tests on 9.5
Related: COMPOSER-2264
2024-06-07 10:31:54 +02:00
Alexander Todorov
ee1f70ec2e Add self-signed CA files to system trust store
in the hope that dnf in 9.5 will actually trust them and not fail
2024-06-07 10:31:54 +02:00
Alexander Todorov
3860f03d68 Update distro aliases in test 2024-06-07 10:31:54 +02:00
Alexander Todorov
e792bd3c13 Update 9.5 repositories in Schutzfile 2024-06-07 10:31:54 +02:00
Alexander Todorov
74421fa647 Generate manifests for RHEL 9.5 2024-06-07 10:31:54 +02:00
Alexander Todorov
537fe1a033 Define 9.5 repositories for gen-manifests 2024-06-07 10:31:54 +02:00
Alexander Todorov
8d0eb8b0cb Replace 9.4 conditions with 9.5 conditions inside test scripts 2024-06-07 10:31:54 +02:00
Alexander Todorov
4bee7cffb9 Replace 9.4 with 9.5 runners 2024-06-07 10:31:54 +02:00
Alexander Todorov
39370abce4 Define rhel-9.5 repos
Related: COMPOSER-2227
2024-06-07 10:31:54 +02:00
Michael Vogt
a691df2353 osbuild-worker-executor: fix order of assert.Equal() in tests
The `assert.Equal()` expects that the "expected" value is put
first. Which is not what I'm used to. It's also slightly inconsistent
because `assert.EqualError()` expects the "actual" err first and
then the expected string. But this commit is not about ranting :)

This commit fixes the order in the tests assert.Equal() so that
mismatches actually are displayed correctly.
2024-06-07 08:17:32 +02:00
Michael Vogt
61bf0c3235 osbuild-worker: do not use error in clienterror.Error.Details
This is an alternative/complementary fix for PR#4137. It is very
simple so should be uncontroverisal.

It fixes an issue that @schuellerf discovered, i.e. that when an error
interface is passed into clienterrors.Error.Details the details get
lost because the json.Marshaler will not know how to handler an
error interface.

To find the problematic uses of `error` a custom vet checker was
build in https://github.com/mvo5/osbuild-cvet. With that the
result is:
```
$ go run github.com/mvo5/osbuild-cvet@latest ./...
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-depsolve.go:93:26: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:404:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:519:31: do not pass 'error' to WorkerClientError() details, use error.Error() instead
/home/mvogt/devel/osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:556:31: do not pass '[]error' to WorkerClientError() details, use []string instead
```
and once this commit is in no more errors.

Just like PR#4137 this is not perfect because it will not do a
recursive check for the passed argument.
2024-06-07 01:19:11 +02:00
Michael Vogt
8ebefbdbc9 main: rework the way the mock logger is passed
Pass the mock logger directly to `run()` instead of mocking
`logrus.New`. Doing the later leads to a data race when multiple
parallel tests modify the (global) `var logrusNew logrus.New`.

Thanks to Tomas Hozza for reporting.
2024-06-06 21:14:31 +02:00
Michael Vogt
95b4a9e250 osbuild-worker-executor: make test output silent again
Do not use the global logger but pass instead the locally created
logger. This means the test output is silent again.

Sadly using the global logger is difficult because it is a global
resource so replacing it in tests means all tests (that are
potentially run in parallel) will write to it which makes testing
specific log output hard.
2024-06-06 16:16:33 +02:00
Michael Vogt
a634868793 all: run ./tools/prepare-source.sh 2024-06-05 18:26:08 +02:00
Michael Vogt
e34728b466 osbuild-worker-executor: appease errcheck 2024-06-05 18:26:08 +02:00
Michael Vogt
138bc73e37 osbuild-worker-executor: appease gosec
Note that gosec IMHO is a bit silly here, the heuristics used are
note very good, i.e. the code is already validating the external
inputs and it's not clear to me that "filepath.Clean()" will help
but it seems to supress the error. I hope gosec provides value
in other places, here it seems to be adding work :/

I also excluded "gosec" from any _test.go files, I do not see
why we should gosec tests?
2024-06-05 18:26:08 +02:00
Sanne Raymaekers
b0543e89f4 osbuild-worker-executor: fix lint warnings/errors
The osbuild-composer linting found a bunch of issues that this
commit fixes.
2024-06-05 18:26:08 +02:00
Sanne Raymaekers
01cfb93956 osbuild-worker-executor: integrate into build system/spec files
Add the osbuild-worker-executor to the build system and add it
to the spec file.
2024-06-05 18:26:08 +02:00
Michael Vogt
0a0e8e27ae vendor: add vendor/github.com/sirupsen/logrus/hooks/test
This is needed by the new `osbuild-worker-executor`.
2024-06-05 18:26:08 +02:00
Michael Vogt
cbb8d79baf c/osbuild-worker-executor: update to match new name
Update the imports/names to match the new name
"osbuild-worker-executor".
2024-06-05 18:26:08 +02:00
Michael Vogt
372d9f07dd cmd/osbuild-worker-executor: import verbatim from mvo5/oaas
This commit imports the repository https://github.com/mvo5/oaas
without keeping the history. The history is largely unimportant
and can be looked up in https://github.com/mvo5/oaas/commits/main
if needed.
2024-06-05 18:26:08 +02:00
Tomáš Hozza
fa416e4545 Test: re-enable snapshot URL check (COMPOSER-2263)
Enumerating snapshots now works after the cleanup of EOL releases.

Fixes https://issues.redhat.com/browse/COMPOSER-2263

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-04 17:17:19 +02:00
Tomáš Hozza
f3e83a3838 Tools/provision.sh: adjust conditional EPEL installation for el10
EPEL-10 can't be installed on any of c10s and el10. Previously, the
script would try to install EPEL repositories package on el10 and fail.

In addition, use our "workaround" repository with c10s builds of some
packages (such as koji) also on el10.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-06-04 17:17:19 +02:00