Commit graph

971 commits

Author SHA1 Message Date
Sanne Raymaekers
536b7d95c5 go.mod: update osbuild/images to v0.123.0
Includes modularity support.
2025-03-10 17:56:19 +01:00
Tomáš Hozza
985e582953 osbuild-composer/config: update distro aliases for RHEL 9 and 10
Bump the `rhel-9` and `rhel-10` distro aliases to the next
in-development releases, because we don't plan to land any new releases
in 9.6 or 10.0.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-02-28 09:35:23 +01:00
Brian C. Lane
0256e09031 lint: Clean up golangci-lint v1.60 complaints
This cleans up the linting results by adding checks for
integer underflow/overflow in several places, suppressing the error in
places where it has been checked, or fixing the types when possible.
2025-02-24 04:20:42 -08:00
Michael Vogt
35167ce83f many: update for new reporegistry.New() api (c.f. pr#1179)
This commit updates osbuild-composer for the new API in images
for the `reporegistry.New()`. The main incompatible change is
that the `/repositories` part is not longer automatically added
inside the library so we need to add it on the call-site.

This needs https://github.com/osbuild/images/pull/1179
2025-02-19 19:19:42 +01:00
Michael Vogt
90f069c2eb many: take repositories from the "images" library
This commit changes the spec file so that most of the repositories
are taken from the `images` library. See images PR#1112 for details.

Note that we still need the -no-auth-keys and the centos-stream
symlinks.
2025-02-13 14:00:45 +01:00
Tomáš Hozza
bb43578d92 osbuild-composer: bump rhel-9 distro alias to rhel-9.6
The alias in the built-in config is meant for on-prem and needs to be
set to the current in-development minor RHEL version.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-02-12 12:53:36 +01:00
Achilleas Koutsou
083d84f70c test: update all calls to image-info
Call the script without a path since it's now in the PATH.

Drop the SELinux labelling workaround (chcon) from filesystem.sh and
image_tests.sh.  The packaged tool is already labelled correctly.

Install osbuild-tools instead of osbuild-composer-tests in
regression-old-worker-new-composer.sh to get osbuild-image-info.
2025-01-31 08:22:29 +01:00
Brian C. Lane
3c2b415e53 jobimpl-depsolve: Always return result
Previously it would not return a result if there was an error. This adds
a deferred function that always returns the current contents of result,
and if there is an error it logs it.

Related: RHEL-60125
2025-01-30 08:00:12 +01:00
Brian C. Lane
d8df7e7cd4 worker: Add search job implementation to worker client
This is similar to the depsolve job, and it shares the solver (which
supports locking, as does DNF itself). This will allow searching for
specific package names, names with globs, or names as substrings of
other names using * as the wildcard.

Related: RHEL-60136
2025-01-30 08:00:12 +01:00
Brian C. Lane
84c0f79286 osbuild-worker: Add helper to setup Repo SSL
This same operation will be done in a few places, turn it into a
function.

Related: RHEL-60136
2025-01-30 08:00:12 +01:00
Michael Vogt
145f8e9f8a osbuild-store-dump: port to new images API 2025-01-29 18:03:11 +01:00
Michael Vogt
e91f4ba961 gen-manifests: port to new images API 2025-01-29 18:03:11 +01:00
Michael Vogt
40011e35a2 composer: use reporegistry.New() again (see PR#4378)
This commit changes cmd/composer to use the `reporegistry.New`
call again. This is a preparation for
https://github.com/osbuild/images/pull/1179

And undoes parts of https://github.com/osbuild/osbuild-composer/pull/4378
but that is no longer necessary because in
https://github.com/osbuild/images/pull/946 the error is now passed
on from `reporegistry.New()` in the same way as from `LoadRepositories()`.
2025-01-29 13:27:32 +01:00
Sanne Raymaekers
a44a499553 worker: support worker server target results 2025-01-24 15:26:15 +01:00
Brian C. Lane
73101d2ff2 Fix non-constant log strings
Newer versions of the go compiler (1.24 in this case) fail when running
go test during a mock rebuild of the srpm created by 'make srpm' on
Fedora 42.

Even though we currently don't support go1.24, fix these so they don't
become an issue when we do.
2025-01-21 16:51:20 -08:00
Michael Vogt
af0543d27c many: update images Manifest() API for PR#1107
This updates composer to use the updated API in images around the
seed handling for manifests, see images PR#1107 for details.

Note that this has no semantic changes yet. We could now simplfy
some things because images will auto-seed but that is for a followup.
2025-01-20 09:50:49 +01:00
Florian Schüller
153bcadf2d osbuild-service-maintenance/aws: merge errors
Collect and merge errors, instead of
nesting errors.
In this case we want to continue execution if only one
cleanup fails.
2024-12-10 11:43:51 +01:00
Florian Schüller
87ef1527fc osbuild-service-maintenance/aws: avoid error on empty list
Passing an empty list to `TerminateInstances` causes an
error message, which is not necessary, as there is
nothing to terminate.
2024-12-10 11:43:51 +01:00
Florian Schüller
24e256c225 osbuild-service-maintenance: add test for allTerminated
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
65b7ee65b2 osbuild-service-maintenance: implement removal of launch templates
Launch templates of instances that are terminated should be removed.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
a96ea533c0 osbuild-service-maintenance: implement removal of security groups
Security groups of instances that are terminated should be removed.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
7ebe266d3c osbuild-service-maintenance: implement removal on invalid parent
Add a safeguard to ensure secure instances without valid
parent instances are terminated, as they are unnecessary to retain.
Typically, the parent does not exist if the secure instance is
older than 2 hours, but this check provides additional validation.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
a7119a4d0f osbuild-service-maintenance/aws: support aws credential file
Support running the maintenance locally with a valid
`~/aws/credentials` file. HMS-3632
2024-12-10 11:43:51 +01:00
Tomáš Hozza
580366d1f3 osbuild-dnf-json-tests: don't set OSTree options for non-OSTree images
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-12-09 09:46:54 +01:00
Lukas Zapletal
e7a7cda3bc cmd: extra env logging for osbuild worker 2024-11-29 11:57:11 +01:00
Michael Vogt
bc7b8355bf worker: report cashes directly to logrus
This is a bit of an RFC commit, I noticed that when we discussed
a crash from the worker we looked at individual message from
syslog/journald for the stacktrace deatils. I was wondering if
having a more direct crash report would be more useful? We can
of course also add more logrus features to flag those with tags
like "crash" or something (I did not do that in this PR, I don't
know much about the operational side, sorry).
2024-11-25 12:02:05 +01:00
Sanne Raymaekers
f672610509 cmd/osbuild-worker: specify hyper v gen for azure images 2024-11-21 11:22:20 +01:00
Lukas Zapletal
03e74e77b2 worker: log proxy setting 2024-11-18 19:33:19 +01:00
Lukas Zapletal
86f903339a worker: parse ostree MTLS proxy early 2024-11-15 10:16:26 +01:00
Lukas Zapletal
2a5d25d9c0 worker: check MTLS config for ostree 2024-11-12 12:12:52 +01:00
Sanne Raymaekers
056b3c5ea6 jobqueue: return if a job was requeued or not 2024-11-07 17:18:48 +01:00
Lukas Zapletal
64f479092d osbuild-worker: use the new ostree resolver API 2024-11-07 16:17:56 +01:00
Florian Schüller
00d3f07d08 Makefile: implement make db-tests
enables the option to run the DB tests locally
that are executed in the github actions
2024-11-06 15:16:42 +01:00
Achilleas Koutsou
af48971981 osbuild-composer: fail weldr init when repos are nil
If weldr tries to initialise when there are no repositories set and
ignore_missing_repos is enabled, return with an error.
2024-11-05 08:21:42 +01:00
Achilleas Koutsou
51287ea57e osbuild-composer/config: new option: ignore_missing_repos
osbuild/images added an error type that's returned when the reporegistry
loader doesn't find any repository configurations to load [1].  This
lets callers decide whether to stop or continue execution based on
whether repository configurations are required.

A new top-level configuration option is added for osbuild-composer that
makes it possible to start the service without having static rpm
repositories configured.  This is useful in certain (SaaS) modes where
build requests specify their own repository configurations.
2024-11-05 08:21:42 +01:00
Sanne Raymaekers
c1b67440c4 cmd/osbuild-service-maintenance: respect dry run
Respect dry run when terminating leftover SIs.
2024-10-28 10:59:25 +01:00
Lukas Zapletal
350ad58c31 worker: use the new resolver API 2024-10-24 11:53:04 +02:00
Sanne Raymaekers
661f39cbb9 cmd/osbuild-service-maintenance: add test for filtering SIs 2024-10-23 10:32:57 +02:00
Sanne Raymaekers
04a5ca6965 cmd/osbuild-service-maintenance: clean up secure instances
Now and then there are leftover secure instances, probably when worker
instances get terminated during builds, this is possible in ASGs. 2
hours as a cutoff should be enough, since the build times out after 60
minutes, and fetching the output archive after 30 minutes, so that
leaves 30 minutes for booting and connection.
2024-10-23 10:32:57 +02:00
Tomáš Hozza
7437770352 composer: don't create RepoRegistry using reporegistry.New()
The `reporegistry.New()` has been enhanced to return an error, in case
there were no repositories loaded. This was to fix the situation in many
unit tests, which were previously not loading any repositories and
silently not running any tests.

This however broke our SaaS deployment, where we actually do not
configure any repositories on the filesystem. As a result,
osbuild-composer started to fail on it.

Workaround this situation in osbuild-composer by reverting to the old
behavior by loading the repo configs separately and then using the
loaded repos (which could be empty map) to initialize the RepoRegistry.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-09-23 18:51:39 +02:00
Tomáš Hozza
71a12742d4 Worker/osbuild/koji: upload SBOM documents
Extend the Koji target handling in the osbuild job, to also upload SBOM
documents attached to the related depsolve job result.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-09-20 17:02:09 +02:00
Tomáš Hozza
1c7462b275 Worker/koji-finalize: import uploaded SBOM documents
If the Koji target result contains information about any uploaded SBOM
documents, import them to Koji as part of the finalize task.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-09-20 17:02:09 +02:00
Tomáš Hozza
4779e90e17 Worker/depsolve: add support for SBOM
Add support to the `DepsolveJob` for requesting SBOM documents and
returning the results from the job.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-09-20 17:02:09 +02:00
Tomáš Hozza
7bdd036395 Update osbuild/images to v0.88.0
Adjust all paces that call `Solver.Depsolve()`, to cope with the changes
that enabled SBOM support.

Fix loading of testing repositories in the CloudAPI unit tests.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-09-20 17:02:09 +02:00
Sanne Raymaekers
22a0452ea9 osbuild-worker: handle error wrapping from dnfjson package
osbuild/images#751 wrapped the errors in the images/dnfjson package to
provide more details, the depsolve job should take this into account to
map the dnfjson error to the correct worker client error.

This caused user input errors errors to be misclassified as internal
errors, triggering depsolve job failure alerts.
2024-09-02 14:39:03 +02:00
Tomáš Hozza
d7e59e6eec Worker: move GCE image guest OS features to upload target options
Previously, the worker was determining the GCE image guest OS Features
on its own, based on the OS name. This caused problems, in case the
osbuild-composer was of a newer version than the worker.

Example:
osbuild-composer contained support for c10s GCE image type and its
implementation also contained the proper guest OS Features list for it.
However, when the worker got the osbuild job, it built it and tried to
fetch the guest OS Features for the distro. Since its implementation was
too old, it didn't contain the code that added the actual support for
c10s GCE images and got no guest OS features list (which is the default
for unsupported distros). The image was successfully uploaded and
shared, but it does not boot in GCP, because it does not know that it
should use UEFI to boot it.

This behavior could be considered a bug. The worker should be dumb. It
should not be making decisions about the image features, but instead it
should take them from the upload target options. And composer should be
the authoritative source of truth for this. Because otherwise, we
basically have two components that need to be updated in sync to add
support for GCE images on a new distro.

Move the GCE image guest OS features to the GCP upload target options.
The worker will just take what is specified there and use it when
importing the image to GCP. As a compatibility layer for the case when
the composer would be older than the worker (unlikely, but still),
worker will try to determine the image guest OS features in case the
list in the upload target options is empty.

Extend the GCP functional tests to check that the imported image has at
least some guest OS features set.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-29 17:37:48 +02:00
Florian Schüller
8d24dcfbde osbuild-worker: add CHANNEL to worker logs
aka "the deployment channel" like "staging" or "production"
2024-08-28 16:41:07 +02:00
Florian Schüller
a4068b328d splunk_logger: move environment hook to splunk_logger pt2
for reusability also in image-builder
2024-08-28 16:41:07 +02:00
Sanne Raymaekers
54820a88df osbuild-worker: switch to aws sdk v2 for errors in ami copy jobs 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
2624516f1a osbuild-worker: use aws sdk v2 for asg scale-in protection 2024-08-20 15:32:40 +02:00