Commit graph

989 commits

Author SHA1 Message Date
Sanne Raymaekers
d7686244cd cmd/osbuild-composer-dbjobqueue-tests: add migration test
Migration 9 alters the result column in the jobs table is relied on for
compose statuses. Because it has to be kept consistent across
migrations, add a test to verify this.

As a side effect, the test itself handles the migration now, so remove
that part from the tests GHA.
2025-07-25 13:10:10 +02:00
Michael Vogt
57813199cf osbuild-dnf-json-tests: use distrofactory to get centos-9
The test was importing `distro/rhel/rhel9` directly instead
of using the `distrofactory`. This is problematic as the go
module for `rhel9` is going away. This commit fixes this
by going via the `distrofactory` which will keep working.
2025-07-23 12:43:12 +02:00
Tomáš Hozza
3e3f9a0789 Many: move to koji upload implementation from osbuild/images
Delete the `internal/upload/koji` package and replace it with
`pkg/upload/koji` package provided by `osbuild/images`.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-07-14 17:20:02 +02:00
Tomáš Hozza
97c3460a94 cmd: delete osbuild-koji
Delete the osbuild-koji tools, which is not used by any code or test
case. This also allows to delete the koji.NewFromPlain() function. This
is to minimize the set of exported functions by the koji package, before
moving it to osbuild/images repository.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-07-09 22:51:53 +02:00
Tomáš Hozza
651946ae54 upload/koji: don't depend on upload target osbuild artifact structure
Modify the Koji image extra metadata to not depend on the upload target
data structure for the OSBuild Artifact. This is the last dependency on
the internal osbuild-composer package, allowing the move of the Koji
upload code to the osbuild/images repository.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-07-09 22:51:53 +02:00
Tomáš Hozza
2ceffb3796 upload/koji: don't depend on upload target results data structures
Modify the Koji image extra metadata to not depend on the upload target
data structures for upload results. The target results are supposed to
be specific to the uploader implementation, which will eventually
change. Moreover, the definition is internal to osbuild-composer, so
this would create a problem once the Koji upload implementation is moved
to osbuild/images.

The Koji upload implementation does not really care about the exact
structure of the upload results. It is just a list of JSON objects.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-07-09 22:51:53 +02:00
Tomáš Hozza
0bafc0915d upload/koji: don't depend on rpmmd.RPM for build metadata
The use of rpmmd.RPM is exclusive to the Koji upload implementation.
Consolidate the metadata structure for RPMs into the koji package
codebase, together with any required functions for converting osbuild
stage metadata. The respective code in osbuild/images will be eventually
removed.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-07-09 22:51:53 +02:00
Tomáš Hozza
ca1e1dce36 Many: factor out logger implementation from upload/koji
The upload/koji package functions were creating a logger and then were
using it. This is not ideal for a library implementation.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-07-09 22:51:53 +02:00
Lukas Zapletal
1cde7e341b common: fix unclosed logrus logging pipes 2025-07-08 11:38:27 +02:00
Sanne Raymaekers
7dea1bcd01 many: remove jobsite code
This was replaced by `cmd/osbuild-worker-executor`.
2025-06-25 03:01:59 +02:00
Lukas Zapletal
f2bf47814f deps: upgrade go-systemd from v1 to v22 2025-06-16 17:46:45 +02:00
Michael Vogt
acec29dea2 osbuild-store-dump: use distrofactory instead of importing fedora
This commit removes the direct use of `pkg/distro/fedora` and
uses `distrofactory` instead. This is the more generic interface,
it also helps with PR#1563 as `pkg/distro/fedora` goes away.

Note that I did not test this but @bcl was kind enough to
mention how this is used:
```
The output from this is used in Test_upgrade in
internal/store/json_test.go, the files there are manually
created when we change the format of the store and add a new
parser to internal/store/json.go -- which hasn't happened in
a long time :)
```
I updated the cmd/README with a snippet of the above.
2025-06-04 12:41:39 +02:00
Lukas Zapletal
0e5d8a94f2 cmd: use log in service-maintenance 2025-05-12 15:20:16 +02:00
Lukas Zapletal
46f0a71053 cmd: use log in jobsite 2025-05-12 15:20:16 +02:00
Lukas Zapletal
71117f3ccc cmd: use log in koji 2025-05-12 15:20:16 +02:00
Lukas Zapletal
650e7b64c2 cmd: use log in upload-gcp 2025-05-12 15:20:16 +02:00
Lukas Zapletal
b87d3d78ec docs: add README to cmd/ 2025-05-12 15:20:16 +02:00
Achilleas Koutsou
cf956ff5a6 Delete internal/blueprint/ and import from osbuild/blueprint
Import osbuild/blueprint v1.6.0
2025-04-17 11:17:58 +02:00
Sanne Raymaekers
536b7d95c5 go.mod: update osbuild/images to v0.123.0
Includes modularity support.
2025-03-10 17:56:19 +01:00
Tomáš Hozza
985e582953 osbuild-composer/config: update distro aliases for RHEL 9 and 10
Bump the `rhel-9` and `rhel-10` distro aliases to the next
in-development releases, because we don't plan to land any new releases
in 9.6 or 10.0.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-02-28 09:35:23 +01:00
Brian C. Lane
0256e09031 lint: Clean up golangci-lint v1.60 complaints
This cleans up the linting results by adding checks for
integer underflow/overflow in several places, suppressing the error in
places where it has been checked, or fixing the types when possible.
2025-02-24 04:20:42 -08:00
Michael Vogt
35167ce83f many: update for new reporegistry.New() api (c.f. pr#1179)
This commit updates osbuild-composer for the new API in images
for the `reporegistry.New()`. The main incompatible change is
that the `/repositories` part is not longer automatically added
inside the library so we need to add it on the call-site.

This needs https://github.com/osbuild/images/pull/1179
2025-02-19 19:19:42 +01:00
Michael Vogt
90f069c2eb many: take repositories from the "images" library
This commit changes the spec file so that most of the repositories
are taken from the `images` library. See images PR#1112 for details.

Note that we still need the -no-auth-keys and the centos-stream
symlinks.
2025-02-13 14:00:45 +01:00
Tomáš Hozza
bb43578d92 osbuild-composer: bump rhel-9 distro alias to rhel-9.6
The alias in the built-in config is meant for on-prem and needs to be
set to the current in-development minor RHEL version.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-02-12 12:53:36 +01:00
Achilleas Koutsou
083d84f70c test: update all calls to image-info
Call the script without a path since it's now in the PATH.

Drop the SELinux labelling workaround (chcon) from filesystem.sh and
image_tests.sh.  The packaged tool is already labelled correctly.

Install osbuild-tools instead of osbuild-composer-tests in
regression-old-worker-new-composer.sh to get osbuild-image-info.
2025-01-31 08:22:29 +01:00
Brian C. Lane
3c2b415e53 jobimpl-depsolve: Always return result
Previously it would not return a result if there was an error. This adds
a deferred function that always returns the current contents of result,
and if there is an error it logs it.

Related: RHEL-60125
2025-01-30 08:00:12 +01:00
Brian C. Lane
d8df7e7cd4 worker: Add search job implementation to worker client
This is similar to the depsolve job, and it shares the solver (which
supports locking, as does DNF itself). This will allow searching for
specific package names, names with globs, or names as substrings of
other names using * as the wildcard.

Related: RHEL-60136
2025-01-30 08:00:12 +01:00
Brian C. Lane
84c0f79286 osbuild-worker: Add helper to setup Repo SSL
This same operation will be done in a few places, turn it into a
function.

Related: RHEL-60136
2025-01-30 08:00:12 +01:00
Michael Vogt
145f8e9f8a osbuild-store-dump: port to new images API 2025-01-29 18:03:11 +01:00
Michael Vogt
e91f4ba961 gen-manifests: port to new images API 2025-01-29 18:03:11 +01:00
Michael Vogt
40011e35a2 composer: use reporegistry.New() again (see PR#4378)
This commit changes cmd/composer to use the `reporegistry.New`
call again. This is a preparation for
https://github.com/osbuild/images/pull/1179

And undoes parts of https://github.com/osbuild/osbuild-composer/pull/4378
but that is no longer necessary because in
https://github.com/osbuild/images/pull/946 the error is now passed
on from `reporegistry.New()` in the same way as from `LoadRepositories()`.
2025-01-29 13:27:32 +01:00
Sanne Raymaekers
a44a499553 worker: support worker server target results 2025-01-24 15:26:15 +01:00
Brian C. Lane
73101d2ff2 Fix non-constant log strings
Newer versions of the go compiler (1.24 in this case) fail when running
go test during a mock rebuild of the srpm created by 'make srpm' on
Fedora 42.

Even though we currently don't support go1.24, fix these so they don't
become an issue when we do.
2025-01-21 16:51:20 -08:00
Michael Vogt
af0543d27c many: update images Manifest() API for PR#1107
This updates composer to use the updated API in images around the
seed handling for manifests, see images PR#1107 for details.

Note that this has no semantic changes yet. We could now simplfy
some things because images will auto-seed but that is for a followup.
2025-01-20 09:50:49 +01:00
Florian Schüller
153bcadf2d osbuild-service-maintenance/aws: merge errors
Collect and merge errors, instead of
nesting errors.
In this case we want to continue execution if only one
cleanup fails.
2024-12-10 11:43:51 +01:00
Florian Schüller
87ef1527fc osbuild-service-maintenance/aws: avoid error on empty list
Passing an empty list to `TerminateInstances` causes an
error message, which is not necessary, as there is
nothing to terminate.
2024-12-10 11:43:51 +01:00
Florian Schüller
24e256c225 osbuild-service-maintenance: add test for allTerminated
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
65b7ee65b2 osbuild-service-maintenance: implement removal of launch templates
Launch templates of instances that are terminated should be removed.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
a96ea533c0 osbuild-service-maintenance: implement removal of security groups
Security groups of instances that are terminated should be removed.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
7ebe266d3c osbuild-service-maintenance: implement removal on invalid parent
Add a safeguard to ensure secure instances without valid
parent instances are terminated, as they are unnecessary to retain.
Typically, the parent does not exist if the secure instance is
older than 2 hours, but this check provides additional validation.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
a7119a4d0f osbuild-service-maintenance/aws: support aws credential file
Support running the maintenance locally with a valid
`~/aws/credentials` file. HMS-3632
2024-12-10 11:43:51 +01:00
Tomáš Hozza
580366d1f3 osbuild-dnf-json-tests: don't set OSTree options for non-OSTree images
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-12-09 09:46:54 +01:00
Lukas Zapletal
e7a7cda3bc cmd: extra env logging for osbuild worker 2024-11-29 11:57:11 +01:00
Michael Vogt
bc7b8355bf worker: report cashes directly to logrus
This is a bit of an RFC commit, I noticed that when we discussed
a crash from the worker we looked at individual message from
syslog/journald for the stacktrace deatils. I was wondering if
having a more direct crash report would be more useful? We can
of course also add more logrus features to flag those with tags
like "crash" or something (I did not do that in this PR, I don't
know much about the operational side, sorry).
2024-11-25 12:02:05 +01:00
Sanne Raymaekers
f672610509 cmd/osbuild-worker: specify hyper v gen for azure images 2024-11-21 11:22:20 +01:00
Lukas Zapletal
03e74e77b2 worker: log proxy setting 2024-11-18 19:33:19 +01:00
Lukas Zapletal
86f903339a worker: parse ostree MTLS proxy early 2024-11-15 10:16:26 +01:00
Lukas Zapletal
2a5d25d9c0 worker: check MTLS config for ostree 2024-11-12 12:12:52 +01:00
Sanne Raymaekers
056b3c5ea6 jobqueue: return if a job was requeued or not 2024-11-07 17:18:48 +01:00
Lukas Zapletal
64f479092d osbuild-worker: use the new ostree resolver API 2024-11-07 16:17:56 +01:00