Commit graph

6622 commits

Author SHA1 Message Date
Alexander Todorov
6aa02bdd4d Start running tests on centos-stream-10 2024-08-12 08:39:05 +03:00
Alexander Todorov
39d91bcb4e There is no EPEL for EL10 yet so use a custom COPR repository 2024-08-12 08:39:05 +03:00
Florian Schüller
9006836afc logrus: add deployment channel as field to the logs 2024-08-07 12:32:57 +02:00
Gianluca Zuccarelli
2da3a73308 cloudapi: json tailoring options
Add support for json tailoring files in the cloudapi. Expand the tests
to check that the options returned are okay and that tailoring options
and json tailoring options can't be provided at the same time.
2024-08-07 12:06:28 +02:00
Gianluca Zuccarelli
16180e9c58 blueprint/customizations: json tailoring
Add support to the blueprints for json tailoring files
2024-08-07 12:06:28 +02:00
Gianluca Zuccarelli
4c225f7d66 cloudapi: openscap tailoring tests
Add an initial test for OpenSCAP tailoring customizations since we will
be expanding the tailoring options to support json tailoring.
2024-08-07 12:06:28 +02:00
Gianluca Zuccarelli
c9972f7da8 go.mod: update osbuild/images to v0.74.0 2024-08-07 12:06:28 +02:00
schutzbot
3789ff4ce8 Post release version bump
[skip ci]
2024-08-07 08:14:59 +00:00
Michael Vogt
86b1143923 osbuildexecutor: show full osbuild exector on json decode errors
This is a short term workaround to get better visibility why
the osbuild executor sometimes sends non-json data. When reading
the result from the executor the entire output is now read and
if the json parsing goes wrong it will use the entire body
in the error message for better debug visibility.
2024-08-05 14:51:40 +02:00
Florian Schüller
0a68fe3005 Makefile: implement helper to process OpenShift templates
just for manual checks if the template syntax is fine
and align with the github action to use the same code
2024-08-02 04:01:02 +02:00
Michael Vogt
1d0232ffc6 osbuild-worker: rework the workerClientErrorFrom() error
The workerClientErrorFrom() was returning an `*clienterrors.Error` and
an `error` (if something with the conversation goes wrong.

But the calling code was expecting that even if an `error` is returned
the `*clienterrors.Error` is still valid. The caller would then just
log the error. As returning a valid `value` even when there is an
`error` is an unexpected pattern this commit changes the code to
always return a `*clienterrors.Error` and log any issue via the
logger.
2024-08-01 17:25:16 +02:00
Sanne Raymaekers
dc389eaa71 cloud/awscloud: fix nil pointer dereference
When the cleanup function gets called, there's a chance the Instnace
field isn't populated yet, so store the instance ID separately and wait
for it to be terminated in case it's present.

The error would produce the following trace:
```
goroutine 1 [running]:
...
main.(*OSBuildJobImpl).Run.func1()
    osbuild/osbuild-composer/cmd/osbuild-worker/jobimpl-osbuild.go:404 +0xc5
panic({0x55e2a76a1e40?, 0x55e2a906d2f0?})
    /usr/lib/golang/src/runtime/panic.go:920 +0x270
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).deleteFleetIfExists(0xc000faa840, 0xc0012718c0)
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:441 +0x175
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).TerminateSecureInstance(0x55e2a90825e0?, 0x2?)
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:192 +0x1d
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).RunSecureInstance.func1()
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:75 +0x69
github.com/osbuild/osbuild-composer/internal/cloud/awscloud.(*AWS).RunSecureInstance(0xc000faa840, {0xc000afeade, 0x10}, {0x0, 0x0}, {0x0, 0x0}, {0xc001120f30, 0x24})
    osbuild/osbuild-composer/internal/cloud/awscloud/secure-instance.go:169 +0x12a7
...
```
2024-08-01 10:58:08 +02:00
Florian Schüller
54904d47da Change log_level for the service to json
This is to be inline with image-builder and to
enable decoding in splunk
2024-07-31 17:46:01 +02:00
Michael Vogt
573b349f16 clienterrors: rename WorkerClientError to clienterrors.New
The usual convention to create new object is to prefix `New*` so
this commit renames the `WorkerClientError`. Initially I thought
it would be `NewWorkerClientError()` but looking at the package
prefix it seems unneeded, i.e. `clienterrors.New()` already
provides enough context it seems and it's the only error we
construct.

We could consider renaming it to `clienterror` (singular) too
but that could be a followup.

I would also like to make `clienterror.Error` implement the
`error` interface but that should be a followup to make this
(mechanical) rename trivial to review.
2024-07-31 17:04:58 +02:00
Sanne Raymaekers
09445a1030 dbjobqueue: correct error wrapping
Preserve context.DeadlineExceeded errors through correct error
wrapping. This will reduce error-level logging noise in the worker.
2024-07-31 13:34:13 +02:00
Achilleas Koutsou
769b04b862 test/api: silent dump_db()
When cleaning up api tests, the build job results are dumped from the
database to store in build-result.txt and are printed to the runner log
at the same time.  This makes it very difficult to read results as the
database dump prints very long lines that flood the job log.  The result
isn't really readable (or useful) unless the file is downloaded
separately.

Silence the `tee` command so that the file is created but no output is
printed from the db dump.
2024-07-29 15:05:15 +02:00
dependabot[bot]
ca2c2dfa4f build(deps): bump the go-deps group across 1 directory with 8 updates
Bumps the go-deps group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [cloud.google.com/go/compute](https://github.com/googleapis/google-cloud-go) | `1.27.3` | `1.27.4` |
| [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) | `1.3.2` | `1.4.0` |
| [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) | `1.54.18` | `1.55.2` |
| [github.com/gophercloud/gophercloud](https://github.com/gophercloud/gophercloud) | `1.13.0` | `1.14.0` |
| [github.com/openshift-online/ocm-sdk-go](https://github.com/openshift-online/ocm-sdk-go) | `0.1.429` | `0.1.432` |
| [github.com/osbuild/images](https://github.com/osbuild/images) | `0.70.0` | `0.72.0` |
| [github.com/vmware/govmomi](https://github.com/vmware/govmomi) | `0.38.0` | `0.39.0` |



Updates `cloud.google.com/go/compute` from 1.27.3 to 1.27.4
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/compute/v1.27.3...compute/v1.27.4)

Updates `github.com/Azure/azure-sdk-for-go/sdk/storage/azblob` from 1.3.2 to 1.4.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/storage/azblob/v1.3.2...sdk/azcore/v1.4.0)

Updates `github.com/aws/aws-sdk-go` from 1.54.18 to 1.55.2
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.54.18...v1.55.2)

Updates `github.com/gophercloud/gophercloud` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/gophercloud/gophercloud/releases)
- [Changelog](https://github.com/gophercloud/gophercloud/blob/v1.14.0/CHANGELOG.md)
- [Commits](https://github.com/gophercloud/gophercloud/compare/v1.13.0...v1.14.0)

Updates `github.com/openshift-online/ocm-sdk-go` from 0.1.429 to 0.1.432
- [Release notes](https://github.com/openshift-online/ocm-sdk-go/releases)
- [Changelog](https://github.com/openshift-online/ocm-sdk-go/blob/main/CHANGES.md)
- [Commits](https://github.com/openshift-online/ocm-sdk-go/compare/v0.1.429...v0.1.432)

Updates `github.com/osbuild/images` from 0.70.0 to 0.72.0
- [Release notes](https://github.com/osbuild/images/releases)
- [Commits](https://github.com/osbuild/images/compare/v0.70.0...v0.72.0)

Updates `github.com/vmware/govmomi` from 0.38.0 to 0.39.0
- [Release notes](https://github.com/vmware/govmomi/releases)
- [Changelog](https://github.com/vmware/govmomi/blob/main/CHANGELOG.md)
- [Commits](https://github.com/vmware/govmomi/compare/v0.38.0...v0.39.0)

Updates `google.golang.org/api` from 0.188.0 to 0.189.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.188.0...v0.189.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/compute
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/gophercloud/gophercloud
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/openshift-online/ocm-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/osbuild/images
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/vmware/govmomi
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-25 15:00:50 +02:00
schutzbot
fd71c9cefa Post release version bump
[skip ci]
2024-07-24 08:14:26 +00:00
Tomáš Hozza
02c35060b9 define-compose-url.sh: support RHEL-10 composes
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
58e2303143 define-compose-url.sh: use download.devel.redhat.com
Use mirror-agnostic URL for composes.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
d0006b0134 define-compose-url.sh: drop RHEL-8 support
There are no more RHEL-8 nightly composes, thus we don't need the code
that defines the repo url for them.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
3e8dc4ca8c define-compose-url.sh: drop condition for RHEL-9.1
We are long time past RHEL-9.1 development and this is therefore not
needed any more.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
ef4bfb0bd5 CI: build c10s RPMs
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
ce1543b39e CI: build RHEL-10.0-nightly RPMs
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
61036b87bb mockbuild.sh: temporarily base missing el10 mock template on el9
There is no RHEL-10 mock template yet. Let's base it on RHEL-9 template
with a few tweaks.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
9c9a3b7937 mockbuild.sh: add workarounds for c10s and el10
Add workarounds needed for mockbuild.sh to work on c10s / el10, due to
non-existent EPEL-10 and issues on c10s.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
c9e9027faa Schutzfile: add RHEL-10.0 repo snapshots
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
ac2b09ffbe Schutzfile: add c10s repo snapshots
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
bb7e095aab Update terraform ref
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tomáš Hozza
d4bbae6f76 Schutzfile: update osbuild ref
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-23 12:14:13 +02:00
Tom Koscielniak
0be8764a79 Fix slack GA pipeline notification job failure 2024-07-23 12:12:36 +02:00
Andrea Waltlova
7c830e90b0 Add password to User schema
Make sure password is encrypted
Update tests

Signed-off-by: Andrea Waltlova <awaltlov@redhat.com>
2024-07-22 09:11:36 +02:00
Sanne Raymaekers
5d0e3ee85b schutzbot/terraform: bump sha
Includes more images with gitlab-runner and dependencies pre-installed.
2024-07-17 19:07:43 +02:00
Tomáš Hozza
c94b6ccde6 Templates: define 'rhel-10' distro alias
Define `rhel-10` distro alias in the OpenShift template. Even though the
same alias is defined in the default configuration, I think that it is
good to also include it in the template to not forget about it in the
future.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-17 11:02:41 +02:00
Tomáš Hozza
286236b698 Config: don't override undefined keys when loading from ENV
Composer can load configuration values defined as map from ENV.
Previously, when loading the configuration from ENV, the whole map would
get overridden, not just values defined in the ENV. This is however not
intended and not consistent with how loading configuration from file
works.

Adjust the configuration loading from ENV and adjust the unit test
accordingly.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-07-17 11:02:41 +02:00
Tom Koscielniak
62ae5aa4ea Create new nightly pipeline for GA/EUS images 2024-07-16 17:00:52 +02:00
schutzbot
0c2336c76b Post release version bump
[skip ci]
2024-07-11 20:58:02 +00:00
Michael Vogt
919b423953 osbuild-worker: tweak error to not include a \n for a failed stage
Small followup for
https://github.com/osbuild/osbuild-composer/pull/4113#discussion_r1670063775

Given that the failed stage is a relatively short string the `\n`
seems unneccessary and quotes are enough.
2024-07-11 09:33:40 +02:00
dependabot[bot]
4ba7085068 build(deps): bump the go-deps group across 1 directory with 10 updates
Bumps the go-deps group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [cloud.google.com/go/compute](https://github.com/googleapis/google-cloud-go) | `1.27.1` | `1.27.3` |
| [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) | `1.42.0` | `1.43.0` |
| [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) | `1.6.0` | `1.7.0` |
| [github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage](https://github.com/Azure/azure-sdk-for-go) | `1.5.0` | `1.6.0` |
| [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) | `1.54.10` | `1.54.18` |
| [github.com/gophercloud/gophercloud](https://github.com/gophercloud/gophercloud) | `1.12.0` | `1.13.0` |
| [github.com/openshift-online/ocm-sdk-go](https://github.com/openshift-online/ocm-sdk-go) | `0.1.425` | `0.1.429` |
| [github.com/osbuild/images](https://github.com/osbuild/images) | `0.69.0` | `0.70.0` |



Updates `cloud.google.com/go/compute` from 1.27.1 to 1.27.3
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.27.1...compute/v1.27.3)

Updates `cloud.google.com/go/storage` from 1.42.0 to 1.43.0
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.42.0...spanner/v1.43.0)

Updates `github.com/Azure/azure-sdk-for-go/sdk/azidentity` from 1.6.0 to 1.7.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.6.0...sdk/azcore/v1.7.0)

Updates `github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage` from 1.5.0 to 1.6.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.5.0...sdk/azcore/v1.6.0)

Updates `github.com/aws/aws-sdk-go` from 1.54.10 to 1.54.18
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.54.10...v1.54.18)

Updates `github.com/gophercloud/gophercloud` from 1.12.0 to 1.13.0
- [Release notes](https://github.com/gophercloud/gophercloud/releases)
- [Changelog](https://github.com/gophercloud/gophercloud/blob/v1.13.0/CHANGELOG.md)
- [Commits](https://github.com/gophercloud/gophercloud/compare/v1.12.0...v1.13.0)

Updates `github.com/openshift-online/ocm-sdk-go` from 0.1.425 to 0.1.429
- [Release notes](https://github.com/openshift-online/ocm-sdk-go/releases)
- [Changelog](https://github.com/openshift-online/ocm-sdk-go/blob/main/CHANGES.md)
- [Commits](https://github.com/openshift-online/ocm-sdk-go/compare/v0.1.425...v0.1.429)

Updates `github.com/osbuild/images` from 0.69.0 to 0.70.0
- [Release notes](https://github.com/osbuild/images/releases)
- [Commits](https://github.com/osbuild/images/compare/v0.69.0...v0.70.0)

Updates `golang.org/x/sys` from 0.21.0 to 0.22.0
- [Commits](https://github.com/golang/sys/compare/v0.21.0...v0.22.0)

Updates `google.golang.org/api` from 0.186.0 to 0.188.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.186.0...v0.188.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/compute
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/gophercloud/gophercloud
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/openshift-online/ocm-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/osbuild/images
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-11 09:31:08 +02:00
Achilleas Koutsou
c6d669f2cc repositories: update Fedora 40 gpg keys
The Fedora 40 gpg keys that we ship in our RPMs were outdated,
essentially making every build of Fedora 40 images impossible with
on-prem osbuild-composer.

Test repository keys were already correct.

Updated all Fedora 40 keys to https://github.com/rpm-software-management/distribution-gpg-keys/blob/main/keys/fedora/RPM-GPG-KEY-fedora-40-primary
2024-07-10 23:57:02 +02:00
schutzbot
dbd1165958 Post release version bump
[skip ci]
2024-07-10 08:16:25 +00:00
Florian Schüller
7cd5abd17c cmd/osbuild-worker/jobimpl-depsolve: show error.Reason only once
as now the .Reason is properly passed over - it was printed twice
2024-07-09 12:12:36 +02:00
Florian Schüller
b0a737421a osbuild-worker: improve error "reason" in case of stage failures 2024-07-09 12:12:36 +02:00
Nicolás M
80a782caac Added arch and storage_account, needed by CIV to deploy the resources without having to check for vhd uri regex.
The regex check is now not needed since we create other shared image metadata on the fly. Plus it was not a good solution for an upstream project, as the regex was only affecting our RH teams internally.
2024-07-08 21:48:26 +09:00
Florian Schüller
4e1ab4cbb5 Makefile: add comment where to see golint-golang mapping 2024-07-05 12:49:26 +02:00
Florian Schüller
1a24aa7d8a Makefile: bump golint version
followup of PR #4241
2024-07-05 12:49:26 +02:00
Achilleas Koutsou
a55bf9a177 test: update path to tailoring file in api tests
See https://github.com/osbuild/images/pull/752
and 7bd70d828e
2024-07-04 19:01:07 +02:00
Achilleas Koutsou
dab896dd64 weldr: update expected error messages
Some depsolve-related error messages in osbuild/images changed.  Update
the expected error messages in our tests.
2024-07-04 19:01:07 +02:00
Achilleas Koutsou
8ac80e8abc go.mod: update osbuild/images to v0.69.0 2024-07-04 19:01:07 +02:00
Achilleas Koutsou
1cc90c6a0b go.mod: update to go v1.21
Go 1.21 is included in all currently supported distro versions.
It is required by new updates to some of our dependencies, including
osbuild/images.
2024-07-04 19:01:07 +02:00