Commit graph

6726 commits

Author SHA1 Message Date
Sanne Raymaekers
0273908fc7 tools/build-rpms: fix ec2 client initialisation
Fixes:
```
Error: Create EC2 instances  failed: 'ec2.ServiceResource' object has no attribute 'describe_images'
Traceback (most recent call last):
  File "/osbuild-composer/tools/build-rpms.py", line 218, in <module>
    stage_generate_rpms(cleanup_actions, args)
  File "/osbuild-composer/tools/build-rpms.py", line 175, in stage_generate_rpms
    create_ec2_instances, cleanup_actions, args, keyname)
  File "/osbuild-composer/tools/build-rpms.py", line 66, in stage
    ret = fun(*args)
  File "/osbuild-composer/tools/build-rpms.py", line 109, in create_ec2_instances
    img = ec2.describe_images(ImageIds=[arch_info[a]["ImageId"]])
AttributeError: 'ec2.ServiceResource' object has no attribute 'describe_images'
```
2024-09-11 17:31:13 +02:00
Sanne Raymaekers
72ed7db62d tools/build-rpms: increase rpm builder instance disk size
The instance created for rpm builds is running out of space on the
default 10G disk, let's double it.
2024-09-11 15:51:04 +02:00
Michael Vogt
3df26ed79c osbuild-worker: fix "crashing" on worker registration issues
When the osbuild worker cannot register itself with the server
on startup the worker will "crash". This is inconsistent with the
existing behavior in `workerHeartbeat()` which deals with connectivity
or other server issue gracefully and retries periodically.

To unify the behavior this commit changes the behavior and only
issues a `logrus.Warnf` instead of the previous `Falalf` when
the registration fails.

Co-authored-by: Florian Schüller <florian.schueller@redhat.com>
2024-09-10 16:19:47 +02:00
Jakub Rusz
751ad6af69 Update Fedora-40 repositories to branched ones 2024-09-10 12:35:19 +02:00
Sanne Raymaekers
d6031ae87a upload/azure: turn off public access on storage accounts
Users might have compliance policies on their azure accounts which
forbid public access on storage accounts.
2024-09-09 12:52:14 +02:00
Jakub Rusz
758982907a tests/cases: Rename upgrade test
And schedule on rhel-10
2024-09-06 14:38:57 +02:00
Jakub Rusz
15c8e83b94 test/cases: adapt upgrade to support rhel9to10 2024-09-06 14:38:57 +02:00
Florian Schüller
bb53f4833f internal/worker/client.go: refactor reading worker ID
Adds a helper function to the worker client instead of
redeclaring the same inline function.
2024-09-06 12:43:05 +02:00
Sanne Raymaekers
a6b87b58f2 test/api: check openscap facts when appropriate 2024-09-06 11:39:35 +02:00
Sanne Raymaekers
b21f24e751 cloudapi/v2: populate openscap related rhsm facts
Also adds the policy id to the blueprint, this doesn't have any effect
on the openscap step, it just puts in place the rhsm fact so instances
registered to insights will appear under that policy.
2024-09-06 11:39:35 +02:00
Sanne Raymaekers
387f971bf0 go.mod: update to osbuild/images@v0.83.0 2024-09-06 11:39:35 +02:00
schutzbot
5b4bbf2e87 Post release version bump
[skip ci]
2024-09-04 08:14:08 +00:00
Tomáš Hozza
d6fd6a9be5 Update osbuild/images to v0.82.0
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-09-03 17:31:21 +02:00
Sanne Raymaekers
6563e98c94 tools/appsre-ansible/rpmbuild: enable cdn repositories
Subscription manager should be configured to manage repositories, and by
disabling rhui-client-config-server-9 rhui repositories don't get
(re-)enabled after updates.
2024-09-03 15:33:48 +02:00
Sanne Raymaekers
8e1d053797 tools/build-rpms: use rhel 9.4 instances as a baseline 2024-09-03 15:33:48 +02:00
dependabot[bot]
76b224c6a9 build(deps): bump the go-deps group across 1 directory with 2 updates
Bumps the go-deps group with 1 update in the / directory: [github.com/osbuild/images](https://github.com/osbuild/images).


Updates `github.com/osbuild/images` from 0.80.0 to 0.81.0
- [Release notes](https://github.com/osbuild/images/releases)
- [Commits](https://github.com/osbuild/images/compare/v0.80.0...v0.81.0)

Updates `google.golang.org/api` from 0.194.0 to 0.195.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.194.0...v0.195.0)

---
updated-dependencies:
- dependency-name: github.com/osbuild/images
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-03 10:26:20 +02:00
Sanne Raymaekers
22a0452ea9 osbuild-worker: handle error wrapping from dnfjson package
osbuild/images#751 wrapped the errors in the images/dnfjson package to
provide more details, the depsolve job should take this into account to
map the dnfjson error to the correct worker client error.

This caused user input errors errors to be misclassified as internal
errors, triggering depsolve job failure alerts.
2024-09-02 14:39:03 +02:00
dependabot[bot]
6bb45ef9d1 build(deps): bump the go-deps group across 1 directory with 13 updates
Bumps the go-deps group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.30.3` | `1.30.4` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.27.9` | `1.27.31` |
| [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.16.9` | `1.17.16` |
| [github.com/aws/aws-sdk-go-v2/service/autoscaling](https://github.com/aws/aws-sdk-go-v2) | `1.43.3` | `1.43.5` |
| [github.com/aws/aws-sdk-go-v2/service/ec2](https://github.com/aws/aws-sdk-go-v2) | `1.173.0` | `1.177.0` |
| [github.com/openshift-online/ocm-sdk-go](https://github.com/openshift-online/ocm-sdk-go) | `0.1.432` | `0.1.438` |
| [github.com/osbuild/images](https://github.com/osbuild/images) | `0.79.0` | `0.80.0` |
| [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) | `1.19.1` | `1.20.2` |



Updates `github.com/aws/aws-sdk-go-v2` from 1.30.3 to 1.30.4
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.30.3...v1.30.4)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.27.9 to 1.27.31
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.9...config/v1.27.31)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.17.9 to 1.17.30
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.17.9...credentials/v1.17.30)

Updates `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` from 1.16.0 to 1.16.12
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.16.0...v1.16.12)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.16.9 to 1.17.16
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.16.9...credentials/v1.17.16)

Updates `github.com/aws/aws-sdk-go-v2/service/autoscaling` from 1.43.3 to 1.43.5
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/fsx/v1.43.3...service/fsx/v1.43.5)

Updates `github.com/aws/aws-sdk-go-v2/service/ec2` from 1.173.0 to 1.177.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/ec2/v1.173.0...service/ec2/v1.177.0)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.51.4 to 1.61.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.51.4...service/s3/v1.61.0)

Updates `github.com/aws/smithy-go` from 1.20.3 to 1.20.4
- [Release notes](https://github.com/aws/smithy-go/releases)
- [Changelog](https://github.com/aws/smithy-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/smithy-go/compare/v1.20.3...v1.20.4)

Updates `github.com/openshift-online/ocm-sdk-go` from 0.1.432 to 0.1.438
- [Release notes](https://github.com/openshift-online/ocm-sdk-go/releases)
- [Changelog](https://github.com/openshift-online/ocm-sdk-go/blob/main/CHANGES.md)
- [Commits](https://github.com/openshift-online/ocm-sdk-go/compare/v0.1.432...v0.1.438)

Updates `github.com/osbuild/images` from 0.79.0 to 0.80.0
- [Release notes](https://github.com/osbuild/images/releases)
- [Commits](https://github.com/osbuild/images/compare/v0.79.0...v0.80.0)

Updates `github.com/prometheus/client_golang` from 1.19.1 to 1.20.2
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.19.1...v1.20.2)

Updates `google.golang.org/api` from 0.193.0 to 0.194.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.193.0...v0.194.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/ec2/imds
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go-v2/service/autoscaling
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ec2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/aws/smithy-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/openshift-online/ocm-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/osbuild/images
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-30 14:19:01 +02:00
Sanne Raymaekers
30ea05dd43 tools/build-rpms: increase size of instances
Mockbuilds are currently timing out after #4287, maybe due to memory
constraints.
2024-08-30 10:49:56 +02:00
Tomáš Hozza
d7e59e6eec Worker: move GCE image guest OS features to upload target options
Previously, the worker was determining the GCE image guest OS Features
on its own, based on the OS name. This caused problems, in case the
osbuild-composer was of a newer version than the worker.

Example:
osbuild-composer contained support for c10s GCE image type and its
implementation also contained the proper guest OS Features list for it.
However, when the worker got the osbuild job, it built it and tried to
fetch the guest OS Features for the distro. Since its implementation was
too old, it didn't contain the code that added the actual support for
c10s GCE images and got no guest OS features list (which is the default
for unsupported distros). The image was successfully uploaded and
shared, but it does not boot in GCP, because it does not know that it
should use UEFI to boot it.

This behavior could be considered a bug. The worker should be dumb. It
should not be making decisions about the image features, but instead it
should take them from the upload target options. And composer should be
the authoritative source of truth for this. Because otherwise, we
basically have two components that need to be updated in sync to add
support for GCE images on a new distro.

Move the GCE image guest OS features to the GCP upload target options.
The worker will just take what is specified there and use it when
importing the image to GCP. As a compatibility layer for the case when
the composer would be older than the worker (unlikely, but still),
worker will try to determine the image guest OS features in case the
list in the upload target options is empty.

Extend the GCP functional tests to check that the imported image has at
least some guest OS features set.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-29 17:37:48 +02:00
Michael Vogt
2442baefde worker: move api.BasePath setup to the start of the funcs
I find it slightly eaiser to read this code when
`api.BasePath = conf.BasePath` is right at the top as it's
unrelated to the parsing code below.

Note that the code itself is problematic:
- api.BasePath is global but client is not, this means that
  multiple client with different configs will result in
  api.BasePath being potentially wrong
- api.BasePath is set in a non-thread safe manner

Changing is a bigger job but we might consider it (IMHO).
2024-08-28 17:17:56 +02:00
Florian Schüller
8d24dcfbde osbuild-worker: add CHANNEL to worker logs
aka "the deployment channel" like "staging" or "production"
2024-08-28 16:41:07 +02:00
Florian Schüller
2519e55ccd tmt/plans/edge-test.fmf: increase disksize for qcow2 test 2024-08-28 16:41:07 +02:00
Florian Schüller
e448fc8665 Makefile: remove old remnant package filter 2024-08-28 16:41:07 +02:00
Florian Schüller
210cfafdd6 Makefile: implement a target to create coverage-report 2024-08-28 16:41:07 +02:00
Florian Schüller
11f707d277 workflows: include splunk_logger sub module in tests 2024-08-28 16:41:07 +02:00
Florian Schüller
a4068b328d splunk_logger: move environment hook to splunk_logger pt2
for reusability also in image-builder
2024-08-28 16:41:07 +02:00
Michael Vogt
43c6bc7abc worker: simplify the POST in workerHeartbeat()
There is no actual payload when posting the worker status so let's
just post `nil` instead of an empty `bytes.Buffer` to make this
explicit.
2024-08-26 16:39:12 +02:00
Michael Vogt
802fbce596 worker: rename server -> serverURL
The current name `Client.server` feels a bit misleading as it is
unclear if this is an abstraction for a "server" object or an
URL. This rename makes this unambiguous.
2024-08-26 16:38:37 +02:00
Sanne Raymaekers
574c9d7d95 tools/appsre-ansible: fix unregister
The community redhat_subscription module calls `subscription-manager
unsusbscribe`, which doesn't exist. Use shell for now.
2024-08-26 14:24:48 +02:00
Michael Vogt
9e5fdc80ee osbuild-composer: add Requires: osbuild-dnf-json-api = 7
Use an exact version dependency on the `osbuild-dnf-json-api` to
ensure incompatible json protocol changes cannot break composer.

See also https://github.com/osbuild/osbuild/pull/1849
2024-08-26 10:05:18 +02:00
schutzbot
972093cc93 Post release version bump
[skip ci]
2024-08-23 11:16:09 +00:00
Tomáš Hozza
a920b39a61 Schutzfile: update osbuild ref to v126
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
f8c5595413 Delete EOL F37 and F38 repos
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
c25400467a Update rpmrepo snapshots
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
82ff759ee1 Test: test GCE image type on el10 / c10s
Enable testing of GCE image type on el10 / c10s. The el10 / c10s image
type temporarily uses cloud-init, because there are no GCP guest tools
for el10 / c10s yet and el9 version can't be installed. This implies
that we need to set the SSH key in the instance metadata and use SSH
directly.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
8035bd2c56 Test: enable image-installer tests on el10 / c10s
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
e53d03fe16 compute/gcp: add guest OS features for el10/c10s
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
a0cad4f9a6 client/compose_test: fix error message checked by the test
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Tomáš Hozza
62d8ad4efe Update osbuild/images to v0.79.0
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-08-23 13:10:53 +02:00
Sanne Raymaekers
9fcbcdb5dc tools/appsre-ansible: retry subscribing rpmbuild machines
Also unsubscribe after the build is done.
2024-08-21 13:42:11 +02:00
schutzbot
2f50ce1b56 Post release version bump
[skip ci]
2024-08-21 08:15:34 +00:00
Sanne Raymaekers
54820a88df osbuild-worker: switch to aws sdk v2 for errors in ami copy jobs 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
2624516f1a osbuild-worker: use aws sdk v2 for asg scale-in protection 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
990ed6a9ad osbuild-uploadgeneric-s3: remove aws sdk v1 dependency 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
cda0db91f5 osbuild-upload-aws: remove aws sdk v1 dependency 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
c90b92f666 cloud/awscloud: test failures when running a secure instance 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
acc415a676 cloud/awscloud: test terminating a secure instance 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
e30fba38fc cloud/awscloud: handle nil describe output when creating LTs/SGs 2024-08-20 15:32:40 +02:00
Sanne Raymaekers
fa3b203178 internal/boot: adapt to aws sdk v2 2024-08-20 15:32:40 +02:00