Commit graph

6900 commits

Author SHA1 Message Date
Sanne Raymaekers
a44a499553 worker: support worker server target results 2025-01-24 15:26:15 +01:00
Sanne Raymaekers
4e803af8cd cloudapi: get rid of localSave check in local target
The local target shouldn't require any specific configuration and should
just be available always.
2025-01-24 15:26:15 +01:00
Sanne Raymaekers
7bfcac30dd cloudapi: support worker server target artifact retrieval
In order to get the artifact location from the cloudapi, add a helper
function in the worker server.
2025-01-24 15:26:15 +01:00
Brian C. Lane
df16f7fc63 v2_test: Add testing for cloudapi /depsolve/blueprint
Test the depsolve using a mocked response, test for mismatched
distributions, and for unsupported architectures.
2025-01-23 11:39:53 -08:00
Brian C. Lane
f377c5e3eb v2_test: Add a test-distro-1 repository
This also adds an actual repository json file for the test-disro.
Without this the repo.ListDistros() function doesn't return any actual
distros.

Related: RHEL-60125
2025-01-23 11:39:53 -08:00
Brian C. Lane
02d0b8ec01 cloudapi: Request depsolve from osbuild-worker
and return the response to the client. This uses the worker to depsolve
the requested packages. The result is returned to the client as a list
of packages using the same PackageMetadata schema as the ComposeStatus
response.  It will also time out after 5 minutes and return an error,
using the same timeout constant as depsolving during manifest
generation.

Related: RHEL-60125
2025-01-23 11:39:53 -08:00
Brian C. Lane
e06e62ca03 cloudapi: Add /depsolve/blueprint route
This will allow depsolving blueprints and returning package metadata for
the dependencies.

Related: RHEL-60125
2025-01-23 11:39:53 -08:00
Brian C. Lane
4f3c93ef1e cloudapi: Make sigmd5 in PackageMetadata optional
In order to reuse PackageMetadata with DepsolveResponse and not include
unused fields this changes the sigmd5 entry to an optional field. This
doesn't effect the use of PackageMetadata in the Compose response since
it is always set, and it allows it to be omitted in the response for
depsolving.

Also adds a basic test for stagesToPackageMetadata

Related: RHEL-60125
2025-01-23 11:39:53 -08:00
Brian C. Lane
08dc5f3041 cloudapi: Move GetCustomizationsFromBlueprintRequest
This function only depends on the Blueprint (cloudapi request type, not
the internal/blueprint) so move it to a function on that so that it can
be reused by other users of the cloudapi Blueprint.

Related: RHEL-60125
2025-01-23 11:39:53 -08:00
Tom Koscielniak
5a9bb8e10a test/cases: Adjust upgrade test to rhel 9.6
Adjust upgrade test to rhel 9.6 in order to meet the rhel CTC2 9.6 to 10.0 upgrade requirements
2025-01-23 16:27:10 +01:00
Sanne Raymaekers
425581fcc1 cloudapi/v2: support local upload target
The target validation rework broke the local upload target, which is
needed for cockpit-image-builder.
2025-01-22 13:54:40 +01:00
schutzbot
7c101fdef7 Post release version bump
[skip ci]
2025-01-22 08:13:06 +00:00
Brian C. Lane
73101d2ff2 Fix non-constant log strings
Newer versions of the go compiler (1.24 in this case) fail when running
go test during a mock rebuild of the srpm created by 'make srpm' on
Fedora 42.

Even though we currently don't support go1.24, fix these so they don't
become an issue when we do.
2025-01-21 16:51:20 -08:00
Michael Vogt
af0543d27c many: update images Manifest() API for PR#1107
This updates composer to use the updated API in images around the
seed handling for manifests, see images PR#1107 for details.

Note that this has no semantic changes yet. We could now simplfy
some things because images will auto-seed but that is for a followup.
2025-01-20 09:50:49 +01:00
Michael Vogt
9dd8b2c50f go.mod,vendor: run go get -u github.com/osbuild/images
This commit updates the "images" dependency.
2025-01-20 09:50:49 +01:00
Florian Schüller
a18ec71dc8 README: remove mailing list
The mailing list was sunset by the IT department and
was rarely used, so we'll replace it with matrix & discussions.
2025-01-17 13:53:53 +01:00
Lukas Zapletal
d531f62488 blueprint: add cacert customization 2025-01-10 10:26:54 +01:00
Lukas Zapletal
f41c764ca7 build(deps): bump deps specifically CVE-2024-45338 2025-01-09 16:30:20 +01:00
schutzbot
bdc755f71b Post release version bump
[skip ci]
2025-01-08 08:13:07 +00:00
Lukas Zapletal
2aaaaf05b3 ci: do not perform linting on vendor/ 2025-01-07 11:57:51 +01:00
Achilleas Koutsou
8b0a1d1714 schutzbot: update Achilleas' ssh keys 2025-01-02 08:03:39 +01:00
Tomáš Hozza
8c99478a69 upgrade_verify.sh: update el10 repo URLs to non-Beta
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2025-01-02 07:33:05 +01:00
Simon Steinbeiss
e90010f153 github/actions: Enable /jira-epic slash commands
This change allows for using the command to create Jira Tasks under a given
Epic both in a pull request comment or in the pull request description.

To trigger the action, you can simply add a comment to a given pull request with
the following content:
/jira-epic ISSUE-1234

This will trigger the action and a bot will create a Jira Task under the
Epic ISSUE-1234. Once this is successful, it will update the pull request
title and description to contain a link to the newly created Jira ticket,
which also means that the two will be linked.
Alternatively, you can also add the command to the pull request description
(if you e.g. want to create the Task at PR creation time).

Note: This is currently only enabled for the HMS project.
2024-12-13 11:27:52 +01:00
Florian Schüller
6fdc6d6a55 go.mod: upgrade golang.org/x/crypto
Upgrade golang.org/x/crypto due to
https://github.com/osbuild/osbuild-composer/security/dependabot/43
2024-12-12 16:03:32 +01:00
schutzbot
f6d320886d Post release version bump
[skip ci]
2024-12-11 08:18:23 +00:00
Florian Schüller
153bcadf2d osbuild-service-maintenance/aws: merge errors
Collect and merge errors, instead of
nesting errors.
In this case we want to continue execution if only one
cleanup fails.
2024-12-10 11:43:51 +01:00
Florian Schüller
87ef1527fc osbuild-service-maintenance/aws: avoid error on empty list
Passing an empty list to `TerminateInstances` causes an
error message, which is not necessary, as there is
nothing to terminate.
2024-12-10 11:43:51 +01:00
Florian Schüller
24e256c225 osbuild-service-maintenance: add test for allTerminated
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
65b7ee65b2 osbuild-service-maintenance: implement removal of launch templates
Launch templates of instances that are terminated should be removed.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
a96ea533c0 osbuild-service-maintenance: implement removal of security groups
Security groups of instances that are terminated should be removed.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
7ebe266d3c osbuild-service-maintenance: implement removal on invalid parent
Add a safeguard to ensure secure instances without valid
parent instances are terminated, as they are unnecessary to retain.
Typically, the parent does not exist if the secure instance is
older than 2 hours, but this check provides additional validation.
HMS-3632
2024-12-10 11:43:51 +01:00
Florian Schüller
a7119a4d0f osbuild-service-maintenance/aws: support aws credential file
Support running the maintenance locally with a valid
`~/aws/credentials` file. HMS-3632
2024-12-10 11:43:51 +01:00
Tomáš Hozza
580366d1f3 osbuild-dnf-json-tests: don't set OSTree options for non-OSTree images
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-12-09 09:46:54 +01:00
Tomáš Hozza
1f590aa232 Weldr/ComposeRequest: OSTree options nil if not set
Previously, the `OSTree` property in the Weldr API `ComposeRequest`
struct was not a pointer to the `ostree.ImageOptions` type. As a result,
it was initialized to an empty struct, even if not set in the client API
call.

As a result, the `OSTree` property in the `distro.ImageOptions` was
always not `nil`, when initializing the osbuild manifest. However, after
a change in `osbuild/images` [0], providing OSTree options for
non-OSTree image types is no longer considered valid. This caused a
failure to submit a new compose for any non-OSTree image type.

Change the `OSTree` property in Weldr `ComposeRequest` to be a pointer
and mark it as optional.

[0] https://github.com/osbuild/images/pull/1071

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-12-09 09:46:54 +01:00
Tomáš Hozza
fdff0a903e Update osbuild/images to v0.105.0
This brings a lot of new stuff, among others:
 - Internal image types for RHEL-10 (COMPOSER-2357)
 - Support for Swap partitions in DiskCustomizations (COMPOSER-2400)
 - Support for new partitioning customizations for CS and RHEL
 - Fix for RHEL 8 and 9 AMI boot mode (RHEL-69628)

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-12-09 09:46:54 +01:00
dependabot[bot]
73f3aa22a2 build(deps): bump codecov/codecov-action from 4 to 5
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4 to 5.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-09 09:46:42 +01:00
Tom Koscielniak
1193187e0a tools/tests: Update rhel10 compose url
Update a rhel 10 compose url to point to nightly instead of public beta.
Fix for a failing rhel 10 nightly pipeline.
2024-12-06 12:06:40 +01:00
Ondřej Budai
3561202acc github: prevent script injections via PR branch names
Prior this commit, ${{ github.event.workflow_run.head_branch }} got
expanded in the bash script. A malicious actor could inject
an arbitrary shell script. Since this action has access to a token
with write rights the malicious actor can easily steal this token.

This commit moves the expansion into an env block where such an
injection cannot happen. This is the preferred way according to the
github docs:
https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
2024-12-05 18:13:17 +01:00
Tom Koscielniak
f5a5705b7e schutzbot/tests: bump rhel10 to nightly, update tf sha and osbuild deps
Bump RHEL 10 from beta to nightly by updating terraform SHA and osbuild dependencies to start testing RHEL 10 nightly and to meet the CTC schedule.
2024-12-05 14:33:12 +01:00
schutzbot
ee41e3dce6 schutzfile: Update snapshots to 20241203 2024-12-04 10:33:09 +01:00
Sanne Raymaekers
f6feb7675b cloud/awscloud: use any instance create fleet returns
Even in case of errors, as long as create fleet returns an instance,
attempt to use it.

In some cases AWS returns `InsufficientInstanceCapacity` but still
creates an instance:
```
msg="Won't retry CreateFleet with OnDemand instance, retry: false, errors: InsufficientInstanceCapacity: There is no Spot capacity available that matches your request.; Already launched instance ([i-...]), aborting create fleet"
msg="doCreateFleetRetry: returning retry: false, msg: [InsufficientInstanceCapacity: There is no Spot capacity available that matches your request. Already launched instance ([i-...]), aborting create fleet]"
msg="doCreateFleetRetry: cancelling retry, instance already exists: [i-...]"
msg="doCreateFleetRetry: setting retry to true"
msg="Checking to retry fleet create on error InsufficientInstanceCapacity (msg: There is no Spot capacity available that matches your request.)"
```
2024-12-03 14:00:12 +01:00
Lukas Zapletal
4b55bc2825 cloudapi: carry ostree MTLS secret over 2024-12-03 13:59:45 +01:00
Sanne Raymaekers
779053d910 cloud/awscloud: give secure instances a name
That way you can just enter the parent instance id into the search bar
and get both the worker and its executor.
2024-12-03 11:56:52 +01:00
Jakub Rusz
f68fcff400 CI: remove jrusz from notifications
Removing myself from the slack notifications as I'm no longer a QE
member.
2024-12-02 14:34:51 +01:00
Sanne Raymaekers
38b799f162 cloud/awscloud: exclude really old instance types
RHEL 10 (nightly) builds fail on stage with "Fatal glibc error: CPU does
not support x86-64-v3", this is most likely due to very old instance
types not supporting a specific instruction set.
2024-11-29 15:42:27 +01:00
Lukas Zapletal
e7a7cda3bc cmd: extra env logging for osbuild worker 2024-11-29 11:57:11 +01:00
Sanne Raymaekers
1372be3f6e test/cases/api/azure: actually verify the HyperV Generation 2024-11-29 10:37:24 +01:00
Lukas Zapletal
6310115629 deps: update images to 0.102 2024-11-27 13:03:29 +01:00
schutzbot
2e1afdc829 Post release version bump
[skip ci]
2024-11-27 08:17:34 +00:00
Ondřej Budai
64ff0e3dad awscloud: add very verbose logging to createFleet creation
We still see this error sometimes:

Unable to start secure instance: Unable to create fleet: InsufficientInstanceCapacity: There is no Spot capacity available that matches your request

This is awkward because the message mentions that there is no spot
capacity, even though the current code should retry on
InsufficientInstanceCapacity. I also confirmed this by searching for
the retries log messages: there are none in the logs.

We need a bigger hammer. Let's log everything that happens in the
createFleet method in order to have better understanding why the
retry logic isn't triggered. We should probably move most of the newly
added logs to the debug level, but let's delay that until we have
more insight into what's happening.
2024-11-26 16:12:09 +01:00