Commit graph

6231 commits

Author SHA1 Message Date
Sanne Raymaekers
e10424de2f osbuildexecutor: introduce osbuildexecutor.Executor interface
Wrap the current osbuildexecutor.Executor in an interface so it's easier
to add different executors, which for instance can run osbuild in a VM.
2024-02-14 09:54:11 +01:00
Sanne Raymaekers
9e85050633 osbuild-worker: add osbuild_executor config option 2024-02-14 09:54:11 +01:00
Diaa Sami
c9c51613a4 composer: glitchtip integration 2024-02-13 14:57:57 +01:00
Simon de Vlieger
95b4979d88 jobsite/builder: move process into builder
Moves the background process into the builder struct in preparation for
using `sync.Once`.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
3cf2531ae2 jobsite: go-staticcheck appeasement
Log and errors should not end with punctuation.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
7b7de50e23 jobsite/builder: enable --json
Use `--json`; this causes `osbuild` to buffer all output and then print
a huge blob of JSON at the end of the command.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
748db5410e jobsite/manager: always retry on some errors
On network layer errors we will retry the request until the timeout
expires.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
46ace22007 jobsite/manager: rename pipeline and export
A better descriptive name externally is to use `export` and
`export-file`.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
42892754fd jobsite/builder: no caching for osbuild.
As builders are not re-used caching has no use and can likely only
lead to disk-size issues. This can be re-enabled when we have an
outside-of-the-builder cache.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
b9b476c430 osbuild/builder: check deferred serve.
Make sure to keep track of the deferred error for serve.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
323d3a1f9c jobsite/builder: rename.
Consistency, the agent is now called builder.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
7548870e91 jobsite/builder: check deferred close.
Check the deffered close by providing a name return value.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
c7788798c6 jobsite/builder: wrap HTTP handlers for errors.
Instead of going directly fatal inside the handlers this commit provides
a wrapper function allowing handlers to instead return an `error`; this
will allow for checking deferred errors.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
401cb01c8d jobsite/manager: PathEscape URL parameter. 2024-02-12 12:03:38 +01:00
Simon de Vlieger
2a422321e5 rpm: add jobsite worker and manager
This adds the `jobsite-worker` and `jobsite-manager` to the `-worker`
package.
2024-02-12 12:03:38 +01:00
Simon de Vlieger
0bd66cf8e8 jobsite: manager and builder
WIP
2024-02-12 12:03:38 +01:00
schutzbot
b6b399e55e Post release version bump
[skip ci]
2024-02-07 08:22:32 +00:00
Xiaofeng Wang
fc81b89cde test: remove workarounds for fixed bugs
Bugs have been fixed, workarounds are not needed anymore.
2024-02-07 08:52:30 +01:00
dependabot[bot]
232bce95b5 build(deps): bump actions/upload-artifact from 3 to 4
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-06 12:28:59 +01:00
dependabot[bot]
7b5c4eb526 build(deps): bump actions/setup-go from 4 to 5
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-06 12:26:58 +01:00
Tomáš Hozza
f6d35ac5ad Update osbuild/images to v0.35.0
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-02-06 12:20:08 +01:00
Sanne Raymaekers
44426bb48f templates/dashboards: add community stage service to orgs 2024-02-05 11:38:53 +01:00
Simon Steinbeiss
add35b86ce DEPLOYING/HACKING.md: Consistently use inline refs
The footnote or endnote style links sometimes don't work well with
DocuSaurus, i.e. they're not resolved correctly and lead to build
errors.
https://github.com/osbuild/osbuild.github.io/actions/runs/7776075785/job/21202788034?pr=36
2024-02-05 10:56:45 +01:00
Simon Steinbeiss
1d1b410654 HACKING/DEPLOYING.md: Markdown syntax fixes 2024-02-04 17:08:57 +01:00
Sanne Raymaekers
de548c36f3 pkg/jobqueue: fix worker status update query
The workers table should be updated, not the heartbeats. Currently every
worker is reregisterig every minute.
2024-02-02 15:24:57 +01:00
Achilleas Koutsou
3e31ebfbb1 osbuild-composer: add iot-bootable-container to weldr deny list
We only want to build this image type in Fedora Koji.
2024-02-02 10:34:41 +01:00
Achilleas Koutsou
bbbb3278f8 test: update fedora-39 test repositories
Required for bootc for iot-bootable-container.
2024-02-02 10:34:41 +01:00
Achilleas Koutsou
76a699653a test: add iot-bootable-container to api tests
The image type supports uploading to both s3 and container registries.
Right now we only care about getting it in Koji, so let's just add it to
the s3 upload section.

The container registry test also embeds containers in the image, which
the iot-bootable-container doesn't support (yet).
2024-02-02 10:34:41 +01:00
Achilleas Koutsou
52d4b196a3 cloudapi: enable iot-bootable-container image type 2024-02-02 10:34:41 +01:00
Tom Koscielniak
d97f54cc5c Add RHEL 9.3 and 8.9 GA to pipeline 2024-02-01 16:07:27 +01:00
dependabot[bot]
b461b1faf3 build(deps): bump the go-deps group with 10 updates
Bumps the go-deps group with 10 updates:

| Package | From | To |
| --- | --- | --- |
| [cloud.google.com/go/compute](https://github.com/googleapis/google-cloud-go) | `1.23.3` | `1.23.4` |
| [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) | `1.36.0` | `1.37.0` |
| [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) | `1.4.0` | `1.5.1` |
| [github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5](https://github.com/Azure/azure-sdk-for-go) | `5.3.0` | `5.5.0` |
| [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) | `1.50.0` | `1.50.8` |
| [github.com/google/uuid](https://github.com/google/uuid) | `1.5.0` | `1.6.0` |
| [github.com/jackc/pgtype](https://github.com/jackc/pgtype) | `1.14.0` | `1.14.1` |
| [github.com/openshift-online/ocm-sdk-go](https://github.com/openshift-online/ocm-sdk-go) | `0.1.390` | `0.1.398` |
| [github.com/osbuild/images](https://github.com/osbuild/images) | `0.33.0` | `0.34.0` |
| [google.golang.org/api](https://github.com/googleapis/google-api-go-client) | `0.160.0` | `0.161.0` |


Updates `cloud.google.com/go/compute` from 1.23.3 to 1.23.4
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/compute/v1.23.3...compute/v1.23.4)

Updates `cloud.google.com/go/storage` from 1.36.0 to 1.37.0
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.36.0...spanner/v1.37.0)

Updates `github.com/Azure/azure-sdk-for-go/sdk/azidentity` from 1.4.0 to 1.5.1
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.4.0...sdk/internal/v1.5.1)

Updates `github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5` from 5.3.0 to 5.5.0
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/resourcemanager/compute/armcompute/v5.3.0...sdk/resourcemanager/compute/armcompute/v5.5.0)

Updates `github.com/aws/aws-sdk-go` from 1.50.0 to 1.50.8
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.50.0...v1.50.8)

Updates `github.com/google/uuid` from 1.5.0 to 1.6.0
- [Release notes](https://github.com/google/uuid/releases)
- [Changelog](https://github.com/google/uuid/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/uuid/compare/v1.5.0...v1.6.0)

Updates `github.com/jackc/pgtype` from 1.14.0 to 1.14.1
- [Changelog](https://github.com/jackc/pgtype/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgtype/compare/v1.14.0...v1.14.1)

Updates `github.com/openshift-online/ocm-sdk-go` from 0.1.390 to 0.1.398
- [Release notes](https://github.com/openshift-online/ocm-sdk-go/releases)
- [Changelog](https://github.com/openshift-online/ocm-sdk-go/blob/main/CHANGES.md)
- [Commits](https://github.com/openshift-online/ocm-sdk-go/compare/v0.1.390...v0.1.398)

Updates `github.com/osbuild/images` from 0.33.0 to 0.34.0
- [Release notes](https://github.com/osbuild/images/releases)
- [Commits](https://github.com/osbuild/images/compare/v0.33.0...v0.34.0)

Updates `google.golang.org/api` from 0.160.0 to 0.161.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.160.0...v0.161.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/compute
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/google/uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: github.com/jackc/pgtype
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/openshift-online/ocm-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-deps
- dependency-name: github.com/osbuild/images
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-01 12:23:01 +01:00
Jakub Rusz
db0e6c9643 Packer: change fedora-38 aarch64 ami
This ami is currently broken, switch to a slightly older one.
2024-01-31 10:11:50 +01:00
Jakub Rusz
9e0edb2f00 CI: update terraform SHA
We had to switch fedora-38-aarch64 image to an older one because the
current one was broken in AWS.
2024-01-31 10:11:50 +01:00
Sanne Raymaekers
e289b763e7 templates/packer: deal with unbound variables
Don't allow unbound variables, but for the variables that are used to
determine whether or not that part of the setup should continue, default
to empty/undefined.
2024-01-30 21:41:31 +01:00
Tomáš Hozza
0c0a758600 Packer: default to RHEL-9.3 Schutzfile config for rhel-9
This is necessary to get the proper osbuild SHA to install the latest
required osbuild version into the image.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
1fa72f262e test/api.sh/vsphere: specify the resource pool for image import
Specify the resource pool when importing the VMDK to the VCenter.
This should prevent the following error:

govc: default resource pool resolves to multiple instances, please specify

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
ae68358a0d worker/osbuild: provide more details and logs when osbuild build fails
Add any errors to job error details when an osbuild build fails.
Otherwise these won't show up in the worker log, which makes
debugging issues harder.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
0690b6f1ba cloudapi: pass depsolve job error details to the manifest job error
If a depsolve job fails, the error details were not passed as details to
the manifest job error details. This may help with debugging failures.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
e664d49541 Test/old-worker-new-composer: don't wait for composer indefinitely
The test would loop for the job timeout limit waiting for composer
container, in case the composer API does not come up for whatever
reason. Modify the test case to wait only for 12x 10s before failing.
In case the composer API does not come up, print the logs from the
composer container and exit with non-zero code.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
a33da152da Test/weldr: add test for dot-notation and distro aliases
Add test case for verifying that distros and repo configurations work
with and without the dot-notation in the on-prem scenario. Run the test
case on the latest RHEL 8 and 9 versions in development and verify the
default distro aliases behavior on them.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
2fe5e01d4d Test/api.sh: test distro aliases and names without dot-notation
Extend the `api.sh` to allow testing compose requests with distro name
which does not use dot-notation (specifically when the dot is removed
from the distro name as it used to be in the past). In addition to that,
allow also testing the distro alias using distro name without the minor
version in compose requests.

Enable these two new test variants in the CI.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
fb6acbbe83 test/data/repositories: update GA releases and add SAP repos
Update RHEL 8.9 and 9.3 repo definitions to use the CDN repos, since
these are already GA.

Add SAP repositories to all RHEL repo configs, to be able to build the
SAP image for testing purposes.

Fix minor issues found in repos (e.g. 8.8 RT repo pointing to 8.7,
etc.).

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
9252e2525c Test/azure.sh: don't use dot-notation for image name
Using the DISTRO_CODE with dot to separate major and minor version is
breaking cloud-image-val testing. Specifically, the tool is using
regular expression to search for an uploaded image, but the expression
is not expecting any dot in the version.

This will be fixed in [1], but merging it will take some time due to CIV
CI being currently broken. Due to this, workaround the problem for now
by making sure that the TEST_ID, which is used to construct the
IMAGE_NAME, does not contain any dot in the DISTRO_CODE.

[1] https://github.com/osbuild/cloud-image-val/pull/290

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
499cda4a57 Test/manifests: rename files to use dot-notation
Otherwise, the image tests will fail on globing for testing manifests.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
66571784d1 Use dot-notation for repo configs and in test cases.
Where applicable, modify all repo config filenames to use a dot
to separate the release major and minor version. Modify test cases
to not remove dot from the distro version any more.

Existing tests will be extended (or new tests added) to explicitly test
backward compatibility and ensure that using old distro names without a
dot still works.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
c82c0ae6b5 Test/api.sh: pass TEST_MODULE_HOTFIXES as an env variable
While I was the one to push for TEST_MODULE_HOTFIXES to be passed as an
argument to `api.sh`, when it was implemented, this is turning our to be
impractical. I would like to extend `api.sh` to test distro names with
and without the dot to separate major and minor version and also to test
the ability to specify distro name explicitly (e.g. a distro alias).
Passing all of these new options and their combinations would make the
CLI go wild.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
17ad9c1165 Makefile/rpm: always copy over SPEC file
Any changes done to the SPEC file were not reflected in the
`rpmbuild/SPECS` directory across runs of `make rpm`. One had to delete
the SPEC file manually to be updated and used for the RPM build.

Mark the target as PHONY to ensure that the SPEC is always updated.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
8ba3426f12 composer: use distro name aliases from configuration
Register the distro name aliases from the configuration on the distro
factory.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
b79eaeb78e weldr: don't determine distro name from image type when depsolving
Weldr API used to determine the distro name from the image type, when it
was getting the repositories to use for depsolving and for the actual
depsolving (solver uses the distro name to namespace cache).

This used to be OK, but with the introduction of distro name aliases,
the distro name used to get the distro object may not be the same as the
name returned by the actual distro object. To preserve the current
behavior, the same name used to get the distro object should be used to
also get the repositories for depsolving and to namespace depsolving
cache.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00
Tomáš Hozza
cdf57e5bc1 osbuild-composer/config: support specifying distro aliases
Add new configuration option `distro_aliases`, which is a map of
strings, allowing to specify distro name alias for supported
distributions.

Define aliases for RHEL major versions without the minor version
specified.

For now, the distro aliases map is not used by any API
implementation and it is ignored.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-01-26 11:32:34 +01:00