Commit graph

4954 commits

Author SHA1 Message Date
Brian C. Lane
a1a3e5ba2d cloudapi: Add azure-sap-rhui support to cloudapi
and Update openapi.v2.gen.go
by running `./tools/prepare-source.sh` which runs `go generate ./...`
among other things.
2022-11-18 16:53:22 +01:00
Brian C. Lane
a00cde7424 test: Add manifest for RHEL 8 azure-sap-rhui image
Generated with:
go run ./cmd/gen-manifests -output ./test/data/manifests -cache /var/tmp/manifest-cache/

Includes some changes to ec2-sap manifests due to using the same repos.
2022-11-18 16:53:22 +01:00
Brian C. Lane
f9b3f95615 distro: Install ansible for sap on 8.5 as well 2022-11-18 16:53:22 +01:00
Brian C. Lane
e4f7e94dd0 distro: Use GibiByte constant for Azure 64GiB Images
instead of using 68719476736, use 64 * common.GibiByte
2022-11-18 16:53:22 +01:00
Brian C. Lane
12c7f12d16 tools: Add new azure-sap-rhui image type
It needs to be listed in format-request-map.json, and the repo snapshots
required to depsolve the packages need to be added to repos.json

NOTE: The snapshots are from https://github.com/osbuild/rpmrepo/
2022-11-18 16:53:22 +01:00
Brian C. Lane
8c07d65fe0 distro: Add azure-sap-rhui image type to RHEL8 2022-11-18 16:53:22 +01:00
Brian C. Lane
a47c239d11 distro: Move alsa-lib out of azureCommonPackageSet Exclude 2022-11-18 16:53:22 +01:00
Brian C. Lane
010d92eed1 distro: Separate out RHEL8 SAP package set into its own function
This also moves cloud specific client packages into that cloud's
PackageSet so that the SAP packages are common to all users.
2022-11-18 16:53:22 +01:00
Brian C. Lane
ed7aaec12d distro: Move RHEL 8 SAP config data into sap.go
This can be shared between cloud providers so move it out of the EC2 SAP
config into its own file and drop the X86_64 from the name (there is
nothing arch specific in it, even if it is only ever used on X86).
2022-11-18 16:53:22 +01:00
Sanne Raymaekers
bff6403b37 cloudapi/v2: pass rhsm requirement to ostree resolve job 2022-11-18 13:07:36 +01:00
Sanne Raymaekers
28319f50d8 worker: log failures in ostree resolve job 2022-11-17 12:55:02 +01:00
Ondřej Budai
422281698a packer: add a workaround for broken podman from 9.1
Podman doesn't work when installed from 9.1 repositories on 9.0 image.

I found that upgrading the whole system helps. Sadly, that requires removing
python-unversioned-command for some reason.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-11-16 22:31:43 +01:00
schutzbot
5669e05264 Post release version bump
[skip ci]
2022-11-16 08:29:16 +00:00
Tomáš Hozza
587e043668 distro/RHEL-9: exclude unwanted packages from the EC2-SAP image
Exclude unwanted packages from the EC2-SAP image. These packages have
been pulled into RHEL-9 image due to the fact that we moved away
from using `@core` package group by default and as a result we dropped
explicit package excludes. However the SAP image includes the
`@Server` package group, which pulls in these unwanted packages, thus
we need to explicitly exclude them in the SAP package set.

Related to COMPOSER-1829

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-15 12:04:42 +01:00
Tomáš Hozza
61478254cd distro/RHEL-9: disable amdgpu module on all AWS images
`amdgpu` module is causing error to be printed in the system log on AWS
instances. After investigation, it turns out that it is not needed.
Disable it by default on all AWS images.

Related to COMPOSER-1807

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-15 12:04:42 +01:00
Tomáš Hozza
a930073e4a distro/RHEL-8: disable amdgpu module on all AWS images
`amdgpu` module is causing error to be printed in the system log on AWS
instances. After investigation, it turns out that it is not needed.
Disable it by default on all AWS images.

Related to COMPOSER-1807

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-15 12:04:42 +01:00
Brian C. Lane
050403fbe4 diff-manifests.sh: Use shared_lib for greenprint and redprint 2022-11-15 12:00:00 +01:00
Paul Whalen
560905108c fedora/package_sets: Update sil-scheherazade-new-fonts
F37 no longer ships sil-scheherazade-fonts, but
sil-scheherazade-new-fonts instead. Let's change this. The repos for
test manifests must have been updated in order to get the new package.

Co-authored-by: Ondřej Budai <ondrej@budai.cz>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-11-14 12:01:47 +01:00
Ondřej Budai
7e97c6475e distro/fedora: print an error if PackageSets cannot initialize a manifest
Previously, this just happened silently and let to extremely odd errors. Let's
just print the error to simplify debugging the next time.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-11-11 18:39:18 +01:00
Ondřej Budai
194351e681 distro/fedora: fix iot-installer package set generation
8fdd158799 modified the Cloud API to resolve
ostree commits using a separate job. This change caused the API handler
to call PackageSets without any ostree options (because they are not resolved
yet).

Unfortunately, the new implementation of PackageSets initializes the manifest.
The initialization checks the options and if the type is iot-installer and
it doesn't have the fetch checksum for IoT, it just returns an error.

To work around this (we need an initialized manifest to create the chains),
this commit just gives the initialization method a dummy checksum. The ostree
options currently don't have any effect on the package sets, so this should
be fine.

In order to make this workaround at least slightly sane, a warning is printed,
there's a new test just for this behaviour and a long comment to remember to
delete these lines.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-11-11 18:39:18 +01:00
Thomas Lavocat
4e7b021514 ci: add tags to AWS instances
The new terraform runners support an extended lists of tags to get a
greater precision on the stats we can extract from AWS cost center.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-11-10 20:56:21 +01:00
Tomáš Hozza
754a9b59b3 tools/provision.sh: copy RHEL repo overrides using wildcard
Don't list specific RHEL repos which should be copied over to the repo
overrides directory, but instead copy all of them, similar to how it's
done for Fedora and CentOS Stream. This is less error prone when support
for a new RHEL release is added to composer.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-09 20:25:01 +01:00
Brian C. Lane
ff823a93e1 distro: Separate out RHEL9 SAP package set into its own function
This also moves cloud specific client packages into that cloud's
PackageSet so that the SAP packages are common to all users.
2022-11-09 20:17:10 +01:00
Brian C. Lane
b785af25ed distro: Move RHEL 9 SAP config data into sap.go
This can be shared between cloud providers so move it out of the EC2 SAP
config into its own file and drop the X86_64 from the name (there is
nothing arch specific in it, even if it is only ever used on X86).
2022-11-09 20:17:10 +01:00
Simon de Vlieger
f6bd62e838 distro: SELinux should be the last stage
SELinux relabels the entire filesystem tree and should always be the
last stage in the os pipelines.
2022-11-09 15:41:40 +01:00
Brian C. Lane
d73171fdde Use go install instead of deprecated go get
In go 1.18 'go install' needs to be used to install an executable. See
https://go.dev/doc/go-get-install-deprecation and
https://go.dev/doc/manage-install
2022-11-09 12:08:43 +01:00
Simon de Vlieger
1612d9cb80 ci: add my SSH keys to the CI ssh keys 2022-11-09 11:31:57 +01:00
Simon de Vlieger
a7f1dd4065 docker-compose: remove unavailable --dnf-json 2022-11-09 11:28:56 +01:00
Sanne Raymaekers
0a9cf9b6a7 dbjobqueue: check context errors after trying to dequeue
This fixes a race condition where the context might have been canceled
or timed out in between the preliminary check and trying to dequeue, and
consequently returning the wrong error.

Instead of doing the preliminary check, just check for the context
errors when trying to dequeue.
2022-11-08 07:37:32 -05:00
Sanne Raymaekers
26b8e2ff6e dbjobqueue: acquire a new connection for each listen query
This fixes a bug where the listen function would keep trying to use a
closed, unrecoverable connection to listen for a notification. This
continued failing, which essentially made the queue instance useless.
2022-11-08 07:37:32 -05:00
Tomáš Hozza
63f1b8d9fb internal/disk: fix potential nil pointer dereference
The `LVMVolumeGroup.Clone()` method could end up dereferencing a `nil`
pointer in the `lv` variable, if there would be a `nil` logical volume
in the LVM volume group. Such situation would be an error of its own.
There is no point in checking if the cloned logical volume is not `nil`
and casting it to another variable. The logic should check if the cloned
logical volume is `nil` and panic in such situation. The following code
can then cast the clone to a different variable without issues and there
is no risk of dereferencing a `nil` pointer.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-07 16:10:41 +01:00
Tomáš Hozza
2735ea5b96 internal/disk: fix potential nil pointer dereference
The `PartitionTable.Clone()` method could end up dereferencing a `nil`
pointer in the `part` variable, if there would be a `nil` partition in
the partition table. Such situation would be an error of its own.
There is no point in checking if the cloned partition is not `nil` and
casting it to another variable. The logic should check if the cloned
partition is `nil` and panic in such situation. The following code can
then cast the clone to a different variable without issues and there is
no risk of dereferencing a `nil` pointer.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-07 16:10:41 +01:00
Tomáš Hozza
baa5e96734 internal/container: delete unreachable code
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-07 16:10:41 +01:00
Tomáš Hozza
5e0058bc98 internal/distro/el7: delete unreachable code
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-07 16:10:41 +01:00
Jakub Rusz
fe1bf7a7bf Update terraform SHA
Fedora-37 was using "ci-ssd" instance in RHOS-01 which does not support
nested KVM. This SHA switches them to use "ci" instances which do
support nested kvm.
2022-11-07 16:10:41 +01:00
Sanne Raymaekers
acc1575cf6 cloudapi/v2: expose ostree contenturl and rhsm options 2022-11-04 11:41:31 +01:00
Brian C. Lane
1096003598 store: Fix loading cross distro compose results
When the store is written to disk it simplifies the ImageBuild details
into a simple image type string. This works fine for composes that match
the host's distro but isn't enough detail to load composes made for
other distros, especially if the image type name isn't supported on the
host. This results in cross distro compose results being lost after a
reboot.

This fix uses the distro information from the compose's blueprint to
determine which distro the image type should be loaded from. It assumes
that the architecture matches the hosts' arch -- this is currently
always true but in the future if cross-arch builds are added it will
need to be addressed in a different way.

newComposeFromV0, newComposesFromV0, and newStoreFromV0 now take a
pointer to the full distro registry instead of an Arch, this allows them
to access the correct image types for the distro selected by the
blueprint. When loading the composes from disk the blueprint distro is
loaded from the registry before checking the image type string.

This means that we do not have to change the store version or on disk
format, the only thing changing is how it decides to populate the
ImageBuild when reloading the store.

A number of tests use a fake test distro using fake architecture names.
These tests have been adjusted to use a fake distro registry with
overridden host architecture that matches the fake one.
2022-11-03 08:39:22 +01:00
Brian C. Lane
bcb927cf63 distroregistry: Add host's architecture name to the registry
The store needs to know what default distribution and arch to use for
images built with a blueprint that has no distro set. Store the host's
architecture name along side the host's distribution details.

SetHostArchName() allows this host arch to be overridden for testing,
which uses arch names that do not match any host.
2022-11-03 08:39:22 +01:00
Tom Gundersen
626530818d worker/server: requeue unresponsive jobs
If a job is unresponsive the worker has most likely crashed or been shut
down and the in-progress job been lost.

Instead of failing these jobs, requeue them up to two times. Once a job is lost
a third time it fails. This avoids infinite loops.

This is implemented by extending FinishJob to RequeuOrFinish job. It takes a
max number of requeues as an argument, and if that is 0, it has the same
behavior as FinishJob used to have.

If the maximum number of requeues has not yet been reached, then the running
job is returned to pending state to be picked up again.
2022-11-02 15:26:00 +01:00
schutzbot
d02f666a4b Post release version bump
[skip ci]
2022-11-02 08:29:15 +00:00
Brian C. Lane
8c78ff8904 test: Add a test for a complete blueprint's TOML output 2022-10-31 15:24:15 +01:00
Brian C. Lane
59178aa72a vendor: Update BurntSushi/toml to v1.2.1
This fixes an error when processing blueprints with customizations.firewall
See https://github.com/BurntSushi/toml/issues/360 for details on the
upstream bug.
2022-10-31 15:24:15 +01:00
dependabot[bot]
159a7e2fe6 build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.5.0 to 1.6.1.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.5.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-31 11:34:44 +01:00
Tomáš Hozza
a001da966a distro: add support for RHEL-9.2
Add relevant repo definitions and generate image test manifests.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-10-31 08:07:09 +01:00
Tomáš Hozza
fc95dac86a distro: add support for RHEL-8.8
Add relevant repo definitions and generate image test manifests.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-10-31 08:07:09 +01:00
Sanne Raymaekers
e94ea7c995 internal/worker: add rhsm to ostree resolve job 2022-10-28 16:14:30 +02:00
Sanne Raymaekers
c865286fb9 internal/ostree: add consumer certs when resolving ostree ref 2022-10-28 16:14:30 +02:00
Sanne Raymaekers
632f272573 internal/rhsm: add consumer secrets to Subscriptions
The consumer key/cert is used to uniquely identify a system against a
candlepin instances. They're useful for any Red Hat (ostree) content
which requires (cert) authentication.
2022-10-28 16:14:30 +02:00
Sanne Raymaekers
500341a25f internal/osbuild: contenturl and rhsm secrets for ostree sources 2022-10-28 16:14:30 +02:00
Tomáš Hozza
5a0d286d6b test/api: don't provide Resource Group Location when testing Azure
Let's default to not providing the Resource Group Location when testing
Cloud API with Azure and leave it up to the implementation to determine
the correct location to use.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-10-27 19:33:43 +02:00