Commit graph

5003 commits

Author SHA1 Message Date
Achilleas Koutsou
a2a3ac8ffe test: regenerate manifests with chrony stages
Regenerate all manifests that used the old chrony stage options with the
'timeservers' field.  This is now replaced by the 'servers' field, which
is a slice of objects with a 'hostname' field instead of a slice of
strings.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
5d760c48ad osbuild: remove Timeservers slice from chrony stage
The org.osbuild.chrony stage was extended to allow additional directives
alongside time servers [1].  The old Timeservers string slice was kept
for backwards compatibility.  Removing support for it in
osbuild-composer makes working with the stage's options simpler.  Using
the new struct slice Servers field and only specifying a Hostname for
each element is equivalent to the old behaviour, so no functionality is
lost.

This simplifies the chrony stage since no validation is required
anymore.
It also simplifies the propagation of configuration options through the
pipeline generation code which doesn't need to check for both types of
stage options.

[1] https://github.com/osbuild/osbuild/pull/692
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
0ba1a5ff73 test: update RHEL 9.x and CS9 vmdk manifests
Regenerated manifests for the vmdk image type with the rewritten
definitions:
- removed RPMs in the build root: packages are added to the build-root
  on-demand by the payload pipelines.  All removed package were
  unnecessary for building the image.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
a027b373a5 distro/rhel9: update vmdk image type to new definitions 2022-11-28 17:20:49 +01:00
Achilleas Koutsou
88ff103621 test: update RHEL 9.x and CS9 edge manifests
Regenerated manifests for the edge image types with the rewritten
definitions:
- removed RPMs in the build root: packages are added to the build-root
  on-demand by the payload pipelines.  All removed package were
  unnecessary for building the image.
- pipeline names changed.
- rpm stage for os pipeline now sets dbpath and ostree-booted option
  (see [1] for original change in Fedora).
- installer pipelines changed to no longer use bootiso.mono stage, but
  instead break it down to component stages (see [2] for original change
  in Fedora).

Fedora IoT manifests updated to include selinux labelling changes.

The skopeo and containers storage stage have been temporarily dropped
since the new pipeline generator functions don't support it yet.

[1] https://github.com/osbuild/osbuild-composer/pull/2914
[2] https://github.com/osbuild/osbuild-composer/pull/3020
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
e8285fb2bf manifest: set selinux labels for cp and tar conditionally
Only label cp and tar if they're found in the list of packages for the
build root.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
ce037e24be distro/rhel9: add image-installer to same platform as edge 2022-11-28 17:20:49 +01:00
Achilleas Koutsou
10544e8016 distro/rhel9: update edge types to new definitions
Updated all edge image types except the simplified installer to the new
image definitions.
Added the image functions, based on the Fedora counterparts, and removed
the pipeline functions.
Removed the build package from the image definition.
Removed firmware packages from platform: not available in repos.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
9d34362bbf test: update RHEL 9.x and CS9 qcow2, oci, and openstack manifests
Regenerated manifests for the QCOW2, OCI, and openstack image types with
the rewritten definitions.

The skopeo stage is not yet added by the live image pipeline.

The following changes in the manifests should affect the final image:
- Removed RPMs in the build root: packages are added to the build-root
  on-demand by the payload pipelines.  All removed packages were
  unnecessary for building the image.
- /usr/bin/tar removed form selinux stage: tar should be added on-demand
  if it's needed by any of the payload pipelines.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
12cbce58cb distro/rhel9: conditionally add RHSM to config
Only add the RHSM options to RHEL and not CS9.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
85a2029b4a manifest: support RHSM stage in OS pipeline 2022-11-28 17:20:49 +01:00
Achilleas Koutsou
89202bdda8 manifest: don't add hostname stage if not specified
Skip the hostname stage if the hostname option is an empty string.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
0c6bbc6dff distro/rhel9: update qcow2 and openstack to new definitions
Add the image function and remove the pipeline function.
Remove the build package set: build packages are added to the pipeline
on-demand based on platform, environment, etc.
Copied the liveImage ImageKind generator from Fedora.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
51ac3e973e distro/rhel9: add support for imageFunc on image types
Add image and environment to the imageType struct.
Also define two new methods:
PackageSetsNew() and ManifestNew()

These get called by the corresponding base methods PackageSets() and
Manifest() when imageType.image is defined.  They will serve to
transition images to the new framework while also supporting the old
until all image types are updated.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
adc7a1cafa distro/rhel9: separate image type definitions
Split image type definitions into separate files by logical groups
(mostly by footprint and cloud platform) for easier navigation.
The image types are now all defined as package globals, like in Fedora.
Image specific package sets are defined in the file for the image type
grouping instead of the package_sets file.

A notable change is in the AWS/AMI types, which have different configs
based on OS version.  To get around this, we create each image type with
a function parameterised by the OS version string.  This is a bit messy
and inconsistent with other image types, but it's a temporary workaround
which will be cleaned up as the definitions get closer to the new
framework used in the fedora package.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
94eb5ca96d distro/rhel9: configure platforms for each image type
Create platform configurations for each image type based on the ones in
Fedora.
Currently this has no effect on the image definition / manifest.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
a0e05ea192 distro/rhel9: add platform to image type
Add platform attribute to imageType like we do in Fedora.
Currently this has no effect.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
1584770307 platform: add configs for ppc64le and s390x 2022-11-28 17:20:49 +01:00
Achilleas Koutsou
fe5835fece distro/rhel9: move imageType implementation to separate file
Move the imageType type and its methods to a separate file to make
them easier to manage.
This might get rewritten to have a default implementation for a lot of
the common functions that have simple, straightforward functionality
(e.g., return the value of an internal field).
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
4ec14ca5da distro/rhel9: move architecture implementation to separate file
Move the architecture type and its methods to a separate file to make
them easier to manage.
This might get merged with the other implementations to make a default
one because they're all (basically) the same.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
7bcbab0ba5 distro/rhel9: update distribution implementation
Update the implementation of the distro.Distro interface to match the
one in Fedora.  The main change is that the runner is a runner.Runner
and not a string.

The distroMap is replaced by two functions that initialise a
distribution struct from a template (one for CentOS and one for RHEL).
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
310578757a runner: add python or platform-python to all build roots
Required for running osbuild modules.
In Fedora, so far, python3 was always pulled by some other build-root
dependency.  We should add it explicitly since it's required by all our
runners.
For RHEL and CentOS, the requirement is platform-python.

For RHEL 8 and CentOS 8, we also need to explicitly add python36 and for
RHEL 9 and CentOS 9 python3 because it's used to run the stages in
osbuild [1].

[1] ea8261cad6/runners/org.osbuild.rhel82 (L61)
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
5592e6a51f runner: add CentOS runner type
New runner type for CentOS 8 and 9.  Copied from RHEL
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
e9746bed32 test: regenerate all edge-commit and qcow2 manifests
Create new manifests for the new edge-commit-with-container
configuration and the qcow2 for RHEL 9.1
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
04a7cd36c7 format-request-map: new manifests for container embedding
Add manifests for container embedding in edge-commits and qcow2s.
Limit qcow2 container embedding to RHEL 9.1 for now since it's not
supported on Fedora or RHEL 8 yet.
2022-11-28 17:20:49 +01:00
Achilleas Koutsou
3fa012b73a runner: fix RHEL runner (was fedora) 2022-11-28 17:20:49 +01:00
Ondřej Budai
97c8622a0c ci: update Fedora 37 runners to GA
We used pre-GA repositories previously. Since GA is now out, let's switch to
it. We need to do two changes:

- use the latest terraform definitions that use the GA images
- update Schutzfile to use GA repositories (and updates)

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-11-28 16:56:44 +01:00
Sanne Raymaekers
7ae79e7786 osbuild-worker: correctly pass ostree error details
Passing an error as details results in the details being empty after
marshaling. Pass the string formatted error instead.
2022-11-26 00:12:52 +00:00
fkolwa
401a59b6e4 distro/rhel7: Add insights-client to Azure package set
Add insights-client package to azureRhuiCommonPackageSet conditionally
Regenerate test cases for RHEL 7 azure-rhui
2022-11-25 13:15:49 +01:00
Xiaofeng Wang
920431a679 test: Disable secure boot for CS9 on libvirt_test.sh 2022-11-25 03:47:30 +01:00
Xiaofeng Wang
422e70c8ce test: Update nightly compose URL to updates URL
Since RHEL 9.1 and 8.7 released, the nightly compose URL moved to
update URL, like /rhel-8/nightly/updates/RHEL-8/latest-RHEL-8.7.0/
2022-11-25 03:47:30 +01:00
Xiaofeng Wang
f732af9e8b test: Do not enable UEFI secure boot on CS9 2022-11-25 03:47:30 +01:00
schutzbot
1c130ef6ed schutzfile: Update snapshots to 20221115 2022-11-25 03:47:30 +01:00
Simon de Vlieger
3443484502 image: image_installer, use gzip compression 2022-11-24 17:59:56 +01:00
Brian C. Lane
0b864c4306 store: Add blueprints to the stored changes
Previously the blueprint change was only present until a reboot. The
change and its version was recorded, but the actual blueprint was not
saved.

This adds saving the blueprint when it is changed, and loading the old
blueprint versions when they are present in the store.

No version change is needed, if the blueprint is not present it is
loaded as an empty blueprint which was already being detected by
checking the length of the Name.

This will allow specific blueprint changes to be recalled after a
restart of the server.

This also includes tests for the new behavior.
2022-11-24 11:19:14 +01:00
Brian C. Lane
088ca6ec72 client: Add GetBlueprintChangeV1
Add a function to recall a specific blueprint change. Also includes
tests.
2022-11-24 11:19:14 +01:00
Brian C. Lane
e2011652e2 weldr: Add test for new /blueprints/change/... route 2022-11-24 11:19:14 +01:00
Brian C. Lane
a4e361ccd2 weldr: Add /blueprints/change/NAME/COMMIT route
This allows clients to request a specific blueprint change. This can
then be used to locally implement diff support.
2022-11-24 11:19:14 +01:00
Antonio Murdaca
a295ed7c90 check FDO re-encryption happens
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2022-11-24 09:33:00 +00:00
Ondřej Budai
ccb3a67d94 ci: use the latest terraform config to fix missing images
cloud-cleaner deleted all our nightly and CS9 testing images, let's replace
them.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-11-24 09:02:29 +01:00
Achilleas Koutsou
5130be4ccc tools: silence version comparison in get_build_info()
The get_build_info() function uses nvrGreaterOrEqual() to determine how
to parse the output from composer-cli.  The function prints the result
of the comparison but is often used in a loop when waiting for a compose
to finish, which makes CI logs very annoying.

Silence the version comparison output when called form get_build_info().
2022-11-23 13:35:57 +01:00
Tomáš Hozza
f24c6ad952 SPEC: run the %preun commands in worker package only on removal
PR#553 [1] introduced a change to the worker subpackage and replaced
the systemd %preun RPM macro with a literal calls to `systemctl` due
to missing functionality in systemd. However, the change omitted the
fact, that the RPM macro executes commands only on package removal,
not upgrade.

As a result a local or remote worker which is running on the system
while the osbuild-composer-worker package is updated, gets stopped.

Fix the scriptlet and run the commands only on package removal.

[1] https://github.com/osbuild/osbuild-composer/pull/553

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-23 10:42:01 +01:00
Sarita Mahajan
dc3c0d9725 simplified-installer: do not mandate FDO section in simplified provisioning
- build simplified installer iso without mentioning FDO section.
  - change done for rhel8 and rhel9
  - add test case for this use case in test/case/ostree-simplified-installer.shovisioning
  - fixed review comments

Signed-off-by: Sarita Mahajan <sarmahaj@redhat.com>
2022-11-22 11:36:54 +01:00
Sanne Raymaekers
44f4225c02 rhsm: remove CA from consumer secrets
The `/etc/rhsm/ca/redhat-uep.pem` CA is not valid for consumer
certificates.

As a result resolving the ostree ref should use the system's CA cert
pool.
2022-11-21 10:27:15 +01:00
Achilleas Koutsou
6f37df7777 tools/define-compose-url: fix RHEL 9.1 compose URL
RHEL 9.1 currently doesn't have a 'latest' compose.  Use most recent
date-stamped build.
2022-11-21 09:45:26 +01:00
Simon de Vlieger
0e2f8bb115 image-installer: move conditionals into distro
Try and keep `image/image_installer.go` as clean as possible and instead
pass any customizations one layer up.
2022-11-21 09:45:26 +01:00
Simon de Vlieger
d4d100383e image: create installer image types for fedora
This adds the `fedora-image-installer` and
`fedora-image-installer-preview` images.

The image installer type installs anaconda-webui on Fedora >= 38 to use
the new UI. It also writes its setting to
`/usr/share/anaconda/interactive-defaults.ks` as the current
anaconda-webui has not yet been tested in kickstart mode.

To do so manifest.Anaconda was expanded to take a (subset) of options
for a KickstartStage which is will write into interactive-defaults.ks.
And to take a list of additional modules to enable, so we can set up
Anaconda with all default modules.
2022-11-21 09:45:26 +01:00
Christian Kellner
9ea58d1486 disk: align LVM2 volumes to the extent size
When the size of a logical volume is not aligned to the extent size of
the volume group, LVM2 will automatically align it by rounding up[1]:
	Rounding up size to full physical extent 29.80 GiB
	Rounding up size to full physical extent <3.82 GiB

Since we don't take that into account when we create a new volume or
set the size of an existing one, the size for the whole volume group
will be short by that amount and thus the creation of the last volume
will fail:
  	Volume group <uuid> has insufficient free space (975 extents): 977 required.

To fix this a new `AlignUp` method is added to the `MountpointCreator`
creator interface. It will align a given size to the requirements of
the implementing container, like e.g. `LVMVolumeGroup`. It is then
used by a new `alignEntityBranch` which takes a size and walks the
entity path, calling `AlignUp` for all entities that implement said
`MountpointCreator` interface; thus the resulting size should fullfil
the alignment requirement for all elements in the path.
NB: `PartitionTable` already had an `AlignUp` method.

Add a corresponding test.

[1]: 8686657664/lib/metadata/metadata.c (L1072)

Co-authored-by: Achilleas Koutsou <achilleas@koutsou.net>
2022-11-18 18:44:12 +01:00
Christian Kellner
93875576e9 disk/lvm: split up CreateMountpoint
Extract a `CreateLogicalVolume` method from `CreateMountpoint`
and implement the latter via the former. This makes it possible
to create a Logical Volume for an existing payload.
2022-11-18 18:44:12 +01:00
Brian C. Lane
a1a3e5ba2d cloudapi: Add azure-sap-rhui support to cloudapi
and Update openapi.v2.gen.go
by running `./tools/prepare-source.sh` which runs `go generate ./...`
among other things.
2022-11-18 16:53:22 +01:00