Don't pass blueprint Users and Groups options all the way down to the
osbuild stage bindings. Instead, convert them to the internal
users.User and users.Group structs.
Ideally we would do this even higher up in the code path, before
reaching the distro, but this is the first step towards that.
- Remove stage-specific input types when they are org.osbuild.tree input
types.
- Use PipelineTreeInputs when stage requires a single tree input
reference with an arbitrary key.
- For Stages that require a specific key with a tree input, make the key
part of the NewXStage() function and only allow specifying the name of
the pipeline from which to copy the tree as part of the function
arguments.
Using new() to create a new struct assigns an empty struct to the
variable, meaning it can never be tested for nil. This means this code
would never detect a missing kernel package.
Extract the non-RHUI specific package set and image configuration into a
common definitions, which will be used by both image types.
Redefine the package sets and default image configuration used by both
image types to inherit from a common definition.
Regenerate image manifests for RHEL-8 / c8s `vhd` and `azure-rhui`
images.
There is no change in the resulting manifest for the `azure-rhui` image
type. However there are substantial changes to the `vhd` image
definition, which is now almost identical to the `azure-rhui` image
type, to provide consistent experience regardless if using RHUI or not.
The default partition table used by the `vhd` image type has been kept
as it was before, since there is yet no consensus on what size to
standardize for both image types.
Move all code related to Azure / VHD images to a separate file,
similarly as it is done in rhel7 distro. This approach makes it easier
to find all the code related to a specific image type family.
Don't redefine the storage unit multiples in each distro, but use the
constants defined in the `common` package. This will make it easier to
split related image type definitions into separate files.
Using basic types as values in the `ImageConfig` structure makes it
impossible to distinguish if the empty value for the type was set
intentionally or if it is just the value the variable was initialized
to. This is very bad especially for `bool` type.
While working on unifying `vhd` and `azure-rhui` image types I found
out, that some newly added variables in the `ImageConfig` structure
were forgotten in the `InheritFrom()` method. This makes it impossible
to inherit their values from a parent configuration. This is however
required for the unification of `vhd` and `azure-rhui` image types. As
described above, it would be impossible to decide whether a `bool` value
should be inherited from the parent configuration or not. The only
solution is to use a pointer to the type. For consistency, use pointer
for all basic types.
Adjust distro implementations accordingly.
Since the oscap remediation stage in osbuild runs
the oscap package in `chroot`, it is necessary to
install the `openscap-scanner` package to the image
itself rather than the build root.
The repo is not needed any more, because the Google Cloud SDK is not
installed in the images by default. If anyone wants to install the SDK,
they can add the appropriate repo definition.
The Google SDK ships pre-compiled binaries. It is undesirable to install
it by default in `gce` and `gce-rhui` in its current shape. Also not
installing it does not anyhow affect the RHEL integration as the guest
OS in GCP.
Since the LVM support was added to all distros, our disk
related code is adaptive, i.e. we will set the correct BLS
and grub2 prefix if there a `boot` partiton is present in
the layout after all customizations happen, which includes
LVMification.
One thing that was not yet fully working was layouts that
do not yet have a `/boot` partition but allow LVMification.
In that case `NewPartitionTable` and if `/boot` was the
first (or only) customization, would LVMify the partition
which in turn would create the `/boot` partition; but after
`newPT.ensureLVM()` the call to `newPT.createFilesystem`
with `/boot` would try to create another `/boot` mountpoint.
In order to deal with this situation correctly we are now
using a two phase approach: 1) enlarge existing mountpoints
and collect new ones. 2) if there are new ones and LMVify
was allowed, switch to LVM layout. Do a second pass and now
create or enlarge existing partitions, handling `/boot` in
the process.
Add basic validation to ensure that the oscap
customizations are valid and required fields
have been provided. The validation also ensures
that the manifest generation errors out if
oscap customization has been enabled for older
or unsupported distros.
Add support for embedding containers in OSTree commits by
storing them in `/usr/share/containers/storage`. The storage
engine is configured accordingly so that this extra location
is automatically taken into account by e.g. `podman`.
Support for adding containers in non-ostree images. The reason we
don't support OSTree artefacts just yet is that the default storage
location for container is `/var/lib/containers/storage`. But for
OSTree images all content in `/var` is discarded, since that is
deployment specific data. We therefore need to store the containers
somewhere else, e.g. `/usr/share/containers/storage`, but then also
need to configure the system to find containers in that location.
osbuild only recently gained the corresponding stage to do so and
thus this will be done in a follow up.
This is the first step to support embedding container images. Here
we add the `containers []container.Spec` argument to supply images
with resolved container specifications. For now all distros will
return an error in case a container is actually supplied since none
of them currently support embedding containers. NB: also no apis or
tools will actually resolve containers.
edge-raw and edge-simplified-installer: only on 8.6+
ec2 and ec2-ha: available on all RHEL 8
ec2-sap: available on 8.4 and 8.6+ (no 8.5)
The ec2-sap image requires ansible, which in 8.4 is called `ansible` and
was replaced by `ansible-core` in 8.6.
The package sets for an image can depend on the blueprint, and
by the same logic there is no reason it should not be able to
depend on the image options.
This is so far a non-functional change, but makes a follow-up
commit simpler (though still without actually depending on
the image options to compute the package sets).
Add a plain `rhel-8` alias as the default distribution name and version
for the `rhel8` package. The `rhel-86` distro is still available via
the NewRHEL86() constructor. These two distributions are identical.
Repositories
------------
The rhel-8 repositories (repositories/rhel-8.json) are now set to the
CDN repositories with no minor version:
https://cdn.redhat.com/content/dist/rhel8/8/...
The rhel-8 test repositories (test/data/repositories/rhel-8.json) were
already set to the plain `8` repositories. The Google repos have been
added.
The test case generator repositories used for `rhel-8` are the rpmrepo
snapshots as for rhel-86.
With 8.3 support dropped, the rhel86 package defines all the supported
RHEL 8 versions except 8.4.
RHEL 8.4 will be merged into the rhel8 package soon.
The rhel8 package represented RHEL 8.3, which is EOL.
The current rhel86 package will be renamed to rhel8 and be responsible
for building all RHEL 8 minor versions.
Move package set chain collation to the distro package and add
repositories to the package sets while returning the package sets from
their source, i.e., the ImageType.PackageSets() method.
This also removes the concept of "base repositories". There are no
longer repositories that are added implicitly to all package sets but
instead each package set needs to specify *all* the repositories it will
be depsolved against.
This paves the way for the requirement we have for building RHEL 7
images with a RHEL 8 build root. The build root package set has to be
depsolved against RHEL 8 repositories without any "base repos" included.
This is now possible since package sets and repositories are explicitly
associated from the start and there is no implicit global repository
set.
The change requires adding a list of PackageSet names to the core
rpmmd.RepoConfig. In the cloud API, repositories that are limited to
specific package sets already contain the correct package set names and
these are now copied to the internal RepoConfig when converting types in
genRepoConfig().
The user-specified repositories are only associated with the payload
package sets like before.
Modify pipelines in all distro definitions to produce stream-optimized VMDK
image.
Regenerate all VMDK test cases.
Bump worker dependency on osbuild to the version supporting VMDK
subformat in both QEMU assembler and stage
Introduce a new method `PackageSetsChains()` to the `ImageType`
interface, which returns a named lists of package sets, which should be
depolved together in a chain.
Extend all distro implementations with the new method.
Add a unit test ensuring that if an image type defines some package set
name chains, that all of the listed package set names are present in the
package set map returned by the image type.
The method is currently not used anywhere. This is a preparation for
switching from current way of depsolving to the chain depsolving.
Use single NewGroupsStageOptions() from osbuild1 and osbuild2 instead of
implementing in each distro.
- Followup from 2eef6e6e2d, copied to the
rest of the RHEL distro definitions.
- Added NewGroupsStageOptions() to osbuild1 for rhel8 and rhel84.
NB: The change was not made in the Fedora distro definitions as they are
currently being rewritten.
Use single NewUsersStageOptions() from osbuild1 and osbuild2 instead of
implementing in each distro.
- Followup from ca8b371142, copied to the
rest of the RHEL distro definitions.
- Added NewUsersStageOptions() to osbuild1 for rhel8 and rhel84.
NB: The change was not made in the Fedora distro definitions as they are
currently being rewritten.
This is needed so that we can do different things depending on the
given layout; this will be used in tests for now only. Only GPT
allows for arbitrary number of partitions and once we assert this
in code we will need to adjust the tests accordingly.
NB: This method might be removed again in the future, once generic
LVM support is added everywhere and the ability to differentiate
between MBR and GPT layouts is not needed anymore.
Do not rely on `distro.imageOptions` having any size information,
i.e. `Size` being `0`. Instead use `imageType.Size()` and the
information in the blueprint customization to calculate the size.
This makes the individual distro definitions idenpendent of the
API entry points that currently calculate the size, e.g.:
internal/cloudapi/v1/v1.go:PostCompose line 184
internal/cloudapi/v2/v2.go:PostCompose line 197
internal/kojiapi/server.go:PostCompose line 135
internal/weldr/api.go:composeHandler line 2289
Each image type now implements BuildPipelines(), which returns a list of
pipeline names that set up the build environment, and
PayloadPipelines(), which returns a list of pipeline names that create
the OS image (all non-build pipeline names).
Older distros that produce v1 manifests should call the distro Fallback
functions to return the common defaults.
A Fallback function for the Exports() method is also added and called by
older distros.
All image types that produce v2 manifests (distros after RHEL 8.4)
should include the information in the image type definition and should
not rely on fallbacks for default values.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
The problem: osbuild-composer used to have a rather uncomplete logic for
selecting client certificates and keys while fetching data from
repositories that use the "subscription model". In this scenario, every
repo requires the user to use a client-side TLS certificate. The problem
is that every repo can use its own CA and require a different pair of
a certificate and a key. This case wasn't handled at all in composer.
Furthermore, osbuild-composer can use remote workers which complicates
things even more.
Assumptions: The problem outlined above is hard to solve in the general
case, but Red Hat Subscription Manager places certain limitations on how
subscriptions might be used. For example, a subscription must be tight to
a host system, so there is no way to use such a repository in osbuild-composer
without it being available on the host system as well.
Also, if a user wishes to use a certain repository in osbuild-composer it
must be available on both hosts: the composer and the worker. It will come
with different pair of a client certificate and a key but otherwise, its
configuration remains the same.
The solution: Expect all the subscriptions to be registered in the
/etc/yum.repos.d/redhat.repo file. Read the mapping of URLs to certificates
and keys from there and use it. Don't change the manifest format and let
osbuild guess the appropriate subscription to use.
