Add a Distro enum to the Manifest struct for selecting package
selection.
Packages are sometimes renamed between distribution versions and since
we do the package selection in the Manifest, we need a way to select
distro-version-specific package names inside the manifest initialiser.
This may change in the future.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
Define a public pipeline implementation that allows initialising with
content, serialising with resolved content, but produces no stages.
This is useful for testing.
Initialise the manifest only once in the enqueue functions
(ImageType.Manifest()) and pass it to the manifest generation function
with the workers and the dependency IDs. The function is renamed from
generateManifest() to serializeManifest() to reflect its new
functionality more accurately. The arguments to the function have also
been trimmed since we no longer need the image type, blueprint, and
image options.
The new functionality of the function is to collect all the resolved
content from the workers and serialize the manifest object.
Use the container sources provided by the content on the initialised
manifest instead of the blueprint to resolve containers. The container
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved containers, but the worker's container resolve job
works on a slice, so we reread the content map to get the pipeline name
(instead of taking the first payload pipeline from the image type).
This requires that there be only one pipeline that embeds containers,
which currently true of all our image types.
Use the commit sources provided by the content on the initialised
manifest instead of the image options to resolve commits. The ostree
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved commit spec, but unlike with the package sets, the
worker's commit resolve job works on a slice, so we reread the content
map to get the pipeline name. This requires that there be only one
pipeline that requires a resolved ostree commit, which is currently true
of all our image types.
Setting the default ostree ref on the image options before calling
Manifest() isn't needed anymore.
OSTree commits are now resolved after manifest initialisation just like
packages and containers. The test commit hash handling is moved to the
resolver function.
Read the ostree options in the compose request into a copy of the struct
that's used to write them.
Read the ostree commits from the content part of the manifest files and
use them in the serialisation function.
In tests (and dev tools) that apply to all image types, set just the
ostree URL instead of all the options.
The default ref is handled by the image functions when needed, so it
doesn't need to be set from the caller.
When initialising a manifest, use the ostree commit source spec only.
Then, when serialising, use the resolved ostree commit.
This is the same process we use for the other content types, packages
and containers.
Two new functions are defined in the distros to handle resolving the
ostree options: one for creating the parent commit source spec and image
ref for image types that have an optional parent commit (ostree commits
and containers) and one for creating the payload commit source spec for
image types that require an ostree payload (ostree raw images and
installers).
The checksum won't be resolved before initialising the manifest (where
checkOptions() is called), so we can't rely on it for validating
options. Instead, we can just make sure that the URL was provided, when
required by the image type.
Also, if a parent ref is specified in the options, a URL *must* be
specified regardless of image type, so verify this combination here
and return the same error message that is returned from
ostree.Resolve().
The parameter combination check will soon be removed from the
ostree.Resolve() function.
Add an ostree commit source (instead of a resolved commit spec) to
pipelines that support ostree commits. Source specs are used when
initialising a manifest for package selection. The resolved commit spec
is added after manifest initialisation through the serialization
function for stage creation.
Pipelines that require or support an ostree commit (either as payload or
a parent) must return the source specs using getOSTreeCommitSources()
after initialisation and the commit specs using getOSTreeCommits()
during serialization.
The FetchChecksum on ostree.ImageOptions was the resolved commit ID of
the parent ref to be pulled (for ostree commits and containers) or the
commit ID of the content ref (for ostree installers and raw images).
With the new process of manifest creation and serialisation, using the
image options to transport resolved content references is bad and
confusing. Image options should only reflect user and image type
options before any references are resolved. With this change, the
ostree.ImageOptions should only reflect the ostree-related options
specified by the user. Commit IDs will only be available after the
manifest is initialised when the commit sources are resolved (before
serialisation).
Pipelines that don't require packages didn't need to implement the
serializeStart() method, but now we need to set the resolved ostree
commit spec when a pipelines requires it.
osbuild can return a json object with details about manifest validation
errors. This adds support for saving those, and printing them when the
Write function is called. eg. when using composer-cli compose log UUID
Includes tests for new behavior.
Creates the 'edge-ami' image type based on edgeRawImage, which generates
a raw image (x86_64, aarch64) ready to upload to AWS EC2.
This 'edge-ami' image type has Ignition support.
Signed-off-by: Irene Diez <idiez@redhat.com>
Our package set was quite outdated. Let's sync it with the Fedora Cloud
kickstart:
https://pagure.io/fedora-kickstarts/blob/c3b160775a3b159f949ba931dcb68e520a460e12/f/fedora-cloud-base.ks
I manually compared built images and our image contain kernel, kernel-modules
and linux-firmware above the official package set:
- removing kernel and kernel-modules is tricky, because we always assume that
the default kernel is called kernel.
- removing linux-firmware is a big hack, since it breaks the RPM dependencies
inside the image.
Thus, let's ignore these for now, it's definitely better than before.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Fedora Cloud SIG considers the qcow2 (cloud base) image as a good fit also for
OpenStack. Let's make our openstack image just an alias of qcow2. Once again,
this removes some duplicated code.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Fedora doesn't build a separate AMI image. The ordinary Cloud Base image
is what's used for AWS.
Here's the code that's responsible for uploading ec2 images - it searches
for Cloud raw.xz images:
fcbface137/fedimg/consumers.py (L114)
And here's the pungi configuration showing that qcow2 and raw.xz are actually
built from the same kickstart:
https://pagure.io/pungi-fedora/blob/e080c0702f38c033025889e5fcc8d9fee5bb2026/f/fedora-cloud.conf#_142
Thus, we can just do the same thing: Let's base the AMI on the qcow2 to save
some bytes.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Before, this was done in the PackageSets() function.
The reason for this is that having an ostree ref affects package
selection (for example, it adds rpm-ostree). At the package selection
phase, it doesn't matter what the ostree ref is; it is just used to
determine if a pipeline is for an ostree-based image type and it doesn't
affect non-ostree-based image types because the image functions ignore
it.
This is only needed in the cloudapi now because other places have
switched to using the new order of operations, where the manifest is
generated after the ostree commit is resolved, so it's always added when
needed.
To add the container specs to Serialize(), we need to map them to the
payload (OS) pipeline. We assume the first name in the image type's
PayloadPipelines() list is the OS pipeline, which is true of all image
types right now but might not be necessarily in the future.
This is a temporary workaround. Eventually, the mapping will be set by
the image type itself when we use the container source specs attached to
the Manifest object.
Still not using the new process for generating the manifest exactly.
This commit only replaces the call to PackageSets() with a call to
Manifest() to get the Content.PackageSets. This is essentially the same
as before, when we were initialising the manifest object twice.
The Manifest() function does a tiny bit more work than PackageSets(),
but it's minimal and we gain the benefit of only having a single code
path and, although it's run twice, it's always run in the same way.
Use the new workflow for generating the manifest before resolving
containers.
The resolver function is adjusted to handle a map of container slices,
but with all current use cases, the map should only ever have one key
for the payload (os) pipeline.
Demonstrate the new workflow for resolving containers.
1. First call Manifest().
2. Get container SourceSpecs from manifest struct.
3. Resolve them.
4. Serialize() with resolved container specs.
The changes in the test manifests are just the information about the
container sources (was a slice but is now a map) and the actual manifest
object isn't affected.
The TestDistro_Manifest test in distro_test_common is adapted
accordingly as well.
When creating a Manifest object, collect container SourceSpecs instead
of resolved Specs.
This is the same way we handle packages: The blueprint option is
converted to source specs and attached to the Manifest object during
creation. Later, the SourceSpecs will be resolved to full container
Specs and used during serialization.
Much like the GetPackageSetChains() manifest method, these two new
methods collect the container and ostree source specifications from the
pipelines that support them. Currently, only one pipeline per manifest
contains references to containers or ostree commits, but we collect them
in a map, keyed by the pipeline name, both for consistency with the
package sets and for any potential future changes that may require
differentiating which pipeline a content source belongs to.
The ImageType.PackageSets() function is going away and instead we will
rely on the ImageType.Manifest() function to both prepare the manifest
and return the package sets. The Manifest() function should never be
called without an ostree ref for ostree type images.
Use the new manifest generation procedure in the distro tests.
Use assert instead of require in TestImageTypePipelineNames to continue
running the rest of the subtests after a failure.
Some initialisations (image options and blueprint customizations) had to
be adjusted to work with the ImageType.Manifest() function.
Some helper functions in distro_test_common are no longer necessary and
have been removed.
The TestPipelineRepositories and TestImageTypePipelineNames tests must
be (partially) skipped for image types which specify a workload
(currently only azure-eap7-rhui), because they ignore payload
repositories.
In getPackageSetChain(), the workload repositories did not include the
ExtraBaseRepos.
In serialize(), when creating the rpm stage options (which collects
repository GPG keys), only the base repos were used, which is why we
previously had to merge repositories. Instead of merging repositories
in the calling function in distro, we should keep them separated so that
we can easily distinguish which repositories are only meant for the
blueprint or workload when we need to.
The merging of payload repositories into the os pipeline had the
unwanted side effect of using the payload repos for the first depsolve
in the os chain when instead they should only be used for the second
(the depsolve for the blueprint or workload packages). This wasn't an
issue before because we didn't do the merging in the PackageSets()
function, but now we rely on the Manifest() function for that
functionality instead.
Before the merging of the two functions, the PackageSets() function did
not merge repositories and the repository-to-package-set mapping was
maintained correctly, but the merging was needed in the Manifest()
function so that rpm stage options were generated for all repositories.
With this change, we are removing the merging so that the mapping is
maintained, and will fix the rpm stage option generation in the pipeline
generator.
Use the new manifest generation procedure in the Weldr API.
Updated test distro to include the same packages from the PackageSets()
method in the Manifest.Content.PackageSets.
Copy the functionality of the ImageType.PackageSets() methods into
ImageType.Manifest() for each distro.
The Manifest() method now collects all package sets and repositories
from the blueprint and image type and after generating the Manifest
instance, calls the GetPackageSetChains() method to attach the computed
package sets to the Manifest before returning it.
The package sets in the call are now renamed to staticPackageSets to
differentiate from the dynamic (computed) package sets that are affected
by the manifest generation.
Pass the entire Blueprint to Manifest() instead of just the
Customizations. The goal is to combine the functionality of the
ImageType.PackageSets() and ImageType.Manifest() methods into one call.
