Edge and IoT manifests are modified from the new option handling. The
"parent" commit ID isn't specified in the options anymore, but it is
(fake) resolved by the manifest generator.
Of particular note is the iot-raw-image manifest that now properly uses
the commit ID in the copy stage for the firmware.
The resolved ostree commits are now stored in the content part of the
manifest metadata alongside package specs and containers.
Add a checksum as a hash of URL + Ref.
Use the parent ref instead of the image ref when it's set. This makes
the test distro always behave like ostree commit and container types
(image types that can use an ostree parent) and not raw image or
installers (that use ostree commits as a payload).
Modify the weldr API test with the expected error message.
Use ostree.ImageOptions for the request parameters instead of a
SourceSpec on the imageRequest.
When preparing the image request, add the ostree values from the API's
compose request to the ostree options on the image options of the image
request.
It's not necessary to create a source spec and it's also not necessary
to add the default ref when it's not specified in the request for an
ostree-based image type. Both of these will be handled by the Manifest
generation based on the ostree options (imageOptions.OSTree). The image
functions will take care of setting any missing parameters or returning
errors if any required parameters are missing.
Change the OSTreeResolveSpec to match the ostree SourceSpec by removing
the Parent field.
Change OSTreeResolveResultSpec to match the CommitSpec by adding the
Secrets field. The RHSM field is kept for backwards compatibility with
older workers.
The Resolve() function is now only responsible for resolving a
SourceSpec to a CommitSpec. It only resolves a checksum if a URL is set
and sets the option for the RHSM secrets.
The Parent has been removed from the SourceSpec. The SourceSpec is a
simple reference to a single ostree ref and has no connection with the
ostree options.
The ostree options for a compose request should be specified using the
ImageOptions struct, not the source spec. The source spec should be
specifying the source parameters for a single ostree commit internally.
Add a Distro enum to the Manifest struct for selecting package
selection.
Packages are sometimes renamed between distribution versions and since
we do the package selection in the Manifest, we need a way to select
distro-version-specific package names inside the manifest initialiser.
This may change in the future.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
Define a public pipeline implementation that allows initialising with
content, serialising with resolved content, but produces no stages.
This is useful for testing.
Initialise the manifest only once in the enqueue functions
(ImageType.Manifest()) and pass it to the manifest generation function
with the workers and the dependency IDs. The function is renamed from
generateManifest() to serializeManifest() to reflect its new
functionality more accurately. The arguments to the function have also
been trimmed since we no longer need the image type, blueprint, and
image options.
The new functionality of the function is to collect all the resolved
content from the workers and serialize the manifest object.
Use the container sources provided by the content on the initialised
manifest instead of the blueprint to resolve containers. The container
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved containers, but the worker's container resolve job
works on a slice, so we reread the content map to get the pipeline name
(instead of taking the first payload pipeline from the image type).
This requires that there be only one pipeline that embeds containers,
which currently true of all our image types.
Use the commit sources provided by the content on the initialised
manifest instead of the image options to resolve commits. The ostree
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved commit spec, but unlike with the package sets, the
worker's commit resolve job works on a slice, so we reread the content
map to get the pipeline name. This requires that there be only one
pipeline that requires a resolved ostree commit, which is currently true
of all our image types.
Setting the default ostree ref on the image options before calling
Manifest() isn't needed anymore.
OSTree commits are now resolved after manifest initialisation just like
packages and containers. The test commit hash handling is moved to the
resolver function.
Read the ostree options in the compose request into a copy of the struct
that's used to write them.
Read the ostree commits from the content part of the manifest files and
use them in the serialisation function.
When a test manifest requires a commit to be resolved for content, fake
the commit ID resolution deterministically by hashing the URL + ref.
Store the resolved commit spec with the manifest metadata alongside the
other content (packages and containers).
Also add the secrets field if RHSM is true, which is now supposed to be
done by the resolver.
In tests (and dev tools) that apply to all image types, set just the
ostree URL instead of all the options.
The default ref is handled by the image functions when needed, so it
doesn't need to be set from the caller.
When initialising a manifest, use the ostree commit source spec only.
Then, when serialising, use the resolved ostree commit.
This is the same process we use for the other content types, packages
and containers.
Two new functions are defined in the distros to handle resolving the
ostree options: one for creating the parent commit source spec and image
ref for image types that have an optional parent commit (ostree commits
and containers) and one for creating the payload commit source spec for
image types that require an ostree payload (ostree raw images and
installers).
The checksum won't be resolved before initialising the manifest (where
checkOptions() is called), so we can't rely on it for validating
options. Instead, we can just make sure that the URL was provided, when
required by the image type.
Also, if a parent ref is specified in the options, a URL *must* be
specified regardless of image type, so verify this combination here
and return the same error message that is returned from
ostree.Resolve().
The parameter combination check will soon be removed from the
ostree.Resolve() function.
Add an ostree commit source (instead of a resolved commit spec) to
pipelines that support ostree commits. Source specs are used when
initialising a manifest for package selection. The resolved commit spec
is added after manifest initialisation through the serialization
function for stage creation.
Pipelines that require or support an ostree commit (either as payload or
a parent) must return the source specs using getOSTreeCommitSources()
after initialisation and the commit specs using getOSTreeCommits()
during serialization.
The FetchChecksum on ostree.ImageOptions was the resolved commit ID of
the parent ref to be pulled (for ostree commits and containers) or the
commit ID of the content ref (for ostree installers and raw images).
With the new process of manifest creation and serialisation, using the
image options to transport resolved content references is bad and
confusing. Image options should only reflect user and image type
options before any references are resolved. With this change, the
ostree.ImageOptions should only reflect the ostree-related options
specified by the user. Commit IDs will only be available after the
manifest is initialised when the commit sources are resolved (before
serialisation).
Pipelines that don't require packages didn't need to implement the
serializeStart() method, but now we need to set the resolved ostree
commit spec when a pipelines requires it.
1. Run RHEL for Edge CI on osbuild/rhel-edge-ci repo
2. Use released RHEL 8.8 and 9.2 boot ISO
3. Extend VM memory to 3072 on ostree.sh to fix error
"Overriding memory to 3072 MiB needed for centos-stream9 network install."
4. Install and start firewalld, configure VM network as trusted zone
osbuild can return a json object with details about manifest validation
errors. This adds support for saving those, and printing them when the
Write function is called. eg. when using composer-cli compose log UUID
Includes tests for new behavior.
Creates the 'edge-ami' image type based on edgeRawImage, which generates
a raw image (x86_64, aarch64) ready to upload to AWS EC2.
This 'edge-ami' image type has Ignition support.
Signed-off-by: Irene Diez <idiez@redhat.com>
cloud-init and bash should be everywhere. Thus, there's no point in specifying
them as a customization. Actually, it might mask error if we ever stop
installing bash/enabling cloud-init.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
