Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
Much like the GetPackageSetChains() manifest method, these two new
methods collect the container and ostree source specifications from the
pipelines that support them. Currently, only one pipeline per manifest
contains references to containers or ostree commits, but we collect them
in a map, keyed by the pipeline name, both for consistency with the
package sets and for any potential future changes that may require
differentiating which pipeline a content source belongs to.
Add Content to the Manifest struct to represent unresolved packages,
container, and ostree commit specs that the caller should resolve before
serialising the manifest.
Copy the Marshal and Unmarshal functions from distro.Manifest to
manifest.OSBuildManifest to keep the same behaviour.
The Version() function isn't used, so let's drop it.
Function that filters a list of repositories to return only the global
repos and any that define a given pipeline or package set name in their
list of PackageSets.
Adding support for container embedding.
The containers need to be specified in the image function (imageFunc)
arguments and when specified, propagate down to the OS pipeline
generator to add the necessary stages.
Support is added for RHEL 9.x and Fedora.
Requires a temporary container spec array with the info from the
blueprint for the first initialization of the manifest that's needed
when collecting required packages.
This should be simplified in the future.
The CommitSource was used to specify the source URL and checksum of a
commit for use in manifest sources. Renaming to CommitSpec and adding a
Ref parameter generalises the type so that we can use it to specify
commits in various situations. This is building towards separating when
ostree parameters are used for fetching a commit, fetching a parent
commit, and building one.
The CommitSpec is (very roughly) analogous to the rpmmd.PackageSpec.
For now this encapsulates osbuild export and filename in that
exported tree. In the future we could add MIME type.
For now this is a concrete type, but should probably be an
interface, so the consumer of artefacts know they are the right
type. Enforcing we only push AMIs to EC2, etc.
Similarly to how checkpoints work, each pipeline can be marked for
export, and the manifest can return all the names of the exported
pipelines, to be passed to osbuild. Additionally, the Export
function returns an artefact object, which can be used to know how
to access the exports once osbuild is done. For now, this is unused.
If Checkpoint() is called on a pipeline, it is marked for
checkpointing. Calling GetCheckpoints() returns the names of all
its pipelines that are marked for checkpointing as a slice of
strings. This can be passed to osbuild by the caller, in which case
the trees produced by each of these pipelines will be checkpointed
to speed up future builds.
Before this can be used in production we need a mechanism for
automatically cleaning up the cache.
The kernel name is optional and can be set later.
The bootloader we skip entirely. Instead, set the architecture, which now becomes
mandatory. Use it to deduce the bootloader, and in the future other pipelines can read
this property from the OS Pipeline, rather than having it passed in.
Rather than providing packageSpecs when constructing the
Manifest, do so at Serialize() time.
This allows the overall flow to be:
```
m := Manifest.New()
m.AddPipeline()
...
c := m.GetPackageSetChains()
p := Depsolve(c)
m.Serialize(p)
```
Before you read this patch, be warned: this is crazy. But rest
assured, this is as bad as it gets, and by the end of this PR it
will all be infinitely simpler than it ever was.
We want Manifests to be self-describing, and in particular, we want
manifest.GetPackageSetChains() to return what
ImageType.GetPackages does today. This should be achieved by
pipelines knowing their required package sets, and the caller
optionally amending additional package sets at Pipeline
initialisation time.
As a first step, while pipelines don't yet know anything about their
required package sets, simply pass in the precomputed
PackageSetChains and set them as optional on each pipeline. Then
we can read the chains back out of t he manifest.
This is not a functional change, but allows us to gradually move
package set definitions from the distros to the pipelines in follow
ups.
Allow manifests to be instantiated without providing packageSpecs.
This allows manifests without packageSpecs to be introspected, but
not serialized.
The only reason we used to require packegaSpecs to be passed at
instantiation time was to compute the kernelVer based on the
kernelName and the NEVRAs in the package set. Now move this to be
done at serialize time.
This means that in order to serialize a manifest, you need to pass
the packageSpecs at initialization time, but if you don't need to
serialize, only to introspect it, then you don't need to pass it
at all.
In a future patch packageSpecs will be passed at serialization time
to not have to make that commitment up front.
The new logic is now that before a pipeline can be serialized, all
the pipelines in the manifest must be "prepared" for serialization
by calling serialize_start() on each of them, which does whatever
would in the past have been done at initialization time which
required the pacakgeSpecs. Once serialization has been finished
serialize_end() must be called on each of the pipelines to undo
any of the preparation, making sure we can serialize repeatedly,
possibly while changing the fields of our pipeline in-between,
and that serialization appears from the outside to not have any
side-effects.
In a future PR we may want to rework this logic to instead use a
serialization context.
This returns the package set chains for a given manifest. A package
chain is a list of depsolve requests (include, exclude, repos) to be
performed in sequence as if packages were incrementally installed
to the same image. Each pipeline has its own package set chain, and
the manifest returns a mapping from pipeline name to their
respective chain.
So far this is not used, and always returns the empty chains.
The intention is to eventually make each pipeline (and hence
manifest) self-describing, so they can return exactly the depsolve
requests needed to serialise them, rather than relying on that
information being maintained externally.
Name it `getPackageSpecs()` instead so we can introduce a new
function `getPackages()` that is about package names rather than
full specs, in a follow-up commit.
Panic in case two pipelines with the same name are added to the
manifest. In the future we should deal with allocating fresh names,
but that is not yet needed.
Make it the responsibility of each pipeline to track its required
sources, and use this to generate a manifest from its pipelines.
This removes the dependency on osbuild2 from distro.go, and the
only other uses of osbuild2 are to specify parameters in
ImageConfig, which will eventually go away too.
As a consequence, this only attaches the required sources for a
given manifest, rather than all known packages in the package set
map.
