Openshift keeps sending requests to composer for a while after it has
sent SIGTERM, this results in requests being reset randomly on
shutdown. Waiting a few seconds before terminating mitigates this.
Weldr makes assumptions about the names of the package sets. This
does not work in all cases, so should be reworked, but for now just
do enough that we don't regress.
This is meant for rapid prototyping of single image types and for
osbuild development, as an alternative to osbuild-mpp. The same
primitives are used as in the image definitions, but without any
policy or inheritance applied.
The user is expected to only edit `playground.go` and then run
the tool to produce osbuild manifests.
A runner is used to run stages in a build pipeline. It is not about
running them on the host, in that case the runner is autodetected.
Not a functional change.
The build packages should always be computed from its
dependents and there should be no need to override it. Drop the
ability, and all the unused code in fedora/*.
NOTE: due to a bug in a previous patch in this PR the extra
package set was never set on the build pipelines.
No longer name the packageSetChains after the package set, but
keep them named after the pipelines. This should be a
non-functional change as dnf-json does not care about what the
chains are called, only that the names are unique.
squashfs-tools is used by the mono stage, and lorax by lorax.
For now copy over the anaconda boot package set as-is, we can
refactor further from the pipelines.
Rather than providing packageSpecs when constructing the
Manifest, do so at Serialize() time.
This allows the overall flow to be:
```
m := Manifest.New()
m.AddPipeline()
...
c := m.GetPackageSetChains()
p := Depsolve(c)
m.Serialize(p)
```
Before you read this patch, be warned: this is crazy. But rest
assured, this is as bad as it gets, and by the end of this PR it
will all be infinitely simpler than it ever was.
We want Manifests to be self-describing, and in particular, we want
manifest.GetPackageSetChains() to return what
ImageType.GetPackages does today. This should be achieved by
pipelines knowing their required package sets, and the caller
optionally amending additional package sets at Pipeline
initialisation time.
As a first step, while pipelines don't yet know anything about their
required package sets, simply pass in the precomputed
PackageSetChains and set them as optional on each pipeline. Then
we can read the chains back out of t he manifest.
This is not a functional change, but allows us to gradually move
package set definitions from the distros to the pipelines in follow
ups.
The package sets for an image can depend on the blueprint, and
by the same logic there is no reason it should not be able to
depend on the image options.
This is so far a non-functional change, but makes a follow-up
commit simpler (though still without actually depending on
the image options to compute the package sets).
Allow manifests to be instantiated without providing packageSpecs.
This allows manifests without packageSpecs to be introspected, but
not serialized.
The only reason we used to require packegaSpecs to be passed at
instantiation time was to compute the kernelVer based on the
kernelName and the NEVRAs in the package set. Now move this to be
done at serialize time.
This means that in order to serialize a manifest, you need to pass
the packageSpecs at initialization time, but if you don't need to
serialize, only to introspect it, then you don't need to pass it
at all.
In a future patch packageSpecs will be passed at serialization time
to not have to make that commitment up front.
The new logic is now that before a pipeline can be serialized, all
the pipelines in the manifest must be "prepared" for serialization
by calling serialize_start() on each of them, which does whatever
would in the past have been done at initialization time which
required the pacakgeSpecs. Once serialization has been finished
serialize_end() must be called on each of the pipelines to undo
any of the preparation, making sure we can serialize repeatedly,
possibly while changing the fields of our pipeline in-between,
and that serialization appears from the outside to not have any
side-effects.
In a future PR we may want to rework this logic to instead use a
serialization context.
Allow the packages returned by a pipeline to be extended by setting
the optional property ExtraPackages.
In the case of OSPipeline, a further UserPackages property is added.
This allows the caller to set the set of packages that should be
depsolevd in a second transaction, on top of the one for the base
packages.
This functionality is currently unused so this is a noop.
A build pipeline now tracks all its dependents, and each pipeline
now implements a `getBuildPackages()` method. For now those
return the empty slice, but will be extended by each pipeline in the
future.
`getPackageSetChain() of the build pipeline simply collects all the
build package sets of its dependents.
This returns the package set chains for a given manifest. A package
chain is a list of depsolve requests (include, exclude, repos) to be
performed in sequence as if packages were incrementally installed
to the same image. Each pipeline has its own package set chain, and
the manifest returns a mapping from pipeline name to their
respective chain.
So far this is not used, and always returns the empty chains.
The intention is to eventually make each pipeline (and hence
manifest) self-describing, so they can return exactly the depsolve
requests needed to serialise them, rather than relying on that
information being maintained externally.
Name it `getPackageSpecs()` instead so we can introduce a new
function `getPackages()` that is about package names rather than
full specs, in a follow-up commit.
Panic in case two pipelines with the same name are added to the
manifest. In the future we should deal with allocating fresh names,
but that is not yet needed.
Make it the responsibility of each pipeline to track its required
sources, and use this to generate a manifest from its pipelines.
This removes the dependency on osbuild2 from distro.go, and the
only other uses of osbuild2 are to specify parameters in
ImageConfig, which will eventually go away too.
As a consequence, this only attaches the required sources for a
given manifest, rather than all known packages in the package set
map.
Add a plain `rhel-8` alias as the default distribution name and version
for the `rhel8` package. The `rhel-86` distro is still available via
the NewRHEL86() constructor. These two distributions are identical.
Repositories
------------
The rhel-8 repositories (repositories/rhel-8.json) are now set to the
CDN repositories with no minor version:
https://cdn.redhat.com/content/dist/rhel8/8/...
The rhel-8 test repositories (test/data/repositories/rhel-8.json) were
already set to the plain `8` repositories. The Google repos have been
added.
The test case generator repositories used for `rhel-8` are the rpmrepo
snapshots as for rhel-86.
With 8.3 support dropped, the rhel86 package defines all the supported
RHEL 8 versions except 8.4.
RHEL 8.4 will be merged into the rhel8 package soon.
The rhel8 package represented RHEL 8.3, which is EOL.
The current rhel86 package will be renamed to rhel8 and be responsible
for building all RHEL 8 minor versions.
Drop the fallback to the `assembler` export if no is specified in the
job and return Job Error in this case.
Remove the constraint to support only a single osbuild export. The job
is now able to use multiple osbuild exports and each target may use a
different one.
The osbuild export is specific to the upload target and different
targets may require using a different export. While osbuild-composer
still does not support multiple exports for osbuild jobs, this prepares
the ground for such support in the future.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
message is always a super-set of the old and new way of providing the
exports to osbuild job.
Weldr API already does not rely on this code and nothing else uses it.
Since the code has been used only on-premise, where we expect the
composer and workers to be always of the same version, there is no need
to keep backward compatibility in the worker.
Stop relying on the server interpreting the set `ImageName` in the
`OSBuildJob` as a signal to upload the image back to the worker server
and add an explicit "Worker Server" upload target to the job.
The uploading of artifacts back to the worker server for the on-premise
(Weldr) use case was signaled to the worker by setting the `ImageName`
in the `OSBuildJob` definition. The code also relies on the osbuild
exports being specified in the `OSBuildJob`, instead of in the target
(this is not implemented yet).
Prepare the ground for moving osbuild export definition from
`OSBuildJob` to `Target` by introducing an explicit `Worker Server"
upload target. This target will signal to the worker that it should
upload the image back to the worker server. The new target is not yet
used by any API implementation.
Extend the worker osbuild job implementation to handle the new upload
target.
The `TestKojiJobTypeValidation` unit test used the `OSBuildJob` and
Koji Target fields in the wrong way, however without any impact on the
testing itself. The reason is that no actual images were built as part
of the test nor the created jobs were ever picked up by a worker.
This change was forgotten in PR#2758.
[1] https://github.com/osbuild/osbuild-composer/pull/2758
The filename of the image as produced by osbuild for a given export is
currently set in each target options type in the `Filename` struct
member. However, the value is not really specific to any target type,
but to the specific export used for the target. For this reason move the
value form target type options to the `Target` struct inside a new
struct `OsbuildArtifact` under the name`ExportFilename`.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
object is always a super-set of the old and new way of providing the
export filename in the Target.
The `Filename` previously set in the `gcpUploadSettings` does not
provide any value. It is the filename of the image as produced by
osbuild for a given export. It may not correspond with the object name
when the image is uploaded to GCP storage and may not even correspond
with the image name after it is imported to GCE. Stop setting the value
and remove the variable from data structures.
This change should not have any impact on backward compatibility,
because the field will be ignored when (Un)Marshalling.
Some variable names used in the `OsBuildJob` `Run()` method were not
very self-explanatory, which made the code harder to understand and
navigate. These were `args`, `options`, `t`. Rename them to be more
self-explanatory of their purpose.
Completely remove the use of `local` target from all code, which is not
required to keep backward compatibility. The target has not been used in
composer for some time already, but some unit tests still used its data
structures. Mark the target as deprecated and adjust all unit tests that
depended on it.
The backward compatibility is kept mostly to enable long running
osbuild-composer instances, which were upgraded to still read old jobs
from the store.
While a target with the same intention will be reintroduced, the current
`local` target data structures contain many fields which would not be
relevant for the new target.
In addition, while the "local" target will be ever used only by Weldr
API, the name would be a bit misleading. Although the worker usually
runs on the same system when using Weldr API, there is no hard
requirement enforcing this setup. In reality, the worker will be
uploading the image back to the worker server, so there is room for a
better name.
A backward compatibility code handling the conversion of VMDK image to
stream-optimized sub-format has been kept in the implementation since
PR#2529 [1] merged on May 4th 2022. Since this change, no API
implementation is submitting jobs, which would hit this conversion code,
because VMDK images are already being produced in the desired
sub-format.
On-premise deployments are expected to use the same composer and worker
versions. There are no composer / worker instances in production, which
are not running the modified code.
Delete the backward compatibility code.
[1] https://github.com/osbuild/osbuild-composer/pull/2529
Modify the `OsBuildJob` implementation to handle multiple upload targets
in a cycle. However, there is still no API implementation, which would be
adding `OsBuildJobs` with multiple targets to the job queue.
The limitations are that only a single osbuild export is supported, and
the same artifact will be used for each target.
At the end of the job, errors from all targets are gathered. In case
there are none, the job succeeds. In case at least one target failed,
the job fails as well. In such a case, a slice of errors from all failed
targets is added to the job error as details.
