Test some valid and invalid combinations for the GetTargets() upload
target selection.
Includes tests with and without the upload options for the default
target.
Read the upload target types and options in the UploadTargets array of
the ImageRequest and initialise the Target array. If the top-level
(old) UploadOptions are also specified, prepend them to the array using
the image type's default target type.
Each upload target type is checked against a support map for
compatibility.
Add an array of targets in the imageRequest and return an array from
ImageRequest.GetTargets() (renamed from GetTarget()). Currently, the
function still only returns one target, the default for the image type
with the top level upload options.
Separate the target selection in GetTarget() into two steps. First
determine the default target name for the image type and then use the
name to initialise the target object. This is a bit more work (and
double switching) but will be needed to support selecting targets
externally.
Separate the handling of each individual target type into its own
function called by GetTarget()'s case switch. This makes the function
more readable and the target object creation reusable.
Added an empty line after each creation of irTarget to make it easier to
visually distinguish the cases that fall through.
Add an upload_targets field to the image request. This lets the API
caller specify multiple upload targets and upload options to be used.
If the upload target type does not match the upload options, the request
is invalid.
For backwards compatibility, the upload targets field is optional. If
it is not specified, the default upload target and upload options for
the image type are assumed, which is the same as the old behaviour.
Adding an explicit selection to the request makes it possible to support
multiple upload targets for the same image type. We plan to support
ostree commits being uploaded to both aws.s3 and pulp.
To report on the multiple upload requests, we add an upload_statuses
field to the ImageStatus response.
There is a lot of repeated checks for bp.Customization != nil, this
simplifies that by creating an empty blueprint.Customizations at the
top, and checking to see if it is still empty at the bottom and setting
it back to nil.
Includes a new test for calling with an empty (not nil)
v2.Customizations set on the request.
Include the osbuild/images module version in the Manifest job result.
The module has direct impact on image definitions and the content of
produced manifest, therefore including this information in the Manifest
job result is very helpful for various purposes (debugging,
traceability).
This will enable to embed this information in the Koji build metadata.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the Manifest job result structure to hold information about
osbuild-composer version, which produced the manifest. This will be
useful for other job types which depend on it and can then push this
information further as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add the information about osbuid artifact to the target result.
Specifically the name of the osbuild pipeline which was exported for the
specific target, and the filename of the exported file.
This will later enable embedding this information in Koji build metadata
to make it easy to reproduce the image build using the attached
manifest.
The `KojiTargetResultOptions` previously contained information only
about the uploaded image file. And even then, some information, such as
the filename, were scattered in other structures such as
`KojiFinalizeJob` struct.
Since the plan is to start uploading also osbuild manifest and osbuild
build log to Koji, we need to extend the result options structure to
hold more information and also make it specific to which file is the
information related.
Rework the `KojiTargetResultOptions` to contain information about:
- the built image
- build log
- osbuild manifest
Information about each file contains:
- filename
- checksum type
- file checksum
- file size
For now, only the built image information is set and consumed by the
worker.
Add custom JSON (un)marshaler for `KojiTargetResultOptions` to handle
backward compatibility when old version of worker or composer server
interact with each other. Cover them with unit tests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
These use 'pkg1' when depsolving, so they need an entry in the manifest
with the mocked checksum:
sha256:e50ddb78a37f5851d1a5c37a4c77d59123153c156e628e064b9daa378f45a2fe
Add support to the cloudapi for generating the tailoring file used
to customize the OpenSCAP remediation. This allows users to select and
unselect rules for the remediation and the `autotailor` stage generates
the tailoring file.
During development it can be very useful to store the results locally
instead of uploading to a remote system. This implements a development
only option to help with that.
To use it you need to add OSBUILD_LOCALSAVE to the server's environment.
This can be done by editing /usr/lib/systemd/system/osbuild-composer.service
and adding:
Environment="OSBUILD_LOCALSAVE=1"
You can then use an 'upload_options' object to skip trying to upload to
the default service for the type of image, eg:
"image_requests": [
{
"architecture": "x86_64",
"image_type": "guest-image",
"upload_options": {
"local_save": true
},
...
}]
The results will be saved to /var/lib/osbuild-composer/artifacts/UUID/
using the default filename for the image type.
If local_save is used without OSBUILD_LOCALSAVE being set it will return
an error with id=36 saying 'local_save is not enabled'.
This moves some of the code from the PostCompose function in handler.go
into methods on the OpenAPI ComposeRequest and ImageRequest structs.
In compose.go I have added several methods.
GetBlueprintWithCustomizations takes the ComposeRequest customizations
and builds a Blueprint struct.
GetPayloadRepositories returns the custom payload repos.
GetSubscription returns the ImageOptions setup with optional
subscription information from the request.
In imagerequest.go I have added GetTarget which takes the upload
options and returns a Target. This moves the giant switch statement,
which may also benefit from further simplification at some point.
GetOSTreeOptions returns the OSTree ImageOptions if there are ostree
settings in the ImageRequest.
GetImageOptions returns the distro.ImageOptions with the size set.
This commit only moves the code, making PostCompose easier to read. All
tests still pass.
cloudapi: Move the size handling to a method on ImageRequest
This adds a 'size' parameter to the image_request object. It can be used
to specify the minimum image size in bytes
This behaves in the same way as the size parameter of the weldr API
For raw images the root partition is grown to fill the available space.
For LVM images the PV uses the available space, but the LV does not,
leaving space available for other LVs to be created after boot.
See COMPOSER-1883
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
The duration middleware should come after the tenant channel middleware,
otherwise the tenant in the context will be empty. The status middleware
can come beforehand because it queries the request context right before
sending a response.
This image type produces the same artifact as the current workstation
installer live media.
During the implementation of this new installer some names have been
changed to make a bit more sense in the source tree. Installer images
now always mention which installer they are (anaconda, etc).
Use ostree.ImageOptions for the request parameters instead of a
SourceSpec on the imageRequest.
When preparing the image request, add the ostree values from the API's
compose request to the ostree options on the image options of the image
request.
It's not necessary to create a source spec and it's also not necessary
to add the default ref when it's not specified in the request for an
ostree-based image type. Both of these will be handled by the Manifest
generation based on the ostree options (imageOptions.OSTree). The image
functions will take care of setting any missing parameters or returning
errors if any required parameters are missing.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
Initialise the manifest only once in the enqueue functions
(ImageType.Manifest()) and pass it to the manifest generation function
with the workers and the dependency IDs. The function is renamed from
generateManifest() to serializeManifest() to reflect its new
functionality more accurately. The arguments to the function have also
been trimmed since we no longer need the image type, blueprint, and
image options.
The new functionality of the function is to collect all the resolved
content from the workers and serialize the manifest object.
Use the container sources provided by the content on the initialised
manifest instead of the blueprint to resolve containers. The container
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved containers, but the worker's container resolve job
works on a slice, so we reread the content map to get the pipeline name
(instead of taking the first payload pipeline from the image type).
This requires that there be only one pipeline that embeds containers,
which currently true of all our image types.
Use the commit sources provided by the content on the initialised
manifest instead of the image options to resolve commits. The ostree
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved commit spec, but unlike with the package sets, the
worker's commit resolve job works on a slice, so we reread the content
map to get the pipeline name. This requires that there be only one
pipeline that requires a resolved ostree commit, which is currently true
of all our image types.
Setting the default ostree ref on the image options before calling
Manifest() isn't needed anymore.
The FetchChecksum on ostree.ImageOptions was the resolved commit ID of
the parent ref to be pulled (for ostree commits and containers) or the
commit ID of the content ref (for ostree installers and raw images).
With the new process of manifest creation and serialisation, using the
image options to transport resolved content references is bad and
confusing. Image options should only reflect user and image type
options before any references are resolved. With this change, the
ostree.ImageOptions should only reflect the ostree-related options
specified by the user. Commit IDs will only be available after the
manifest is initialised when the commit sources are resolved (before
serialisation).
