During development it can be very useful to store the results locally
instead of uploading to a remote system. This implements a development
only option to help with that.
To use it you need to add OSBUILD_LOCALSAVE to the server's environment.
This can be done by editing /usr/lib/systemd/system/osbuild-composer.service
and adding:
Environment="OSBUILD_LOCALSAVE=1"
You can then use an 'upload_options' object to skip trying to upload to
the default service for the type of image, eg:
"image_requests": [
{
"architecture": "x86_64",
"image_type": "guest-image",
"upload_options": {
"local_save": true
},
...
}]
The results will be saved to /var/lib/osbuild-composer/artifacts/UUID/
using the default filename for the image type.
If local_save is used without OSBUILD_LOCALSAVE being set it will return
an error with id=36 saying 'local_save is not enabled'.
This moves some of the code from the PostCompose function in handler.go
into methods on the OpenAPI ComposeRequest and ImageRequest structs.
In compose.go I have added several methods.
GetBlueprintWithCustomizations takes the ComposeRequest customizations
and builds a Blueprint struct.
GetPayloadRepositories returns the custom payload repos.
GetSubscription returns the ImageOptions setup with optional
subscription information from the request.
In imagerequest.go I have added GetTarget which takes the upload
options and returns a Target. This moves the giant switch statement,
which may also benefit from further simplification at some point.
GetOSTreeOptions returns the OSTree ImageOptions if there are ostree
settings in the ImageRequest.
GetImageOptions returns the distro.ImageOptions with the size set.
This commit only moves the code, making PostCompose easier to read. All
tests still pass.
cloudapi: Move the size handling to a method on ImageRequest
This adds a 'size' parameter to the image_request object. It can be used
to specify the minimum image size in bytes
This behaves in the same way as the size parameter of the weldr API
For raw images the root partition is grown to fill the available space.
For LVM images the PV uses the available space, but the LV does not,
leaving space available for other LVs to be created after boot.
See COMPOSER-1883
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
The duration middleware should come after the tenant channel middleware,
otherwise the tenant in the context will be empty. The status middleware
can come beforehand because it queries the request context right before
sending a response.
This image type produces the same artifact as the current workstation
installer live media.
During the implementation of this new installer some names have been
changed to make a bit more sense in the source tree. Installer images
now always mention which installer they are (anaconda, etc).
Use ostree.ImageOptions for the request parameters instead of a
SourceSpec on the imageRequest.
When preparing the image request, add the ostree values from the API's
compose request to the ostree options on the image options of the image
request.
It's not necessary to create a source spec and it's also not necessary
to add the default ref when it's not specified in the request for an
ostree-based image type. Both of these will be handled by the Manifest
generation based on the ostree options (imageOptions.OSTree). The image
functions will take care of setting any missing parameters or returning
errors if any required parameters are missing.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
Initialise the manifest only once in the enqueue functions
(ImageType.Manifest()) and pass it to the manifest generation function
with the workers and the dependency IDs. The function is renamed from
generateManifest() to serializeManifest() to reflect its new
functionality more accurately. The arguments to the function have also
been trimmed since we no longer need the image type, blueprint, and
image options.
The new functionality of the function is to collect all the resolved
content from the workers and serialize the manifest object.
Use the container sources provided by the content on the initialised
manifest instead of the blueprint to resolve containers. The container
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved containers, but the worker's container resolve job
works on a slice, so we reread the content map to get the pipeline name
(instead of taking the first payload pipeline from the image type).
This requires that there be only one pipeline that embeds containers,
which currently true of all our image types.
Use the commit sources provided by the content on the initialised
manifest instead of the image options to resolve commits. The ostree
sources on the manifest are a map keyed by the name of the pipeline that
will use the resolved commit spec, but unlike with the package sets, the
worker's commit resolve job works on a slice, so we reread the content
map to get the pipeline name. This requires that there be only one
pipeline that requires a resolved ostree commit, which is currently true
of all our image types.
Setting the default ostree ref on the image options before calling
Manifest() isn't needed anymore.
The FetchChecksum on ostree.ImageOptions was the resolved commit ID of
the parent ref to be pulled (for ostree commits and containers) or the
commit ID of the content ref (for ostree installers and raw images).
With the new process of manifest creation and serialisation, using the
image options to transport resolved content references is bad and
confusing. Image options should only reflect user and image type
options before any references are resolved. With this change, the
ostree.ImageOptions should only reflect the ostree-related options
specified by the user. Commit IDs will only be available after the
manifest is initialised when the commit sources are resolved (before
serialisation).
Before, this was done in the PackageSets() function.
The reason for this is that having an ostree ref affects package
selection (for example, it adds rpm-ostree). At the package selection
phase, it doesn't matter what the ostree ref is; it is just used to
determine if a pipeline is for an ostree-based image type and it doesn't
affect non-ostree-based image types because the image functions ignore
it.
This is only needed in the cloudapi now because other places have
switched to using the new order of operations, where the manifest is
generated after the ostree commit is resolved, so it's always added when
needed.
To add the container specs to Serialize(), we need to map them to the
payload (OS) pipeline. We assume the first name in the image type's
PayloadPipelines() list is the OS pipeline, which is true of all image
types right now but might not be necessarily in the future.
This is a temporary workaround. Eventually, the mapping will be set by
the image type itself when we use the container source specs attached to
the Manifest object.
Still not using the new process for generating the manifest exactly.
This commit only replaces the call to PackageSets() with a call to
Manifest() to get the Content.PackageSets. This is essentially the same
as before, when we were initialising the manifest object twice.
The Manifest() function does a tiny bit more work than PackageSets(),
but it's minimal and we gain the benefit of only having a single code
path and, although it's run twice, it's always run in the same way.
Pass the entire Blueprint to Manifest() instead of just the
Customizations. The goal is to combine the functionality of the
ImageType.PackageSets() and ImageType.Manifest() methods into one call.
Return manifest.Manifest from the Manifest() function without
serializing. The caller then has to call the manifest.Serialize()
function using the depsolved packages.
This moves towards changing the order of actions required to generate a
manifest. With this change, the manifest creation and depsolving can be
done independently, but this still requires instantiating the manifest
object twice (InstantiateManifest() is called in PackageSets() and
Manifest()), which we don't want to have to do.
Move the FactsImageOptions from distro to the new rhsm/facts package.
At the same time define the values we use as an enum, including the
"test-manifest" value.
Though the values don't really matter, the test value is defined first
so it takes the 0 value, which feels nicer conceptually.
The field in the distro.ImageOptions is changed to be a pointer to allow
for nil values.
Same as with the container SourceSpec, the struct specifies the required
information to resolve an ostree commit from a source (URL, ref, and
optional parent).
Renaming for consistency.
Move the subscription options from distro to its own package.
Now we can import the manifest package into the distro package (instead
of the other way around) so we can work with the manifest.Manifest type
in distro.
Explicitly specify the AMI boot mode in AWS upload target in Cloud API
compose handler. The value is determined based on image type's boot
mode.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Convert some of the fields in the `RepoConfig` struct
to pointers. Since `RepoConfig` will be used to convert
custom repositories to an array of `osbuild.YumRepository`,
we need to ensure that fields that are not set explicitly
are not saved to the `/etc/yum.repos.d` repository files.
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].
Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.
[1] See dnf.config
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.
Copy the value whenever a spec is moved to a different representation.
This changes the `Manifest` function of the `ImageType`
interface so that any warnings detected during the
`checkOptions` step of the manifest initialization can
be propagated back to the Weldr-API (see next commit).
Signed-off-by: Irene Diez <idiez@redhat.com>
