Previously, nil values in the conversion source were in some cases converted
to empty arrays or empty objects. This is undesirable, because it can be in
certain cases changing the semantics of the blueprint. See e.g.
f317064da5/pkg/distro/rhel7/imagetype.go (L239C7-L239C7)
This commit modifies the conversion process so nil values are converted
without any changes. Also, the `Convert` function was covered with a unit
test.
During development it can be very useful to store the results locally
instead of uploading to a remote system. This implements a development
only option to help with that.
To use it you need to add OSBUILD_LOCALSAVE to the server's environment.
This can be done by editing /usr/lib/systemd/system/osbuild-composer.service
and adding:
Environment="OSBUILD_LOCALSAVE=1"
You can then use an 'upload_options' object to skip trying to upload to
the default service for the type of image, eg:
"image_requests": [
{
"architecture": "x86_64",
"image_type": "guest-image",
"upload_options": {
"local_save": true
},
...
}]
The results will be saved to /var/lib/osbuild-composer/artifacts/UUID/
using the default filename for the image type.
If local_save is used without OSBUILD_LOCALSAVE being set it will return
an error with id=36 saying 'local_save is not enabled'.
Also extend the Koji test case to verify that the boot mode information
is in the build extra metadata and that it contains valid value.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add the information about the image boot mode to the OSBuildJob options
as well as to the OSBuildJobResult options. The intention is that the
worker will simply copy the value from the job options to job result,
so that the `KojiFinalize` job can then access this information and
upload it to Koji as extra metadata.
This information is required in Koji in order for Red Hat's SP tooling
to know how to import image to the cloud environment in case the boot
mode affects the import parameters.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Expose the extra metadata information for each image output stored in
`ImageExtraInfo` also in the build extra metadata. The extra metadata
for each image is nested under key corresponding to the image filename.
Extend the Koji test case to check information in the buildinfo output
and specifically check for all expected metadata in the build Extra
metadata field.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The original Koji implementation expected that the output of a content
generator is only an image. While in reality, we will eventually upload
other types of files as outputs to Koji, such as logs and osbuild
manifest.
Rename Koji structures and their members to better map to the upstream
Koji documentation and their JSON representation. Add comments to
structures. Define type aliases and constants for string values which
are more like enums, than a free-form values.
These changes have no effect on the actual JSON representation of any of
the structures
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This moves some of the code from the PostCompose function in handler.go
into methods on the OpenAPI ComposeRequest and ImageRequest structs.
In compose.go I have added several methods.
GetBlueprintWithCustomizations takes the ComposeRequest customizations
and builds a Blueprint struct.
GetPayloadRepositories returns the custom payload repos.
GetSubscription returns the ImageOptions setup with optional
subscription information from the request.
In imagerequest.go I have added GetTarget which takes the upload
options and returns a Target. This moves the giant switch statement,
which may also benefit from further simplification at some point.
GetOSTreeOptions returns the OSTree ImageOptions if there are ostree
settings in the ImageRequest.
GetImageOptions returns the distro.ImageOptions with the size set.
This commit only moves the code, making PostCompose easier to read. All
tests still pass.
cloudapi: Move the size handling to a method on ImageRequest
This adds a 'size' parameter to the image_request object. It can be used
to specify the minimum image size in bytes
This behaves in the same way as the size parameter of the weldr API
For raw images the root partition is grown to fill the available space.
For LVM images the PV uses the available space, but the LV does not,
leaving space available for other LVs to be created after boot.
See COMPOSER-1883
UBI and the oldest support Fedora (37) now all have go 1.19, so we are
cleared to switch.
gofmt now reformats comments in certain cases, so that explains the formatting
changes in this commit.
See https://go.dev/doc/go1.19#go-doc
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
The duration middleware should come after the tenant channel middleware,
otherwise the tenant in the context will be empty. The status middleware
can come beforehand because it queries the request context right before
sending a response.
The duration middleware should come after the tenant channel middleware,
otherwise the tenant in the context will be empty. The status middleware
can come beforehand because it queries the request context right before
sending a response.
Set fallback datastreams for openscap if no datastream is provided by
the user. This will also simplify the cloudapi experience by not
exposing the `datastream`
osbuild-composer no longer supports Fedora 36, so we are safe to drop these
now.
iotCommitPackageSet would really benefit from sorting, but that would
create hard to handle conflicts with other PRs upstream, so I decided
not to do it in this commit.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Update Fedora 37, 38, and 39 repositories for the test case generator.
Changes are only package versions and dependencies.
For Fedora 39, a package (lklug-fonts) had to be removed from the
Anaconda package set since it is no longer available.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This image type produces the same artifact as the current workstation
installer live media.
During the implementation of this new installer some names have been
changed to make a bit more sense in the source tree. Installer images
now always mention which installer they are (anaconda, etc).
Make sure checksums used in the file input helper functions contain only
1 colon delimiter and it is not at the start of the string.
Adjusted tests to work with new restriction.
Helper functions that create stage input objects with references always
hard-coded `sha256:` as a prefix/algorithm for the checksum. This
prevents the functions from being used in cases where other algorithms
are use, like sha1, which is possible with (perhaps older) RPM
repositories. The inputs in osbuild a number of hashing algorithms and
we should be able to generate stages with other prefixes when necessary.
Remove the `sha256:` prefix in the helper functions and assume all
arguments to these functions provide the correct prefix.
Update tests to match.
PR#3421 [1] unintentionally removed the `rhui-azure-rhel8` package from
the Azure EAP7 RHUI image base package set. As a result, the image
manifest can't be built successfully. The reason is that the removed
package installs a RPM GPG key, which is hard-coded in the image manifest
to be imported as part of the image build.
Add the package back to the image base package set and regenerate all
affected test manifests.
[1] https://github.com/osbuild/osbuild-composer/pull/3421
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Since we only need to retrieve the file names, we can use
`(*os.File).Readdirnames` to avoid reading the whole file info for
better performance.
Sample benchmark:
func Benchmark_Readdir(b *testing.B) {
for i := 0; i < b.N; i++ {
f, err := os.Open("/")
if err != nil {
b.Fatal(err)
}
_, err = f.Readdir(-1)
if err != nil {
f.Close()
b.Fatal(err)
}
f.Close()
}
}
func Benchmark_Readdirnames(b *testing.B) {
for i := 0; i < b.N; i++ {
f, err := os.Open("/")
if err != nil {
b.Fatal(err)
}
_, err = f.Readdirnames(-1)
if err != nil {
f.Close()
b.Fatal(err)
}
f.Close()
}
}
goos: linux
goarch: amd64
pkg: github.com/osbuild/osbuild-composer/internal/jsondb
cpu: AMD Ryzen 7 PRO 4750U with Radeon Graphics
Benchmark_Readdir-16 31304 33551 ns/op 5638 B/op 70 allocs/op
Benchmark_Readdirnames-16 128443 12124 ns/op 1228 B/op 30 allocs/op
PASS
ok github.com/osbuild/osbuild-composer/internal/jsondb 3.098s
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add a checksum as a hash of URL + Ref.
Use the parent ref instead of the image ref when it's set. This makes
the test distro always behave like ostree commit and container types
(image types that can use an ostree parent) and not raw image or
installers (that use ostree commits as a payload).
Modify the weldr API test with the expected error message.
Use ostree.ImageOptions for the request parameters instead of a
SourceSpec on the imageRequest.
When preparing the image request, add the ostree values from the API's
compose request to the ostree options on the image options of the image
request.
It's not necessary to create a source spec and it's also not necessary
to add the default ref when it's not specified in the request for an
ostree-based image type. Both of these will be handled by the Manifest
generation based on the ostree options (imageOptions.OSTree). The image
functions will take care of setting any missing parameters or returning
errors if any required parameters are missing.
Change the OSTreeResolveSpec to match the ostree SourceSpec by removing
the Parent field.
Change OSTreeResolveResultSpec to match the CommitSpec by adding the
Secrets field. The RHSM field is kept for backwards compatibility with
older workers.
The Resolve() function is now only responsible for resolving a
SourceSpec to a CommitSpec. It only resolves a checksum if a URL is set
and sets the option for the RHSM secrets.
The Parent has been removed from the SourceSpec. The SourceSpec is a
simple reference to a single ostree ref and has no connection with the
ostree options.
The ostree options for a compose request should be specified using the
ImageOptions struct, not the source spec. The source spec should be
specifying the source parameters for a single ostree commit internally.
Add a Distro enum to the Manifest struct for selecting package
selection.
Packages are sometimes renamed between distribution versions and since
we do the package selection in the Manifest, we need a way to select
distro-version-specific package names inside the manifest initialiser.
This may change in the future.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
