Also extend the Koji test case to verify that the boot mode information
is in the build extra metadata and that it contains valid value.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Copy the image boot mode information from the job options to the
job result, so that the `KojiFinalize` job can later access it
and import as an extra metadata to Koji.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Expose the extra metadata information for each image output stored in
`ImageExtraInfo` also in the build extra metadata. The extra metadata
for each image is nested under key corresponding to the image filename.
Extend the Koji test case to check information in the buildinfo output
and specifically check for all expected metadata in the build Extra
metadata field.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The original Koji implementation expected that the output of a content
generator is only an image. While in reality, we will eventually upload
other types of files as outputs to Koji, such as logs and osbuild
manifest.
Rename Koji structures and their members to better map to the upstream
Koji documentation and their JSON representation. Add comments to
structures. Define type aliases and constants for string values which
are more like enums, than a free-form values.
These changes have no effect on the actual JSON representation of any of
the structures
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
UBI and the oldest support Fedora (37) now all have go 1.19, so we are
cleared to switch.
gofmt now reformats comments in certain cases, so that explains the formatting
changes in this commit.
See https://go.dev/doc/go1.19#go-doc
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Manifest tests can no longer run and also aren't needed.
All image type manifests are generated and built in the obsuild/images
CI tests and the test function in images
(distro_test_common.TestDistro_Manifest()) has been removed.
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
Change the OSTreeResolveSpec to match the ostree SourceSpec by removing
the Parent field.
Change OSTreeResolveResultSpec to match the CommitSpec by adding the
Secrets field. The RHSM field is kept for backwards compatibility with
older workers.
The Resolve() function is now only responsible for resolving a
SourceSpec to a CommitSpec. It only resolves a checksum if a URL is set
and sets the option for the RHSM secrets.
The Parent has been removed from the SourceSpec. The SourceSpec is a
simple reference to a single ostree ref and has no connection with the
ostree options.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
When a test manifest requires a commit to be resolved for content, fake
the commit ID resolution deterministically by hashing the URL + ref.
Store the resolved commit spec with the manifest metadata alongside the
other content (packages and containers).
Also add the secrets field if RHSM is true, which is now supposed to be
done by the resolver.
In tests (and dev tools) that apply to all image types, set just the
ostree URL instead of all the options.
The default ref is handled by the image functions when needed, so it
doesn't need to be set from the caller.
The FetchChecksum on ostree.ImageOptions was the resolved commit ID of
the parent ref to be pulled (for ostree commits and containers) or the
commit ID of the content ref (for ostree installers and raw images).
With the new process of manifest creation and serialisation, using the
image options to transport resolved content references is bad and
confusing. Image options should only reflect user and image type
options before any references are resolved. With this change, the
ostree.ImageOptions should only reflect the ostree-related options
specified by the user. Commit IDs will only be available after the
manifest is initialised when the commit sources are resolved (before
serialisation).
Demonstrate the new workflow for resolving containers.
1. First call Manifest().
2. Get container SourceSpecs from manifest struct.
3. Resolve them.
4. Serialize() with resolved container specs.
The changes in the test manifests are just the information about the
container sources (was a slice but is now a map) and the actual manifest
object isn't affected.
The TestDistro_Manifest test in distro_test_common is adapted
accordingly as well.
Use the new manifest generation procedure in the cmd line tools. The
new procedure doesn't rely on ImageType.PackageSets() to compute the
packages for the depsolving. Instead, it calls Manifest() and depsolves
the packages attached to the returned object
(manifest.Content.PackageSets).
Pass the entire Blueprint to Manifest() instead of just the
Customizations. The goal is to combine the functionality of the
ImageType.PackageSets() and ImageType.Manifest() methods into one call.
Return manifest.Manifest from the Manifest() function without
serializing. The caller then has to call the manifest.Serialize()
function using the depsolved packages.
This moves towards changing the order of actions required to generate a
manifest. With this change, the manifest creation and depsolving can be
done independently, but this still requires instantiating the manifest
object twice (InstantiateManifest() is called in PackageSets() and
Manifest()), which we don't want to have to do.
Move the FactsImageOptions from distro to the new rhsm/facts package.
At the same time define the values we use as an enum, including the
"test-manifest" value.
Though the values don't really matter, the test value is defined first
so it takes the 0 value, which feels nicer conceptually.
The field in the distro.ImageOptions is changed to be a pointer to allow
for nil values.
Same as with the container SourceSpec, the struct specifies the required
information to resolve an ostree commit from a source (URL, ref, and
optional parent).
Renaming for consistency.
Removing the dependence of the manifest package on the distro package to
import manifest into distro.
Wherever arch names are needed, we use the enums from the platform
package instead.
Add an optional `BootMode` field to the AWS target options.
This allows to signal to worker the intended boot mode to use when
registering the AMI in AWS. If not specified, the default behavior is
preserved, specifically that the boot mode will be determined by the
default boot mode of the instance provisioned from the AMI.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When the AMI is being registered from a snapshot, the caller can
optionally specify the boot mode of the AMI. If no boot mode is
specified, then the default behavior is to use the boot type of the
instance that is launched from the AMI.
The default behavior (no boot type specified) is preserved after this
change.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Convert some of the fields in the `RepoConfig` struct
to pointers. Since `RepoConfig` will be used to convert
custom repositories to an array of `osbuild.YumRepository`,
we need to ensure that fields that are not set explicitly
are not saved to the `/etc/yum.repos.d` repository files.
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].
Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.
[1] See dnf.config
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.
Copy the value whenever a spec is moved to a different representation.
This adds a function, CleanupOldCacheDirs, that checks the dirs under
/var/cache/osbuild-composer/rpmmd/ and removes files and directories
that don't match the current list of supported distros.
This will clean up the cache from old releases as the are retired, and
will also cleanup the old top level cache directory structure after an
upgrade.
NOTE: This function does not return errors, any real problems it
encounters will also be caught by the cache initialization code and
handled there.
This causes dnf-json to use separate caches, allowing them to run in
parallel, with one lock per distribution. Multiple depsolves with the
same distribution in the blueprint will continue to be serial.
These RequiredSizes are a map that is passed on to the partition table
logic which had hardcoded defaults. This makes it possible to define
either no RequiredSizes (`nil`) or empty RequiredSizes which means no
further constraint checks or partition resizes will be done.
ioutil has been deprecated since go 1.16, this fixes all of the
deprecated functions we are using:
ioutil.ReadFile -> os.ReadFile
ioutil.ReadAll -> io.ReadAll
ioutil.WriteFile -> os.WriteFile
ioutil.TempFile -> os.CreateTemp
ioutil.TempDir -> os.MkdirTemp
All of the above are a simple name change, the function arguments and
results are exactly the same as before.
ioutil.ReadDir -> os.ReadDir
now returns a os.DirEntry but the IsDir and Name functions work the
same. The difference is that the FileInfo must be retrieved with the
Info() function which can also return an error.
These were identified by running:
golangci-lint run --build-tags=integration ./...
The default number of threads (16) is OK for general use case. However,
we are being asked by RH IT to lower the number of threads when
uploading the image to Azure using proxy server.
Make the number of threads configurable in the worker configuration and
default to the currently used value if it is not provided.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
