Update the osbuild/images to the version which introduces "dot notation"
for distro release versions.
- Replace all uses of distroregistry by distrofactory.
- Delete local version of reporegistry and use the one from the
osbuild/images.
- Weldr: unify `createWeldrAPI()` and `createWeldrAPI2()` into a single
`createTestWeldrAPI()` function`.
- store/fixture: rework fixtures to allow overriding the host distro
name and host architecture name. A cleanup function to restore the
host distro and arch names is always part of the fixture struct.
- Delete `distro_mock` package, since it is no longer used.
- Bump the required version of osbuild to 98, because the OSCAP
customization is using the 'compress_results' stage option, which is
not available in older versions of osbuild.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Allow passing module_hotfixes flag through the cloudapi.
This will enable depsolving on repositories that might be affected by modularity filtering.
Refs HMS-3202
Unresponsive workers (>=1 hour of no status update) are cleaned up.
Several things are enabled by keeping track of workers, in future the
worker server could:
- keep track of how many workers are active
- see if a worker for a specific architecture is available
This handles corrupt job json files by skipping them. They still exist,
and errors are logged, but the system keeps working.
If one or more of the json files in /var/lib/osbuild-composer/jobs/
becomes corrupt they can stop the osbuild-composer service from
starting, or stop commands like 'composer-cli compose status' from
working because they quit on the first error and miss any job that
aren't broken.
Commit 312d87c6d5fab7ffd085a303e27b8db41111c86e adds validation of the
checksums to the ImageType.Manifest call, so it requires a valid looking
checksum, otherwise it will fail with a 'ManifestCreationFailed' error
when running the unit tests.
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
Add a checksum as a hash of URL + Ref.
Use the parent ref instead of the image ref when it's set. This makes
the test distro always behave like ostree commit and container types
(image types that can use an ostree parent) and not raw image or
installers (that use ostree commits as a payload).
Modify the weldr API test with the expected error message.
The Resolve() function is now only responsible for resolving a
SourceSpec to a CommitSpec. It only resolves a checksum if a URL is set
and sets the option for the RHSM secrets.
The Parent has been removed from the SourceSpec. The SourceSpec is a
simple reference to a single ostree ref and has no connection with the
ostree options.
The ostree options for a compose request should be specified using the
ImageOptions struct, not the source spec. The source spec should be
specifying the source parameters for a single ostree commit internally.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
OSTree commits are now resolved after manifest initialisation just like
packages and containers. The test commit hash handling is moved to the
resolver function.
The FetchChecksum on ostree.ImageOptions was the resolved commit ID of
the parent ref to be pulled (for ostree commits and containers) or the
commit ID of the content ref (for ostree installers and raw images).
With the new process of manifest creation and serialisation, using the
image options to transport resolved content references is bad and
confusing. Image options should only reflect user and image type
options before any references are resolved. With this change, the
ostree.ImageOptions should only reflect the ostree-related options
specified by the user. Commit IDs will only be available after the
manifest is initialised when the commit sources are resolved (before
serialisation).
Use the new workflow for generating the manifest before resolving
containers.
The resolver function is adjusted to handle a map of container slices,
but with all current use cases, the map should only ever have one key
for the payload (os) pipeline.
Use the new manifest generation procedure in the Weldr API.
Updated test distro to include the same packages from the PackageSets()
method in the Manifest.Content.PackageSets.
Pass the entire Blueprint to Manifest() instead of just the
Customizations. The goal is to combine the functionality of the
ImageType.PackageSets() and ImageType.Manifest() methods into one call.
Return manifest.Manifest from the Manifest() function without
serializing. The caller then has to call the manifest.Serialize()
function using the depsolved packages.
This moves towards changing the order of actions required to generate a
manifest. With this change, the manifest creation and depsolving can be
done independently, but this still requires instantiating the manifest
object twice (InstantiateManifest() is called in PackageSets() and
Manifest()), which we don't want to have to do.
Move the FactsImageOptions from distro to the new rhsm/facts package.
At the same time define the values we use as an enum, including the
"test-manifest" value.
Though the values don't really matter, the test value is defined first
so it takes the 0 value, which feels nicer conceptually.
The field in the distro.ImageOptions is changed to be a pointer to allow
for nil values.
Same as with the container SourceSpec, the struct specifies the required
information to resolve an ostree commit from a source (URL, ref, and
optional parent).
Renaming for consistency.
When a blueprint containing package name globs is frozen it was failing
because it could not find the string in the dependency list. This fixes
it by replacing the package glob with all of the matching packages from
the dependency list.
This removes the setPkgEVRA function and adds a new expandBlueprintGlobs
function that handles the package name glob expansion, and replacement of
the version globs with the dependency EVRA.
Also includes testing for the new function.
Explicitly specify the AMI boot mode in AWS upload target in Weldr API
compose handler. The value is determined based on image type's boot
mode.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Convert some of the fields in the `RepoConfig` struct
to pointers. Since `RepoConfig` will be used to convert
custom repositories to an array of `osbuild.YumRepository`,
we need to ensure that fields that are not set explicitly
are not saved to the `/etc/yum.repos.d` repository files.
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].
Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.
[1] See dnf.config
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.
Copy the value whenever a spec is moved to a different representation.
This adds a new field `Warnings` to the `ComposeReply`
struct, allowing to send back any warnings (e.g. deprecation
notices) generated during the `checkOptions` step of the
manifest initialization.
See also https://github.com/osbuild/weldr-client/pull/99 which
handles the weldr-client side of things.
Signed-off-by: Irene Diez <idiez@redhat.com>
This causes dnf-json to use separate caches, allowing them to run in
parallel, with one lock per distribution. Multiple depsolves with the
same distribution in the blueprint will continue to be serial.
It is difficult to tell if these are really running in parallel or not,
even with loggin, but it helps. They will all always start at the same
time (because they are run concurrently with goroutines) and if things
work right should be finished at about the same time.
ioutil has been deprecated since go 1.16, this fixes all of the
deprecated functions we are using:
ioutil.ReadFile -> os.ReadFile
ioutil.ReadAll -> io.ReadAll
ioutil.WriteFile -> os.WriteFile
ioutil.TempFile -> os.CreateTemp
ioutil.TempDir -> os.MkdirTemp
All of the above are a simple name change, the function arguments and
results are exactly the same as before.
ioutil.ReadDir -> os.ReadDir
now returns a os.DirEntry but the IsDir and Name functions work the
same. The difference is that the FileInfo must be retrieved with the
Info() function which can also return an error.
These were identified by running:
golangci-lint run --build-tags=integration ./...
DNF supports more than one GPG key. It is possible that one may be used for
signing packages, and another to sign the repository metadata. This
renamed GPGKey to GPGKeys internally. It does not change the on-disk
repository json format.
After introducing Go 1.18 to a project, it's required by law to convert at
least one method to a generic one.
Everyone hates IntToPtr, StringToPtr, BoolToPtr and Uint64ToPtr, so let's
convert them to the ultimate generic ToPtr one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
