Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
Add a checksum as a hash of URL + Ref.
Use the parent ref instead of the image ref when it's set. This makes
the test distro always behave like ostree commit and container types
(image types that can use an ostree parent) and not raw image or
installers (that use ostree commits as a payload).
Modify the weldr API test with the expected error message.
The Resolve() function is now only responsible for resolving a
SourceSpec to a CommitSpec. It only resolves a checksum if a URL is set
and sets the option for the RHSM secrets.
The Parent has been removed from the SourceSpec. The SourceSpec is a
simple reference to a single ostree ref and has no connection with the
ostree options.
The ostree options for a compose request should be specified using the
ImageOptions struct, not the source spec. The source spec should be
specifying the source parameters for a single ostree commit internally.
Do not expose the content of the manifest statically and instead rely on
the public methods to retrieve source specifications dynamically.
Since the methods require iterating through the pipelines to collect
source specifications, we should avoid calling the function multiple
times when we can reuse the returned values.
The new test_distro's manifest produces a slightly different empty
manifest when serialized even without content. Cloud API and Koji tests
have been adapted to match.
Weldr tests have been updated in several ways:
- The test_distro content resolver is used to resolve manifest content
before serializing.
- The test scenarios in TestCompose have been named for easier
troubleshooting (easier to identify a failing test by name).
- Manifests that work with the secondary ostree repo (the "other") use
the appropriate URL and ref and create a secondary "other" serialized
manifest.
The weldr API's test flag for resolving ostree commits does not produce
the same, fixed hash every time but instead computes a sha256 from the
URL + ref, like we do in the test manifests.
OSTree commits are now resolved after manifest initialisation just like
packages and containers. The test commit hash handling is moved to the
resolver function.
The FetchChecksum on ostree.ImageOptions was the resolved commit ID of
the parent ref to be pulled (for ostree commits and containers) or the
commit ID of the content ref (for ostree installers and raw images).
With the new process of manifest creation and serialisation, using the
image options to transport resolved content references is bad and
confusing. Image options should only reflect user and image type
options before any references are resolved. With this change, the
ostree.ImageOptions should only reflect the ostree-related options
specified by the user. Commit IDs will only be available after the
manifest is initialised when the commit sources are resolved (before
serialisation).
Use the new workflow for generating the manifest before resolving
containers.
The resolver function is adjusted to handle a map of container slices,
but with all current use cases, the map should only ever have one key
for the payload (os) pipeline.
Use the new manifest generation procedure in the Weldr API.
Updated test distro to include the same packages from the PackageSets()
method in the Manifest.Content.PackageSets.
Pass the entire Blueprint to Manifest() instead of just the
Customizations. The goal is to combine the functionality of the
ImageType.PackageSets() and ImageType.Manifest() methods into one call.
Return manifest.Manifest from the Manifest() function without
serializing. The caller then has to call the manifest.Serialize()
function using the depsolved packages.
This moves towards changing the order of actions required to generate a
manifest. With this change, the manifest creation and depsolving can be
done independently, but this still requires instantiating the manifest
object twice (InstantiateManifest() is called in PackageSets() and
Manifest()), which we don't want to have to do.
Move the FactsImageOptions from distro to the new rhsm/facts package.
At the same time define the values we use as an enum, including the
"test-manifest" value.
Though the values don't really matter, the test value is defined first
so it takes the 0 value, which feels nicer conceptually.
The field in the distro.ImageOptions is changed to be a pointer to allow
for nil values.
Same as with the container SourceSpec, the struct specifies the required
information to resolve an ostree commit from a source (URL, ref, and
optional parent).
Renaming for consistency.
When a blueprint containing package name globs is frozen it was failing
because it could not find the string in the dependency list. This fixes
it by replacing the package glob with all of the matching packages from
the dependency list.
This removes the setPkgEVRA function and adds a new expandBlueprintGlobs
function that handles the package name glob expansion, and replacement of
the version globs with the dependency EVRA.
Also includes testing for the new function.
Explicitly specify the AMI boot mode in AWS upload target in Weldr API
compose handler. The value is determined based on image type's boot
mode.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Convert some of the fields in the `RepoConfig` struct
to pointers. Since `RepoConfig` will be used to convert
custom repositories to an array of `osbuild.YumRepository`,
we need to ensure that fields that are not set explicitly
are not saved to the `/etc/yum.repos.d` repository files.
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].
Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.
[1] See dnf.config
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.
Copy the value whenever a spec is moved to a different representation.
This adds a new field `Warnings` to the `ComposeReply`
struct, allowing to send back any warnings (e.g. deprecation
notices) generated during the `checkOptions` step of the
manifest initialization.
See also https://github.com/osbuild/weldr-client/pull/99 which
handles the weldr-client side of things.
Signed-off-by: Irene Diez <idiez@redhat.com>
This causes dnf-json to use separate caches, allowing them to run in
parallel, with one lock per distribution. Multiple depsolves with the
same distribution in the blueprint will continue to be serial.
It is difficult to tell if these are really running in parallel or not,
even with loggin, but it helps. They will all always start at the same
time (because they are run concurrently with goroutines) and if things
work right should be finished at about the same time.
ioutil has been deprecated since go 1.16, this fixes all of the
deprecated functions we are using:
ioutil.ReadFile -> os.ReadFile
ioutil.ReadAll -> io.ReadAll
ioutil.WriteFile -> os.WriteFile
ioutil.TempFile -> os.CreateTemp
ioutil.TempDir -> os.MkdirTemp
All of the above are a simple name change, the function arguments and
results are exactly the same as before.
ioutil.ReadDir -> os.ReadDir
now returns a os.DirEntry but the IsDir and Name functions work the
same. The difference is that the FileInfo must be retrieved with the
Info() function which can also return an error.
These were identified by running:
golangci-lint run --build-tags=integration ./...
DNF supports more than one GPG key. It is possible that one may be used for
signing packages, and another to sign the repository metadata. This
renamed GPGKey to GPGKeys internally. It does not change the on-disk
repository json format.
After introducing Go 1.18 to a project, it's required by law to convert at
least one method to a generic one.
Everyone hates IntToPtr, StringToPtr, BoolToPtr and Uint64ToPtr, so let's
convert them to the ultimate generic ToPtr one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When the store is written to disk it simplifies the ImageBuild details
into a simple image type string. This works fine for composes that match
the host's distro but isn't enough detail to load composes made for
other distros, especially if the image type name isn't supported on the
host. This results in cross distro compose results being lost after a
reboot.
This fix uses the distro information from the compose's blueprint to
determine which distro the image type should be loaded from. It assumes
that the architecture matches the hosts' arch -- this is currently
always true but in the future if cross-arch builds are added it will
need to be addressed in a different way.
newComposeFromV0, newComposesFromV0, and newStoreFromV0 now take a
pointer to the full distro registry instead of an Arch, this allows them
to access the correct image types for the distro selected by the
blueprint. When loading the composes from disk the blueprint distro is
loaded from the registry before checking the image type string.
This means that we do not have to change the store version or on disk
format, the only thing changing is how it decides to populate the
ImageBuild when reloading the store.
A number of tests use a fake test distro using fake architecture names.
These tests have been adjusted to use a fake distro registry with
overridden host architecture that matches the fake one.
If params.Ref is an empty string, it's set to the distro's default
ref. The only difference here is that the default ref also gets
verified.
It makes splitting out resolving ostree refs to a new job easier.
In the weldr and cloud apis, ostree.ResolveParams always got executed,
also for non-ostree image types. Make it more explicit by only resolving
if the image type is actually an ostree image.
Previously, it was expected from the user to provide the Object name
when uploading image to GCP. The object name does not matter much,
because the object is deleted once image import finishes. Make
the specification of the object name optional and generate it if not
provided.
Adjust the GCP Weldr test case to not provide the Object name when
uploading the image.
The user can still provide the Object name if needed.
There is a desire to make the worker as "dumb" as possible. Therefore it
is not desired to generate the AWS object key names in the worker if it
was not provided in the job.
Modify the worker code to not generate the AWS object key in any case
and instead set an error in case the object key was not provided.
Modify Weldr API implementation to generate the object key, if it was
not provided by the user. This is consistent with Cloud API
implementation.
Instead of using the ostree.RequestParams in the OSTReeImageOptions,
define a new struct specific to ImageOptions for the ostree parameters.
This is almost identical to the new ostree.CommitSpec but the meaning of
the parameters changes based on image type and it would not be clear if
the CommitSpec was used in all cases. For example, the parameters of
the new OSTreeImageOptions do not always refer to the same commit. The
URL and Checksum may point to a parent commit to be pulled in to base
the new commit on, while the Ref refers to the new commit that will be
built (which may have a different ref from the parent).
The ostree.ResolveParams() function now returns two strings, the
resolved ref, which is replaced by the defaultRef if it's not specified
in the request, and the resolved parent checksum if a URL is specified.
The URL does not need to be returned since it's always the same as the
one specified in the request.
The function has been rewritten to make the logic more clear.
The docstring for the function has been rewritten to cover all use cases
and error conditions.
With an empty or missing version number the commit message would not
include the version (which is set to 0.0.0 by calling Initialize). This
adds a call to Initialize() in the API code before constructing the
commit message. It also moves the check for non-empty blueprint name
into the Initialize call where it belongs.
This satisfies the linter complaint about potential Slowloris attack
where headers are read slowly in an attempt to DoS the server.
The uses of ListenAndServe are only for testing purposes and are not run
in the production server so ignore the lint errors in
osbuild-mock-openid-provider.
