Currently the responses are all embedded in the weldr API functions.
They need to be usable by the client helper functions so I'm copying
them here and giving them names.
A later commit will go through the API can refactor it to use these
instead of the embedded ones.
This package will contain functions for communicating with the API that
can be used in both the integration tests as well as in a future cmdline
tool similar to composer-cli
When possible the client functions will return the same structures used
by weldr/api.go which have been exported in weldr/json.go
Return errors from all distro's New() functions instead of logging and
returning nil. Also, return errors instead of panicking from
NewRegistry() and NewDefaultRegistry().
These packages (and their tests) shouldn't access the distro package,
because that's cyclic.
Also, these packages should only test the objects they expose.
WithSingleDistro() doesn't follow go's naming convention for creating
objects (New*). Rename it to NewRegistry() and rename the old
NewRegistry() to NewDefaultRegistry().
The idea is that NewRegistry() can be used to create full Registry
objects from outside the package. NewDefaultRegistry() is a convenience
function that creates a Registry with all known distros.
rpmmd looked at the CACHE_DIRECTORY environment variable to set a path
for the dnf repository cache. Aside from being a smelly thing to do
from a library, this breaks osbuild-pipeline and osbuild-dnf-json-tests,
which don't run as systemd services and thus don't have CACHE_DIRECTORY
set.
Explicitly pass the cache directory to rpmmd. Keep using a path based on
CACHE_DIRECTORY for osbuild-composer. Use the user's `.cache` directory
for osbuild-pipeline and a temporary directory for the tests.
Images can be built for rhel 8.1. The pipeline generation and distro
tests are based off of the rhel 8.2 ones. Repository information as been
added for rhel 8.1. The repo urls are internal ones and will only work
if the user is on the Red Hat vpn.
If no packages are included in a blueprint, the slice remains `nil`,
which translates to `null` in json. Always initialize the slice by
pointing it to an empty array.
This can happen when CacheDirectory= is missing from the service file.
That's unlikely to happen, but it's hard to figure out what caused the
failure when it does. Be explicit and panic.
This changes osbuild-composer's behavior to match lorax-composer when
encountering invalid versions. Instead of leaving them as-is it will
return a BlueprintError explaining the problem. eg.
"errors": [
{
"id": "BlueprintsError",
"msg": "Invalid 'version', must use Semantic Versioning: is not in dotted-tri format"
}
]
This is enforced on new blueprints (including the workspace). If a
previously stored blueprint has an invalid version and a new one is
pushed it will use the new version number instead of trying to bump the
invalid one.
This also moves the version bump logic into blueprint instead of store,
and adds an Initialize function that will make sure that the blueprint
has sane default values for any missing fields.
This includes tests for the Initialize and BumpVersion functions.
This adds returning errors from the store PushBlueprint* functions, and
adds handling of the errors to the API code in preparation for new code
to check the blueprint before saving it.
The LoadRepositories function interates over a list of paths and expects
to find a distro configuration in one of them. The case when no path
with valid configuration is found was not handled. This patch introduces
the check.
Without passing in a cachedir, dnf would create a random one for every
invocation. This meant that caches were never reused, nor cleaned up
properly.
Let systemd create a cache directory for us in /var/cache/ and use
that via the environment variable systemd sets for us.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is needed for unit tests, because it wasn't possible to mock the
rpmmd module before. This also requires that the checksum is moved to
the compose request and evaluated in the endpoint handler instead of
push compose. I think it makes sense to have the checksum in the compose
request directly.
Also a "module platform ID" is required now, but we don't have the
"global" distribution any more, so this patch introduces mapping from a
distribution to the module platform ID.
the name was misleading because the function could do more than just
download package list. In PushComposeRequest it is also used to fetch
checksums for the repositories, therefore I decided to rename it to
reflect this usage.
The change also requires customizations in the error handling, as some
errors are now handled automatically by the custom unmarshaler.
Include a note about HTTP return types.
There was a bug in the previous implementation which used to pass the
argument as a value but that does not work because we need to change the
value of it. The new implementation uses pass by reference.
Create a test to cover this scenario.
If the Epoch is > 0 the it should be added to the front of the version,
separated by a colon.
Also include a depsolve package with a non-zero Epoch and adjust the
tests accordingly.
Without making a deep copy of the blueprint the changes made to the
package and module versions will persist in memory, causing it to lose
the package and module version globs.
This can be seen by executing a freeze request and then a depsolve. The
blueprint included in the depsolve had the version globs replaced by the
frozen EVRA values.
The Blueprint struct is complex, deep, and full of references. This
means that any changes to it in memory will persist. Sometimes you need
an actual copy of it, so this adds DeepCopy which uses the json.Marshal
and Unmarshal functions to create a deep copy with no references to the
original.
This is not very efficient, but the alternative is adding Copy functions
to all the member structs and then calling them to build the copy.
This adds the modules to the list of package specs to be depsolved. It
includes a new function to build the version glob package string, as
well as tests for the new function and for depsolving with modules in
the blueprint.
This adds support for the modules field. It moves the version
replacement into a separate function, setPkgEVRA, and adds tests for the
new function as well as for blueprints with packages in both the
packages and modules lists.
The dependencies are not sorted, so depending on what order they were
returned in the freeze route would or would not return the correct
results (exhibited by the version being the original glob instead of the
EVRA).
This also fixes the tests so that the depsolve results are slightly
unsorted by adding a dep-package3 to the start of the list.
The current `NewRegistry` implementation allows for nil values in the
map, but this leads to subtle bugs when using the registry. This patch
enforces non-nil values by introducing additional checks before we
insert the value into the map.
The change unfortunately breaks a lot of tests and therefore it is
necessary to create additional mock: distro.
The new mock is used instead of the previous "real" implementation,
which used to contain nil values.
Images can be built for fedora 32. The pipeline generation and distro
tests are based off of the fedora 30 ones. Repository information has
also been added for the fedora 32 repos.
Images can be built for fedora 31. The pipeline generation and distro
tests are based off of the fedora 30 ones. Repository information has
also been added for the fedora 31 repos.
I wanted to create a unit test for this method but then I decided not to.
The reason is that if we add another field to ImageBuild but fail to
modify the test it won't catch the bug. I think higher level testing is
needed to cover this function.
This outputs the sources needed for the pipeline generated for the
distro. At the moment no pipelines require sources, and so this
always returns the empty list.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is the replacement for the DNF stage, containing only GPG
keys and package checksums. It is meant to be used together with
the files source to actually fetch the packages. Depsolving must
be done in composer and the full package list inserted into
the pipeline.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is unused for now, but will allow us to generate pipelines with
the pre-depsolved NEVRAs, so osbuild does not need to depsolve again.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The packag selection shown in the UI does not include the base
packages that will be included in the image, and they cannot,
because the base packages depends on the output type, and the UI
packages shown in the UI are independent of the output type.
It is possible to select packages incompatible with the base
packages. Discover this sooner rather than later, by including
the base packages in the final depsolve before creating the
pipeline.
In the future the result of the depsolve will be used to create
the pipeline, so this is another prerequisite for moving from
the dnf to the rpm stage.
Also depsolve the build packages for the same reason. Note that we
always set clean to false in this case, as the depsolving of the
main packages would have performed any cleaning necessary.
Also extend dnf-json to support excluding packages from depsolving.
Signed-off-by: Tom Gundersen <teg@jklm.no>
