This commit introduces basic support for upload API. Currently, all the routes
required by cockpit-composer are supported (except for /compose/log).
Also, ComposeEntry struct is moved outside of the store package. I decided
to do it because it isn't connected in any way to store, it's more connected
to API. Due to this move there's currently a known bug that image size is
not returned. This should be solved by moving Image struct inside Compose
struct by follow-up PR.
Make osbuild-composer use FromHost() directly. Everywhere else needs to
specify the distro explicitly.
Also don't panic when a distro doesn't exist. Instead, return nil. Make
sure all callers check for that.
Automatically registering on `init()` is clever, but a bit too magical
and easy to get wrong, because every binary must include all distros
somewhere.
Flip this inside out: distros now have a `New()`, which returns
something that implements the `Distro` interface. The distro package
explicitly creates all of them.
This means that distros cannot import distro itself anymore, because go
forbids import cycles. This only affected `InvalidOutputFormatError`.
Return a generic error for now.
Prior this commit there wasn't an easy to populate the store. The only way
was to call the weldr API or store methods. This design made testing of
various edges quite hard.
This commit adds store fixtures - an easy way how to define store state
before each test case.
In addition, the fixtures were refactored so that new instances are created
prior each test. Before this change the tests were in some cases dependant
on each other.
The helper functions in both api packages were more or less same. However,
over time they have been slowly diverging. This commit extract the helpers
into one common package to make the tests more maintainable and
to deduplicate the code.
lorax-composer recently introduced API version 1. This commit introduces
very basic support for it. This implementation tries to deduplicate code
for routes with the same behaviour as much as possible. All the differences of
v1 API are marked as TODOs for now and will be implemented in follow-ups PRs.
Make distros export repository information and use those in the weldr
API. This means that repos are only specified once and that the API
returns the right packages when we allow different distros.
Split the error case (no sources specified) into its own function, so
that we can use `source/info/:sources` (note the colon) to get the list
of sources without the leading `/`. This gets rid of two special cases
which made the previous implementation hard to parse.
The naming is confusing: repositories have an `id` and a human readable
`name`. Weldr's sources also have a field called `name`, but
lorax-composer uses that as a way to identify repositories by their id.
Use `id` consistently here as well.
The blueprints freeze test now creates the blueprint with the package
dep-package1 which is mocked and will properly depsolve when the
blueprint is frozen. This test can no longer run externally against
lorax-composer.
This test creates a new blueprint with libsemanage. Libsemanage is
already in the mock rpmmd so when we test the freeze route on this
blueprint, the blueprint will properly depsolve and return the package
with the depsolved version.
The blueprint freeze route returns the blueprint info but each
package will be the package selected by depsolving. So, instead of the
version being the version number with optional wildcards as
/blueprints/info would provide, the version is of the form
`Version-Release.Arch`.
These endpoint are similar in many ways, therefore just one commit. Their
functionality is basically same as in lorax except for error messages and
weird edge cases when handling trailing slashes.
closes#64, closes#65
Add a route to set a blueprint back to its state at a particular change.
The route `blueprints/undo/:blueprint/:commit` requires the blueprint
name and the commit hash for the change that the blueprint should be
reverted too. Also, the commit message for the change created when a
blueprint is pushed is now passed from the api to the store's
PushBlueprint funtion.
The package list is generated on each request for a package so there is
no longer a need to generate the package list in main or to store these
packages in the API object.
The package list is now generated from the base repo and the
user-defined repos. This allows users to add packages not found in the
base repo to their blueprints.
We want to test API methods which calls dnf. Unfortunately, calling dnf
is expensive operation - it requires network access and downloading
a lot of (meta)data. This commit changes the rpmmd implementation
so that it can be mocked.
For each blueprint name passed to the route, a list of the changes to
that blueprint will be returned.
weldr/tests: add blueprint changes test
In order to test blueprint changes a blueprint must be created with a
unique id. Blueprint changes are not deleted when the blueprint is
deleted so in order to test this against lorax the blueprint must have
not been used/tested before. This id is created from a random int. The
test creates and deletes the same blueprint twice to check that each
creation updates the list of changes.
The delete source handler now removes the leading "/" from the
parameters passed to it. Not removing the "/" caused sources to not
be deleted from the store since they could not be found when their name
contained a "/" as the first character.
lorax-composer uses the host's repositories for building images. This is
prone to accidental configuration errors and duplicates functionality
(adding custom repositories via the source API is much more explicit).
However, blueprints don't specify the distribution they're based on.
This is something they should do in the future to enable cross-distro
image builds. Until then, read `/etc/os-release` to detect the host OS
and use that as the base distro for a blueprint.
Detection works by concatenating the ID field with a `-` and the
VERSION_ID field. This mandates how additional distributions should
register themselves to the `distro` package.
If composer cannot build the detected distro, fall back to fedora-30.
Use this detection everywhere that fedora-30 was hard-coded before.
Introduce the `distro` package, which contains an interface for OS
implementations. Its main purpose is to convert a blueprint to a
distro-specific pipeline.
Also introduce the `distro/fedora30` package. It is the first
implementation of the distro interface. Most of its code has been copied
with minimal modifications from the blueprint package.
The `blueprint` package is now back to serving a single purpose:
representing a weldr blueprint. It does not depend on the `pipeline`
package anymore.
Change osbuild-composer and osbuild-pipeline to use the new API,
hard-coding "fedora-30". This looks a bit weird now, but is the same
behavior as before.
All test cases now also take an "distro" key in the "compose" object.
Add tests for adding and deleting user defined sources. Update existing
tests for source info. The source being added is the fish repository.
This is the same source that the cockpit-composer tests attempt to add.
The api now supports user defined sources alongside the existing base
repository. The routes /projects/source/new and
/projects/source/delete/<name> have been added.
/projects/source/info/<names> and /projects/source/list have been
updated.
The SourceConfig type is now called from the store instead of being
declared in the api.
Make each command accept a `repos` key containing repository
descriptions.
Make weldr API pass the repository like this. Nothing should change,
because the repos were the same (Fedora 30).
There was missing condition for the case when compose push fails.
As the previous commit introduces customizations which can return even more
errors it is important to report such errors.
For now the API is returning only 500 HTTP code. In future we should
distinguish between failures and blueprint validation errors.
Compose tests ignore the id and timestamps and verify that the rest
of the response is as expected. The testRoute function now accepts
fields to ignore and uses dropFields to remove them from the test and
response objects.
Certain fields (timestamps, uuids, etc.) are difficult to test for. The
dropFields function allows specific fields to be removed from an
unmarshalled json response body.
All composes now include the image size in their status response. The
image size will be 0 except for finished composes where it will be the
file length in bytes.
This can serve as a starting point, but it shows there are a few
problems to solve: we need to verify json that depends on the setup,
in particular, the json the queue contains will contain UUID's that
are generated out of our control.
Moreover, the setup for this test only makes sense for internal test,
so I think we may want to change the logic for whether or not a test
sholud be supported to be run externally to be per test-function,
rather than per call to sendHTTP().
Signed-off-by: Tom Gundersen <teg@jklm.no>
This does not change the behavior, but refactors according to these principles:
1) No two routes are tested in the same function (but it would be ok to split
tests for one route over several funcions)
2) At most one testRoute() call is made per API object, and the state is
completely set up and tore down between tets.
On top of this we should add more test cases to each of the tables, but
I'm leaving this to future PRs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This simplifies the code a bit, and will be used in follow-up patches
to distinguish between setup calls and explicit tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Not all tests will be compatible with external APIs such as Lorax. When
calling testRoute each test now declares if it can run against an
external API or not. This change allows us to test against Lorax but
skip the cases that will be invalid when not run against
osbuild-composer's API.
In order to maintain parity with lorax the api needs to reply with an
error message equivalent to that used by lorax. Error messages are now
returned inside an error object that contains an id, message, and
optional status code.
For some routes, there are errors when no url parameters are passed. The
httprouter was using named parameters of the form /:param which does not
match for empty parameters. Now, it has been updated to use the
catch-all parameters of the form /*param. This change allows the case of
no parameters. However, parameters will now include a "/" as
their first character. This needs to be removed from the string in the
route handler.
In order to provide the proper error message for
/modules/list/<modules>, searching for the modules needed to be updated.
The requested modules and known packages are iterated over and if there
is a match the module is added to the response. Also, the found module
is dropped from the list of requested modules. If this list is not empty
after searching all of the modules then an error is returned containing
the name of the non-existant module.
At the end of each set of tests the blueprint created is deleted.
This tests the delete route as well as removing blueprints
created while testing external apis such as lorax.
This hides the state hanlding in the store package. As before, it
can be disabled by passing `nil` instead of the path to the state
file.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Wrap the channel in Pop and Push methods, so it is not exposed to
the callers. PushCompose replaces the old AddCompose for consistency,
and PopCompose simply reads from the other end of the channel.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Drop the jobUpdates channel, and instead add an UpdateCompose method
to the store, which updates the status of a compose directly.
This allows us to report back errors directly, rather than having to
mirror the staet in the jobqueue API.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Let the weldr API take the store as an argument, rather than create it
itself. This will allow us to share the store with the jobqueue API in
follow-up patches.
Signed-off-by: Tom Gundersen <teg@jklm.no>
