Introduce the `distro` package, which contains an interface for OS
implementations. Its main purpose is to convert a blueprint to a
distro-specific pipeline.
Also introduce the `distro/fedora30` package. It is the first
implementation of the distro interface. Most of its code has been copied
with minimal modifications from the blueprint package.
The `blueprint` package is now back to serving a single purpose:
representing a weldr blueprint. It does not depend on the `pipeline`
package anymore.
Change osbuild-composer and osbuild-pipeline to use the new API,
hard-coding "fedora-30". This looks a bit weird now, but is the same
behavior as before.
All test cases now also take an "distro" key in the "compose" object.
Add tests for adding and deleting user defined sources. Update existing
tests for source info. The source being added is the fish repository.
This is the same source that the cockpit-composer tests attempt to add.
The api now supports user defined sources alongside the existing base
repository. The routes /projects/source/new and
/projects/source/delete/<name> have been added.
/projects/source/info/<names> and /projects/source/list have been
updated.
The SourceConfig type is now called from the store instead of being
declared in the api.
Make each command accept a `repos` key containing repository
descriptions.
Make weldr API pass the repository like this. Nothing should change,
because the repos were the same (Fedora 30).
There was missing condition for the case when compose push fails.
As the previous commit introduces customizations which can return even more
errors it is important to report such errors.
For now the API is returning only 500 HTTP code. In future we should
distinguish between failures and blueprint validation errors.
Compose tests ignore the id and timestamps and verify that the rest
of the response is as expected. The testRoute function now accepts
fields to ignore and uses dropFields to remove them from the test and
response objects.
Certain fields (timestamps, uuids, etc.) are difficult to test for. The
dropFields function allows specific fields to be removed from an
unmarshalled json response body.
All composes now include the image size in their status response. The
image size will be 0 except for finished composes where it will be the
file length in bytes.
This can serve as a starting point, but it shows there are a few
problems to solve: we need to verify json that depends on the setup,
in particular, the json the queue contains will contain UUID's that
are generated out of our control.
Moreover, the setup for this test only makes sense for internal test,
so I think we may want to change the logic for whether or not a test
sholud be supported to be run externally to be per test-function,
rather than per call to sendHTTP().
Signed-off-by: Tom Gundersen <teg@jklm.no>
This does not change the behavior, but refactors according to these principles:
1) No two routes are tested in the same function (but it would be ok to split
tests for one route over several funcions)
2) At most one testRoute() call is made per API object, and the state is
completely set up and tore down between tets.
On top of this we should add more test cases to each of the tables, but
I'm leaving this to future PRs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This simplifies the code a bit, and will be used in follow-up patches
to distinguish between setup calls and explicit tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Not all tests will be compatible with external APIs such as Lorax. When
calling testRoute each test now declares if it can run against an
external API or not. This change allows us to test against Lorax but
skip the cases that will be invalid when not run against
osbuild-composer's API.
In order to maintain parity with lorax the api needs to reply with an
error message equivalent to that used by lorax. Error messages are now
returned inside an error object that contains an id, message, and
optional status code.
For some routes, there are errors when no url parameters are passed. The
httprouter was using named parameters of the form /:param which does not
match for empty parameters. Now, it has been updated to use the
catch-all parameters of the form /*param. This change allows the case of
no parameters. However, parameters will now include a "/" as
their first character. This needs to be removed from the string in the
route handler.
In order to provide the proper error message for
/modules/list/<modules>, searching for the modules needed to be updated.
The requested modules and known packages are iterated over and if there
is a match the module is added to the response. Also, the found module
is dropped from the list of requested modules. If this list is not empty
after searching all of the modules then an error is returned containing
the name of the non-existant module.
At the end of each set of tests the blueprint created is deleted.
This tests the delete route as well as removing blueprints
created while testing external apis such as lorax.
This hides the state hanlding in the store package. As before, it
can be disabled by passing `nil` instead of the path to the state
file.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Wrap the channel in Pop and Push methods, so it is not exposed to
the callers. PushCompose replaces the old AddCompose for consistency,
and PopCompose simply reads from the other end of the channel.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Drop the jobUpdates channel, and instead add an UpdateCompose method
to the store, which updates the status of a compose directly.
This allows us to report back errors directly, rather than having to
mirror the staet in the jobqueue API.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Let the weldr API take the store as an argument, rather than create it
itself. This will allow us to share the store with the jobqueue API in
follow-up patches.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This encapsulates all the state of osbuild, no longer just the state specific
to the weldr API. Make it a first class type, preparing for direct access from
both the weldr and jobqueue APIs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Each output type will have its own translator from a blueprint to
a pipeline.
In the future, we may also want to distinguish between different
base distributions and architectures, but for now we keep it
simple and hardcode to a given one.
Some placeholder output formats are added, these have not been
tested or reserached, but are included to show how the it will
all fit together.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This will contain all the translation logic from blueprints to pipelines,
which will constitute the main part of composer, so let's separate that
out from the API.
Signed-off-by: Tom Gundersen <teg@jklm.no>
osbuild-composer's tests can be run against lorax-composer's api in
order to ensure parity with lorax. This can be accomplished by passing
the environment variable LORAX=true into the tests. Otherwise, the
tests run as usual against osbuild-composer's api.
Use the exact same status strings as is used in the API,
making it clearer that they are the same (and avoiding any
translation). Remember the creation/start/finish timestamps.
And store the output type.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Go doesn't really do variants, so we must somehow emulate it. The
json objects we use are essentially tagged unions, with a `name`
field in reverse domain name notation identifying the type and a
type specific 'options' object.
In Go we represent this by having an BarOptions interface, which
implements a private method `isBarOptions()`, making sure that only
types in the same package are able to implement it. Each type FooBar
that should belong to the variant implements the interface, and a
constructor `NewFooBar(options *FooBarOptions) *Bar` that makes sure
the `name` field is set correctly.
This would be enough to represent our types and marshal them into
JSON, but unmarshalling would not work (json does not know about
our tags, so would not know what concrete types to demarshal to).
We therefore must also implement the Unmarshall interface for Bar,
to select the right types for the Options field.
We implement his logic for Target, Stage and Assembler. A handful
of concrete types are also implemented, matching what osbuild
supports.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This way it can be correctly exposed in the API. We listen on a channel
from the job-queue, where status updates are pushed when the worker is
running/finished (or, in the future, failed).
Signed-off-by: Tom Gundersen <teg@jklm.no>
This means they are serialised and restarted in case composer is
crashed/restarted. We also need this in the future to track their
state, and allow them to be listed in the UI.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For now we will hardcode the org.osbuild.local target, so we might
as well fix this up front.
We do not yet support any target types, but for testing purposes we
claim to support 'tar', and we pass a noop tar pipeline to the worker.
This makes introspecting the job-queu api using curl a bit more
pleasant.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is by no means done, and needs more tests, docs and bugfixes,
but push it early so we have a common base to work on.
Based on work by Martin Sehnoutka.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The list of changes between the committed blueprint and its workspace
are returned when a user requests the diff. Each change includes the
new(added) or the old(removed) package.
If a package had version "*" and its name-version would be "name-*".
This would match all packages containing "name" in their name. Instead
the name-version is "name-*-*" so it matches any version of the package.
Keep main.go files under cmd/ and internal libraries under internal/.
This will allow us to add more exutables under cmd/ (whereas only one
was possible when main.go was kept in the root).
Signed-off-by: Tom Gundersen <teg@jklm.no>
