Imho it makes more sense from REST perspective. Also, in the future there
will be ROUTE for uploading image to image build. As it's not a good idea
transport file inside JSON, all the parameters (compose id and image
build id) need to be inside the URL. Therefore for the sake of consistency,
all these routes should have compose id and image build id in the URL.
There is another solution to embedding multiple values inside http body
which allows file transport - multipart/form-data. I think using form-data
is worth when doing more complex stuff, for our usecase transporting all
the metadata in the URL is more appropriate solution.
The compose will soon move to a concept of including multiple image
builds per one compose, we need to accommodate extra identifier to
handle this scenario.
We were using fedora-30 as a test-distro and tar as test-output, but
that causes lots of churn in the tests when we refactor things. Use
the test distro instead, when generic functionality is being tested
and restrict testing of the individual distros to the distro-specific
tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Introduce a DistroRegister object. For now this does not introduce
any functional changes, as the object is always instantited to be
the same. However, in follow-up patches it will get options.
Signed-off-by: Tom Gundersen <teg@jklm.no>
On architectures that require EFI, we must create the ESP partition
and use a GPT partition table. We must also install either the UEFI
or the legacy version of GRUB2 in the image.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The pipeline generation now takes the architecture as an argument.
Currently only x86_64 is supported. The architecture is detected
at start-up, and passed down to each pipeline translation.
For osbuild-pipeline we now requrie the architecture to be passed
in.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Instead of having a static repository checksum, set it dynamically from
the metadata that osbuild-composer last saw. This is implemented in
dnf-json, which returns the checksums for each repository on every call.
This enables the use of repositories that change over time, such as
fedora-updates. Note that the osbuild pipeline will break when such a
repository changes. This is intentional: pipelines have to be
reproducible.
Prior to this commit outputs directory used by local target was owned by root.
This made impossible for osbuild-composer to delete images. (osbuild-composer
doesn't run as root).
This commit introduces state directory in which osbuild-composer creates
outputs directory. Because this directory is owned by osbuild-composer, it's
able to delete files inside.
As a part of f4991cb1 ComposeEntry struct was removed from store package.
This change made sense because this struct is connected more with API than
with store - store uses its own Compose struct. In addition, converters
between Compose and ComposeEntry were added. Unfortunately, ComposeEntry
contains ImageSize which was not stored in Compose but retrieved from store
using GetImage method. This made those converters dependent on the store,
which was messy.
To solve this issue this commit adds image struct into Compose struct.
The content of image struct is generated on the worker side - when the worker
sets the compose status to FINISHED, it also sends Image struct with detailed
information about the result.
This commit introduces basic support for upload API. Currently, all the routes
required by cockpit-composer are supported (except for /compose/log).
Also, ComposeEntry struct is moved outside of the store package. I decided
to do it because it isn't connected in any way to store, it's more connected
to API. Due to this move there's currently a known bug that image size is
not returned. This should be solved by moving Image struct inside Compose
struct by follow-up PR.
Make osbuild-composer use FromHost() directly. Everywhere else needs to
specify the distro explicitly.
Also don't panic when a distro doesn't exist. Instead, return nil. Make
sure all callers check for that.
The helper functions in both api packages were more or less same. However,
over time they have been slowly diverging. This commit extract the helpers
into one common package to make the tests more maintainable and
to deduplicate the code.
osbuild has a concept of runners now: scripts that set up a build
environment. Update the osbuild submodule to latest master, change
`Pipeline` to to the new buildroot description format, and use the
`org.osbuild.fedora30` runner from the fedora30 distro.
This slightly changes the customizations logic. We now make sure
that each stage is appended exactly once.
customizations.go are now responsible only for the things that are
completely generic, and not per-ouput-type. helpers.go contain more
high-level helpers that combine customziations and per-output-type
defaults.
This does not change the behaviour, though some pipelines are slightly
reordered to make them consistent.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The main difference (according to `rpm -qa`) is an additional package
containing a gpg key which was used to verify packages. The one
generated by lorax-composer doesn't have this, because it doesn't verify
signatures.
A job is now in "WAITING" state exactly when it is in the channel,
once it is popped it enters "RUNNING" state. It is only possible
to update the state of a job that is in the running state.
This mean that updating to "RUNNING" is entirely optional, but in
the future we may want to use this as a watchdog logic, and require
the worker to update at regular intervals to avoid being restarted.
The job queue API is updated to require a POST followed by one
or several PATCH messages to the returned ID. If a patch is sent
to an ID before the POST it is as if the object does not exist
(regarldess of it being in the queue in WAITING state or not).
Once a job has been POSTed it can be PATCHed to update it zero or
more times with (still) RUNNING before exactly oncee with either
FINISHED or FAILED.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This hides the state hanlding in the store package. As before, it
can be disabled by passing `nil` instead of the path to the state
file.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Wrap the channel in Pop and Push methods, so it is not exposed to
the callers. PushCompose replaces the old AddCompose for consistency,
and PopCompose simply reads from the other end of the channel.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Drop the jobUpdates channel, and instead add an UpdateCompose method
to the store, which updates the status of a compose directly.
This allows us to report back errors directly, rather than having to
mirror the staet in the jobqueue API.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Let the store in weldr be the only one that keeps state, and push
updates directly there. This fixes a bug where there was an ID mismatch.
Change the API to not let the caller pick the UUID, but provide it
in the response. Use the same UUID as is used to identify composes,
this makes it simpler to trace what is going on.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use the exact same status strings as is used in the API,
making it clearer that they are the same (and avoiding any
translation). Remember the creation/start/finish timestamps.
And store the output type.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Go doesn't really do variants, so we must somehow emulate it. The
json objects we use are essentially tagged unions, with a `name`
field in reverse domain name notation identifying the type and a
type specific 'options' object.
In Go we represent this by having an BarOptions interface, which
implements a private method `isBarOptions()`, making sure that only
types in the same package are able to implement it. Each type FooBar
that should belong to the variant implements the interface, and a
constructor `NewFooBar(options *FooBarOptions) *Bar` that makes sure
the `name` field is set correctly.
This would be enough to represent our types and marshal them into
JSON, but unmarshalling would not work (json does not know about
our tags, so would not know what concrete types to demarshal to).
We therefore must also implement the Unmarshall interface for Bar,
to select the right types for the Options field.
We implement his logic for Target, Stage and Assembler. A handful
of concrete types are also implemented, matching what osbuild
supports.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This way it can be correctly exposed in the API. We listen on a channel
from the job-queue, where status updates are pushed when the worker is
running/finished (or, in the future, failed).
Signed-off-by: Tom Gundersen <teg@jklm.no>
For now we will hardcode the org.osbuild.local target, so we might
as well fix this up front.
We do not yet support any target types, but for testing purposes we
claim to support 'tar', and we pass a noop tar pipeline to the worker.
This makes introspecting the job-queu api using curl a bit more
pleasant.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is by no means done, and needs more tests, docs and bugfixes,
but push it early so we have a common base to work on.
Based on work by Martin Sehnoutka.
Signed-off-by: Tom Gundersen <teg@jklm.no>
