The `jobs/:job_id/builds/:build_id/image` route was awkward: the
`:jobid` was actually weldr's compose id and `:build_id` was always `0`.
Change it to `jobs/:job_id/artifacts/:name`, where `:job_id` is now a
job id, and `:name` is the name of the artifact to upload. In the
future, it could support uploading more than one artifact.
This allows removing outputs from `store`, which is now back to being a
pure JSON-store. Take care that `weldr` returns (and deletes) images
from the new (or for backwards compatibility, the old) location.
The `org.osbuild.local` target continues to exist as a marker for the
worker to know whether it should upload artifacts.
Similarly to how some image types accept a size-argument, some image
types require ostree-specific options. For now this is only used by
Fedora IoT on Fedora 32.
composer-cli nor cockpit-composer currently has the ability to pass in
these options, but the change is backwards compatible and existing image
types are unaffected. If the options are omitted for image types that
require them, sensible defaults are chosen.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When generating an osbuild manifest for an image type, we take a
customizations struct, which specifies the image-type-independent
customizations to apply. We also take the size argument, which is
specific to the image build and not part of the blueprint.
Introduce a new argument ImageOptions, which for now just wraps the size
argument. These options are specific to the image build/type, and
therefore does not belong with the other customizations.
For now this is a non-functional change, but follow-up commits will
introduce more types of image options.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The store-jobqueue split broke the retrieval of logs in weldr API. Weldr API
still retrieved the logs from store but they are now stored in the jobqueue.
This commit fixes that by taking advantage of the fact that getComposeStatus()
now returns the compose result from which the logs can be generated.
When a job doesn't exist for a compose, just return an empty result.
In the future this will cause logs disappear from jobs created before the
store-jobqueue split but we decided we can live with that, they are still
in /var for those who really want them.
The reasoning is the same as in one of the previous commits. I think that
returning multiple values from one method can be confusing and potentially
error-prone. Also, this method will return ComposeResult soon too.
Also, I chose to create a new composeStatus struct instead of reusing
worker.JobStatus. I think that's a good thing in terms of consistency and
reducing the dependencies between packages.
Not a functional change.
Personally, I don't like methods returning too much values. I think it's
easy to assign the values in an incorrect order and thus interpreting them
in a wrong way. Also, the following commits will make JobStatus() return
also the JobResult, which would make the number of returned values even
higher.
Not a functional change.
The store only serves the weldr API, and that hard-codes the assumption
of only one image build per compose all over the place. Move this
assumption into the json serialization handler.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This reduces the amount of resolving and error checking we have to do.
This exposed a bug in weldr's ComposeEntry type, which will be fixed in
a follow-up commit.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The types exposed by the compose package are only used in the store API,
so move them there where they belong.
Making the ownership of the types clear, rather than having them live in
a package for themselves, will make it clearer how the types can be
modified in follow-up commits.
Also remove the JSON annotations, as these types are no longer used for
serialization.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Empty names are not allowed, and blueprint names should only contain
characters matching: ^[a-zA-Z0-9._-]+$
This also adds tests for the various places where the blueprint name
could potentially be wrong.
The store is responsible for two things: user state and the compose queue. This
is problematic, because the rcm API has slightly different semantics from weldr
and only used the queue part of the store. Also, the store is simply too
complex.
This commit splits the queue part out, using the new jobqueue package in both
the weldr and the rcm package. The queue is saved to a new directory `queue/`.
The weldr package now also has access to a worker server to enqueue and list
jobs. Its store continues to track composes, but the `QueueStatus` for each
compose (and image build) is deprecated. The field in `ImageBuild` is kept for
backwards compatibility for composes which finished before this change, but a
lot of code dealing with it in package compose is dropped.
store.PushCompose() is degraded to storing a new compose. It should probably be
renamed in the future. store.PopJob() is removed.
Job ids are now independent of compose ids. Because of that, the local
target gains ComposeId and ImageBuildId fields, because a worker cannot
infer those from a job anymore. This also necessitates a change in the
worker API: the job routes are changed to expect that instead of a
(compose id, image build id) pair. The route that accepts built images
keeps that pair, because it reports the image back to weldr.
worker.Server() now interacts with a job queue instead of the store. It gains
public functions that allow enqueuing an osbuild job and getting its status,
because only it knows about the specific argument and result types in the job
queue (OSBuildJob and OSBuildJobResult). One oddity remains: it needs to report
an uploaded image to weldr. Do this with a function that's passed in for now,
so that the dependency to the store can be dropped completely.
The rcm API drops its dependencies to package blueprint and store, because it
too interacts only with the worker server now.
Fixes#342
Code that's calling PushCompose() had to depsolve packages and fetch the
right ImageType from a distro, but not create the osbuild manifest. That
was left for PushCompose to do. Move it out of there to the callers, so
that the store is mainly concerned with storing things.
This also simplifies the argument list of PushCompose().
This function had two modes (`uuids == nil` and `uuids != nil`). Instead
of splitting it up into one function for each mode, inline it at call
sites. It was only used three times.
Instead of iterating over all composes, iterate over the passed UUIDs
(which amount to all composes when `*` is passed). This is more
straight-forward to read and more efficient in the case of requesting
only one (or a few) statuses.
`uuids` was initialized with a length, but only appended to. This lead
to a slice with lots of empty uuids, followed by the actual ones.
Fix this by initializing to an empty slice.
This happened to work because there's a non-fatal check for non-existing
composes further down, and the empty string fails that test.
The UUID list should only include UUIDs that are successfully deleted.
If there was an error it should only be listed in the errors list. This
matches the behavior of lorax-composer.
Yes. This goes against my desire not to change code to accommodate
tests. But there is no other good way to test compose results without
long running, and possibly fragile, composes. And this matches lorax's
behavior.
The change adds support for the ?test=1|2 query parameter to the compose
POST, and a new store function - PushTestCompose that handles creating
the fake compose results.
Passing ?test=1 will create a failed compose. Passing ?test=2 will
create a successful compose, but one without any files.
The purpose of these is to be able to test the compose result API
responses like compose/failed, etc.
Currently, if a TOML source is added with no name, or the source is
incorrectly inside a [section] it will add an empty source, causing
depsolving to crash.
This adds tests for 'name' and 'type' fields as a minimum requirement,
and returns an API error if they are empty or missing.
This also includes unit and integration tests.
Closes PR#462
The method is available in the Distro interface, but the distro does not
have the information needed to provide this kind of information any
more. The logic is now split into Arch and ImageType interfaces. This
patch will allow us to get rid of some old code and move forward.
The automatic local target is only needed when accessing the API via
weldr.
In the store, the target was only added when `stateDir` was not `nil`.
This is only used for testing which doesn't exercise the branch in
weldr. Thus, the same check is not needed there.
The following commit will introduce support for forced architecture in
dnf-json. The APIs already have this kind of information, so we can
simply pass it to the Depsolve and FetchMetadata functions.
Currently all image types are supported on all arches, but in the
future we may want to restrict this. In that case, return the
image types that are valid for the arch in qusetion, rather than
all the possible ones.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Check for errors and return early if they are found, rather than
check for the absence of errors.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
weldr needs to know the host architecture. Rather than pinning
a string, pin a real Arch object, and query its name when we
need it.
This verifies the validitiy of the architecture for the given
distro before it is passed to weldr, rather than lazily on
demand.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than having to assume that we only ever produce one
artifact, have each upload target contain the filename it expects
to upload from the osbuild output.
An image file is always explicitly named in the manifest, and we
leave it up to each distro to decide how this is done, but the
convention is to use the same image filename as used when
downloading the image through weldr.
Now make this policy explicit, by quering the distro for the image
name and inserting it into each upload target.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For serializeation, make our own private structs. The structs
in the target package are not exactly the same as the ones used by
weldr, so in order to avoid too many compromises, let's just do
an explicity translation.
As a general principle, we aim to only use private types for
serialization and rather translate than reuse for different
purposes.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Mixing the way to build a distribution with where to get the source
packages from is wrong: it breaks pre-release repos, local mirrors, and
other use cases. To accommodate those, we introduced
`/etc/osbuild-composer/repositories`.
However, that doesn't work for the RCM API, which receives repository
URLs to use from outside requests. This API has been wrongly using the
`additionalRepos` parameter to inject those repos. That's broken,
because the resulting manifests contained both the installed repos and
the repos from the request.
To fix this, stop exposing repositories from the distros, but require
passing them on every call to `Manifest()`. This makes `additionalRepos`
redundant.
Fixes#341
