fsjobqueue_test contained tests that are generically testing the
JobQueue interface. Split those out into its own package `jobqueuetest`.
These tests will be useful when implementing a new package that conforms
to the JobQueue interface.
An occupied worker checks about every 15 seconds if it's current job was
cancelled. Use this to introduce a heartbeat mechanism, where if
composer hasn't heard from the worker in 2 minutes, the job times out
and is set to fail.
JobArgs() function replaced with more general Job() function that
returns all the parameters used to originally define a job during
Enqueue(). This new function enables access to the type of a job in the
queue, which wasn't available until now (except when Dequeueing).
JobArgs() returns the arguments submitted with a job in raw form.
Since the structure of the args are opaque to job queue, it's the
responsibility of the caller to deserialize the arguments.
Args retrieval is added to the existing TestArgs() function.
testjobqueue did not implement the JobQueue interface correctly (noted
in its package comment), making it impossible to write tests for
JobQueue itself.
Replace its use everywhere with fsjobqueue operating on a temporary
directory.
Soon, we want to begin tagging the jobs with the name of its submitter.
The simplest way to add a tag to a job is to put it into its type string.
However, as we don't know (and don't want to know) the submitters' names when
osbuild-composer is initialized, we need to be able to push arbitrary job
types into the jobqueue.
This commit therefore lifts the restriction that a jobqueue accepts only
a predefined set of job types. Now, jobqueue clients can push jobs of
arbitrary names.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The status of a job may depend on the status of its dependenices,
as we do not repeat for instance the failed state in each dependent
job.
Return also the list of dependencies so these can be queried too.
Similarly to the recent changes to Dequeue(), let the caller unmarshal the
return JSON. This allows us to pass the result on without being able
to unmarshal it.
In follow-up patches, we will pass results of jobs to dependent jobs,
but the worker API does not know about the different job types, nor how
to unmarshal them.
Once a job has been enqueued, there is no way to query its dependencies.
This makes dequeue more symmetric to enqueue by returning the
dependencies that were passed to enqueue, allowing the caller to
query the dependencies and their results.
Signed-off-by: Tom Gundersen <teg@jklm.no>
While dependencies are purely internal, sorting and pruning them is a
reasonable optimization. However, we wish to expose them in follow-up
commits and then we want them to remain unchanged from the input.
Nothing in the internal logic seems to rely on the fact the dependencies
were sorted.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The worker server was heavily tied to OSBuildJob(Result). Untie it so
that it can deal with different job types in the future.
This necessitates a change in the jobqueue: Dequeue() now returns the
job type, as well as job arguments as json.RawMessage. This is so that
the server can wait on multiple job types with different argument
types.
The weldr, composer, and koji APIs continue to use only "osbuild" jobs.
This makes the queue more type safe and allows to get rid of the
`pendingChannel` and `pendingChannels` helpers, which only existed to
create not-yet-existing pending channels.
The enum is redundant information that can be deduced from the job's
times: queuedAt, startedAt, and finishedAt. Not having it reduces the
potential for inconsistent state.
Now that the "old" `jobqueue` package was renamed to `worker`, add a new
package that contains an interface to an actual job queue. Also add two
implementations: fsjobqueue, a job queue backed by the file system, and
testjobqueue, which can be used as a mock implementation for testing.
These packages are not yet used.
This package does not contain an actual queue, but a server and client
implementation for osbuild's worker API. Name it accordingly.
The queue is in package `store` right now, but about to be split off.
This rename makes the `jobqueue` name free for that effort.
The API is always advertising a content type of application/json, but
not sending JSON on errors.
Change it to send simple JSON objects like this:
{ "message": "something went wrong" }
This can be extended to include more structured information in the
future.
Also return an (for now) empty JSON object from `addJobHandler()`. It
returned nothing before, which is invalid JSON.
Stop testing for the actual error strings in `api_test.go`. These are
meant for humans and can change. Only check what a client could
meaningfully check for, which is only the HTTP status code right now.
Don't panic when the error message cannot be written to the connection.
Ignore it, because there's nothing we can do in this case. The standard
library has the same behavior.
This makes the jobqueue package independent of forking osbuild, the
choices for which (exact invocation, location of the cache directory)
should be made in the worker.
The usual style is three blocks: standard library, external
dependencies, internal dependencies.
Also run through `go fmt`, which sorts each of those blocks
alphabetically.
Similar to `weldr/json.go`, this file will contain structs that are
shared between client and server and only meant to be used to
serialize from and to JSON.
This immediate change allows us to make fields in `Job` private in a
future commit.
This moves the client code into the same package as the server code,
which makes it easier to change (and version) the two in sync. Also, it
will allow to make some structs private to the jobqueue package and to
test `Client`.
Also rename it to jobqueue.Client.
Rather than having to assume that we only ever produce one
artifact, have each upload target contain the filename it expects
to upload from the osbuild output.
An image file is always explicitly named in the manifest, and we
leave it up to each distro to decide how this is done, but the
convention is to use the same image filename as used when
downloading the image through weldr.
Now make this policy explicit, by quering the distro for the image
name and inserting it into each upload target.
Signed-off-by: Tom Gundersen <teg@jklm.no>
A job's purpose is to build an osbuild manifest and upload the results
somewhere. It should not know about which distro was used to generate
the pipeline.
Workers depended on the distro package in two ways:
1. To set an osbuild `--build-env`. This is not necessary anymore in new
versions of osbuild. More importantly, it was wrong: it passed the
runner from the distro that is being built, instead of one that
matches the host.
This patch simply removes that logic.
2. To fetch the output filename with `Distro.FilenameFromType()`. While
that is useful, I don't think it warrants the dependency.
This patch uses the fact that all current pipelines output exactly
one file and uploads that. This should probably be extended in the
future to upload all output files, or to name them explicitly in the
upload target.
The worker should now compile to a smaller binary and do less
unnecessary work on startup (like reading repository files).
Only the weldr API has the concept of a default distro. Pass that distro
explicitly to `PushCompose()` and fetch the distro from the compose in
all other functions that accessed Store.Distro.
