particle-os/debian-forge-composer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

store: move queue out of the store

Browse source 
The store is responsible for two things: user state and the compose queue. This
is problematic, because the rcm API has slightly different semantics from weldr
and only used the queue part of the store. Also, the store is simply too
complex.

This commit splits the queue part out, using the new jobqueue package in both
the weldr and the rcm package. The queue is saved to a new directory `queue/`.

The weldr package now also has access to a worker server to enqueue and list
jobs. Its store continues to track composes, but the `QueueStatus` for each
compose (and image build) is deprecated. The field in `ImageBuild` is kept for
backwards compatibility for composes which finished before this change, but a
lot of code dealing with it in package compose is dropped.

store.PushCompose() is degraded to storing a new compose. It should probably be
renamed in the future. store.PopJob() is removed.

Job ids are now independent of compose ids. Because of that, the local
target gains ComposeId and ImageBuildId fields, because a worker cannot
infer those from a job anymore. This also necessitates a change in the
worker API: the job routes are changed to expect that instead of a
(compose id, image build id) pair. The route that accepts built images
keeps that pair, because it reports the image back to weldr.

worker.Server() now interacts with a job queue instead of the store. It gains
public functions that allow enqueuing an osbuild job and getting its status,
because only it knows about the specific argument and result types in the job
queue (OSBuildJob and OSBuildJobResult). One oddity remains: it needs to report
an uploaded image to weldr. Do this with a function that's passed in for now,
so that the dependency to the store can be dropped completely.

The rcm API drops its dependencies to package blueprint and store, because it
too interacts only with the worker server now.

Fixes #342
This commit is contained in:
Lars Karlitski 2020-05-03 17:44:22 +02:00 • committed by Tom Gundersen
parent 64011e3cba
commit b5769add2c
18 changed files with 415 additions and 443 deletions
Download patch file Download diff file Expand all files Collapse all files

74
internal/compose/compose.go
View file

@ -5,6 +5,7 @@ package compose
import (
"time"
"github.com/google/uuid"
"github.com/osbuild/osbuild-composer/internal/blueprint"
"github.com/osbuild/osbuild-composer/internal/common"
"github.com/osbuild/osbuild-composer/internal/osbuild"
@ -21,15 +22,20 @@ func (ste *StateTransitionError) Error() string {
// ImageBuild represents a single image build inside a compose
type ImageBuild struct {
Id int `json:"id"`
QueueStatus common.ImageBuildState `json:"queue_status"`
ImageType common.ImageType `json:"image_type"`
Manifest *osbuild.Manifest `json:"manifest"`
Targets []*target.Target `json:"targets"`
JobCreated time.Time `json:"job_created"`
JobStarted time.Time `json:"job_started"`
JobFinished time.Time `json:"job_finished"`
Size uint64 `json:"size"`
Id int `json:"id"`
ImageType common.ImageType `json:"image_type"`
Manifest *osbuild.Manifest `json:"manifest"`
Targets []*target.Target `json:"targets"`
JobCreated time.Time `json:"job_created"`
JobStarted time.Time `json:"job_started"`
JobFinished time.Time `json:"job_finished"`
Size uint64 `json:"size"`
JobId uuid.UUID `json:"jobid,omitempty"`
// Kept for backwards compatibility. Image builds which were done
// before the move to the job queue use this to store whether they
// finished successfully.
QueueStatus common.ImageBuildState `json:"queue_status,omitempty"`
}
// DeepCopy creates a copy of the ImageBuild structure
@ -55,6 +61,7 @@ func (ib *ImageBuild) DeepCopy() ImageBuild {
JobStarted: ib.JobStarted,
JobFinished: ib.JobFinished,
Size: ib.Size,
JobId: ib.JobId,
}
}
@ -94,55 +101,6 @@ func (c *Compose) DeepCopy() Compose {
}
}
func anyImageBuild(fn func(common.ImageBuildState) bool, list []common.ImageBuildState) bool {
acc := false
for _, i := range list {
if fn(i) {
acc = true
}
}
return acc
}
func allImageBuilds(fn func(common.ImageBuildState) bool, list []common.ImageBuildState) bool {
acc := true
for _, i := range list {
if !fn(i) {
acc = false
}
}
return acc
}
// GetState returns a state of the whole compose which is derived from the states of
// individual image builds inside the compose
func (c *Compose) GetState() common.ComposeState {
var imageBuildsStates []common.ImageBuildState
for _, ib := range c.ImageBuilds {
imageBuildsStates = append(imageBuildsStates, ib.QueueStatus)
}
// In case all states are the same
if allImageBuilds(func(ib common.ImageBuildState) bool { return ib == common.IBWaiting }, imageBuildsStates) {
return common.CWaiting
}
if allImageBuilds(func(ib common.ImageBuildState) bool { return ib == common.IBFinished }, imageBuildsStates) {
return common.CFinished
}
if allImageBuilds(func(ib common.ImageBuildState) bool { return ib == common.IBFailed }, imageBuildsStates) {
return common.CFailed
}
// In case the states are mixed
// TODO: can this condition be removed because it is already covered by the default?
if anyImageBuild(func(ib common.ImageBuildState) bool { return ib == common.IBRunning }, imageBuildsStates) {
return common.CRunning
}
if allImageBuilds(func(ib common.ImageBuildState) bool { return ib == common.IBFailed || ib == common.IBFinished }, imageBuildsStates) {
return common.CFailed
}
// Default value
return common.CRunning
}
// UpdateState changes a state of a single image build inside the Compose
func (c *Compose) UpdateState(imageBuildId int, newState common.ImageBuildState) error {
switch newState {