The store is responsible for two things: user state and the compose queue. This
is problematic, because the rcm API has slightly different semantics from weldr
and only used the queue part of the store. Also, the store is simply too
complex.
This commit splits the queue part out, using the new jobqueue package in both
the weldr and the rcm package. The queue is saved to a new directory `queue/`.
The weldr package now also has access to a worker server to enqueue and list
jobs. Its store continues to track composes, but the `QueueStatus` for each
compose (and image build) is deprecated. The field in `ImageBuild` is kept for
backwards compatibility for composes which finished before this change, but a
lot of code dealing with it in package compose is dropped.
store.PushCompose() is degraded to storing a new compose. It should probably be
renamed in the future. store.PopJob() is removed.
Job ids are now independent of compose ids. Because of that, the local
target gains ComposeId and ImageBuildId fields, because a worker cannot
infer those from a job anymore. This also necessitates a change in the
worker API: the job routes are changed to expect that instead of a
(compose id, image build id) pair. The route that accepts built images
keeps that pair, because it reports the image back to weldr.
worker.Server() now interacts with a job queue instead of the store. It gains
public functions that allow enqueuing an osbuild job and getting its status,
because only it knows about the specific argument and result types in the job
queue (OSBuildJob and OSBuildJobResult). One oddity remains: it needs to report
an uploaded image to weldr. Do this with a function that's passed in for now,
so that the dependency to the store can be dropped completely.
The rcm API drops its dependencies to package blueprint and store, because it
too interacts only with the worker server now.
Fixes#342
Now that the "old" `jobqueue` package was renamed to `worker`, add a new
package that contains an interface to an actual job queue. Also add two
implementations: fsjobqueue, a job queue backed by the file system, and
testjobqueue, which can be used as a mock implementation for testing.
These packages are not yet used.
Code that's calling PushCompose() had to depsolve packages and fetch the
right ImageType from a distro, but not create the osbuild manifest. That
was left for PushCompose to do. Move it out of there to the callers, so
that the store is mainly concerned with storing things.
This also simplifies the argument list of PushCompose().
This function had two modes (`uuids == nil` and `uuids != nil`). Instead
of splitting it up into one function for each mode, inline it at call
sites. It was only used three times.
Instead of iterating over all composes, iterate over the passed UUIDs
(which amount to all composes when `*` is passed). This is more
straight-forward to read and more efficient in the case of requesting
only one (or a few) statuses.
`uuids` was initialized with a length, but only appended to. This lead
to a slice with lots of empty uuids, followed by the actual ones.
Fix this by initializing to an empty slice.
This happened to work because there's a non-fatal check for non-existing
composes further down, and the empty string fails that test.
Image size should not be specified for a tar image. When the tar
assembler was being created image size was not passed to it. However,
the TarAssemblerOptions contained a Size field which would default to 0.
This field is removed so that the tar assembler options no longer
specify a size. The assembler tests are also updated to reflect this
change.
os.Exit() doesn't execute defer-ed functions, OTOH we call panic()
in case of setup errors, which does run defer-ed functions. So move
the actual test setup and execution in a separate function which will
execute all defer-ed functions and return the exit code from the
test suite to TestMain.
Prior this commit the ami image type produced raw.xz images. This was bad for
two reasons:
- The upload was broken because AWS doesn't support tar.xz format
- XZ compression is terribly slow
This commit changes the format to vhdx, which is supported by AWS and also
quite quick. See https://github.com/osbuild/osbuild-composer/issues/257
why vhdx was chosen.
Fixes#257
The old aws unit test is not very useful now, because there's the new
integration test which covers a lot more. Therefore, this commit
removes it.
Fixes#512
This package does not contain an actual queue, but a server and client
implementation for osbuild's worker API. Name it accordingly.
The queue is in package `store` right now, but about to be split off.
This rename makes the `jobqueue` name free for that effort.
We need to make partition uuids stable to be able to run image-info
tests on images using gpt. For example all aarch64 images use gpt.
Also change the type of the (part) UUID to string because that's what
the other UUIDs use and it is easier to work with.
Also adds new types to weldr/json.go to support them.
ComposeEntry had to be duplicated instead of used as-is because it
enforces image type strings that do not match what the API uses (the API
types are all lower case, the internal names are capitalized).
The UUID list should only include UUIDs that are successfully deleted.
If there was an error it should only be listed in the errors list. This
matches the behavior of lorax-composer.
Yes. This goes against my desire not to change code to accommodate
tests. But there is no other good way to test compose results without
long running, and possibly fragile, composes. And this matches lorax's
behavior.
The change adds support for the ?test=1|2 query parameter to the compose
POST, and a new store function - PushTestCompose that handles creating
the fake compose results.
Passing ?test=1 will create a failed compose. Passing ?test=2 will
create a successful compose, but one without any files.
The purpose of these is to be able to test the compose result API
responses like compose/failed, etc.
us-east-1 seems to very slow these days, some imports can take up to
15 minutes. This commit raises the number of attempts before we give up.
Previously, we did 40 attempts, each delayed by 15 seconds, making the total
timeout equal to 10 minutes. Now we do 80 attempts with the same delay,
making the total timeout 20 minutes.
The partuuid is a uuid of a specific GPT partition as opposed to the ptuuid
which is associated with the partition table and filesystem uuid which
is associated with a filesystem on a partition.
The support for partuuid was introduced in osbuild 11, see this PR for
reference: https://github.com/osbuild/osbuild/pull/306
We were using dnf's default of 48h, but that does not work for
updates repositories, as they depend on an expiration time of 6h.
Allow the metadata_expire value to be configured per repository.
If the value is unset, then never expire the metadata. Set the
value to 6h for all the fedora testing repos.
This fixes issue #476.
Signed-off-by: Tom Gundersen <teg@jklm.no>
There is no point in having the grub2 stage in pipelines for image types
that are not bootable. The current version is probably a result of
previous refactoring where the member variable was named `IncludeFSTab`.
Moving the grub2 stage into the conditional branch should also fix test
generation on aarch64.
Finally it is necessary to regenerate test cases for non-bootable image
types.
If you don't set the time it ends up being the default Go time which is
1/1/1 and when you convert that using UnixNano() you get a nice big
negative number (-6795364578871345152) which then eventually shows up in
the queue, finished, and failed output as 'Fri Aug 30 17:47:39 1754'
and since the Time Travel feature is not yet complete this is
impossible.
The fix is to set it when the job is started.
