Prior this commit local target copied the image from a worker to a composer
using cp(1) command. This prevented the local target to work on remote
workers.
This commit switches the local target implementation to using the jobqueue
API introduced in the previous commit. I had some concerns about speed
of this solution (imho nothing can beat pure cp(1) implementation) but
ad hoc sanity tests showed the copying of the image using the jobqueue API
when running the worker on the same machine as the composer is still
more or less instant.
In #221 Compose was refactored: Now it can have multiple image builds. More
image builds result in more jobs. Each job has its own result (logs from
osbuild). Additionally, also targets are now a part of image build. With
local target this effectively means we can have multiple images per compose.
However, these artifacts (images & results) were stored only per compose
prior this commit, thus rendering the behaviour of composes with multiple
image builds undefined and racy.
This commit fixes it by storing all the artifacts per image build instead of
per compose. To achieve this feature, getComposeDirectory and
getImageBuildDirectory methods were created to centralize the path assembly.
Paths to artifacts prior this commit:
${COMPOSER_STATE_DIR}/outputs/${COMPOSE_ID}/*
Paths to artifacts after this commit:
${COMPOSER_STATE_DIR}/outputs/${COMPOSE_ID}/${IMAGE_BUILD_ID}/*
Imho it makes more sense from REST perspective. Also, in the future there
will be ROUTE for uploading image to image build. As it's not a good idea
transport file inside JSON, all the parameters (compose id and image
build id) need to be inside the URL. Therefore for the sake of consistency,
all these routes should have compose id and image build id in the URL.
There is another solution to embedding multiple values inside http body
which allows file transport - multipart/form-data. I think using form-data
is worth when doing more complex stuff, for our usecase transporting all
the metadata in the URL is more appropriate solution.
Everything that this field contained can be computed in another way:
- path: just lookup the local target and read the path from there
- mime: can be derived from distribution and compose output type
- size: can be derived from the path
Therefore it imho doesn't make much sense to store these information multiple
times.
There's no connection between user-specified image size and actual image
size (which might differ du to a compression). Therefore, it's not needed
to check if the actual image exists if we want to return just the
user-specified size.
When requesting the compose status, a user may want to filter the list
of composes by blueprint name, compose status, and/or compose type. These
filters can now be set in the /compose/status route's url as the queries
blueprint, status, and type.
The compose now contains multiple image builds, but Weldr API does not
support this feature. Use the first image build every time.
Also start using the new types instead of plain strings.
All the preceding changes in jobqueue, compose package, common types
etc. were a base for refactoring the store so that a compose can handle
multiple image builds. This commit introduces the change in a backwards
compatible way, so that the weldr api don't change.
The compose will soon move to a concept of including multiple image
builds per one compose, we need to accommodate extra identifier to
handle this scenario.
The problem with the previous one was that we used artifical image
types, architectures, and distribution but that is not possible any more
because we don't want to use plain strings any more. This commit
introduces a new testing distro which uses proper, existing types.
The store package is getting too big and convoluted, this new package
will help with separation of the compose logic from our current
implementation of the store.
The current implementation will be removed in following commits.
These states will be used for tracking the image builds and compose
states in the rest of our codebase. There should be no change in the
behavior. It is a 1 to 1 replacement with the only difference of using
type alias instead of plain string.
we want to move to a state where composer is a stateless service,
therefore we cannot take blueprints by name (and fetch them from a
storage) or assume repositories by the distribution. This commit
introduces both of these parameters as part of the structure.
The repositories used by RCM never change, there is no need for them to
include checksum of the repository.
Also rework the tests as the store will change in following commits
(store now needs to include a registry of all distros because we can do
cross-distro builds)
When a user does not define the image size for a compose the default
image size of that image type is used. In order to properly store the
compose's image size even if the default is used the store calls the
distro function GetSizeForOutputType. This function accepts an output
format and image size. If the image size is 0 then the default
value for the output format will be returned. Also, for vhd images the
size must be rounded. This is now handled in the distro function instead
of the api.
When a use defines the image size for a compose this size is stored in
the compose struct so that the virtual image size can be returned by the
api instead of the file size of the image.
In the future remote workers will be introduced. Obviously, the remote worker
cannot support the local target. Unfortunately, the current implementation of
storing the osbuild result is dependant on it.
This commit moves the responsibility of storing osbuild result to the
composer process instead of the worker process. The result is transferred from
a worker to a composer using extended HTTP API.
Generating a SHA-1 based on time is not safe. A collision can easily be
generated, and if parallel operations are used they will eventually
collide. This reads random bytes and uses them for the SHA-1 hash. It
will return an error if the rand.Read() fails.
Otherwise if there is an error creating result.json or if the Run
command failed it will remain owned by root.root and cannot be deleted
via the API.
Closes#204
We treat the osbuild-pipeline as a tool run only within the source, therefore
we should search for the repository configs only in the current directory.
In the current state, osbuild-pipeline exits with random golang error,
such as goroutine failed, which is not at all helpful. This PR
introduces CLI arguments validation and helful error messages that use
the newly introduced types so that we don't waste time guessing what
was the right way to invoke this tool.
We currently use strings for passing arches and image types around,
which is not ideal. We should have a finite set of supported image types
and architectures as well as upload targets. This PR introduces custom
types to make the code base more readable and possibly also more
correct.
I considered some alternatives like a struct with private fields, but struct cannot
be const, so that does not help either. Eventually I think this is the "get s**t done"
solution.
The package also includes unit-tests which try to convert string to
structure and the other way around to make sure it all works properly.
Our current crypt implmentation requires glibc to be available on the
developer's system. But it is not strictly necessary, because I'd like
to run at least unit tests on mac. This PR makes it possible to compile
and run unit-tests on mac, but it will panic if someone tries to run
osbuild-composer on mac.
When creating a compose the desired image size can be set. If the image
type is a VHD the image size is rounded up to the nearest MB since all
VHDs on Azure must have a virtual size aligned to 1 MB.
When creating a pipeline the assembler includes an image size. This
image size can be set when creating the pipeline but if it is 0 then a
default image size will be used. The default is 2 GB except for ami
images which are 6 GB.
Osbuild-composer expects two or three listeners and fails if there is
an unexpected number of listening sockets. Checking if there are not two
or if there are not three listeners always returns true even if there are
the desired number of listeners. Therefore, osbuild-composer always
crashes. The check now only crashes if there are a number of listeners
other than 2 or 3.
It contains two basic endpoints:
* POST /v1/compose
* GET /v1/compose/<uuid>
It passes all the tests, but cannot be used for the intended use case
because the store API does not (yet) support distributions and
architectures as a parameters.
Introduce a new osbuild-tests command and ship it in the -tests
sub-package.
The intended usecase is to install the -tests subpackage into an
otherwise pristine VM, and call the osbuild-tests binary over ssh
from the outside of the booted VM. If the binary exits with a return
code of 0, the tests passed, otherwise they failed. The VM should
not be reused after running the tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
During development of a new distro, we need to test composer against
nightly or beta repositories, but we cannot ship composer itself
with the nightly repository information hardcoded in. At the same
time, we want to distinguish between the system repositories of the
host and the repositories we use to generate images (the host may not
use the same distro/version/architecture as the target, and it may
include custom repositories that the target should not).
We therefore ship per distro repository information that can be
overriden (typically in testing) by dropping files in /etc.
For now use the latest nightlies for RHEL-8.2, we may want to
replace these with the official mirrors for GA eventually.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We were using fedora-30 as a test-distro and tar as test-output, but
that causes lots of churn in the tests when we refactor things. Use
the test distro instead, when generic functionality is being tested
and restrict testing of the individual distros to the distro-specific
tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Introduce a DistroRegister object. For now this does not introduce
any functional changes, as the object is always instantited to be
the same. However, in follow-up patches it will get options.
Signed-off-by: Tom Gundersen <teg@jklm.no>
