The compose now contains multiple image builds, but Weldr API does not
support this feature. Use the first image build every time.
Also start using the new types instead of plain strings.
All the preceding changes in jobqueue, compose package, common types
etc. were a base for refactoring the store so that a compose can handle
multiple image builds. This commit introduces the change in a backwards
compatible way, so that the weldr api don't change.
The compose will soon move to a concept of including multiple image
builds per one compose, we need to accommodate extra identifier to
handle this scenario.
The problem with the previous one was that we used artifical image
types, architectures, and distribution but that is not possible any more
because we don't want to use plain strings any more. This commit
introduces a new testing distro which uses proper, existing types.
The store package is getting too big and convoluted, this new package
will help with separation of the compose logic from our current
implementation of the store.
The current implementation will be removed in following commits.
These states will be used for tracking the image builds and compose
states in the rest of our codebase. There should be no change in the
behavior. It is a 1 to 1 replacement with the only difference of using
type alias instead of plain string.
we want to move to a state where composer is a stateless service,
therefore we cannot take blueprints by name (and fetch them from a
storage) or assume repositories by the distribution. This commit
introduces both of these parameters as part of the structure.
The repositories used by RCM never change, there is no need for them to
include checksum of the repository.
Also rework the tests as the store will change in following commits
(store now needs to include a registry of all distros because we can do
cross-distro builds)
When a user does not define the image size for a compose the default
image size of that image type is used. In order to properly store the
compose's image size even if the default is used the store calls the
distro function GetSizeForOutputType. This function accepts an output
format and image size. If the image size is 0 then the default
value for the output format will be returned. Also, for vhd images the
size must be rounded. This is now handled in the distro function instead
of the api.
When a use defines the image size for a compose this size is stored in
the compose struct so that the virtual image size can be returned by the
api instead of the file size of the image.
In the future remote workers will be introduced. Obviously, the remote worker
cannot support the local target. Unfortunately, the current implementation of
storing the osbuild result is dependant on it.
This commit moves the responsibility of storing osbuild result to the
composer process instead of the worker process. The result is transferred from
a worker to a composer using extended HTTP API.
Generating a SHA-1 based on time is not safe. A collision can easily be
generated, and if parallel operations are used they will eventually
collide. This reads random bytes and uses them for the SHA-1 hash. It
will return an error if the rand.Read() fails.
Otherwise if there is an error creating result.json or if the Run
command failed it will remain owned by root.root and cannot be deleted
via the API.
Closes#204
We treat the osbuild-pipeline as a tool run only within the source, therefore
we should search for the repository configs only in the current directory.
In the current state, osbuild-pipeline exits with random golang error,
such as goroutine failed, which is not at all helpful. This PR
introduces CLI arguments validation and helful error messages that use
the newly introduced types so that we don't waste time guessing what
was the right way to invoke this tool.
We currently use strings for passing arches and image types around,
which is not ideal. We should have a finite set of supported image types
and architectures as well as upload targets. This PR introduces custom
types to make the code base more readable and possibly also more
correct.
I considered some alternatives like a struct with private fields, but struct cannot
be const, so that does not help either. Eventually I think this is the "get s**t done"
solution.
The package also includes unit-tests which try to convert string to
structure and the other way around to make sure it all works properly.
Our current crypt implmentation requires glibc to be available on the
developer's system. But it is not strictly necessary, because I'd like
to run at least unit tests on mac. This PR makes it possible to compile
and run unit-tests on mac, but it will panic if someone tries to run
osbuild-composer on mac.
When creating a compose the desired image size can be set. If the image
type is a VHD the image size is rounded up to the nearest MB since all
VHDs on Azure must have a virtual size aligned to 1 MB.
When creating a pipeline the assembler includes an image size. This
image size can be set when creating the pipeline but if it is 0 then a
default image size will be used. The default is 2 GB except for ami
images which are 6 GB.
Osbuild-composer expects two or three listeners and fails if there is
an unexpected number of listening sockets. Checking if there are not two
or if there are not three listeners always returns true even if there are
the desired number of listeners. Therefore, osbuild-composer always
crashes. The check now only crashes if there are a number of listeners
other than 2 or 3.
It contains two basic endpoints:
* POST /v1/compose
* GET /v1/compose/<uuid>
It passes all the tests, but cannot be used for the intended use case
because the store API does not (yet) support distributions and
architectures as a parameters.
Introduce a new osbuild-tests command and ship it in the -tests
sub-package.
The intended usecase is to install the -tests subpackage into an
otherwise pristine VM, and call the osbuild-tests binary over ssh
from the outside of the booted VM. If the binary exits with a return
code of 0, the tests passed, otherwise they failed. The VM should
not be reused after running the tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
During development of a new distro, we need to test composer against
nightly or beta repositories, but we cannot ship composer itself
with the nightly repository information hardcoded in. At the same
time, we want to distinguish between the system repositories of the
host and the repositories we use to generate images (the host may not
use the same distro/version/architecture as the target, and it may
include custom repositories that the target should not).
We therefore ship per distro repository information that can be
overriden (typically in testing) by dropping files in /etc.
For now use the latest nightlies for RHEL-8.2, we may want to
replace these with the official mirrors for GA eventually.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We were using fedora-30 as a test-distro and tar as test-output, but
that causes lots of churn in the tests when we refactor things. Use
the test distro instead, when generic functionality is being tested
and restrict testing of the individual distros to the distro-specific
tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Introduce a DistroRegister object. For now this does not introduce
any functional changes, as the object is always instantited to be
the same. However, in follow-up patches it will get options.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In RHEL golang dependencies must be vendored, whereas on Fedora they
must be packaged separately. Add conditionals accordingly.
Some macros are not yet available in RHEL, so fall back to older
versions. We may want to just use macros avilable everywhere
unconditionally, but I left that for a follow-up.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Commit b1c5ef2a introduced support for retrieving logs from osbuild.
This commit finishes the second part - actually returning the logs
from /compose/logs route.
dnf-json relies on dnf's ability to cache repository metadata. This is
important, because the API calls it quite often to serve requests for
package lists and depsolves.
However, osbuild's dnf stage always fetches new metadata, because it
doesn't have access to the host's cache. Since metadata is valid for
some time, even after a repository changed, the checksum we put in
the pipeline might be old.
Force a new metadata download when producing the pipeline. This is still
not perfect, but greatly reduces the probability of putting stale
metadata into the pipeline.
When restarting composer, we were not handling the compose states
correctly.
This resolves that as follows:
* any running composes are marked as failed,
* any waiting composes are put back in the pending jobs queue
As a consequence of needing the ability to reinitialize the job
queue, we must include the depsolved pipeline in the compose object.
This is the correct thing to do, as the semantics we currently
adhere to is that pipelines are depsolved when the compose is
started (and restarting composer should not affect this by for
instance re-depsolve the pipeline).
Resolves rhbz#1784062.
Signed-off-by: Tom Gundersen <teg@jklm.no>
