copy() will not append to a slice, so if the length is 0 nothing will be
copied. Initialize the slice with the length instead of the capacity.
This fixes a problem with blueprint commits vanishing when the server is
restarted.
This commit also includes a new test in json_test for the
newCommitsFromV0() function.
Initial definition for RHEL for edge package set on x86.
This is based marginally on the upstream Fedora IoT package set
but slimmed down and adjusted for RHEL package naming.
Based on Tom's patch.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
The sources weldr API already supports this, so hook it up to be
represented on disk and in our internal state tracking too.
This does not yet hook this up to be respected by osbuild, which
currently takes this to be unconditionally set to true.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This changes store.DeleteSource to DeleteSourceByName for v0 use and
DeleteSourceByID for v1 usage.
It includes a new client function DeleteSourceV1, adds a new test, and
converts the tests for the previous Source V1 API commits to use
DeleteSourceV1.
This commit changes the store.GetAllSources to distinguish between
getting the source by the Name field, or by the ID (the key to the map)
using GetAllSourcesByName and ...ByID.
SourceConfig.RepoConfig() now takes an id parameter because SourceConfig
only stores the Name, not the ID.
In weldr I split the sourceInfoHandler into 2 separate functions for v0
and v1 behavior, with the core of the old function refactored as
getSourceConfigs and used by both of them.
This also adds new structs for the SourceResponseV0 and SourceResponseV1
as well as helper functions for converting to/from store.SourceConfig
This commit changes the store.PushSource function to take the key as
well as the SourceConfig so that it can be used for v0 or v1.
It adds helper functions for decoding the toml/json into a new
SourceConfig interface type which lets the core source/new code be
shared between the versions.
It also adds tests for the new API behavior.
This is the first patch in a series to add APIv1 support to the
/projects/source routes. The change involves using the store.Sources key
in a different way (as an id instead of as a duplicate of the struct's
Name field) but does not actually involve changing the Sources json in
the store.
In the V0 API the name of the source was used as the identifier, and
there was no short id. In V1 the source is identified by the API using
a short id, and the Name is just a field in the struct to describe the
source. This will become more obvious with the /projects/source/info
response.
This commit changes the following:
Changes store.ListSources to ListSourcesByName and explicitly pulls the
name from the source struct instead of the key. v0 will use this
function call.
Adds store.ListSourcesById which returns the source key as the
identifier. This is used by v1.
Adds a new weldr.SourcesListV1 response type, even though it is exactly
the same as the V1 response in this specific case. I thought it would be
better to have one called V1 than to reuse the V0 struct and possibly
confuse people.
The /projects/source/list API now lists the sources by name for v0 and id for v1.
A test has been added. You will notice it still uses v0 to push and
delete the sources. These will be updated when the new version of the
functions are added in subsequent commits.
The new tool osbuild-store-dump saves store.json to the current working
directory, with more or less arbitrary data in it.
This has been executed on osubild-composer-{12,13} (mutatis mutandis),
and the results are saved in `internal/store/test`. A new test is added
which loads these stores and does very basic verification on them having
been loaded correctly.
This is mostly meant to catch regressions that means old stores are able
to make composer crash, or lose all its data. It would not catch minor
errors that leave the stores syntactically correct.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than Manifest() returning an osbuild.Manifest object, introduce a
new distro.Manifest object which represents it as an opaque, JSON
serializable object. This new type has the following properties:
1) its serialization is compatible with the input to osbuild,
2) any valid osbuild input can be deserialized into it, and
3) marshalling and unmarshaling to and from JSON is lossless.
This means that even as we change the subset of valid osbulid manifests
that we support, we can still load any previous state from disk, and it
will continue to work just as before, even though we can no longer
deserialize it into our internal notion of osbuild.Manifest.
This fixes the underlying problem of which #685 was a symptom.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Any valid ImageBuild must contain a Manifest, so don't allow this to be
nil, simplifying the code a bit in the process.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The manifests we support marshalling/unmarshalling in composer are a
subset of the ones supported by osbuild. osbuild needs to preserve
backwards compatibility, but composer only needs to be concerned with
producing valid manifests at a given time.
A problem with this is that when we marshal our manifests to disk, and
read them out with a future version of composer, we would have to make
sure that the serialization is compatible, or unmarshalling the entire
store will fail. This is what happened when commit 5938276d3b
changed the way we marshal the file source.
Improve on this by only dropping those composes from the store on
unmarshal, whose manifests we are no longer able to unmarshal.
In future commits we will drop that requirement too.
Signed-off-by: Tom Gundersen <teg@jklm.no>
`log` is `nil`, unless we are in verbose mode. Skip logging if no logger
was passed.
Long-term I think we should reconsider how we do our logging, and always
log these type of unexpected errors, and hence always have the logger be
non-nil. For now, fix the immediate problem.
The fact that loading the store from disk fails on upgrade is
unexpected, and should be fixed separately.
Fixes#685.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is how it is used in the rest of the code, as a name to represent
the repository in the weldr API. Rename to match its use, and avoid
confusion with the ID passed to dnf-json, which is not the same.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The `jobs/:job_id/builds/:build_id/image` route was awkward: the
`:jobid` was actually weldr's compose id and `:build_id` was always `0`.
Change it to `jobs/:job_id/artifacts/:name`, where `:job_id` is now a
job id, and `:name` is the name of the artifact to upload. In the
future, it could support uploading more than one artifact.
This allows removing outputs from `store`, which is now back to being a
pure JSON-store. Take care that `weldr` returns (and deletes) images
from the new (or for backwards compatibility, the old) location.
The `org.osbuild.local` target continues to exist as a marker for the
worker to know whether it should upload artifacts.
GetImageBuildResult() method is not called anywhere, so we can drop it. As
it was the only reader of result.json file, we can also drop all the code
which wrote to that file.
By the same logic as the previous patch.
This exposed an apparent bug, where we are sorting by the version field
of the blueprint in the change objects, but these were never
(un)marshalled, so would always be set to zero in the past.
Remove the no-op sorting for now, as it never had an effect, and
put a note whether we should instantiate the blueprint in the change
object somehow.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Apply the backwards compatibility where the ImageBuild struct is
assembled. This should make the code simpler to reason about, by
reducing the structs we produce and later fix up.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The store only serves the weldr API, and that hard-codes the assumption
of only one image build per compose all over the place. Move this
assumption into the json serialization handler.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Serialize to a plain string, and do the translation for compatibility
from string to string.
This removes a lot of code, and if we change the store format, we should
just skip the conversion altogether.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We may want to move this to the test_distro package in the future, but
calling it something else than its real name is just confusing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This reduces the amount of resolving and error checking we have to do.
This exposed a bug in weldr's ComposeEntry type, which will be fixed in
a follow-up commit.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The types exposed by the compose package are only used in the store API,
so move them there where they belong.
Making the ownership of the types clear, rather than having them live in
a package for themselves, will make it clearer how the types can be
modified in follow-up commits.
Also remove the JSON annotations, as these types are no longer used for
serialization.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Distinguish between the types used at runtimes and the types used for
(un)marshaling. Also make the types private to the store package.
This should allow us to reduce the interdependencies between the
packages and more easily change things without accidentally breaking
backwards compatibility.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This way we can make more of the store fields and types private in
follow up commits.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
After the queue rework, TestCompose() was the only user of
UpdateImageBuildInCompose() and Compose.UpdateState(). Set all required
fields in TestCompose() directly instead of going through those.
The store is responsible for two things: user state and the compose queue. This
is problematic, because the rcm API has slightly different semantics from weldr
and only used the queue part of the store. Also, the store is simply too
complex.
This commit splits the queue part out, using the new jobqueue package in both
the weldr and the rcm package. The queue is saved to a new directory `queue/`.
The weldr package now also has access to a worker server to enqueue and list
jobs. Its store continues to track composes, but the `QueueStatus` for each
compose (and image build) is deprecated. The field in `ImageBuild` is kept for
backwards compatibility for composes which finished before this change, but a
lot of code dealing with it in package compose is dropped.
store.PushCompose() is degraded to storing a new compose. It should probably be
renamed in the future. store.PopJob() is removed.
Job ids are now independent of compose ids. Because of that, the local
target gains ComposeId and ImageBuildId fields, because a worker cannot
infer those from a job anymore. This also necessitates a change in the
worker API: the job routes are changed to expect that instead of a
(compose id, image build id) pair. The route that accepts built images
keeps that pair, because it reports the image back to weldr.
worker.Server() now interacts with a job queue instead of the store. It gains
public functions that allow enqueuing an osbuild job and getting its status,
because only it knows about the specific argument and result types in the job
queue (OSBuildJob and OSBuildJobResult). One oddity remains: it needs to report
an uploaded image to weldr. Do this with a function that's passed in for now,
so that the dependency to the store can be dropped completely.
The rcm API drops its dependencies to package blueprint and store, because it
too interacts only with the worker server now.
Fixes#342
Code that's calling PushCompose() had to depsolve packages and fetch the
right ImageType from a distro, but not create the osbuild manifest. That
was left for PushCompose to do. Move it out of there to the callers, so
that the store is mainly concerned with storing things.
This also simplifies the argument list of PushCompose().
Yes. This goes against my desire not to change code to accommodate
tests. But there is no other good way to test compose results without
long running, and possibly fragile, composes. And this matches lorax's
behavior.
The change adds support for the ?test=1|2 query parameter to the compose
POST, and a new store function - PushTestCompose that handles creating
the fake compose results.
Passing ?test=1 will create a failed compose. Passing ?test=2 will
create a successful compose, but one without any files.
The purpose of these is to be able to test the compose result API
responses like compose/failed, etc.
If you don't set the time it ends up being the default Go time which is
1/1/1 and when you convert that using UnixNano() you get a nice big
negative number (-6795364578871345152) which then eventually shows up in
the queue, finished, and failed output as 'Fri Aug 30 17:47:39 1754'
and since the Time Travel feature is not yet complete this is
impossible.
The fix is to set it when the job is started.
If a user creates a compose with a size of 0, the default image size for
the image type should be used. Also, certain image types have
requirements for the image size. In order to ensure that the proper image
size is stored in the compose object, the compose's ImageBuild object
uses ImageType.Size() to get the correct image size for the image type.
