Workers don't report status for the osbuild run and the upload targets
separately. Before the move to the jobqueue, we explicitly set the
status of all targets when a compose finished. When I removed that,
the image status broke.
Set the status from what's returned by api.getComposeStatus() to restore
the original behavior.
Fixes#702
The new tool osbuild-store-dump saves store.json to the current working
directory, with more or less arbitrary data in it.
This has been executed on osubild-composer-{12,13} (mutatis mutandis),
and the results are saved in `internal/store/test`. A new test is added
which loads these stores and does very basic verification on them having
been loaded correctly.
This is mostly meant to catch regressions that means old stores are able
to make composer crash, or lose all its data. It would not catch minor
errors that leave the stores syntactically correct.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than Manifest() returning an osbuild.Manifest object, introduce a
new distro.Manifest object which represents it as an opaque, JSON
serializable object. This new type has the following properties:
1) its serialization is compatible with the input to osbuild,
2) any valid osbuild input can be deserialized into it, and
3) marshalling and unmarshaling to and from JSON is lossless.
This means that even as we change the subset of valid osbulid manifests
that we support, we can still load any previous state from disk, and it
will continue to work just as before, even though we can no longer
deserialize it into our internal notion of osbuild.Manifest.
This fixes the underlying problem of which #685 was a symptom.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Any valid ImageBuild must contain a Manifest, so don't allow this to be
nil, simplifying the code a bit in the process.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The manifests we support marshalling/unmarshalling in composer are a
subset of the ones supported by osbuild. osbuild needs to preserve
backwards compatibility, but composer only needs to be concerned with
producing valid manifests at a given time.
A problem with this is that when we marshal our manifests to disk, and
read them out with a future version of composer, we would have to make
sure that the serialization is compatible, or unmarshalling the entire
store will fail. This is what happened when commit 5938276d3b
changed the way we marshal the file source.
Improve on this by only dropping those composes from the store on
unmarshal, whose manifests we are no longer able to unmarshal.
In future commits we will drop that requirement too.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Specifying the image format in the `DiskContainers` part of the snapshot
import process sometimes causes AWS to fail the import due to formatting
issues. However, AWS seems to do a decent job at determining the image
format on its own without specifying the format explicitly.
Signed-off-by: Major Hayden <major@redhat.com>
`log` is `nil`, unless we are in verbose mode. Skip logging if no logger
was passed.
Long-term I think we should reconsider how we do our logging, and always
log these type of unexpected errors, and hence always have the logger be
non-nil. For now, fix the immediate problem.
The fact that loading the store from disk fails on upgrade is
unexpected, and should be fixed separately.
Fixes#685.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This adds placeholder architecture structs, but no image types are
hooked up.
This will allow osbuild-composer to run on these architectures, and
serve the weldr API, but without support for building any images.
Image type support will be added in the future.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is not a functional change, all produced manifests are as before.
This is in preparation for adding support for more architectures in a
follow-up patch.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rename the `Fedora32` type to simply `distribution`, to avoid the
stutter. Move `New()` to the bottom of the file, which is the only
non-generic part. Also make the linter happy.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is currently only used in tests. We broke backwards compatibility
when we split the repo name and the repo id. The tests expect this to be
indexed by name, not index.
This should probably be reworked or dropped, for now revert to the old
behavior.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The 'id' field in the json files is exposed as 'name' in the weldr API,
and that is now its only use. Rename it to match.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is how it is used in the rest of the code, as a name to represent
the repository in the weldr API. Rename to match its use, and avoid
confusion with the ID passed to dnf-json, which is not the same.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We don't have to go via a map, and also don't have to rely on the ID's
in the passed in repos are the same, by instead relying on RepoID being
the offset into the passed in repos array, as set in the previous patch.
Signed-off-by: Tom Gundersen <teg@jklm.no>
rpmmd now contains three repo config structs. RepoConfig is for use
throughout osbuild-composer. repository is for reading from our
internal json repository files before creating a RepoConfig.
dnfRepoConfig is is for use within rpmmd and it matches what dnf-json
expects from a repo.
Repos now contain support for rhsm. In order to connect to rhel's
cdn, dnf must pass an sslcacert, sslcakey, and sslclientcert. When a
repo is used for fetching metadata or depsolving it is checked for
rhsm secrets. If secrets are needed they are retrieved from the host
system. Packages requiring rhsm are marked as requiring
"org.osbuild.rhsm" secrets.
We do not properly test, and do not have properly defined use-cases for
the ext4-filesystem, partitioned-disk, nor tar image types. Drop them to
focus on delivering the things we car properly test.
Signed-off-by: Tom Gundersen <teg@jklm.no>
There are times where it would be good to monitor that osbuild-composer
is up and running. Add a very simple status check that always returns
200/OK. This can be expanded later to verify that other parts of
osbuild-composer are working properly.
Signed-off-by: Major Hayden <major@redhat.com>
The `jobs/:job_id/builds/:build_id/image` route was awkward: the
`:jobid` was actually weldr's compose id and `:build_id` was always `0`.
Change it to `jobs/:job_id/artifacts/:name`, where `:job_id` is now a
job id, and `:name` is the name of the artifact to upload. In the
future, it could support uploading more than one artifact.
This allows removing outputs from `store`, which is now back to being a
pure JSON-store. Take care that `weldr` returns (and deletes) images
from the new (or for backwards compatibility, the old) location.
The `org.osbuild.local` target continues to exist as a marker for the
worker to know whether it should upload artifacts.
Similarly to how some image types accept a size-argument, some image
types require ostree-specific options. For now this is only used by
Fedora IoT on Fedora 32.
composer-cli nor cockpit-composer currently has the ability to pass in
these options, but the change is backwards compatible and existing image
types are unaffected. If the options are omitted for image types that
require them, sensible defaults are chosen.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When generating an osbuild manifest for an image type, we take a
customizations struct, which specifies the image-type-independent
customizations to apply. We also take the size argument, which is
specific to the image build and not part of the blueprint.
Introduce a new argument ImageOptions, which for now just wraps the size
argument. These options are specific to the image build/type, and
therefore does not belong with the other customizations.
For now this is a non-functional change, but follow-up commits will
introduce more types of image options.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This adds the wrapper object and a simple test for the rpm-ostree stage.
See the osbuild documentation for details.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This adds the wrapper object and a simple test for the ostree.commit
assembler.
See the osbuild documentation for details.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Exactly one of the UUID or the label must be set. The helpers are kept
the same and only supports the UUID for now, but these should likely be
dropped in favor of open-coding the structs anyway.
We do not enforce that the UUID or label is set, but osbuild will.
Signed-off-by: Tom Gundersen <teg@jklm.no>
As it turns out, the default expectation is not to distinguish between
these. We will now produce whatever is the most recent minor release by
default, and image tests will still be pinned at a given snapshot to be
reproducible.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Importing an image into an EBS snapshot can sometimes take a long time.
The customer experience for a timeout is not great.
Allow the import to continue running for as long as needed without
timing out. This allows a customer to monitor the process of their
import until it has a success or failure status.
Fixesosbuild/osbuild-composer#632.
Signed-off-by: Major Hayden <major@redhat.com>
GetImageBuildResult() method is not called anywhere, so we can drop it. As
it was the only reader of result.json file, we can also drop all the code
which wrote to that file.
The store-jobqueue split broke the retrieval of logs in weldr API. Weldr API
still retrieved the logs from store but they are now stored in the jobqueue.
This commit fixes that by taking advantage of the fact that getComposeStatus()
now returns the compose result from which the logs can be generated.
Let's speak why I merged JobResult and JobStatus together:
Both methods actually called jobqueue.JobStatus(), so performance-wise
there's no real difference and it feels to me that it makes the code simpler:
You don't have to decide which method to call, you just get all the data
about a job in one call. We could split those again when we see some perf
issues with retrieving logs on each status check but I don't think we should
optimize prematurely. Let's leave some work for the future us.
When a job doesn't exist for a compose, just return an empty result.
In the future this will cause logs disappear from jobs created before the
store-jobqueue split but we decided we can live with that, they are still
in /var for those who really want them.
The reasoning is the same as in one of the previous commits. I think that
returning multiple values from one method can be confusing and potentially
error-prone. Also, this method will return ComposeResult soon too.
Also, I chose to create a new composeStatus struct instead of reusing
worker.JobStatus. I think that's a good thing in terms of consistency and
reducing the dependencies between packages.
Not a functional change.
