Let's keep this on the same filesystem as the osbuild store, and
in particular stay away from /var/tmp and its scary semantics.
We are not aware of any issues caused by /var/tmp, but getting
rid of it means we don't have to think about that when debugging,
if nothing else.
Signed-off-by: Tom Gundersen <teg@jklm.no>
VMDK image has default name 'disk.vmdk' and there is no option to change the name when uploading to vSphere,
so I'm using symlink so that uploaded image has the name user specified instead of the default one.
New upload target for VMWare, similar to the ones for AWS and Azure,
allowing users to set credentials for their vSphere instance.
Commit also includes function that performs the actual upload.
Expose a more detailed job status result - specifically, include upload status
alongside image status. Expand openapi.yml accordingly and add an UploadStatus
field to the OSBuildJobResult struct. At the moment, only represent the
"success" and "failure" states of UploadStatus - to differentiate between
"pending" and "running" would involve significant design decisions and should be
addressed in a separate commit.
Previously, the checks that dependencies were successful were all over the
Run() method. This led to a issue #1101 (lovely binary number btw).
This commit rewrites the Run() method to:
1) Extract dynamic args. Return an error if they cannot be unmarshalled.
2) Check if dependencies were successful. If not, call kojiFail, update the
job and return.
3) Create the CGImport metadata and call kojiImport.
Fixes#1101
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
osbuild output should be always JSON-parsable. However, if a user has a weird
installation of osbuild, it can return malformed JSON. In this case, it's
very hard to debug what happened because the worker doesn't provide any
useful information. This commit adds the non-parsable osbuild output to the
error so there's at least some clue in the logs what happened.
Let me reiterate: In 99% these cases, this shouldn't be needed. It will give
us some hints in these 1% cases though.
You may want to ask if using json.Decoder isn't better because it doesn't
buffer the input. However, it does buffer. Firstly, json.Decoder.Decode()
buffers the whole JSON value (it contains kinda cool state machine to do it)
and in the second run, it parses the buffer. Therefore, the only thing that
this commit actually does is that it moves the buffer out of json.Decoder.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Serializing an interface does not work, let us simply use the string
representation and treat the empty string as no error. This is
compatible with the current API in the success case, and fixes the
error case, which is currently broken.
Also extend the test matrix for the kojiapi to ensure that all the
different kinds of errors can be serialized correctly and leads to
the correct status being returned.
Fixes#1079 and #1080.
The three new job types osbuild-koji, koji-init, and koji-finalize
allows the different tasks to be split appart and in particular for
there to be several builds on different architectures as part of a
given compose.
Introduce JobImplementation and turn the current RunJob() into
OSBuildJobImpl. Make main() select a job impl based on job type.
This is in preparation to add additional impls.
Move the fact that the worker is requesting jobs of type "osbuild" out
of the client library.
For one, require consumers to pass accepted job types to RequestJobs()
and allow querying for the job type with the new Type() function.
Also, make OSBuildArgs() and Update() generic, requiring to pass an
argument that matches the job type.
Now, main() does not deal with OSBuildJobResult anymore, and RunJob()
doesn't return it. This means we can add more job types (i.e., different
RunJob()s) now.
WatchJob() regularly checks if a job was canceled in a goroutine. It
does so by accessing composer's `/jobs/{token}` route. However, once the
main goroutine marks the job as done (by sending PATCH to that same
route), the `token` is no longer valid and thus the route not accessible
anymore.
main() does cancel the goroutine running WatchJob, but it's not
guaranteed that it gets scheduled in time to actually stop watching the
job.
Thus, don't cancel the job when fetching the `/jobs/{token}` fails. This
means that it won't cancel the job anymore when the connection to
composer goes down.
Also, we will be able to move job.Update() into RunJob().
Workers reported status via an `osbuild.Result`, which only includes
osbuild output. Make it report OSBuildJobResult instead, which was meant
to be used for this purpose and is already used as the result type in
the jobqueue.
While at it, add any errors produced by targets into this struct, as
well as an overall success flag.
Note that this breaks older workers returning the result of an osbuild
job to a new composer. I think this is fine in this case, for two
reasons:
1. We don't support running different versions of the worker and
composer in the weldr API, and remote workers aren't widely used yet.
2. Both osbuild.Result and worker.OSBuildJobResult have a top-level
`Success` boolean. Thus, logs are lost in such cases, but the overall
status of the compose is not.
This function is almost the same as the koji uploader, except that it
calls `CGFailBuild` instead of `CGImport` at the end.
Don't exit early from RunJob() when the job failed. Instead, go through
all the uploaders anyway. All the others don't do anything when the job
fails, but now we have the chance to do the necessary `CGFailBuild` call
for koji.
This moves more logic from main() into RunJob(), so that we can support
different job kinds in the future.
Add "image_name" and "stream_optimized" fields to the osbuild job as
replacement for the local target options. The former signifies the name
of the uploaded artifact and whether an artifact should be uploaded at
all (only weldr API). The latter will be deprecated at some point, when
osbuild itself can make streamoptimized vmdk images.
This change separates what have always been two distinct concepts:
artifacts that are reported back to the composer node (in practice
always running on the same machine), and upload targets to clouds and
such. Separating them makes it easier to add job types that only allow
one upload target while keeping artifacts.
Keep the local target around, so that jobs that are scheduled can still
be run after an upgrade.
The server hasn't used common.ImageBuildState to mark a job as
successful or failed for a long time. Instead, it's using the job's
return argument for that. (Jobs don't have a high-level concept of
failing).
Drop the check in the server, and always send "FINISHED" from the client
for backwards compatibility.
osbuild reports failing builds in two ways: it sets the "success" field
in its output to `false` and it returns with a non-zero exit status. The
worker used both, returning an `OSBuildError` when osbuild return
non-zero, but also forwarding the resulting object with the "success"
field.
Change this to only use the "success" field and ignore the return value.
The latter is useful for people running osbuild in a terminal or script,
but is redundant for this use-case.
This makes error reporting more consistent: `RunOSBuild` only returns an
error when *running* osbuild failed, not when the build fails.
We have the same thing for AWS. The AWS target also specifies under what name
should be the image available in EC2.
As requested by Brew maintainers Tomáš Kopeček and Lubomír Sedlář.
osbuild runs directly on the host, there's no intermediate container,
therefore we should set the container type to none.
As suggested by Brew maintainers Tomáš Kopeček and Lubomír Sedlář.
Contrary to our assumption, we cannot initialize the build with the
link to the task. We can only update the link once the build has
completed.
This seems like a bug in koji, but we keep it like this for now.
Move to requiring CGInitBuild to be called before CGImport. In the
future we could make the former optional again, but for now we want to
allow the caller to have done CGInitBuild and for composer only to do
the CGImport using the passed in build_id and token.
Also rename and document some struct fields in the metadata struct to
make them more specific to our use-case and hopefully easier to read.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Introduce a target for Koji and hooked it up in the worker, so if koji
target is specified, the image is uploaded to koji.
[teg: use system kerberos config rather than reading from env]
Instead of sending a `token` to workers, send back to URLs:
1. "location": URL at which the job can be inspected (GET) and updated
(PATCH).
2. "artifact_location": URL at which artifacts should be uploaded to.
The actual URLs remain the same, but a client does not need to stitch
them together manually (except appending the artifact's name).
Unfortunately, the client code generated by `deepmap` does not lend
itself to this style of APIs. Use standard http.Client again, which is a
partial revert of 0962fbd30.
The job token will be deprecated in favor of URLs.
If a key is not set, use a new random UUID. Also, don't overwrite the
options struct with that new key.
Don't give out job ids to workers, but `tokens`, which serve as an
indirection. This way, restarting composer won't confuse it when a stray
worker returns a result for a job that was still running. Also,
artifacts are only moved to the final location once a job finishes.
This change breaks backwards compatibility, but we're not yet promising
a stable worker API to anyone.
This drops the transition tests in server_test.go. These don't make much
sense anymore, because there's only one allowed transition, from running
to finished. They heavily relied on job slot ids, which are not easily
accessible with the `TestRoute` API. Overall, adjusting this seemed like
too much work for their benefit.
In the same way `osbuild.Manifest` is the input to the osbuild API,
`osbuild.Result` is the output. Move it to the `osbuild` package where
it belongs.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
vCenter requires images to be uploaded as vmdk StreamOptimized. Lorax
always produced images on this format, so we should make sure to do the
same for our VMWare images.
Allow LocalTarget to request the images produced by osbuild be converted
to be streamOptimized before saving in composer, and hook the weldr API
up to enable this option for vmdk images.
Ideally this should simply be an option in osbuild, but that would
require some more work, which we will not manage in time for RHEL8.3.
Therefore do this minimal fix.
Note that that means the images produced by our manifests (including in
our image-test test cases) are not on the format that the weldr API
returns, so the tests we run on them would also, for now, need to
convert before uploading to vCenter.
Signed-off-by: Tom Gundersen <teg@jklm.no>
composer uses the success field to decide whether a build succeeded or failed.
This is bad.
Unfortunately, fixing this requires kinda big code changes. This commit
changes the worker's behaviour to set the osbuild success flag to false
even on errors which weren't caused by osbuild (e.g. an upload error).
This is certainly hacky but I think it's still essential to tell the user
that an error occurred.
Fixes#789
We print messages to the log when the build fails, but we don't print
one when the build is successful.
I really want to celebrate our successes more often, so let's print a
success message when osbuild completes successfully. Since success
doesn't feel like success without an emoji, let's add one of those, too. 🎉
Signed-off-by: Major Hayden <major@redhat.com>
