CHANGE_ID is set for PRs, which is why it worked during the review
process, but in the master branch it is not set. BRANCH_NAME is set for
both PRs and master branch. In case of PRs it is in form `PR-<pull
request number>`.
In the same way we require authentication for the worker API, require
clients of the koji API to authenticate using SSL client certificates.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Contrary to our assumption, we cannot initialize the build with the
link to the task. We can only update the link once the build has
completed.
This seems like a bug in koji, but we keep it like this for now.
Move to requiring CGInitBuild to be called before CGImport. In the
future we could make the former optional again, but for now we want to
allow the caller to have done CGInitBuild and for composer only to do
the CGImport using the passed in build_id and token.
Also rename and document some struct fields in the metadata struct to
make them more specific to our use-case and hopefully easier to read.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Introduce a target for Koji and hooked it up in the worker, so if koji
target is specified, the image is uploaded to koji.
[teg: use system kerberos config rather than reading from env]
Add a systemd socket for Koji API. If enabled when osbuild-composer.service
is started, the service will also listen on the socket and serve Koji API
there.
Note that Koji API doesn't upload to Koji yet, this still needs to be hooked
up.
Based on a patch from Tom Gundersen, thanks!
When using random names for artifacts like AWS snapshots, or Azure
images, it becomes hard to clean them up in case of CI failure. See this
issue for more details:
https://github.com/osbuild/osbuild-composer/issues/942
This PR introduces predictable names so that we can easily determine
which artifact belongs to which PR and therefore we can decide to wipe
all resources that are not needed any more.
Instead of sending a `token` to workers, send back to URLs:
1. "location": URL at which the job can be inspected (GET) and updated
(PATCH).
2. "artifact_location": URL at which artifacts should be uploaded to.
The actual URLs remain the same, but a client does not need to stitch
them together manually (except appending the artifact's name).
Unfortunately, the client code generated by `deepmap` does not lend
itself to this style of APIs. Use standard http.Client again, which is a
partial revert of 0962fbd30.
The job token will be deprecated in favor of URLs.
If a key is not set, use a new random UUID. Also, don't overwrite the
options struct with that new key.
Don't give out job ids to workers, but `tokens`, which serve as an
indirection. This way, restarting composer won't confuse it when a stray
worker returns a result for a job that was still running. Also,
artifacts are only moved to the final location once a job finishes.
This change breaks backwards compatibility, but we're not yet promising
a stable worker API to anyone.
This drops the transition tests in server_test.go. These don't make much
sense anymore, because there's only one allowed transition, from running
to finished. They heavily relied on job slot ids, which are not easily
accessible with the `TestRoute` API. Overall, adjusting this seemed like
too much work for their benefit.
More specifically only those that are needed in
/cmd/osbuild-image/tests.
This patch can be merged with the previous one if we want to make sure
every commit can be built, but I'm going to keep it like this for now so
that we can easily see the changes.
explicitly specify the cluster and the default resource pool
when importing b/c the import process creates a temporary VM,
which requires a ResourcePool to provision. Same thing when
provisioning a VM.
Prior this commit we only had support for username/password authentication
in the koji integration. This wasn't particularly useful because this
auth type isn't used in any production instance.
This commit adds the support for GSSAPI/Kerberos authentication.
The implementation uses kerby library which is very lightweight wrapper
around C gssapi library.
Also, the koji unit test and the run-koji-container script were modified
so the GSSAPI auth is fully tested.
Previously, Koji instance could be both logged-in and not logged-in.
This change disallows it: Now, the Koji instance is created by calling
koji.Login, so it must be always logged-in. This change should lead to more
robust code.
In the same way `osbuild.Manifest` is the input to the osbuild API,
`osbuild.Result` is the output. Move it to the `osbuild` package where
it belongs.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The tests are no longer run on Travis, therefore we don't need the special
setup to run them there.
This change should also fix#929 that is probably caused due to osbuild
executed in a weird way.
Fixes#929
vCenter requires images to be uploaded as vmdk StreamOptimized. Lorax
always produced images on this format, so we should make sure to do the
same for our VMWare images.
Allow LocalTarget to request the images produced by osbuild be converted
to be streamOptimized before saving in composer, and hook the weldr API
up to enable this option for vmdk images.
Ideally this should simply be an option in osbuild, but that would
require some more work, which we will not manage in time for RHEL8.3.
Therefore do this minimal fix.
Note that that means the images produced by our manifests (including in
our image-test test cases) are not on the format that the weldr API
returns, so the tests we run on them would also, for now, need to
convert before uploading to vCenter.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is not shipped in RHEL, so use the library directly to query the IP
address. This is a massive hack, but let us revisit this after the next
release.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We need the same RPMs to work equally well on a host running a beta
release (pulling beta content) as on a machine running GA (pulling GA
content). Detect this at run-time and point at the right repository.
Testing this is a bit hairy as we are building 8.3 images, but obviously
there is currently no 8.3 content at the GA URLs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The osbuild-composer-rcm package was never finished, not in use and will be replaced by osbulid-composer-koji.
Signed-off-by: Tom Gundersen <teg@jklm.no>
`systemd-tmpfiles` will helpfully delete "old" files in /var/tmp at regular
intervals. The files installed from rpm has the timestamps from when they
were packaged, which causes some to be cleaned up when the timer triggers.
The first timer triggers 15 minutes after boot, so we were sometimes hit
by this when our CI was under load.
Fixes#839 and #862.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Create a GetImageInfoCommand analogous to GetOsbuildCommand that
will adjust the PYTHONPATH for image-info on travis so that the
osbuild python module is accessible.
Previously, all resouces were created with a certain tag. When the cleanup
phase came, the Resources - List route[1] was used to get all resources with
the tag. Then, they were deleted in the right order.
Sadly, the Resources - List API has issues with listing disks. Sometimes,
it returns the virtual disks 15 minutes after they were created. As the
result, the disks have been left behind quite often and our bill was higher
than necessary.
This commit uses a different method - the Go code now knows all resource names
(see the previous commit), so it can delete all resources without listing them
using the "broken" API route.
[1]: https://docs.microsoft.com/en-us/rest/api/resources/resources/list
Prior this commit the resource names were generated in the deployment
template, so the Go code actually didn't know them. This commit generates
all names in the Go code, so they can be used in the future commits.
Rather than getting a set of base packages from the ImageType, and then
appending the requested packages from the blueprint, pass the blueprint
into the new Packages() function, and return the full set of packages to
be depsolved.
This allows us to also append packages based on other customizations
too, and use that to append chrony when the timezone is set. This
matches the behavior anaconda had, and there was a TODO item to do this,
which had been overlooked.
Fixes#787.
Signed-off-by: Tom Gundersen <teg@jklm.no>
composer uses the success field to decide whether a build succeeded or failed.
This is bad.
Unfortunately, fixing this requires kinda big code changes. This commit
changes the worker's behaviour to set the osbuild success flag to false
even on errors which weren't caused by osbuild (e.g. an upload error).
This is certainly hacky but I think it's still essential to tell the user
that an error occurred.
Fixes#789
We print messages to the log when the build fails, but we don't print
one when the build is successful.
I really want to celebrate our successes more often, so let's print a
success message when osbuild completes successfully. Since success
doesn't feel like success without an emoji, let's add one of those, too. 🎉
Signed-off-by: Major Hayden <major@redhat.com>
When osbuild fails, osbuild-composer's image tests print the output as
ugly JSON and that makes it difficult to determine why osbuild failed.
Instead, print the json in pretty format. 🦋Fixes#782.
Signed-off-by: Major Hayden <major@redhat.com>
When osbuild crashes (e.g. when cp fails because of the machine running
out of disk space), it doesn't produce a machine-readable result. Due to
our suboptimal handling of the result struct (this is my fault), this can
lead to result == nil. However, composer expects that result != nil in all
cases because it uses the Success flag to assess the compose state. If
result == nil, it just crashes terribly.
Exit the whole worker process when a job was canceled, because osbuild
does not clean up all child processes when receiving SIGKILL.
Change the service to restart osbuild-worker also on success, and
decrease the restart timeout.
