This just translates between the OpenAPI spec and our internal
API.
This still lacks tests, but a follow-up commit adds integration tests.
`internal/kojiapi/openapi.gen.go` was automatically generated from
`internal/kojiapi/openapi.yml`. To regenerate use `go generate ./...`.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This adds the OpenAPI spec for the new composer-koji API. This API is meant
to expose expose just the functionality needed to generate images and push
them to koji.
Each compose may consist of several images, each image may have a
different architecture and image type, and the set of repositories with
their contents may be distinct. However, a compose is restricted to one
distro and one koji transaction.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When using random names for artifacts like AWS snapshots, or Azure
images, it becomes hard to clean them up in case of CI failure. See this
issue for more details:
https://github.com/osbuild/osbuild-composer/issues/942
This PR introduces predictable names so that we can easily determine
which artifact belongs to which PR and therefore we can decide to wipe
all resources that are not needed any more.
The worker API returns errors of the form:
{ "message": "..." }
Print the message of those errors when receiving an error on the client.
This adds an `Error` type to openapi.yml and marks all routes as
returning it on 4XX and 5XX.
Instead of sending a `token` to workers, send back to URLs:
1. "location": URL at which the job can be inspected (GET) and updated
(PATCH).
2. "artifact_location": URL at which artifacts should be uploaded to.
The actual URLs remain the same, but a client does not need to stitch
them together manually (except appending the artifact's name).
Unfortunately, the client code generated by `deepmap` does not lend
itself to this style of APIs. Use standard http.Client again, which is a
partial revert of 0962fbd30.
Don't give out job ids to workers, but `tokens`, which serve as an
indirection. This way, restarting composer won't confuse it when a stray
worker returns a result for a job that was still running. Also,
artifacts are only moved to the final location once a job finishes.
This change breaks backwards compatibility, but we're not yet promising
a stable worker API to anyone.
This drops the transition tests in server_test.go. These don't make much
sense anymore, because there's only one allowed transition, from running
to finished. They heavily relied on job slot ids, which are not easily
accessible with the `TestRoute` API. Overall, adjusting this seemed like
too much work for their benefit.
The code generator uses the `operationID` field to generate server
handlers, client functions, and types. Use simpler names to make the
generated code easier to read.
This kind of common base path is better set in the top-level
`server.url` field, so that it can be adjusted.
For now, drop it completely, as we already broke the consistency when
introducing the `/status` route.
This change breaks backwards compatibility, but we're not yet promising
a stable worker API to anyone.
Write an openapi spec for the worker API and use `deepmap/oapi-codegen`
to generate scaffolding for the server-side using the `labstack/echo`
server.
Incidentally, echo by default returns the errors in the same format that
worker API always has:
{ "message": "..." }
The API itself is unchanged to make this change easier to understand. It
will be changed to better suit our needs in future commits.
More specifically only those that are needed in
/cmd/osbuild-image/tests.
This patch can be merged with the previous one if we want to make sure
every commit can be built, but I'm going to keep it like this for now so
that we can easily see the changes.
We now have greenboot available so update the packages and services
also add exclude sections as subscription-manager is for some reason
getting pulled in which brings dnf and other deps we explicitly don't
want.
Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
[teg: minor fixes and squash several fixup commits]
Prior this commit we only had support for username/password authentication
in the koji integration. This wasn't particularly useful because this
auth type isn't used in any production instance.
This commit adds the support for GSSAPI/Kerberos authentication.
The implementation uses kerby library which is very lightweight wrapper
around C gssapi library.
Also, the koji unit test and the run-koji-container script were modified
so the GSSAPI auth is fully tested.
In the near future, we will need to communicate with Koji using HTTPS.
This will surely bring the need for ignoring bad certificates/providing
our own self-signed ones. Thus, this commit prepares the Koji integration
by adding a way to accept a custom http transport which can be used to
customize the TLS settings.
Previously, Koji instance could be both logged-in and not logged-in.
This change disallows it: Now, the Koji instance is created by calling
koji.Login, so it must be always logged-in. This change should lead to more
robust code.
run-koji-container has now two actions: start and stop:
- ./run-koji-container.sh start
- ./run-koji-container.sh stop
The start action starts all containers. When it exits, all containers are
started and running in the background. To stop and removethem, use the stop
action.
This change is needed so we're able to easily use this script also in the CI
environment.
The setup should be container engine agnostic. This changes allows this script
to be run on systems which prefer docker over podman (e.g. Github Actions).
- inside RunJob() there is a deferred function which will remove
the entire temporary directory in which images are created, including
the streamOptimized file
- inside testBootUsingVMware(), which wants to use this function,
there is already a deferred function which removes the converted
image
osbuild support returning metadata about each of the stages/assembler
runs. Parse the results from the rpm stage, which contains the header
fields from the installed RPMs, in particular the MD5 sum of the RPMs in
question. This information is needed to be passed as metadata to koji
when uploading images.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In the same way `osbuild.Manifest` is the input to the osbuild API,
`osbuild.Result` is the output. Move it to the `osbuild` package where
it belongs.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This test is not run anywhere because it was surpassed by image tests with
azure boot type which perform more than just uploading a randomly generated
file to Azure. Let's delete dead code.
Applying a `Name` tag to the snapshot makes the name of the image appear
in the snapshot listing in EC2's console. It also makes it easier to
remove a snapshot after deregistering the AMI.
Signed-off-by: Major Hayden <major@redhat.com>
vCenter requires images to be uploaded as vmdk StreamOptimized. Lorax
always produced images on this format, so we should make sure to do the
same for our VMWare images.
Allow LocalTarget to request the images produced by osbuild be converted
to be streamOptimized before saving in composer, and hook the weldr API
up to enable this option for vmdk images.
Ideally this should simply be an option in osbuild, but that would
require some more work, which we will not manage in time for RHEL8.3.
Therefore do this minimal fix.
Note that that means the images produced by our manifests (including in
our image-test test cases) are not on the format that the weldr API
returns, so the tests we run on them would also, for now, need to
convert before uploading to vCenter.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We need the same RPMs to work equally well on a host running a beta
release (pulling beta content) as on a machine running GA (pulling GA
content). Detect this at run-time and point at the right repository.
Testing this is a bit hairy as we are building 8.3 images, but obviously
there is currently no 8.3 content at the GA URLs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Change the translation from our internal structs to the structs used for
weldr serialization to drop account details. These must obviously be
passed in to configure an upload, but exposing them in the logs may be
surprising.
There is no notion of user accounts in the weldr API, and the state
should not be considered private. However, this is likely to take people
by surprise, so let us guard the secrets entrusted to us.
Fixes#907.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Improve the message returned by osbuild-composer when a user asks for
logs of a compose that is still running.
Signed-off-by: Major Hayden <major@redhat.com>
