From Koji Content Generator Metadata[1]:
"maven, win, or image: Legacy build type names which appear at this level
instead of inside typeinfo."
=> see, it's legacy
"typeinfo: A map whose entries are the names of the build types used for
this build, which are free form maps containing type-specific information
for this build."
=> struct{} is used for typeinfo.image because the docs says it should contain
"a free form map", null apparently isn't an option.
[1]: https://docs.pagure.org/koji/content_generator_metadata/
As suggested by the Brew maintainers Tomáš Kopeček and Lubomír Sedlář.
Fedora 33 ships the new API so let's do the switch now.
But... this would break older Fedoras because they only have the old API,
right?
We have the following options:
1) Ship xmlrpc compat package to Fedora 33+. This would mean that we delay the API switch till F32 EOL. This would be the most elegant solution, yet it has two issues: a) We will surely not be able to deliver the compat package before F33 Final Freeze. b) It's an extra and annoying work.
2) Downstream patch. No.
3) Use build constraints and have two versions of our code for both different
API.
I chose solution #3. It has an issue though:
%gobuild macro already passes -tags argument to go build. Therefore the
following line fails because it's not possible to use -tags more than once:
%gobuild -tags kolo_xmlrpc_oldapi ...
Therefore I had to come up with manual tinkering with the build constraints
in the spec file. This is pretty ugly but I like that:
1) Go code is actually clean, no weird magic is happening there.
2) We can still ship our software to Fedora/RHEL as we used to
(no downstream patches)
3) All downstreams can use the upstream spec file directly.
Note that this doesn't affect RHEL in any way as it uses vendored libraries.
Fedora 33 ships kolo/xmlrpc with a different API. This commit extracts the
affected code so we can use build flags in the future allowing us to use
both API versions.
Also install it is part of he tests subpackage. This a helper-tool, not
golang code, so should not live in `internal`. We need access to this
from the integration tests, so install it onto the tests system.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than using the arbitrary port 8701, use the standard 443. The
worker API will remain on a separate port, and as long as the two APIs
are exposed by the same binary that will have to remain separate at
8700.
Move the test instance of koji on localhost from 443 to 4343, to avoid a
conflict.
In a follow-up we should also give this API a prefix, so the cloud API
can share the same port with it.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This sets up containers running koji and supporting infrastructure, and
calls the osbuild-composer-koji API to build and image and push it into
our testing instance.
koji-compose.py and various fixes by Christian Kellner.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Contrary to our assumption, we cannot initialize the build with the
link to the task. We can only update the link once the build has
completed.
This seems like a bug in koji, but we keep it like this for now.
The API of kolo/xmlrpc changed after the commit that is shipped in
Fedora. Pin the vendored version to that and adjust the API usage.
This should make the RPM compile in both RHEL and Fedora.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add support for both cancelling and failing a build. This is tested, but
not hooked up, as we need some more architecture work before that makes
sense.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Move to requiring CGInitBuild to be called before CGImport. In the
future we could make the former optional again, but for now we want to
allow the caller to have done CGInitBuild and for composer only to do
the CGImport using the passed in build_id and token.
Also rename and document some struct fields in the metadata struct to
make them more specific to our use-case and hopefully easier to read.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Prior this commit we only had support for username/password authentication
in the koji integration. This wasn't particularly useful because this
auth type isn't used in any production instance.
This commit adds the support for GSSAPI/Kerberos authentication.
The implementation uses kerby library which is very lightweight wrapper
around C gssapi library.
Also, the koji unit test and the run-koji-container script were modified
so the GSSAPI auth is fully tested.
In the near future, we will need to communicate with Koji using HTTPS.
This will surely bring the need for ignoring bad certificates/providing
our own self-signed ones. Thus, this commit prepares the Koji integration
by adding a way to accept a custom http transport which can be used to
customize the TLS settings.
Previously, Koji instance could be both logged-in and not logged-in.
This change disallows it: Now, the Koji instance is created by calling
koji.Login, so it must be always logged-in. This change should lead to more
robust code.
run-koji-container has now two actions: start and stop:
- ./run-koji-container.sh start
- ./run-koji-container.sh stop
The start action starts all containers. When it exits, all containers are
started and running in the background. To stop and removethem, use the stop
action.
This change is needed so we're able to easily use this script also in the CI
environment.
The setup should be container engine agnostic. This changes allows this script
to be run on systems which prefer docker over podman (e.g. Github Actions).
- inside RunJob() there is a deferred function which will remove
the entire temporary directory in which images are created, including
the streamOptimized file
- inside testBootUsingVMware(), which wants to use this function,
there is already a deferred function which removes the converted
image
This test is not run anywhere because it was surpassed by image tests with
azure boot type which perform more than just uploading a randomly generated
file to Azure. Let's delete dead code.
Applying a `Name` tag to the snapshot makes the name of the image appear
in the snapshot listing in EC2's console. It also makes it easier to
remove a snapshot after deregistering the AMI.
Signed-off-by: Major Hayden <major@redhat.com>
The AWS API always uses pointers to strings everywhere, rather than
plain strings. The log messages got this wrong and printed the pointers
rather than dereferencing them.
Don't require users to do a memory dump to get at the AMIs, but instead
print the actual strings. Also change to %s from %v to let the linter
warn us in the future. %s accepts only strings, whereas %v prints
whatever you throw at it.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When ImportSnapshot fails, it goes into the deleted state. However, the code
didn't consider this state as a failure. This commit fixes that.
Fixes#706
The AWS upload process takes time and has multiple steps that can fail
or take an extended time to run.
Add some basic logging to various parts of the AWS upload process to
assist with troubleshooting in CI and production.
Signed-off-by: Major Hayden <major@redhat.com>
Specifying the image format in the `DiskContainers` part of the snapshot
import process sometimes causes AWS to fail the import due to formatting
issues. However, AWS seems to do a decent job at determining the image
format on its own without specifying the format explicitly.
Signed-off-by: Major Hayden <major@redhat.com>
Importing an image into an EBS snapshot can sometimes take a long time.
The customer experience for a timeout is not great.
Allow the import to continue running for as long as needed without
timing out. This allows a customer to monitor the process of their
import until it has a success or failure status.
Fixesosbuild/osbuild-composer#632.
Signed-off-by: Major Hayden <major@redhat.com>
This does not yet actually upload the image, and it only supports empty
images. You need to place a an empty file named <filename>, with a valid
extension (e.g., .qcow2) in /mnt/koji/work/<directory>/.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Prior this commit the ami image type produced raw.xz images. This was bad for
two reasons:
- The upload was broken because AWS doesn't support tar.xz format
- XZ compression is terribly slow
This commit changes the format to vhdx, which is supported by AWS and also
quite quick. See https://github.com/osbuild/osbuild-composer/issues/257
why vhdx was chosen.
Fixes#257
The old aws unit test is not very useful now, because there's the new
integration test which covers a lot more. Therefore, this commit
removes it.
Fixes#512
us-east-1 seems to very slow these days, some imports can take up to
15 minutes. This commit raises the number of attempts before we give up.
Previously, we did 40 attempts, each delayed by 15 seconds, making the total
timeout equal to 10 minutes. Now we do 80 attempts with the same delay,
making the total timeout 20 minutes.
After the snapshot is imported to EC2, there's no point in having the s3
object anymore. If the s3 object is not deleted, the user will be charged
for it periodically, while the object doesn't provide any value.
