Commit graph

64 commits

Author SHA1 Message Date
Christian Kellner
0d9dc2b72d builder: break list comprehension over two lines
Small syntactic change to make it easier to read.
2020-11-13 11:06:10 +01:00
Christian Kellner
cedb982593 builder: support compose status json serialization
Add a '.as_dict()' method to ComposeStatus, which can be used to
serialize the object as JSON or properly log it. Indeed use it
in the debug log to avoid a useless message that contains:
<_koji_plugin__osbuild.ComposeStatus object at 0x7fb80172b820>
2020-11-13 11:06:10 +01:00
Christian Kellner
5d309bd86d plugin/builder: more generic upload_json method
Rename "upload_meta_data" to "upload_json" and re-use that method
from "attach_logs", removing the duplicated code there.
2020-11-13 11:06:10 +01:00
Christian Kellner
e81ac2d946 builder: fetch and attach build logs
Use the new log API introduced in osbuild-composer >= 24, to fetch
the logs for the compose and attach them to the task. It is a non-
fatal error if fetching the logs fails; in that case a warning is
emitted. NB: logs are attached no matter the compose result.
Logs are per image-request, the content is JSON, but otherwise not
further specified.
2020-11-13 11:06:10 +01:00
Christian Kellner
9e10eb58eb builder: keep order of requested architectures
Preserve the order of architectures in the image requests inside
the compose requests, i.e. don't transform 'arches' into a set,
which has random order. It is not that anyone should really
depend on it, but there is also no need to mess with the order,
potentially making it harder for humans to match requested arches
with compose request content and logs.
2020-11-13 11:06:10 +01:00
Tomas Kopecek
9d6d278ffb builder: invert missing arch support logic
The builder plugin checks that all of the requested architectures
are indeed supported, which is determined via the build tag. It
does that by constructing two sets, the requested architectures
and the supported architectures, and then constructing the set
of (asymmetric) differences between the one and the other. This
difference was meant to be "requested" - "supported", which then
will contain architectures that have been requested but are not
supported, or be empty in case all requested architectures are
indeed supported (the good case). However, previously the diff
was done the other way around ("supported" - "requested"), which
will then return a set of architectures that are supported but
were not requested, which is not what we want to check for.

Invert that the argument of the difference to indeed end up with
"requested" - "supported", which results in the correct check.
2020-11-03 10:54:04 +01:00
Christian Kellner
7ecf7447e4 builder: use 'server' instead of 'url' for config
Mostly to be more in line with how things are called in all other
koji configuration files.
2020-09-26 11:37:02 +01:00
Christian Kellner
b9c3a3d2f9 all: sync with composer changes
Upstream composer has introduce a few changes that we need to
adapt for:

 - the koji composer API is now exposed on the standard https
   port (443). Thus koji hub and web need to move to a different
   pair: 8080 (http) and 4343 (https). Change the scripts and
   tests for that

 - the koji API gained a prefix 'api/composer-koji/v1/'. Change
   client and unit tests to use that prefix. Use urljoin to
   create new APIs

 - composer configuration format (osbuild-composer.toml) has
   changed and now also includes configuration for the CA
   and allowed domains

 - update the composer RPM repositories to the commit for the
   21 upstream release.
2020-09-25 22:33:31 +01:00
Christian Kellner
20abf9a142 builder: tag the build on success
When a compose / build was successful, tag it with the destination
tag. This will create a sub-task and wait for it. The individual
parameter are the same used for standard "image" koji tasks.
2020-09-25 15:26:36 +01:00
Christian Kellner
d300c96f1f builder: save the compose request
After creating the compose request object, save it as meta-data of
the task. This should come in especially handy when a compose gets
rejected by osbuild-composer.
2020-09-21 13:01:04 +02:00
Christian Kellner
d0499f4ef4 builder: create_compose takes a ComposeRequest obj
Instead of having the individual components for a compose request as
argument, make create_compose just take a ComposeRequest object.
This will pave the way to save the compose request as meta-data for
the task, which can be useful if something goes wrong, e.g. creating
the request.
2020-09-21 13:01:04 +02:00
Christian Kellner
3342f88f3b builder: properly decode the http response body
What is returned by content.decode() is a binary string and needs
to be properly decoded before we use it, especially in the error
messages.
2020-09-19 22:41:26 +02:00
Christian Kellner
807a4937ee builder: remove status member from result dict
Now that an error is thrown for failing builds, the status will
always be "success" and thus is redundant.
2020-09-19 22:41:26 +02:00
Christian Kellner
f2f34a3b76 builder: raise an exception on compose failures
Instead of a normal task return with a result dictionary, raise an
exception when the compose failed.
2020-09-19 22:41:26 +02:00
Christian Kellner
2dbeab79a5 plugin/builder: more logging
Add more debug logging and always print the compose result status.
2020-09-19 22:41:26 +02:00
Christian Kellner
a1b03e2b80 builder: better and unified result value
When returning the result from the task handler function, return
a more complete and structured object in all cases. The name of
the sub-object is named after the service the item belongs to;
specifically composer is used for what belongs to (osbuild)-composer.
2020-09-19 22:41:26 +02:00
Christian Kellner
e2d0f49610 plugin/builder: remove unused to_json method
Since the switch to requests, with its support for dictionary to
json conversion, this method is no longer necessary.
2020-09-19 22:41:26 +02:00
Christian Kellner
24850d3303 plugin/hub: specify type of architectures array
In the json schema used to validate the input, specify the type of
the architectures array as a string.
2020-09-17 16:33:14 +01:00
Christian Kellner
1d9612ca05 plugin: pass repos as arrays
Currently we were passing the repo information as a comma
separated string, which is fragile, since urls can contain
commas. Just transfer them as arrays of strings.
2020-09-17 16:33:14 +01:00
Christian Kellner
bae564a1ab plugin/builder: new helper to parse cert strings
The requests cert argument accepts a tuple with the key and the
cert, or one string as a path to a combination of both. Create
a new helper and use that to parse the "ssl_cert" configuration
option as well as the command line option. Change the latter
into a string.
2020-09-17 10:48:37 +02:00
Christian Kellner
9eeb810e3f plugin/builder: use global constant for cfg files
Use a global constant `DEFAULT_CONFIG_FILES` for the default
plugin configuration files. Besides giving it more visibility
it also makes it easier to test, because in the unit test,
this can easily be overwritten.
2020-09-17 10:48:37 +02:00
Christian Kellner
f2d73f018b plugin/builder: return 0 from sub-commands
In the functions that implement the sub-commands (for stand-alone
use), always explicitly return 0, so that main() will always
return an int.
2020-09-17 10:48:37 +02:00
Christian Kellner
db82105eb0 plugin/builder: TLS/SSL support (client side certs)
Add support for client side certificates via a new configuration
option: "ssl_cert", which should be a path to the certificate or
a comma separated list of paths to certificates.
Additionally, the "ssl_verify" option controls how the server
certificate is validated. This can either be a boolean ("True",
"False") or a path to a CA file.
NB: The url for the composer host must contain "https" for the
ssl/tls engine to kick in.

When used as a stand-alone koji API client, two new command
line options are provided, --cert and --ca. See their help
text for a description.
2020-09-17 10:48:37 +02:00
Christian Kellner
790462e967 plugin/builder: sync status enum with upstream
Align our status enums with the one from upstream, i.e. composer's
koji API openAPI spec as of commit 8ccbde8.
2020-09-17 10:48:37 +02:00
Christian Kellner
d0d167d2f8 plugin/hub: fix typo
It is schema, not schmea. That is not even a word.
2020-09-17 10:48:37 +02:00
Christian Kellner
d4b5cae604 plugin/builder: global constants for default urls
Use a global constant for the composer and kojib hub URLs instead
of defining them in the source twice.
2020-09-17 10:48:37 +02:00
Christian Kellner
a0c562e5c6 fixup! plugin/builder: use requests python package 2020-09-15 18:59:03 +02:00
Christian Kellner
569d53c2d9 plugin/builder: use requests python package
Use the requests python package, which is more convenient to use,
and also supports sessions, where global options, like TLS certs
can be set.
Adapt the container and the spec file to install and require
the python package.
2020-09-15 18:59:03 +02:00
Christian Kellner
c39f1d6d59 plugin/builder: properly raise exceptions
In two cases the exception was merely instantiated but not
actually thrown.
2020-09-14 10:00:56 +02:00
Christian Kellner
5ba7028340 plugin/hub: raise ParameterError for invalid input
Convert the jsonschema.exceptions.ValidationError into the more
appropriate koji.ParameterError.
2020-09-13 13:20:19 +02:00
Christian Kellner
0bf3c47113 plugin/hub: clean up imports
Remove an unused import and sort the imports properly.
Also silence the for the kojihub import.
2020-09-11 23:18:04 +01:00
Christian Kellner
899c2233c7 plugin/cli: silence pylint warning
We use `_running_in_bg` for now, as other plugins do. Silence the
warning for that.
2020-09-11 23:18:04 +01:00
Christian Kellner
01cf643516 plugin/cli: return a value in all code paths
Be consistent in all code paths.
2020-09-11 23:18:04 +01:00
Christian Kellner
4eeb998a60 plugin/cli: provide a bit documentation
Better describe the plugin in the file level doc-string.
2020-09-11 23:18:04 +01:00
Christian Kellner
e3ce5f7c7b plugin/builder: silence some pylint warnings
The 'koji' parameter for the compose request is called that way to
follow the OpenAPI spec, so silence the "redefined-outer-name"
warning.

The handler function needs have the actual task arguments and thus
on purpose deviates from the argument list of the base class'
handler method. So silence that.
2020-09-11 23:18:04 +01:00
Christian Kellner
6f1d685d9b plugin/builder: move argparse into main
The argparse import is only needed when the plugin is used as a
stand alone composer koji API client, therefore only include it
in main().
2020-09-11 23:18:04 +01:00
Christian Kellner
f7794daa13 plugin/builder: fix indent in cmd line arguments 2020-09-11 23:18:04 +01:00
Christian Kellner
1715558c8f plugin/builder: add more documentation
Describe the plugin itself and how it can be used as a stand-alone
executable.
2020-09-11 23:18:04 +01:00
Christian Kellner
20831a99bb plugin/builder: fix BuildError msg to include name
Fix the error message string to be a proper f-string in the
OSBuildImage.arches_for_config method, so it actually includes
the name of the tag.
2020-09-11 23:18:04 +01:00
Christian Kellner
928d2278be plugin/cli: return task result when waiting
When waiting for the task, return the result from the `watch_task`
call, which will indicate if the task was successful. This will
then be reflected in the error code of the koji call.
2020-09-10 18:41:13 +01:00
Christian Kellner
af4e66d2b2 plugin/hub: use jsonschema to validate input
Use jsonschema to validate the input to the XMLRPC call to catch
mistakes early, i.e. before creating the task.
2020-09-10 18:41:13 +01:00
Christian Kellner
3dc463c8dd plugin/cli: pretty-print the options dict
Also show the options that we are using.
2020-09-10 18:41:13 +01:00
Christian Kellner
3f5de8a68d plugin/cli: eager check of target input
Check the target with name exists and has a destination tag, so
we can abort early.
2020-09-09 15:56:35 +02:00
Christian Kellner
3934382337 plugin/cli: honor global --quiet flag
If --quiet was requested globally, then don't print anything.
2020-09-09 15:51:11 +02:00
Christian Kellner
8099d79aa1 plugin/cli: support --wait, --nowait option
Can be used to explicitly opt in or out of waiting for the task to
finish. The help text is taken from the standard koji client.
2020-09-09 15:28:36 +02:00
Christian Kellner
79387f1f2c plugins/cli: use upstream description help text
For --repo and --release, use the help text from koji build-image.
2020-09-09 15:07:36 +02:00
Christian Kellner
81aebbc681 plugin/cli: use koji_cli.lib.OptionParser
Use koji's OptionParser instead of argparse to be better aligned
with the formatting of standard koji options.
2020-09-09 15:06:53 +02:00
Christian Kellner
dda203419c plugin/osbuild: adapt for composer API changes
The task id can and must now be passed to composer via the `koji`
object in the `ComposeRequest`. Do that.
2020-09-09 14:12:25 +02:00
Christian Kellner
49a6974e18 plugins/cli: initial version of the cli plugin
Re-uses code from compose.py. Needs to be installed into
  /usr/lib/python3.$/site-packages/koji_cli_plugins
Yes indeed.
2020-09-08 17:46:21 +02:00
Christian Kellner
8fb1342631 Make image_type and distro required arguments
Distro, in composer terms "distribution", and "image_type" are
required for composer, so make that explicit everywhere in the
code.
2020-09-08 16:13:48 +02:00