Commit graph

373 commits

Author SHA1 Message Date
Christian Kellner
20abf9a142 builder: tag the build on success
When a compose / build was successful, tag it with the destination
tag. This will create a sub-task and wait for it. The individual
parameter are the same used for standard "image" koji tasks.
2020-09-25 15:26:36 +01:00
Christian Kellner
f39f7ef4b0 test/builder: mock session.host for tagging
Create a MockHost class that mocks a small subset of the HostExport
koji class, i.e. the builder specific XML RCP interface. All the
methods needed for tagging a build, including waiting for tasks,
are mocked. MockHost will keep a list of tagged builds, to check
in the unit test functions.
2020-09-25 15:26:36 +01:00
Christian Kellner
65bc71c9d1 make-tags.sh: add packages to tag
Add the "Fedora-Cloud" and "RHEL-Cloud" package to the f32-candidate
tag, so that builds can be tagged properly.
2020-09-25 15:26:36 +01:00
Christian Kellner
8274b0250e compose.py: remove, unused
Replaced by the builder plugin, in standalone use.
2020-09-25 15:26:36 +01:00
Christian Kellner
aa7df2efc5 test/builder: assert the compose request is saved
Assert, via the new UploadTracker, that the "compose-request.json"
is always saved, especially in the case where composer refuses the
compose (via bad request).
2020-09-21 13:01:04 +02:00
Christian Kellner
d300c96f1f builder: save the compose request
After creating the compose request object, save it as meta-data of
the task. This should come in especially handy when a compose gets
rejected by osbuild-composer.
2020-09-21 13:01:04 +02:00
Christian Kellner
d0499f4ef4 builder: create_compose takes a ComposeRequest obj
Instead of having the individual components for a compose request as
argument, make create_compose just take a ComposeRequest object.
This will pave the way to save the compose request as meta-data for
the task, which can be useful if something goes wrong, e.g. creating
the request.
2020-09-21 13:01:04 +02:00
Christian Kellner
521bee4700 test/builder: mock koji file uploads
Replace the "fast_incremental_upload" of the plugin with a custom
one that will keeps track of all uploaded files through it. Can
be used to ensure that certain uploads happen.
NB: this assumes that "fast_incremental_upload" was or will be
directly imported into the plugin namespace.
2020-09-21 13:01:04 +02:00
Christian Kellner
3342f88f3b builder: properly decode the http response body
What is returned by content.decode() is a binary string and needs
to be properly decoded before we use it, especially in the error
messages.
2020-09-19 22:41:26 +02:00
Christian Kellner
807a4937ee builder: remove status member from result dict
Now that an error is thrown for failing builds, the status will
always be "success" and thus is redundant.
2020-09-19 22:41:26 +02:00
Christian Kellner
825584966e test/builder: test failing composes
Check that an exception is thrown on compose failures, i.e. when
the compose status indicates that the compose has failed.
2020-09-19 22:41:26 +02:00
Christian Kellner
f2f34a3b76 builder: raise an exception on compose failures
Instead of a normal task return with a result dictionary, raise an
exception when the compose failed.
2020-09-19 22:41:26 +02:00
Christian Kellner
c70ec02fba test/builder: ability to set compose status
Can be used to mock a failing build, but setting the overall
status to "failure".
2020-09-19 22:41:26 +02:00
Christian Kellner
2dbeab79a5 plugin/builder: more logging
Add more debug logging and always print the compose result status.
2020-09-19 22:41:26 +02:00
Christian Kellner
a1b03e2b80 builder: better and unified result value
When returning the result from the task handler function, return
a more complete and structured object in all cases. The name of
the sub-object is named after the service the item belongs to;
specifically composer is used for what belongs to (osbuild)-composer.
2020-09-19 22:41:26 +02:00
Christian Kellner
e2d0f49610 plugin/builder: remove unused to_json method
Since the switch to requests, with its support for dictionary to
json conversion, this method is no longer necessary.
2020-09-19 22:41:26 +02:00
Christian Kellner
f930691d86 run-tests: remove the container after use
Pass `--rm` to automatically remove the container after it exited.
2020-09-17 19:53:06 +02:00
Christian Kellner
5c6605b974 test: shellcheck support for run-tests.sh
The severity is set to "warning" by default, but can be controlled
via the SHELLCHECK_SEVERITY env variable.
2020-09-17 19:53:06 +02:00
Christian Kellner
f4c850d7d0 ci: enable ShellCheck warnings
Enable ShellCheck at the warning level.
2020-09-17 19:53:06 +02:00
Christian Kellner
b8f991e386 ci: fix typo in job name
It is linting not "liniting". So many iiiis.
2020-09-17 19:53:06 +02:00
Christian Kellner
2b7741519b run-tests: double quote $(pwd)
This could indeed also be anything.
2020-09-17 19:53:06 +02:00
Christian Kellner
2e24a5506f test: use "bash" for entrypoint.sh
Also double quote GITHUB_WORKSPACE because that could indeed by
anything.
2020-09-17 19:53:06 +02:00
Christian Kellner
b44ceb19ef test/integration: remove unused variable
Don't need ARCH.
2020-09-17 19:53:06 +02:00
Christian Kellner
2bdfcfe0ae test/make-certs: remove unused variable
ALT_NAMES is now defined in test/data/composer.ssl.conf.
2020-09-17 19:53:06 +02:00
Christian Kellner
24850d3303 plugin/hub: specify type of architectures array
In the json schema used to validate the input, specify the type of
the architectures array as a string.
2020-09-17 16:33:14 +01:00
Christian Kellner
9ad7faf298 test/hub: check optional arguments
Include the release and repo arguments in the basic compose test,
so that if their type was to change, we were to catch it here.
2020-09-17 16:33:14 +01:00
Christian Kellner
1d9612ca05 plugin: pass repos as arrays
Currently we were passing the repo information as a comma
separated string, which is fragile, since urls can contain
commas. Just transfer them as arrays of strings.
2020-09-17 16:33:14 +01:00
Christian Kellner
c6f1fa0a6f ci: configure coverage thresholds
Taken from osbuild-composer. What is good for them is good for us.
2020-09-17 16:33:14 +01:00
Christian Kellner
78d9bea92d container: use latest brew staging repos
This is what is deployed in the brew staging area. Be as close as
possible to that.
2020-09-17 15:40:33 +02:00
Christian Kellner
5ab3acd7b1 container+test: match container base to host
On Fedora use containers based on Fedora, on RHEL use containers
based on RHEL, so we test the correct integration of the plugins
into the corresponding distribution.
2020-09-17 10:48:37 +02:00
Christian Kellner
2aae1242f8 container: add RHEL container variants
Add a RHEL version of container/*/Dockerfile. Uses the internal
repos.
2020-09-17 10:48:37 +02:00
Christian Kellner
0df038c8c5 container: add internal repos
One for the latest koji and the other one for RHEL. They are only
accessible from within the internal network.
2020-09-17 10:48:37 +02:00
Christian Kellner
14003137fc run-builder: supply --rm only for "fg"
When run in the background, remove the container in "builder_stop",
this should make it easier to print logs between starting and
stopping the container.
2020-09-17 10:48:37 +02:00
Christian Kellner
5b237ffe84 run-builder: set sensible bash flags
Do the usual "set -euo pipefail".
2020-09-17 10:48:37 +02:00
Christian Kellner
955c531000 test/builder: check --cert command line option
Check that passing `--cert` as a command line option works, for
this we use real working certificates because this will actually
be parsed by requests.
2020-09-17 10:48:37 +02:00
Christian Kellner
15f645e6fa test/builder: more ssl_cert config check
Check we can handle a single cert properly, i.e. pass it as a
plain string to requests. Also check that if three components
are specified, an ValueError is thrown.
2020-09-17 10:48:37 +02:00
Christian Kellner
94f7b35967 test/builder: check ssl_verify can be a string
The ssl_verify config option can be a string, i.e. a path to the
certificate authority for the server side certificate. Check
that we handle that properly.
2020-09-17 10:48:37 +02:00
Christian Kellner
bae564a1ab plugin/builder: new helper to parse cert strings
The requests cert argument accepts a tuple with the key and the
cert, or one string as a path to a combination of both. Create
a new helper and use that to parse the "ssl_cert" configuration
option as well as the command line option. Change the latter
into a string.
2020-09-17 10:48:37 +02:00
Christian Kellner
c2d73b431a test/builder: check config parsing
Check that options defined in the config file will be reflected
in the plugin. test all known options.
2020-09-17 10:48:37 +02:00
Christian Kellner
9eeb810e3f plugin/builder: use global constant for cfg files
Use a global constant `DEFAULT_CONFIG_FILES` for the default
plugin configuration files. Besides giving it more visibility
it also makes it easier to test, because in the unit test,
this can easily be overwritten.
2020-09-17 10:48:37 +02:00
Christian Kellner
cde65d1f74 CI: run rhel8 integration test in PSI
We need access to internal repos, only available in PSI.
2020-09-17 10:48:37 +02:00
Christian Kellner
78cef84f67 test/builder: check repo configuration
In the test_compose_success check, after the build is "done",
check the repositories in compose request match with what
we expect.
2020-09-17 10:48:37 +02:00
Christian Kellner
41ee1f1708 test/integration: build image for current distro
Add a separate unit test to do a compose for a RHEL based image,
but then restrict the execution of test_compose_{disto} to the
distribution the test is running on, since currently cross-
distribution builds can be problematic.
2020-09-17 10:48:37 +02:00
Christian Kellner
565cecc07f test/integration: wait for compose result
In the "test_compose" integration test, explicitly wait for the
compose result, instead of being ok with having created the
request for it.
2020-09-17 10:48:37 +02:00
Christian Kellner
cf56696d90 test/builder: check stand-alone usage
Check the basic usage of the plugin as a stand-alone client for
the osbuild-composer API.
2020-09-17 10:48:37 +02:00
Christian Kellner
f2d73f018b plugin/builder: return 0 from sub-commands
In the functions that implement the sub-commands (for stand-alone
use), always explicitly return 0, so that main() will always
return an int.
2020-09-17 10:48:37 +02:00
Christian Kellner
d4f0b169a0 composer: update to master/8ccbde8
This is right when PR-952 landed, which is what we are updating
from. This will use SSL certification and also re-enable auto
kerberos authorization via config files for composer.
2020-09-17 10:48:37 +02:00
Christian Kellner
a660b39e00 test/integration: use copy-cred.sh
Instead of manually copying credentials around, use the new
copy-creds.sh script. Also don't manually initialize kerberos,
because that is now done automatically by the worker and the
builder.
2020-09-17 10:48:37 +02:00
Christian Kellner
c782f7f350 test/integration: make composer certificates
Use test/make-certs.sh to generate osbuild composer ssl/tls certs.
2020-09-17 10:48:37 +02:00
Christian Kellner
660713a39b container/builder: configure ssl for plugin
Configure the ssl/tls cert verification. Use make-certs.sh and
copy-creds.sh from test/ to put all the ducks in the row for
this to work.
2020-09-17 10:48:37 +02:00