Commit graph

409 commits

Author SHA1 Message Date
Christian Kellner
50a0b09b5d README.md: fix links to example configs
Remove extra "`" chars that broke the link.
2020-11-03 11:31:29 +01:00
Christian Kellner
0b4a13e8c7 test/builder: more architectures in build tag
Add more supported build architectures to the standard build tag.
This will check the logic for (un-)supported build architectures
in the builder plugin. If we only have one arch, which is the
requested arch, it will not catch logic errors, that invert the
supported vs requested architectures.
2020-11-03 10:54:04 +01:00
Tomas Kopecek
9d6d278ffb builder: invert missing arch support logic
The builder plugin checks that all of the requested architectures
are indeed supported, which is determined via the build tag. It
does that by constructing two sets, the requested architectures
and the supported architectures, and then constructing the set
of (asymmetric) differences between the one and the other. This
difference was meant to be "requested" - "supported", which then
will contain architectures that have been requested but are not
supported, or be empty in case all requested architectures are
indeed supported (the good case). However, previously the diff
was done the other way around ("supported" - "requested"), which
will then return a set of architectures that are supported but
were not requested, which is not what we want to check for.

Invert that the argument of the difference to indeed end up with
"requested" - "supported", which results in the correct check.
2020-11-03 10:54:04 +01:00
Christian Kellner
12a7793900 spec: require python3-jsonschema (hub plugin)
The hub plugin uses jsonschema, so it must explicitly depend on
it. The dependency is installed "by hand" in the container since
by default, the plugin is not installed via the RPM, but copied
from the source repository and thus all dependencies must be
present already.
2020-11-03 09:20:03 +01:00
Christian Kellner
8d40baedf8 test/builder: avoid using list in keyword argument
Although in that specific use case it was fine, since the list
was copied before usage, pylint in newer versions was unhappy.
2020-10-29 17:12:03 +01:00
Christian Kellner
f5564c5f1f ci: explicitly install the dnsname podman plugin
On RHEL a pre-compiled version is currently installed from the
vendor directory (see the README.md there). On Fedora it is
packaged in podman-plugins and that used to be installed
without explicitly asking for it, but that seems to have changed.
Explicitly install it on Fedora to ensure it is present.

Symptom of the dns plugin missing is that the koji hub container
will not start because it will block waiting for the postgres
container forever.
2020-10-28 21:48:44 +01:00
Christian Kellner
768c25744f NEWS.md: add news entries for release 1
More of a description of the project and existing functionality
than highlights.
2020-09-30 15:17:03 +02:00
Christian Kellner
58f23204fc README.md: Fix cli plugin sub-command in intro
It is `osbuild-image` not `osbuild-command`.
2020-09-30 15:17:03 +02:00
Christian Kellner
4a23652ed8 README.md: add installation instructions 2020-09-30 15:17:03 +02:00
Christian Kellner
8772ee5acf README.md: explain what this is about a bit more
Also some white-space changes to the source more readable.
2020-09-30 15:17:03 +02:00
Christian Kellner
8fccf0857f spec: change upstream URL to new location
The project now is located at https://github.com/gicmo/koji-osbuild
2020-09-30 15:17:03 +02:00
Christian Kellner
57347fbbb7 editorconfig: add markdown settings 2020-09-30 15:17:03 +02:00
Christian Kellner
e6f9721276 test/run-builder: use command -v instead of which
The latter is non-standard according to ShellCheck. Let's use the
standard one.
2020-09-29 21:40:25 +01:00
Christian Kellner
b1b7425ca6 test/make-tags.sh: KOJI_SERVER env variable
Introduce KOJI_SERVER that can be set outside, defaulting to
http://localhost:8080/kojihub for local testing.
2020-09-29 21:40:25 +01:00
Christian Kellner
30aa71563c test: DATA_DIR can be externally defined
Change the use of DATA_DIR across all scripts so that it will
only be set if it does not yet exist. This makes it possible
to change the location of it across all scripts at once.
2020-09-29 21:40:25 +01:00
Christian Kellner
4dd2e6d814 test: SHARE_DIR can be externally defined
Change the use of SHARE_DIR across all scripts so that it will
only be set if it does not yet exist. This makes it possible
to change the location of it across all scripts at once.
2020-09-29 21:40:25 +01:00
Christian Kellner
aa8513c20d test: move container/ into test/
Move the container directory, containing the container definitions
for all the test containers, to test/, where all the other test-
related files are located (with the exception of `Schutzbot`).
Use `test/build-container.sh` to build the container, instead of
replicating that in `test-integration.sh`.
2020-09-29 21:40:25 +01:00
Christian Kellner
b625f96c2b test: move run-builder into test/
Move the run-builder.sh helper into the `test/` sub-directory. It
is mainly needed for testing together with the other test scripts.
2020-09-29 21:40:25 +01:00
Christian Kellner
660f0f2700 test: move run-koji-container.sh to test/
Move the run-koji-container.sh helper into the `test/` sub-directory.
It is mainly needed for testing together with the other test scripts.
2020-09-29 21:40:25 +01:00
Christian Kellner
05f1ecac8e test: move make-tags.sh into test/
Move the make-tags.sh helper into the `test/` sub-directory. It is
mainly needed for testing together with the other test scripts.
2020-09-29 21:40:25 +01:00
Christian Kellner
00c2313226 test/builder: use handler factory everywhere
Convert all tests so they use the new handler factory method.
2020-09-29 21:40:25 +01:00
Christian Kellner
4325ad00b6 test/builder: use handler factory in config checks
Use the new generic handler factory method in the configuration
data checks, eliminating a lot of duplicated code. No semantic
change intended.
2020-09-29 21:40:25 +01:00
Christian Kellner
d9c0f9e316 test/builder: add handler factory helper
Add a new helper method that can be used to create an instance of
OSBuildImage. Optionally, the session and options can be passed
in as well as a custom config data, otherwise the default session
and object are used. If custom configuration data was specified,
a temporary configuration file with the config data is created
and set plugin-wide so it is picked up by the object constructor.
2020-09-29 21:40:25 +01:00
Christian Kellner
7ecf7447e4 builder: use 'server' instead of 'url' for config
Mostly to be more in line with how things are called in all other
koji configuration files.
2020-09-26 11:37:02 +01:00
Christian Kellner
1b8277c078 README.md: include sample builder configuration
To highlight how the builder plugin is configured and where to put
the configuration files.
2020-09-26 11:37:02 +01:00
Christian Kellner
3ee5966b79 HACKING.md: add Patches, Coding Style sections
Small hint where to file patches and what the coding style is.
2020-09-26 11:37:02 +01:00
Christian Kellner
01de94bc44 HACKING.md: small formatting adjustmensts
Declare code blocks to be shell, add a newline here and there.
2020-09-26 11:37:02 +01:00
Christian Kellner
b9c3a3d2f9 all: sync with composer changes
Upstream composer has introduce a few changes that we need to
adapt for:

 - the koji composer API is now exposed on the standard https
   port (443). Thus koji hub and web need to move to a different
   pair: 8080 (http) and 4343 (https). Change the scripts and
   tests for that

 - the koji API gained a prefix 'api/composer-koji/v1/'. Change
   client and unit tests to use that prefix. Use urljoin to
   create new APIs

 - composer configuration format (osbuild-composer.toml) has
   changed and now also includes configuration for the CA
   and allowed domains

 - update the composer RPM repositories to the commit for the
   21 upstream release.
2020-09-25 22:33:31 +01:00
Christian Kellner
3b9980df50 test/build-container: simple helper script
Simple helper to build the two container for the hub and the builder.
2020-09-25 22:33:31 +01:00
Christian Kellner
b2a62d4dcf HACKING.md: troubleshooting and cleanup sections
Some more help if stuff goes wrong and some cleanup instructions.
2020-09-25 22:33:31 +01:00
Christian Kellner
add4f77e68 container/hub: disable notification
We don't ever want to send out email notifications for anything
during testing.
2020-09-25 15:26:36 +01:00
Christian Kellner
ec16d4fb14 HACKING.md: move useful links from README.md
There are not of general interest, but are specific to plugin
development. Move it to HACKING.md.
2020-09-25 15:26:36 +01:00
Christian Kellner
66aebcd0d4 README.md: refer to HACKING.md for development
HACKING.md replaced the development section of the README.md.
Refer to that new document.
2020-09-25 15:26:36 +01:00
Christian Kellner
3be53c1849 HACKING.md: added with testing instructions
Add instructions how to run the unit tests as well as the full
local test environment.
2020-09-25 15:26:36 +01:00
Christian Kellner
e9e23b63e5 test/builder: ensure failed builds are not tagged
Since for a failed task, the build id is not returned, we check
that no tags exist at all (for any build).
2020-09-25 15:26:36 +01:00
Christian Kellner
fb9c6d9777 test/builder: ensure a successful build is tagged
Use the new MockHost class and its `tags` property to ensure that
a successful build is indeed tagged.
2020-09-25 15:26:36 +01:00
Christian Kellner
20abf9a142 builder: tag the build on success
When a compose / build was successful, tag it with the destination
tag. This will create a sub-task and wait for it. The individual
parameter are the same used for standard "image" koji tasks.
2020-09-25 15:26:36 +01:00
Christian Kellner
f39f7ef4b0 test/builder: mock session.host for tagging
Create a MockHost class that mocks a small subset of the HostExport
koji class, i.e. the builder specific XML RCP interface. All the
methods needed for tagging a build, including waiting for tasks,
are mocked. MockHost will keep a list of tagged builds, to check
in the unit test functions.
2020-09-25 15:26:36 +01:00
Christian Kellner
65bc71c9d1 make-tags.sh: add packages to tag
Add the "Fedora-Cloud" and "RHEL-Cloud" package to the f32-candidate
tag, so that builds can be tagged properly.
2020-09-25 15:26:36 +01:00
Christian Kellner
8274b0250e compose.py: remove, unused
Replaced by the builder plugin, in standalone use.
2020-09-25 15:26:36 +01:00
Christian Kellner
aa7df2efc5 test/builder: assert the compose request is saved
Assert, via the new UploadTracker, that the "compose-request.json"
is always saved, especially in the case where composer refuses the
compose (via bad request).
2020-09-21 13:01:04 +02:00
Christian Kellner
d300c96f1f builder: save the compose request
After creating the compose request object, save it as meta-data of
the task. This should come in especially handy when a compose gets
rejected by osbuild-composer.
2020-09-21 13:01:04 +02:00
Christian Kellner
d0499f4ef4 builder: create_compose takes a ComposeRequest obj
Instead of having the individual components for a compose request as
argument, make create_compose just take a ComposeRequest object.
This will pave the way to save the compose request as meta-data for
the task, which can be useful if something goes wrong, e.g. creating
the request.
2020-09-21 13:01:04 +02:00
Christian Kellner
521bee4700 test/builder: mock koji file uploads
Replace the "fast_incremental_upload" of the plugin with a custom
one that will keeps track of all uploaded files through it. Can
be used to ensure that certain uploads happen.
NB: this assumes that "fast_incremental_upload" was or will be
directly imported into the plugin namespace.
2020-09-21 13:01:04 +02:00
Christian Kellner
3342f88f3b builder: properly decode the http response body
What is returned by content.decode() is a binary string and needs
to be properly decoded before we use it, especially in the error
messages.
2020-09-19 22:41:26 +02:00
Christian Kellner
807a4937ee builder: remove status member from result dict
Now that an error is thrown for failing builds, the status will
always be "success" and thus is redundant.
2020-09-19 22:41:26 +02:00
Christian Kellner
825584966e test/builder: test failing composes
Check that an exception is thrown on compose failures, i.e. when
the compose status indicates that the compose has failed.
2020-09-19 22:41:26 +02:00
Christian Kellner
f2f34a3b76 builder: raise an exception on compose failures
Instead of a normal task return with a result dictionary, raise an
exception when the compose failed.
2020-09-19 22:41:26 +02:00
Christian Kellner
c70ec02fba test/builder: ability to set compose status
Can be used to mock a failing build, but setting the overall
status to "failure".
2020-09-19 22:41:26 +02:00
Christian Kellner
2dbeab79a5 plugin/builder: more logging
Add more debug logging and always print the compose result status.
2020-09-19 22:41:26 +02:00