Move the last remaining test into the correct subdir. With this done,
all our tests run in one of the 3 groups:
* `make test-src`
Run tests against the source-code, including linters.
* `make test-mod`
Run unit-tests on the individual python modules. This needs no
special permissions (unless noted in each test) or runtime
environments. It is meant to be fast and easy to run in all
circumstances.
* `make test-run`
Run tests that execute the osbuild pipeline. This requires
superuser privileges and will likely take a while. Furthermore,
this might produce large artifacts.
Move the stage-tests over to the new test-infrastructure. This moves
the test invocation into `./test/run/test_stages.py`, so it is invoked
as part of the runtime-tests. Secondly, the test-data is stored in
./test/data/stages/ so the path is relative to
TestBase.locate_test_data().
While at it, this also drops the dynamic class modifications and instead
uses subTest(). This simplifies the code quite a bit and avoids
dynamically creating python code.
Move the `test_osbuild.py` test into the module-test directory. This
test contains just a bunch of basic functionality tests for a selection
of osbuild modules. Hence, it can be run together with the other module
tests.
Move `test_objectstore` into the module-level tests. This allows us to
run it as part of `make test-module.
Make sure to properly guard it as root-only module.
This moves the `sources_tests` into ./test/data/sources/ and makes the
entire test use `locate_test_data()` to get relative paths for their
accesses.
This further improves our test cases to support running from
installments rather than local checkouts. We need access to ./test/data
guarded, so we can install packages and still have the tests access the
correct paths.
This also adjusts the HTTP-Server we use in the test to serve data
relative to a path it is handed. I now chose `./test/data`, which will
easily allow us to re-use the same HTTP-Server in the future for other
tests that require it.
We want to extend our base-class to support extensions to
unittest.TestCase, so make sure we inherit from it.
Adjust all callers to no longer inherit from TestCase, since this is now
done automatically by TestBase.
Move the `tree-diff` tool into ./tools, which is our new place for tools
used by the test-suite or during development.
The only hard-coded user is the TestBase, so fix its path to the tool
so the test-suite will continue to find it.
Add a new trivial runtime-test which simply runs a no-op pipeline. This
is a fast, trivial test that simply verifies osbuild is properly setup
and accessible.
Remove the explicit no-op test from the CI, now that the test-suite has
it as well.
Recently downloading rpms from the fau.de mirror is failing quite
often with timeouts. Replace its only usage in the rpm stages
test with the kernel.org mirror.
We currently run pylint on all files we find in the checkout. This is
particularly annoying when we have osbuild-stores in the checkout, which
then contain lots of python files in their stored trees.
Change the pylint test to use `git ls-tree` to find all files in the
index and then only run pylint against the subset that we are interested
in.
Convert our custom code to list modules to the new ModuleInfo
method list_modules_for_class that does the same thing. This
is then indeed also testing that new function.
This was superseded by test_osbuild's test_moduleinfo. It also
seems to be non-functional do to assuming `properties` in all
the STAGE_OPTS. Removing this.
The are converging on a nomenclature where the sum of Stages,
Assemblers, Sources (and future entities like those) together
are called 'Modules'.
Thus rename StageInfo to ModuleInfo and the corresponding
variables and methods.
Using `[]` as default value for arguments makes `pylint` complain. The
reason is that it creates an array statically at the time the function
is parsed, rather than dynamically on invocation of the function. This
means, when you append to this array, you change the global instance and
every further invocation of that function works on this modified array.
While our use-cases are safe, this is indeed a common pitfall. Lets
avoid using this and resort to `None` instead.
This silences a lot of warnings from pylint about "dangerous use of []".
We use comments in all other tests, rather than doc-strings. Convert the
os-release test to do the same. If we wanted doc-strings, we can convert
all tests over. This commit just tries to keep the tests in-sync.
Note that doc-strings cause `unittest` to print the doc-strings to
stdout during test-execution, making it overly verbose (especially for
multiline docs). By converting it to comments, this behavior is
suppressed.
This adds F32 manifests in ./test/data/. To avoid magically deducing the
package list out of the void, this adds a ManifestPreProcessor (MPP)
called `./tools/mpp-depsolve.py`. What this does is it takes a manifest
on stdin, modifies it, and produces a manifest on stdout.
The `mpp-depsolve.py` preprocessor takes a manifest and modifies all the
`org.osbuild.rpm` stages. It parses a new option to that stage called
`mpp-depsolve`, which contains a package-list, a repo-list, and dnf
metadata. It then drops this `mpp-depsolve` option (since it would be an
invalid manifest otherwise), depsolves the packages, inserts a proper
"packages" option as well as appends the correct paths to the sources
entry.
With this in place, this adds `mpp-f32-base.json` and
`mpp-f32-build.json` in ./test/data/manifests/. These will then be used
as base F32 manifests for our test-suite.
Lastly, this adds `./test/data/README.md` as a place to document the
files we place in `./test/data/`, since most of the files do not allow
for comments.
The `tree-diff` tool currently requires access to our local checkout,
since we do not install the tool. Provide accessors in `TestBase` so we
do not hard-code the path everywhere.
Add a new OSBuild class to `./test/test.py`. This class is an extension
of `./test/osbuildtest.py`, but no longer requires the `output_id` and
`tree_id` identifiers of osbuild.
Furthermore, this new executor uses context-managers to make sure any
temporary object is only accessed for a contained time-frame.
The idea is that source can themselves spawn other modules, esp.
new secrets modules. For this they need to know the library dir,
aka 'libdir' throughout the osbuild source. Therefore change the
SourceServer to directly get the library directory instead of
just the sub-directory to the sources. Then pass the library
directory to via the JSON API to the source.
Adjust all usage of the SourceServer, including the tests.
This changes the `modprobe nbd` invocation to be non-fatal on failure,
since it might very well fail on reasonable setups. `modprobe` fails if
it cannot find a module in `/lib/modules`, even if it could reasonably
well figure out whether a module is already loaded. The reason is that
it needs the metadata from the module file to find the required modules
parameters.
If you have `nbd` already loaded but not mapped in `/lib/modules`, the
current call will cause test failures, even though the test would run
smoothly.
Fix this by never requiring `modprobe nbd` to succeed, but instead rely
on the tests failing if accessing `nbd` fails.
This drops the directory './test/testing-rpms'. The directory was
introduced in:
commit d975effc42
Author: Martin Sehnoutka <sehnoutka.martin@gmail.com>
Date: Thu Jul 25 11:12:27 2019 +0200
improve vagrant test and its documentation
It used to be the automatic target directory to store rpms created via
`make copy-rpms-to-test`. This target no longer exists. It was dropped
in:
commit 59b7b545b2
Author: Lars Karlitski <lars@karlitski.net>
Date: Fri Mar 6 11:07:52 2020 +0100
Makefile: remove vagrant rules
Introduce a third test-group called `src` alongside `mod` and `run.
This will contain tests that run against the source code of osbuild.
This initial commit introduces `test/src/test_pylint.py` which will run
the python linter against all our sources.
Use the new `locate_test_data()` helper to get access to test-data.
Guard the test with `have_test_data()` to skip it in case test-data
access is not available.
Use the `can_modify_immutable()` helper from the TestBase parent class
so we do not duplicate the code in multiple places. Similarly, make use
of the `have_rpm_ostree()` helper.
Add a new base class called `TestBase` to our test-suite. This allows
sharing common code between our tests without requiring them to import
each other. Furthermore, it paves the way towards executing all our
tests as part of the `unittest` framework, including pylint and others.
For now, this adds the following features to `TestBase`:
* Common test-guards that are shared between our tests, like
`can_modify_immutable()` or `have_rpm_ostree()`.
* Accessors to the test-checkout. This is `have_test_checkout()` to
check whether the running test has a repository checkout, and
`locate_test_checkout()` to get a path to the repository checkout.
This will allow us to put pylint and friends into the unittest
framework, guard them properly, and still allow running the tests
from a global install which might not have access to a checkout.
For now, we always assume we run from a checkout.
* Accessors to test-data. If we start installing tests as a module
into the system, we cannot bundle test-data together with code.
Therefore, two accessors `have_test_data()` and `locate_test_data()`
are implemented to guard access to test data. If a checkout is
available, it will be used to locate test-data.
In the future, we want to be able to pass a separate path to the
test-data, thus allowing us to install tests into a system.
Extract the `suppress_oserror()` function from the ObjectManager and
make it available as utility for other code as well.
This also adds a bunch of tests that verify it works as expected.
This changes `-chardev stdio` to `-chardev file` and uses a temporary
file to communicate with QEMU.
This fixes an issue where `-chardev stdio` hangs if `STDIN` is not a
TTY. I could not figure out how to make it work without a TTY, and it
does not print any meaningful diagnostics. Problem is, in CI and other
automated runners, we do not necessarily have a TTY as STDIN.
This just switches to a temporary file, which seems to work under all
circumstances.
Up until now it was verified that loading the `STAGE_OPTS` data
works, i.e. that it contains valid JSON, but the schema itself
was not verified. Use the new `Schema.check` method to do that.
In the "test_tar" test the compression can be None, which in turn
will be serialized as "compression: null" in the options dict.
However, this is not a valid option according to the schema. The
schema could be adapted to allow for this but it is probably
better to just omit empty or null values.
Add checks for the new validation API, specifically check that
errors in the main manifest, the stages and stage options,
assembler and assembler options. Also ensure that there are no
duplicated errors due to pipeline nesting (e.g. build pipelines).
