Align the makefile targets with the test-targets (`module` -> `mod`,
etc.). This way, we have consistent names everywhere.
While at it, move the `make test-run` invocation closer to the others.
Move the `test_osbuild.py` test into the module-test directory. This
test contains just a bunch of basic functionality tests for a selection
of osbuild modules. Hence, it can be run together with the other module
tests.
Move `test_objectstore` into the module-level tests. This allows us to
run it as part of `make test-module.
Make sure to properly guard it as root-only module.
Run the MPP tools in the CI and verify the committed test-data did not
change and is up-to-date.
This runs `make test-data` and then simply uses `git diff --exit-code`
to trigger a CI failure if there are any differences in ./test/data.
We are now building RPMs via mock in Jenkins, so we don't need these RPM
build jobs in GitHub Workflows.
Signed-off-by: Major Hayden <major@redhat.com>
Work around the quay.io issues by using the standard Fedora containers.
Also, make most of the dnf operations a little quieter to make it easier
to find problems.
Signed-off-by: Major Hayden <major@redhat.com>
Introduce a third test-group called `src` alongside `mod` and `run.
This will contain tests that run against the source code of osbuild.
This initial commit introduces `test/src/test_pylint.py` which will run
the python linter against all our sources.
This new module contains utilities that help to introspect parts
that constitute the inner parts of osbuild, i.e. its stages
and assembler (which is also considered a type of stage in
this context). It contains the `StageInfo` class that can that
contains meta-information about the individual stage, such as
a short information (`info`), a longer description (`desc`) and
its JSON schema. A new Schema class represents schema data and
has a `validation` method that can be used to validate that json
data conforms to said schema.
A `Index` class can be used to obtain `StageInfo` and `Schema`
for entities identified via `klass` and `name`.
A top level `validate` method is introduced that can validate
manifest data.
Internally it uses the `jsonschema` package so add that as a
requirement and Install this dependency in the CI.
Move the 'test_util_selinux.py' test into the module-unittest
subdirectory.
Drop the '__main__' hookup while at it. `python -m unittest --help`
explains how you can run individual tests.
Add convenience targets to `Makefile` which can run common sets of
tests. For now, add a target for pylint, module-unittests,
pipeline-runtime-tests, as well as all tests.
Currently, it is quite cumbersome to run a reasonable test-setup
locally. Pylint invokation is rather complex, the unittests and runtime
tests in ./test are mixed, and not all tests in ./test can necessarily
be run from a development system.
This commit prepares for a simpler setup:
* Add `make test-pylint` to run pylint as it is run by CI.
* Add `make test-module` to run all module-unittests. This is meant to
be fast (preferably close to instant) and easy to run during
development to do a short check whether there are obvious typos or
other errors in local changes.
If we can keep these tests to machine-local requirements, if we
avoid any sleeps or heavy computations, then this will remain a
convenient test-suite to run locally without having to wait for
30min. In other words: We should be able to keep this under 10s (and
for the long term under 1min) easily.
* Add `make test-runtime` to run all osbuild pipeline executions. This
is not meant to be fast, but thorough. This will require external
sources (preferably limited to a suitable container image with
everything embedded). This will very likely not be run during
development, but rather by the CI.
* Add `make test-all` to run all tests. Very handy for shy people when
the chance of embarrassing copy-paste mistakes is too high to push
publicly.
Additionally to these new targets, this PR introduces 2 new directories
in ./test: ./test/mod/ and ./test/run/
These are meant as equivalent to `test-module` and `test-runtime`. The
reason is that preferably we stick to the auto-discovery of `unittest`
to enumerate tests, rather than enrolling our own or having to enumerate
them explicitly somewhere.
However, we need some way to tell `unittest` which test belongs into
which group. The easiest setup is likely to just use sub-directories.
Note that `test-all` picks all tests independently of where they are
put, even if they are in further different sub-modules under ./test.
For now, no tests are moved into the new directories. I expect this to
take a bit, since there are several out-standing PRs that modify ./test.
I intend to do the final move once we agreed on this and we synchronized
our test-modifications.
Make sure we always pylint all python sources. We currently skip tests
as well as osbuild/util. Fix this by always recursively looking for all
python sources and then linting them.
Now that containers are being built daily in GitHub Actions from
the container[0] repository, we can use those containers. That will
speed up the GitHub Actions for osbuild and reduce errors from
unreleased Fedora versions.
[0] https://github.com/osbuild/containers
Signed-off-by: Major Hayden <major@redhat.com>
Drop the filter on `master`. This prevents us from running the CI on
forked repositories. In particular, it prevents us from pushing
branches to github-forks and have the CI run on them. This is very nice
to have as development tool, as it allows running the real CI without
opening a PR.
This adds another job to the CI runner. This builds and tests the
documentation. The tests are currently reduced to just verifying the
respective man-pages are actually generated. This can be extended on in
the future.
Verify the copy on write semantics of `objectstore.Object`, i.e.
content will only be copied at the moment a client wants to write
to `Object`. This also checks that `Object.base` works.
Modify the CI to execute the unit tests in a privileged container
because `Object.read()` works internally by bind mounting a path.
The mount operation needs at least CAP_SYS_ADMIN and overwriting
the file permissions CAP_DAC_OVERRIDE.
A single workflow allows us to re-use artifacts throughout all jobs.
Also, we can fail the tests early if there's a linting issue before
we spend time waiting for RPMs to build.
Signed-off-by: Major Hayden <major@redhat.com>
