ff8cd76def
Add convenience targets to `Makefile` which can run common sets of
tests. For now, add a target for pylint, module-unittests,
pipeline-runtime-tests, as well as all tests.
Currently, it is quite cumbersome to run a reasonable test-setup
locally. Pylint invokation is rather complex, the unittests and runtime
tests in ./test are mixed, and not all tests in ./test can necessarily
be run from a development system.
This commit prepares for a simpler setup:
* Add `make test-pylint` to run pylint as it is run by CI.
* Add `make test-module` to run all module-unittests. This is meant to
be fast (preferably close to instant) and easy to run during
development to do a short check whether there are obvious typos or
other errors in local changes.
If we can keep these tests to machine-local requirements, if we
avoid any sleeps or heavy computations, then this will remain a
convenient test-suite to run locally without having to wait for
30min. In other words: We should be able to keep this under 10s (and
for the long term under 1min) easily.
* Add `make test-runtime` to run all osbuild pipeline executions. This
is not meant to be fast, but thorough. This will require external
sources (preferably limited to a suitable container image with
everything embedded). This will very likely not be run during
development, but rather by the CI.
* Add `make test-all` to run all tests. Very handy for shy people when
the chance of embarrassing copy-paste mistakes is too high to push
publicly.
Additionally to these new targets, this PR introduces 2 new directories
in ./test: ./test/mod/ and ./test/run/
These are meant as equivalent to `test-module` and `test-runtime`. The
reason is that preferably we stick to the auto-discovery of `unittest`
to enumerate tests, rather than enrolling our own or having to enumerate
them explicitly somewhere.
However, we need some way to tell `unittest` which test belongs into
which group. The easiest setup is likely to just use sub-directories.
Note that `test-all` picks all tests independently of where they are
put, even if they are in further different sub-modules under ./test.
For now, no tests are moved into the new directories. I expect this to
take a bit, since there are several out-standing PRs that modify ./test.
I intend to do the final move once we agreed on this and we synchronized
our test-modifications.
118 lines
3 KiB
YAML
118 lines
3 KiB
YAML
name: Runtime Tests
|
|
|
|
on: [pull_request, push]
|
|
|
|
#
|
|
# We set `PYTHONUNBUFFERED` to get more immediate and ordered output from
|
|
# python programs. This should not be necessary if all relevant output used the
|
|
# unbuffered stderr, but that is not entirely under our control.
|
|
#
|
|
env:
|
|
PYTHONUNBUFFERED: 1
|
|
|
|
jobs:
|
|
runtime_tests:
|
|
name: "Runtime Pipeline Execution Tests"
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: "Clone Repository"
|
|
uses: actions/checkout@v2
|
|
- name: "Install Dependencies"
|
|
run: |
|
|
sudo apt-get update
|
|
sudo apt-get -y install \
|
|
nbd-client \
|
|
qemu-utils \
|
|
rpm \
|
|
systemd-container \
|
|
tar \
|
|
yum
|
|
- name: "Set up Python"
|
|
uses: actions/setup-python@v1
|
|
with:
|
|
python-version: 3.7
|
|
- name: "Run Pipeline Tests"
|
|
run: sudo env "PATH=$PATH" make test-runtime
|
|
|
|
noop_pipeline_tests:
|
|
name: "Noop-Pipeline Tests"
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: "Clone Repository"
|
|
uses: actions/checkout@v2
|
|
- name: "Install Dependencies"
|
|
run: |
|
|
sudo apt-get update
|
|
sudo apt-get -y install \
|
|
systemd-container
|
|
- name: "Set up Python"
|
|
uses: actions/setup-python@v1
|
|
with:
|
|
python-version: 3.7
|
|
- name: "Run Noop-Pipeline Tests"
|
|
run: |
|
|
for i in {0..2} ; do
|
|
sudo env "PATH=$PATH" python3 -m osbuild --libdir . samples/noop.json
|
|
done
|
|
|
|
source_tests:
|
|
name: "Source Tests"
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: "Clone Repository"
|
|
uses: actions/checkout@v2
|
|
- name: "Install Dependencies"
|
|
run: |
|
|
sudo apt-get update
|
|
sudo apt-get -y install \
|
|
rpm \
|
|
systemd-container
|
|
- name: "Set up Python"
|
|
uses: actions/setup-python@v1
|
|
with:
|
|
python-version: 3.7
|
|
- name: "Run Source Tests"
|
|
run: sudo env "PATH=$PATH" python3 -m unittest -v test.test_sources
|
|
|
|
assembler_tests:
|
|
name: "Assembler Tests"
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: "Clone Repository"
|
|
uses: actions/checkout@v2
|
|
- name: "Install Dependencies"
|
|
run: |
|
|
sudo apt-get update
|
|
sudo apt-get -y install \
|
|
nbd-client \
|
|
qemu-utils \
|
|
rpm \
|
|
systemd-container \
|
|
tar \
|
|
yum
|
|
- name: "Set up Python"
|
|
uses: actions/setup-python@v1
|
|
with:
|
|
python-version: 3.7
|
|
- name: "Run Assembler Tests"
|
|
run: sudo env "PATH=$PATH" python3 -m unittest -v test.test_assemblers
|
|
|
|
stage_tests:
|
|
name: "Stage Tests"
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: "Clone Repository"
|
|
uses: actions/checkout@v2
|
|
- name: "Install Dependencies"
|
|
run: |
|
|
sudo apt-get update
|
|
sudo apt-get -y install \
|
|
systemd-container \
|
|
tar \
|
|
yum
|
|
- name: "Set up Python"
|
|
uses: actions/setup-python@v1
|
|
with:
|
|
python-version: 3.7
|
|
- name: "Run Stage Tests"
|
|
run: sudo env "PATH=$PATH" python3 -m unittest -v test.test_stages
|