The netns() function sets up a new namespace for tests. The function is
also used to determine whether those tests can be run (using
unittest.skipUnless()). A bug in the function made the changes stick if
the function failed early. Specifically, when the "ip link" line fails,
the function exits without reverting to the old namespace.
Since the code is used in "skipUnless()", it's run during test
collection, which means that even if the relevant tests aren't selected,
they affect the environment for other tests.
Add two simple tests to check that the osbuild executable fails with the
right exit codes when passed an invalid manifest or checkpoint.
This reuses test.OSBuild, which is extended to raise CalledProcessError
if needed.
Create a cache directory with the scope of the stage test suit and
after each test is run cache the downloaded files for the `files`
source via the new `OSBuild.copy_source_data` method. Initialize
OSBuild with that cache directory, so previously downloaded files
get used between different stage tests.
If the stage test folder contains a `metadata.json` file, it will
contain a dictionary where the keys are stage ids and the values
are dictionaries containing the metadata to verify. For each of
those the stage will be looked up in the pipeline result of 'b'
and verified that the metadata matches.
640k ought to be enough for anybody!
Err... I mean...
The assembler tests now install only the filesystem and selinux packages and
their dependencies. For this, we don't need the luxury of 2 GiB.
This commit changes the image size to 512 MiB. This has some advantages:
- the tests are faster - I measured the qemu assembler test and the running
time went down from 290s to 260s.
- the tests can be run in environments with smaller disk space
Previously, the osbuild executor had its internal temporary directory that
served as the output directory. However, this approach gives no power to
the caller to control the lifetime of the produced artifacts. When more
images are built using one executor, the results will accumulate in one
place possibly leading to exhaustion of disk space.
This commit removes the executor's internal output directory. The output
directory can now be passed to osbuild.compile, so the caller can control
its lifetime. If no directory is passed in, the compile method will use
its own temporary directory - this is useful in cases when the caller
doesn't care about the built artifacts or the manifest doesn't have any
outputs.
This way the test can benefit from osbuild's internal cache:
The first subtest builds all the stages and runs the assembler
The next subtests can reuse the built stages and just run the assembler
Some data from my machine running the qemu test:
Building the manifest takes about 120 seconds
Running just the assembler on the cache's content takes 30 seconds.
Before this change, the whole manifest was built 3 times:
3 * 120 = 360 seconds
After this change, the whole manifest is built once and the cache
is reused 2 times:
1 * 120 + 2 * 30 = 180 seconds
Let the caller decide which executor instance should be used to build
the manifest. This change allows us to use osbuild's built-in cache
in the following commit.
Rely on the ability of `BaseAPI` to auto-generate socket addresses
when no one was provided. The `BuildRoot` does not rely on the
sockets being created in the `BuildRoot.api` directory anymore and
will instead bind-mount each individual socket address to the well
known location via the `BaseAPI.endpoint` identifier.
Convert all API providers to take the `socket_address` as an
optional keyword argument.
#471 extends the assembler test suite to also test xfs and btrfs filesystems
in raw and qemu assemblers. However, this change leads to long running times
of this suite.
The running time of these test consist of 3 main steps:
1) Building the build pipeline
2) Building the stages
3) Running the assembler
There are two optimization approaches:
1) Caching
OSBuild supports caching, therefore it's possible to cache results of first
two steps.
2) Minimizing the operating system tree
Assemblers don't care about the image contents. Therefore, it's possible
to create just a small tree which would be used to test the assemblers.
This should lead to speed up in the step 2 (smaller tree should be built
quicker) and in step 3 (big part of assembling is just copying files over
to the image).
This commit implements the second approach. A new test manifest is now added,
which just installs the filesystem package and its dependencies and this tree
is then labeled. This solution was chosen, so that the assemblers get
something that looks as a proper filesystem tree but also can be built pretty
quickly.
Before this change, the test_rawfs method with #471 merged ran for 842 seconds.
After this change, it ran for 391 seconds.
Add a simple check for the selinux check by building the f32-base
image with an added selinux stage. Use the options from a test
json file and verify the labels against a set of labels given in
the aforementioned test file.
Only generate stage tests for sub-directories in stages_tests
that contain a diff.json. This should allow us to have specialized
stage tests that don't use the current {a, b}.json & diff.json
pattern.
Verify the rpm-ostree.input hash is set correctly for the repository
itself as well. This will in turn also verify that the repository
is existent and can be accessed.
Using the network block device (nbd) kernel module to test all
the non-raw image formats often caused tests to fail due to nbd
not being stable itself (see below).
Instead convert non-raw images to the raw format via qemu-img
convert and mount those with loop-back devices. All the testing
code itself stays the same.
Example nbd error messages:
kernel: block nbd15: NBD_DISCONNECT
kernel: block nbd15: Disconnected due to user request.
kernel: print_req_error: 89 callbacks suppressed
kernel: blk_update_request: I/O error, dev nbd15, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
kernel: buffer_io_error: 134 callbacks suppressed
kernel: Buffer I/O error on dev nbd15, logical block 0, async page read
kernel: blk_update_request: I/O error, dev nbd15, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
Split up the partition table test into reading the partition table
and then asserting it has the correct entries. Prepares the usage
of the partition information later.
Move the last remaining test into the correct subdir. With this done,
all our tests run in one of the 3 groups:
* `make test-src`
Run tests against the source-code, including linters.
* `make test-mod`
Run unit-tests on the individual python modules. This needs no
special permissions (unless noted in each test) or runtime
environments. It is meant to be fast and easy to run in all
circumstances.
* `make test-run`
Run tests that execute the osbuild pipeline. This requires
superuser privileges and will likely take a while. Furthermore,
this might produce large artifacts.
Move the stage-tests over to the new test-infrastructure. This moves
the test invocation into `./test/run/test_stages.py`, so it is invoked
as part of the runtime-tests. Secondly, the test-data is stored in
./test/data/stages/ so the path is relative to
TestBase.locate_test_data().
While at it, this also drops the dynamic class modifications and instead
uses subTest(). This simplifies the code quite a bit and avoids
dynamically creating python code.
This moves the `sources_tests` into ./test/data/sources/ and makes the
entire test use `locate_test_data()` to get relative paths for their
accesses.
This further improves our test cases to support running from
installments rather than local checkouts. We need access to ./test/data
guarded, so we can install packages and still have the tests access the
correct paths.
This also adjusts the HTTP-Server we use in the test to serve data
relative to a path it is handed. I now chose `./test/data`, which will
easily allow us to re-use the same HTTP-Server in the future for other
tests that require it.
Add a new trivial runtime-test which simply runs a no-op pipeline. This
is a fast, trivial test that simply verifies osbuild is properly setup
and accessible.
Remove the explicit no-op test from the CI, now that the test-suite has
it as well.
The idea is that source can themselves spawn other modules, esp.
new secrets modules. For this they need to know the library dir,
aka 'libdir' throughout the osbuild source. Therefore change the
SourceServer to directly get the library directory instead of
just the sub-directory to the sources. Then pass the library
directory to via the JSON API to the source.
Adjust all usage of the SourceServer, including the tests.
