Move from using 'zram' to 'zram-generator-defaults' in the ostree bootiso
testing manifest. More information is available in Fedora 33 Change
document [1].
Add org.osbuild.kernel-cmdline stage to fedora-boot.json manifest
because of change in how grub handles the kernel command line arguments
[2].
GRUB2 Stage 2 checksums in assemblers test are updated. The change have
been verified by building the fedora-boot.json manifest with each checked
filesystem and booting the image in QEMU with legacy mode.
[1] https://fedoraproject.org/wiki/Changes/SwapOnZRAM
[2] https://github.com/osbuild/osbuild-composer/pull/982#issuecomment-697356929
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Parameterize `test_sources` via `pytest.mark.parameterize`, so that
now the product of source and test cases for that source is visible
to pytest and thus also the caller.
Instead of testing the legacy `SourcesServer` and `sources.get`,
test the `Source.download` method, which is the new and exclusive
way how sources are used in osbuild. For this, the complete info
for a specific source is now included the specific test case.
For the `org.osbuild.curl` source this means that the respective
information is moved from `sources.json`, which is then unused
and thus removed. The test case that checks for an unknown
checksum is also removed because `Source.download` just fetches
everything instead of a subset.
Use `pytest`, which is a dependency of osbuild since a few versions,
instead of `unittest` for the source tests. The main reason is to be
able to use `pytest.mark.parameterize` in the near future to easily
generate a test matrix.
When the expected metadata does not match the produced metadata,
write the latter to `/tmp` for closer inspection; additionally
this should help update the metadata file in case the changes
are expected.
This is a partial revert of commit d584a1e, which converted the
dynamically generate stage tests to sub tests. The problem with
the latter is that they can't be individually run, which makes
testing changes to specific stages cumbersome to develop.
Therefore switch back to a model where the stage tests are
dynamically generated via a class decorator.
Extend the current ostree container test case to use the new
bootiso manifest to build a bootable iso. This is done in the
existing container test case in order to share the build root
and ostree commit results and speed up testing. In the future,
the test infrastructure should be extended so that the cache
can be optionally shared between test cases.
Add a new basic test for the dracut stage. It uses a osbuild
pipeline to build an initrd and inspects it via the initrd.py
module. The content is compared to a reference located in the
same directory as the pipeline (test/data/stages/dracut/).
Add a v2 manifest to builds an ostree commit, then builds a small
container with a webserver and puts that commit into it.
Start a new test suit where this manifest is built and then checked
that the artifact exists. In the future the artifact should also be
inspected and run by a container engine and the commit be pulled
via ostree.
Add a simple noop pipeline, with a noop stage and a noop input,
which all in all does nothing much; but it will validate, load
exectue the pipeline, stage and inputs. So maybe not really
"nothing" in the strictest sense.
In the test case for the tar assembler, actually verify the
content by un-tar-ing the result again and comparing it to
the tree. This would have spotted missing SELinux labels.
The netns() function sets up a new namespace for tests. The function is
also used to determine whether those tests can be run (using
unittest.skipUnless()). A bug in the function made the changes stick if
the function failed early. Specifically, when the "ip link" line fails,
the function exits without reverting to the old namespace.
Since the code is used in "skipUnless()", it's run during test
collection, which means that even if the relevant tests aren't selected,
they affect the environment for other tests.
Add two simple tests to check that the osbuild executable fails with the
right exit codes when passed an invalid manifest or checkpoint.
This reuses test.OSBuild, which is extended to raise CalledProcessError
if needed.
Create a cache directory with the scope of the stage test suit and
after each test is run cache the downloaded files for the `files`
source via the new `OSBuild.copy_source_data` method. Initialize
OSBuild with that cache directory, so previously downloaded files
get used between different stage tests.
If the stage test folder contains a `metadata.json` file, it will
contain a dictionary where the keys are stage ids and the values
are dictionaries containing the metadata to verify. For each of
those the stage will be looked up in the pipeline result of 'b'
and verified that the metadata matches.
640k ought to be enough for anybody!
Err... I mean...
The assembler tests now install only the filesystem and selinux packages and
their dependencies. For this, we don't need the luxury of 2 GiB.
This commit changes the image size to 512 MiB. This has some advantages:
- the tests are faster - I measured the qemu assembler test and the running
time went down from 290s to 260s.
- the tests can be run in environments with smaller disk space
Previously, the osbuild executor had its internal temporary directory that
served as the output directory. However, this approach gives no power to
the caller to control the lifetime of the produced artifacts. When more
images are built using one executor, the results will accumulate in one
place possibly leading to exhaustion of disk space.
This commit removes the executor's internal output directory. The output
directory can now be passed to osbuild.compile, so the caller can control
its lifetime. If no directory is passed in, the compile method will use
its own temporary directory - this is useful in cases when the caller
doesn't care about the built artifacts or the manifest doesn't have any
outputs.
This way the test can benefit from osbuild's internal cache:
The first subtest builds all the stages and runs the assembler
The next subtests can reuse the built stages and just run the assembler
Some data from my machine running the qemu test:
Building the manifest takes about 120 seconds
Running just the assembler on the cache's content takes 30 seconds.
Before this change, the whole manifest was built 3 times:
3 * 120 = 360 seconds
After this change, the whole manifest is built once and the cache
is reused 2 times:
1 * 120 + 2 * 30 = 180 seconds
Let the caller decide which executor instance should be used to build
the manifest. This change allows us to use osbuild's built-in cache
in the following commit.
Rely on the ability of `BaseAPI` to auto-generate socket addresses
when no one was provided. The `BuildRoot` does not rely on the
sockets being created in the `BuildRoot.api` directory anymore and
will instead bind-mount each individual socket address to the well
known location via the `BaseAPI.endpoint` identifier.
Convert all API providers to take the `socket_address` as an
optional keyword argument.
#471 extends the assembler test suite to also test xfs and btrfs filesystems
in raw and qemu assemblers. However, this change leads to long running times
of this suite.
The running time of these test consist of 3 main steps:
1) Building the build pipeline
2) Building the stages
3) Running the assembler
There are two optimization approaches:
1) Caching
OSBuild supports caching, therefore it's possible to cache results of first
two steps.
2) Minimizing the operating system tree
Assemblers don't care about the image contents. Therefore, it's possible
to create just a small tree which would be used to test the assemblers.
This should lead to speed up in the step 2 (smaller tree should be built
quicker) and in step 3 (big part of assembling is just copying files over
to the image).
This commit implements the second approach. A new test manifest is now added,
which just installs the filesystem package and its dependencies and this tree
is then labeled. This solution was chosen, so that the assemblers get
something that looks as a proper filesystem tree but also can be built pretty
quickly.
Before this change, the test_rawfs method with #471 merged ran for 842 seconds.
After this change, it ran for 391 seconds.
Add a simple check for the selinux check by building the f32-base
image with an added selinux stage. Use the options from a test
json file and verify the labels against a set of labels given in
the aforementioned test file.
Only generate stage tests for sub-directories in stages_tests
that contain a diff.json. This should allow us to have specialized
stage tests that don't use the current {a, b}.json & diff.json
pattern.
Verify the rpm-ostree.input hash is set correctly for the repository
itself as well. This will in turn also verify that the repository
is existent and can be accessed.
Using the network block device (nbd) kernel module to test all
the non-raw image formats often caused tests to fail due to nbd
not being stable itself (see below).
Instead convert non-raw images to the raw format via qemu-img
convert and mount those with loop-back devices. All the testing
code itself stays the same.
Example nbd error messages:
kernel: block nbd15: NBD_DISCONNECT
kernel: block nbd15: Disconnected due to user request.
kernel: print_req_error: 89 callbacks suppressed
kernel: blk_update_request: I/O error, dev nbd15, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
kernel: buffer_io_error: 134 callbacks suppressed
kernel: Buffer I/O error on dev nbd15, logical block 0, async page read
kernel: blk_update_request: I/O error, dev nbd15, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
Split up the partition table test into reading the partition table
and then asserting it has the correct entries. Prepares the usage
of the partition information later.
Move the last remaining test into the correct subdir. With this done,
all our tests run in one of the 3 groups:
* `make test-src`
Run tests against the source-code, including linters.
* `make test-mod`
Run unit-tests on the individual python modules. This needs no
special permissions (unless noted in each test) or runtime
environments. It is meant to be fast and easy to run in all
circumstances.
* `make test-run`
Run tests that execute the osbuild pipeline. This requires
superuser privileges and will likely take a while. Furthermore,
this might produce large artifacts.
Move the stage-tests over to the new test-infrastructure. This moves
the test invocation into `./test/run/test_stages.py`, so it is invoked
as part of the runtime-tests. Secondly, the test-data is stored in
./test/data/stages/ so the path is relative to
TestBase.locate_test_data().
While at it, this also drops the dynamic class modifications and instead
uses subTest(). This simplifies the code quite a bit and avoids
dynamically creating python code.
This moves the `sources_tests` into ./test/data/sources/ and makes the
entire test use `locate_test_data()` to get relative paths for their
accesses.
This further improves our test cases to support running from
installments rather than local checkouts. We need access to ./test/data
guarded, so we can install packages and still have the tests access the
correct paths.
This also adjusts the HTTP-Server we use in the test to serve data
relative to a path it is handed. I now chose `./test/data`, which will
easily allow us to re-use the same HTTP-Server in the future for other
tests that require it.
Add a new trivial runtime-test which simply runs a no-op pipeline. This
is a fast, trivial test that simply verifies osbuild is properly setup
and accessible.
Remove the explicit no-op test from the CI, now that the test-suite has
it as well.
The idea is that source can themselves spawn other modules, esp.
new secrets modules. For this they need to know the library dir,
aka 'libdir' throughout the osbuild source. Therefore change the
SourceServer to directly get the library directory instead of
just the sub-directory to the sources. Then pass the library
directory to via the JSON API to the source.
Adjust all usage of the SourceServer, including the tests.
