Make sure we catch expected errors and fail gracefully. Also, make
sure the output id is printed on successufl osbuild run so it can
be introspected from outside the test suite.
Handle the case where osbuild suceeds but does not return an
output_id.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When we get an ssh connection, before the image is fully booted,
systemctl is-system-started returnse "starting", treat this as a
failed connection, and keep retrying.
Some distros may not support the wait switch correctly.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Currently we still only build for x86_64, but now the test suite is
prepared for hooking up other architectures.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This way we can test the distros on their respective CI, as not
all distros can be built in all environment. In particular RHEL
needs to be on a subscribed host.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We can now select specific cases, but whether or not to check image-info
or boot the image is determined purely by the contents of the json test
case.
We still run the tests as two travis workers just to avoid the timeout,
this should clearly be reworked.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We were supporting downloading an image and checking its image info. We don't
want to rely on external resources, and we should not test images made by others.
Drop it.
We were failing on name reuse. We should be able to not name these
at all, but nspawn is not happy with that.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We now have three top-level maps, that can be combined in any way:
boot-test: information about how to boot the image
compose: information about how to generate the pipeline
pipeline: the pipeline to generate the image
expected: the expected image-info
This creates compose entries for all the boot tests, but the blueprints
are named 'blueprint-draft', as we are not yet verifynig that the pipeline
is correct.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This passes the redhat user with ssh key as an ISO image to our
qemu instances, making sure images relying on cloud-init rather than
hardcoded user credentials can be used in our tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For the qemu tests this makes no difference as we are anyway forwarding
the ports. But the nspawn tests share the same network namespace between
the image and the ssh client running the test without any forwarding. In
order for that to work we had to modify the image to use a non-standard
port.
We don't want this for two reasons: we want to make sure we test our images
unmodified, and also this meant that when we changed our pipeline generation
we were not verifying that the boot test cases were updated accordingly. As
a result they have now drifted.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This makes sure we can use any port to connect to sshd, without
worrynig about clashes with parallel tests, or an sshd instance
running on the host.
Signed-off-by: Tom Gundersen <teg@jklm.no>
osbuild has a concept of runners now: scripts that set up a build
environment. Update the osbuild submodule to latest master, change
`Pipeline` to to the new buildroot description format, and use the
`org.osbuild.fedora30` runner from the fedora30 distro.
1) additional qemu tests for ami, vmdk, vhd, and openstack image types
2) new type of systemd-nspawn tests for tar, ext4, and parititioned disk
types
the systemd-nspawn tests use loopback network interface directly from
the host so it is necessary to tweak the settings of its SSH server.
This is done in a "script" stage using simple "sed" command.
The tests works by executing osbuild with predefined pipeline. Then the
image boots and the testing script creates SSH connection to the running
VM. If everything goes fine `systemctl is-system-running` is executed
with result `running` and the test case passed.
The JSON definition of the test case contains also a blueprint that
should generate the desired pipeline, but it didn't work for me, so I'm
including it for future use from the golang unit tests.
These tests (will) test more than just image-info: they'll take a
blueprint, verify that `osbuild-pipeline` generates the correct
pipeline, run osbuild with that pipeline and verify that the resulting
image has the expected image-info output.
This change only includes the latter half (i.e., only moves the already
existing tests).
Also drop python's unittest. It was hard to control output (important
for quickly spotting failures and to make travis happy). This introduces
test/run, which runs all test cases in test/cases or the ones given on
the command line.
When a failure occurs, it prints a diff of the actual and the expected
image info.
