History

Ondřej Budai e235fdedb3 tests/image: boot the vhd images on Azure Previously, vhd images were tested using QEMU. This commit changes that to boot them in the actual Azure infrastructure. Azure VMs have quite a lot of dependencies - a network interface, a virtual network, a network security group, a public ip address and a disk. Azure CLI and Azure Portal handle the creation of all these resources internally. However, when using the API, the caller is responsible to create all these resources before creating an actual VM. To handle the creation of all the resources in the right order, a deployment is used. A deployment is a set of resources defined in a JSON document. It can optionally take parameters to customize each deployment. After the deployment is finished, the VM is up and ready to be tested using SSH. Sadly, the deployments are a bit hard to clean-up. One would expect that deleting a deployment removes all the deployed resources. However, it doesn't work this way and therefore it's needed to clean up all resources "manually". For this reason, our deployment sets a unique tag on all the resources created by the deployment. After this test is finished, the API is queried for all the resources with the tag and then, they're deleted in the right order.		2020-04-27 20:34:20 +02:00
..
cases	tests/image: boot the vhd images on Azure	2020-04-27 20:34:20 +02:00
cloud-init	test/run: pass cloud-init data to qemu	2019-12-10 02:47:35 +01:00
keyring	Introduce local boot test case for QCOW2	2019-11-11 15:47:01 +01:00
azure-deployment-template.json	tests/image: boot the vhd images on Azure	2020-04-27 20:34:20 +02:00
README.md	.github: switch to codecov.io	2020-03-30 20:43:34 +02:00

README.md

osbuild-composer testing information

Test binaries, regardless of their scope/type (e.g. unit, API, integration) must follow the syntax of the Go testing package, that is implement only TestXxx functions with their setup/teardown when necessary in a yyy_test.go file.

Test scenario discovery, execution and reporting will be handled by go test.

Some test files will be executed directly by go test during rpm build time and/or in CI. These are usually unit tests. Scenarios which require more complex setup, e.g. a running osbuild-composer are not intented to be executed directly by go test at build time. Instead they are intended to be executed as stand-alone test binaries on a clean system which has been configured in advance (because this is easier/more feasible). These stand-alone test binaries are also compiled via go test -c -o during rpm build or via make build. See Integration testing for more information.

Notes on asserts and comparing expected values

When comparing for expected values in test functions you should use the testify/assert or testify/require packages. Both of them provide an impressive array of assertions with the possibility to use formatted strings as error messages. For example:

assert.Nilf(t, err, "Failed to set up temporary repository: %v", err)

If you want to fail immediately, not doing any more of the asserts use the require package instead of the assert package, otherwise you'll end up with panics and nil pointer memory problems.

Stand-alone test binaries also have the -test.failfast option.

Notes on code coverage

Code coverage is recorded in codecov.io. This information comes only from unit tests and for the time being we're not concerned with collecting coverage information from integration tests, see .github/workflows/tests.yml.

Integration testing

This will consume the osbuild-composer API surface via the composer-cli command line interface. Implementation is under cmd/osbuild-tests/.

The easiest way to get started with integration testing from a git checkout is:

dnf -y install rpm-build
dnf -y builddep osbuild-composer.spec
make rpm to build the software under test
dnf install rpmbuild/RPMS/x86_64/osbuild-composer-*.rpm - this will install both osbuild-composer, its -debuginfo, -debugsource and -tests packages
systemctl start osbuild-composer
/usr/libexec/tests/osbuild-composer/osbuild-tests to execute the test suite. It is best that you use a fresh system for installing and running the tests!

NOTE:

The easiest way to start osbuild-composer is via systemd because it takes care of setting up the UNIX socket for the API server.

If you are working on a pull request that adds more integration tests (without modifying osbuild-composer itself) then you can execute the test suite from the local directory without installing it:

make build - will build everything under cmd/
./osbuild-tests - will execute the freshly built integration test suite