Previously, all resouces were created with a certain tag. When the cleanup
phase came, the Resources - List route[1] was used to get all resources with
the tag. Then, they were deleted in the right order.
Sadly, the Resources - List API has issues with listing disks. Sometimes,
it returns the virtual disks 15 minutes after they were created. As the
result, the disks have been left behind quite often and our bill was higher
than necessary.
This commit uses a different method - the Go code now knows all resource names
(see the previous commit), so it can delete all resources without listing them
using the "broken" API route.
[1]: https://docs.microsoft.com/en-us/rest/api/resources/resources/list
Prior this commit the resource names were generated in the deployment
template, so the Go code actually didn't know them. This commit generates
all names in the Go code, so they can be used in the future commits.
When osbuild fails, osbuild-composer's image tests print the output as
ugly JSON and that makes it difficult to determine why osbuild failed.
Instead, print the json in pretty format. 🦋Fixes#782.
Signed-off-by: Major Hayden <major@redhat.com>
On some environments (like RHEL gating) there's no virtualization available.
This commit adds -disable-local-boot argument to osbuild-image-tests. When
this argument is present, the local booting is skipped. This doesn't affect
the cloud booting, the test binary still tries to do that. If no credentials
are available, the fall back to local booting will be skipped if
-disable-local-boot is given.
This makes it easier to use the test binary with the `-run` argument.
Instead of the full path:
-test.run TestImages//usr/share/tests/osbuild-composer/cases/rhel_8.2-x86_64-openstack-boot.json
this only requires the actual name:
-test.run TestImages/rhel_8.2-x86_64-openstack-boot.json
When edd7b37ea added `--output-directory` to the invocation of osbuild,
it also removed `--store`.
This was a mistake: osbuild's default store is `.osbuild`, which is not
what we want. Restore the old behavior of passing a temporary directory,
but use the same for each test run.
Treating stdout and stderr separately makes it hard to match what
happened when. It's also easy to miss when `-v` is passed to the test
binary.
Print the output to stdout when osbuild fails, because the test
framework we're using does not print errors if they're too large.
Also, don't special-case exec.ExitError. Output might be useful in any
case.
Schutzbot currently runs all the same tests an all arches/distro
combinations. This will not work as we introduce image types only on
some distro/arches.
In the future we should make the image-test binary more clever, so
Schutzbot won't have to tell it which cases to run at all. For now,
simply don't fail if the specified test-case does not exist.
Signed-off-by: Tom Gundersen <teg@jklm.no>
RHEL 8 doesn't have qemu-system-x86_64 in PATH. Instead, it is in
/usr/libexec/qemu-kvm. This commit fixes that by detecting the host
distro and choosing the right binary.
-cpu host cannot be used with anything else than kvm. This commit removes
hvf and tcg because it doesn't make any sense with -cpu host.
If this causes some issues for anyone, we can revert back and remove -cpu
host.
Running qemu with -accel accel= results in the following error:
qemu-system-x86_64: -accel accel=kvm:hvf:tcg: Don't use ':' with -accel,
use -M accel=... for now instead
Qemu 4.2 deprecated the -accel accel= argument. When the arg is passed in,
qemu exists status code of 1.
This commit changes the qemu command to use the recommended way of specifying
the acceleration options.
See:
3d5e90a50b
Previously, vhd images were tested using QEMU. This commit changes that to
boot them in the actual Azure infrastructure.
Azure VMs have quite a lot of dependencies - a network interface, a virtual
network, a network security group, a public ip address and a disk. Azure CLI
and Azure Portal handle the creation of all these resources internally.
However, when using the API, the caller is responsible to create all these
resources before creating an actual VM.
To handle the creation of all the resources in the right order, a deployment
is used. A deployment is a set of resources defined in a JSON document.
It can optionally take parameters to customize each deployment. After the
deployment is finished, the VM is up and ready to be tested using SSH.
Sadly, the deployments are a bit hard to clean-up. One would expect that
deleting a deployment removes all the deployed resources. However, it doesn't
work this way and therefore it's needed to clean up all resources "manually".
For this reason, our deployment sets a unique tag on all the resources created
by the deployment. After this test is finished, the API is queried for all
the resources with the tag and then, they're deleted in the right order.
The cmd/osbuild-image-tests package is becoming bigger than I would like to.
It will be nice to split it to some smaller pieces at some point.
This commit does the first step - splits off the first subpackage containing
all the constants.
This commits enables the parallelism for the image tests. However, there's
a catch. Osbuild cannot be reliably run in parallel, so the code uses
a mutex to ensure there's always only one osbuild instance for now. Even
with this limitation, there's a significant speed-up of the tests:
Prior this commit, the image tests run in 40 minutes on Travis. After this
commit, the time is reduced to 32 minutes.
The speed-up will have an even bigger effect when more cloud-upload tests are
added to the test suite.
Speed up the boot tests by allowing qemu to use all of the available
CPUs on the system. Our CI VMs have at least 2 CPUs and this shortens
the time required for a boot test.
Signed-off-by: Major Hayden <major@redhat.com>
The osbuild-image-tests don't currently support boot test for any
alternative architecture because the qemu-system-x86_64 command is
hardcoded. This patch introduces a branch specific to aarch64, but
without a KVM support as I was unable to make it run in Beaker, which is
currently the only offering we have with ARM machines. As a workaround
the boot tests will be skipped if kvm kernel module is not found and only
image-info tests will run.
Prior this commit the ami images were tested locally using qemu. This does
not reflect at all how they're used in practice. This commit introduces
the support for running them in the actual AWS. Yay!
The structure of code reflects that we want to switch to osbuild-composer
to build the images soon.
It's not very clear that the constants are indeed constants. This commit moves
them to a new struct. This way it should be more clear that those values are
constants.
Soon, images will be run non-locally (AWS, Azure). For the remote ones it's
potentially dangerous to use the publicly available key-pair. This change
prepares the codebase for specifying different keys than the pre-generated
one.
Soon, images will be run non-locally (AWS, Azure). For those boot types
there's no need to have an unshared network namespace. This commit prepares
the code for that.
Soon, images will be run non-locally (AWS, Azure). For this, we cannot
use ssh in a namespaced environment. This commit makes the namespace for ssh
optional.
This is not a functional change (except for the slightly different output).
NoError() is more understandable (both in the code and in the output) than
Nil() when asserting errors. Also the rest of the tests uses mostly NoError()
when asserting errors, this change should lead to a better consistency.
testify library cannot deal with error messages with length > 64k. Sadly,
osbuild output is very long one line. This commits formats the output before
making the error from it.
Build constraints must be added to every file. We only want to build
this package when the `integration` tag is set.
Without this, every build prints this warning:
# github.com/osbuild/osbuild-composer/cmd/osbuild-image-tests
runtime.main_main·f: function main is undeclared in the main package
When using %#v instead of %v, printing an error gives a slightly more
informative error message, therefore this commit switches to %#v in all
error prints.
The distro argument and restrictions to run only tests for the same distro
as the host's one were confusing. This commit removes them. Now the behaviour
is following:
By default all the test cases in test case directory are run.
If test cases are given by arguments, they are all run, and test case
directory is ignored.
Previously we had only tests for Fedora 30, therefore I added a hack to treat
every Fedora as Fedora 30, so the tests would run. Now, we have tests for
multiple Fedora versions, therefore the hack is no longer needed.
We're currently rewriting all the integration tests to use the Go
testing framework. This commit does the switch for the image tests.
I decided not to use the testing framework in functions which are
not directly tight to testing (booting images, running osbuild). I think
it's reasonable to use classic error handling there and propagate the errors
to places directly tight to testing and use the testing library.
This enables us to reuse the code in different part of projects if needed.
By default, image test executable runs only test cases for the same distro
as the host's one. On Travis there's Ubuntu, so we need to adjust the
behaviour and run the cases for a distro specified by command line
arguments.
We need to use different values for path constants when running the tests
on the Travis CI. This is the first step to achieve this.
Note that this commit may be reverted when Travis CI is dropped.
This commit makes the osbuild-image-tests binary doing the same set of tests
like the old test/run script.
Changes from test/run:
- qemu/nspawn are now killed gracefully. Firstly, SIGTERM is sent.
If the process doesn't exit till the timeout, SIGKILL is sent.
I changed this because nspawn leaves some artifacts behind when killed
by SIGKILL.
- the unsharing of network namespace now works differently because of
systemd issue #15079
A manifest is struct made up of a pipeline and a sources object. So
far all our sources objects are empty, but we have moved from
using pipelines to manifests everywhere, in preparation for
generating pipelines that require sources.
Make the same change in the test cases.
Signed-off-by: Tom Gundersen <teg@jklm.no>
./test/run test suite has served us well over the last months. However,
there is currently a major effort to run the better defined integration
test suite on a CI. Nonetheless, two very important parts are still missing
from the integration test suite: inspecting the image with image-info
and booting the image. This commit begins the work on this matter by porting
a part of ./test/run suite to Go. Currently, only image-info tests work, the
rest will come in the following commits.
