144570026d
coveralls doesn't work from GitHub actions. Its "github" service type uses the GITHUB_TOKEN from the action, which only has read access when invoked from a forked repository. codecov gets this right: it validates that an uploaded coverage file originated from a GitHub action run by asking GitHub, and then uses its OAuth credentials (through the Marketplace App) to comment and set status. Also, coveralls requires a third-party package to convert go's coverage report to a format it understands. codecov detects the format server-side. It also handles go's coverage format better: it highlights lines with "some coverage" in yellow (coveralls has no concept of this).
76 lines
3.2 KiB
Markdown
76 lines
3.2 KiB
Markdown
# osbuild-composer testing information
|
|
|
|
Test binaries, regardless of their scope/type (e.g. unit, API, integration)
|
|
must follow the syntax of the Go
|
|
[testing package](https://golang.org/pkg/testing/), that is implement only
|
|
`TestXxx` functions with their setup/teardown when necessary in a `yyy_test.go`
|
|
file.
|
|
|
|
Test scenario discovery, execution and reporting will be handled by `go test`.
|
|
|
|
Some test files will be executed directly by `go test` during rpm build time
|
|
and/or in CI. These are usually unit tests. Scenarios which require more complex
|
|
setup, e.g. a running osbuild-composer are not intented to be executed directly
|
|
by `go test` at build time. Instead they are intended to be executed as
|
|
stand-alone test binaries on a clean system which has been configured in
|
|
advance (because this is easier/more feasible). These stand-alone test binaries
|
|
are also compiled via `go test -c -o` during rpm build or via `make build`.
|
|
See *Integration testing* for more information.
|
|
|
|
|
|
## Notes on asserts and comparing expected values
|
|
|
|
When comparing for expected values in test functions you should use the
|
|
[testify/assert](https://godoc.org/github.com/stretchr/testify/assert) or
|
|
[testify/require](https://godoc.org/github.com/stretchr/testify/require)
|
|
packages. Both of them provide an impressive array of assertions with the
|
|
possibility to use formatted strings as error messages. For example:
|
|
|
|
```
|
|
assert.Nilf(t, err, "Failed to set up temporary repository: %v", err)
|
|
```
|
|
|
|
If you want to fail immediately, not doing any more of the asserts use the
|
|
require package instead of the assert package, otherwise you'll end up with
|
|
panics and nil pointer memory problems.
|
|
|
|
Stand-alone test binaries also have the `-test.failfast` option.
|
|
|
|
|
|
## Notes on code coverage
|
|
|
|
Code coverage is recorded in
|
|
[codecov.io](https://codecov.io/github/osbuild/osbuild-composer).
|
|
This information comes only from unit tests and for the time being
|
|
we're not concerned with collecting coverage information from integration
|
|
tests, see `.github/workflows/tests.yml`.
|
|
|
|
|
|
## Integration testing
|
|
|
|
This will consume the osbuild-composer API surface via the `composer-cli`
|
|
command line interface. Implementation is under `cmd/osbuild-tests/`.
|
|
|
|
The easiest way to get started with integration testing from a git
|
|
checkout is:
|
|
|
|
* `dnf -y install rpm-build`
|
|
* `dnf -y builddep osbuild-composer.spec`
|
|
* `make rpm` to build the software under test
|
|
* `dnf install rpmbuild/RPMS/x86_64/osbuild-composer-*.rpm` - this will
|
|
install both osbuild-composer, its -debuginfo, -debugsource and -tests packages
|
|
* `systemctl start osbuild-composer`
|
|
* `/usr/libexec/tests/osbuild-composer/osbuild-tests` to execute the test suite.
|
|
It is best that you use a fresh system for installing and running the tests!
|
|
|
|
**NOTE:**
|
|
|
|
The easiest way to start osbuild-composer is via systemd because it takes care
|
|
of setting up the UNIX socket for the API server.
|
|
|
|
If you are working on a pull request that adds more integration tests
|
|
(without modifying osbuild-composer itself) then you can execute the test suite
|
|
from the local directory without installing it:
|
|
|
|
* `make build` - will build everything under `cmd/`
|
|
* `./osbuild-tests` - will execute the freshly built integration test suite
|