Add a new option `--repofrompath` allowing to specify additional DNF
repositories, which will be used on the Runner when installing any
packages (such as osbuild).
Extend the `test/README.md` to mention the new option. In addition,
specify some aspects of the script in more detail, because some of
them were not easy to figure out by users.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When `image-info` inspects ostree images, the `/usr/etc` is bind-mounted
to `/etc`. This results in conflicting SELinux policy specification for
these files and makes the outcome dependent on the `setfiles` build.
All the files in `/etc` have mismatch in the expected vs. actual SELinux
context.
Exclude `/etc` from the check of SELinux ctx mismatches in case the
analysed tree is from an ostree-based image.
Sort the list returned `read_selinux_ctx_mismatch()` based on the item's
`filename` key, to make the result consistent across runs.
`image-info` can not read SELinux labels from the images, which are not
known to the host. This makes the report content depend on the host
environment. As a temporary workaround, relabel the image-info script with
osbuild_exec_t label to allow it to read unknown SELinux labels.
Modify documentation in `test/README.md` to explain the issue with
`image-info` and unknown SELinux labels.
Modify the `generate-all-test-cases` to relabel `image-info` before
generating test cases.
Modify the `image_tests.sh` to relabel `image-info` before running image
test cases.
Add 'tar' image for 'rhel-8' on 's390x' back to the matrix of generated
test cases, as it was removed by mistake. Regenerate the image test
case. Remove 'tar' image from 'rhel-84' on 's390x' from the matrix of
generated test cases, as it is not supported.
Regenerate all affected image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
by defining the COMPOSE_URL environment variable! This will allow
testing more flavors of internal releases.
The rest is renaming files and variables to reflect the fact that
we're running tests against internal trees, not only nightlies.
Split the integration testing section into two parts. One for testing of
Weldr API and another one for Cloud API. Cloud API integration testing
part has two sections, one for testing integration with AWS and another
one for GCP.
Fix path to osbuild-composer-cli-tests tool.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add the `generate-all-test-cases` tool, which generates all defined image
test cases based on provided "distro x arch x image-type" combination
matrix and CLI arguments. Test cases are generated by running the
`generate-test-cases` tool inside an architecture-specific VM.
The tool requires a qcow2 image to be provided for each requested
architecture. These images are used to create ephemeral VMs used
for test case generation.
There appears to be an issue with entropy on all VMs for other
architecture than the host's one. It makes the generation of the first
test case usually fail due to the `go get` command failing with a "TLS
handshake timeout" error. There is a workaround to retry the generation
of a test case a maximum of 3 times in this case. In the usual case,
the test case generation passes on the second try. All subsequent test
case generations should complete successfully on the first try.
Add the `distro-arch-imagetype-map.json` file as the default source of
the "distro x arch x image-type" combination matrix for generating test
cases. Its content is based on the current content of the
`test/data/manifests` directory.
Update `test/README.md` to mention the `generate-all-test-cases` tool.
Add a note, that `generate-test-cases` must be run on a host with the
same architecture, as the intended test case's.
Known issues:
- The tool does not work with RHEL images because of missing "9p"
filesystem support.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The directory with image-tests test cases has been renamed from `cases`
to `manifests`. This has not been previously reflected in the test/README.md
and osbuild-image-tests code. osbuild-image-tests hardcodes the test
cases directory path and uses it in case no test case are passed
to it on the command line. Since the image_tests.sh CI test case looks
for image-tests test cases in the correct directory and passes the
relevant ones to osbuild-image-tests, the CI didn't detect this issue.
Running osbuild-images-tests without any argument and let it run all
test cases from the default test cases directory as part of CI probably
does not make sense. Due to this reason, I'm not adding any new test.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Attempt to clarify the structure of our tests. Each test case is now
encapsulated in a script in `test/cases`. Each of these scripts should
be runnable on a pristine machine and be independent of each other. It
is up to the test-orchestractor to decide if they should be run
consequtively instance, or in parallel on separate instances. Each
script can execute several tests and call whatever helper binaries
is desired. However, each case should be assumed to always run as one.
No tests should be run directly from git, but should rather be installed
onto the test system using rpm and run from there. This moves towards
unifying our two types of test cases.
The new structure of is now:
`test/cmd`: the executors, one for each test-case. This is installed
into `/usr/libexec/test/osbuild-composer`.
`test/data`: data and config used by the tests. This is installed into
`/usr/share/tests/osbuild-composer`.
`schutzbot`: configuration of the actual test run. In particular, this
is where the distros and repositories to test against are
configured.
This is very much still work-in-progress, and is only the first step
towards simplifying schutzbot. Apart from moving files around, this
should be a noop.
Signed-off-by: Tom Gundersen <teg@jklm.no>
These had the generic name osbulid-tests, rename the binary to more
closely describe what is under test: weldr API as composed by
composer-cli.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The image tests are now able to upload images to Azure and boot them there.
However, the documentation on how to set up the required resources in Azure
was missing. This commit adds it.
coveralls doesn't work from GitHub actions. Its "github" service type
uses the GITHUB_TOKEN from the action, which only has read access when
invoked from a forked repository.
codecov gets this right: it validates that an uploaded coverage file
originated from a GitHub action run by asking GitHub, and then uses its
OAuth credentials (through the Marketplace App) to comment and set
status.
Also, coveralls requires a third-party package to convert go's coverage
report to a format it understands. codecov detects the format
server-side. It also handles go's coverage format better: it highlights
lines with "some coverage" in yellow (coveralls has no concept of this).
