As it turns out, the default expectation is not to distinguish between
these. We will now produce whatever is the most recent minor release by
default, and image tests will still be pinned at a given snapshot to be
reproducible.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Previously, vhd images were tested using QEMU. This commit changes that to
boot them in the actual Azure infrastructure.
Azure VMs have quite a lot of dependencies - a network interface, a virtual
network, a network security group, a public ip address and a disk. Azure CLI
and Azure Portal handle the creation of all these resources internally.
However, when using the API, the caller is responsible to create all these
resources before creating an actual VM.
To handle the creation of all the resources in the right order, a deployment
is used. A deployment is a set of resources defined in a JSON document.
It can optionally take parameters to customize each deployment. After the
deployment is finished, the VM is up and ready to be tested using SSH.
Sadly, the deployments are a bit hard to clean-up. One would expect that
deleting a deployment removes all the deployed resources. However, it doesn't
work this way and therefore it's needed to clean up all resources "manually".
For this reason, our deployment sets a unique tag on all the resources created
by the deployment. After this test is finished, the API is queried for all
the resources with the tag and then, they're deleted in the right order.
Requests to download.fedoraproject.org are redirected to a regional
mirror that is fairly close to the system that is making the request.
This could speed up downloads slightly since it avoids using the
Fedora proxies.
Signed-off-by: Major Hayden <major@redhat.com>
Prior this commit the ami image type produced raw.xz images. This was bad for
two reasons:
- The upload was broken because AWS doesn't support tar.xz format
- XZ compression is terribly slow
This commit changes the format to vhdx, which is supported by AWS and also
quite quick. See https://github.com/osbuild/osbuild-composer/issues/257
why vhdx was chosen.
Fixes#257
go mod works differently in go 1.12 and go 1.13. When someone uses the
prepare-source tool with go >= 1.13, the ci complains because it uses
go 1.12. This commit changes the script to use a well-defined go version.
In PR #400:
https://github.com/osbuild/osbuild-composer/pull/400
we changed the boot type of AWS images from qemu-extract to aws, which
resulted in a bug in the generate-test-case script. It tried to attach
xz archive using qemu-nbd which works just fine from qemu-nbd
perspective but sfdisk and blkid of course failed, because the xz
archive is not a disk image.
This patch makes sure we extract the image before attaching it to
/dev/nbdX.
The current implementation does not report any errors because it uses
the subprocess.check_output function from Python std library. The
function is convenient to use but it hides all the errors from us which
makes it very inconvenient when the test generation often fails (like
developing support for alternative architecture). This patch introduces
a wrapper around the subprocess.run function which dumps the
subprocess's stdout to stderr in case of failure and forwards it to the
caller in case of success. This way the generation code is still very
straight forward, but the errors are reported properly.
Prior this commit the ami images were tested locally using qemu. This does
not reflect at all how they're used in practice. This commit introduces
the support for running them in the actual AWS. Yay!
The structure of code reflects that we want to switch to osbuild-composer
to build the images soon.
Rather than use the round-robin mirror, use a fixe stable one.
We should not over-use this, but it seems the round-robin one is
simply too unstable for our tests.
Eventually, we want to use metalink here instead, but we need to
rework the file source first, so we can do that in a way that
does not make the tests dependent on the time/place the test
case was generated (currently the URL to a specific mirror ends
up pinned in the test-case).
Signed-off-by: Tom Gundersen <teg@jklm.no>
Require repository informaiton to be passed as input, rather than
read from the current directory.
Reading from the repository informaiton meant to be used by weldr
has several drawbacks.
- it makes it impractical to use the tool outside a git checkout
- it makes it awkward to adapt the repositories to different use
cases
- it means that the shipped repositories cannot be extended with
update repos, as the same repos are used for testing, and that
would render our tests non-reproducible.
Overall, we are moving towards making repositories something the
caller must always pass in, rather than something that composer
maintains. For the weldr API we need to keep working as before,
but for new APIs we are avoiding that.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This means that the unit tests no longer need to load the
repositories from the git repo, and in a follow-up, osbuild-composer
won't need to either.
By splitting the repositories used for testing from the system
repositories available through the weldr API we are able to extend
the system repositories without affecting the reproducibility of
the tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
systemctl list-unit-files doesn't produce machine readable output.
parse_unit_files isn't very good at reading it and can produce duplicate
records.
This commit fixes it by deduplicating and sorting the units. This is a bit
hacky solution, but should work just alright. In the future we might
dump list-unit-files and do the job ourself, but let's not recreate the
systemd logic for now.
This script generates json test cases for each supported output format.
It requires an architecure, distro, osbuild store, and output directory
as input. There is also a json object which maps from output format to a
test case request which includes the compose request and boot type.
The script uses these test case requests to call the generate-test-case
script and then outputs the resulting json test cases to files in the
specified output directory.
The generate-test-cases no longer maps from output format to boot
type. Instead, the boot data is passed in through the standard input
alongside the composer-request object.
The script generate-test-cases only creates a single test case per run.
Therefore, the name should be singular. Also, this script has been moved
to a directory which will contain all test case generators.
This includes the packgase and build-packages used by each pipeline.
For now, this information is not used anywhere, but when we move
from dnf to rpm-based pipelines, this is what will be used instead
of the repo metadata checksum.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For tarballs, this is currently not supported, so no point in generating
the images in the first place. This will still be done during testing to
boot-test them.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The test case json files will increase in complexity with the move from
dnf to json. They quantity of them will also continue to grow as new
distros, architectures, boot methods, image types, and blueprint
customizations become available. The generate-test-cases script
simplifies the process of creating new test cases. It accepts a compose
request and boot method as input and then uses osbuild-pipeline,
osbuild, and image-info to generate the test case.
[tomegun: some clean-ups and allow store to be reused]
A script that runs various go tools (mod tidy, mod vendor, and fmt for
now).
The idea is that it prepares the source to be ready for master. As such,
running it on master shouldn't modify any files. Make sure of that by
adding a test.
If we detect a ESP (via its type UUID) remember it and mount it at
any filesystem that has a /boot/efi (there should in theory be
only one). This is needed so grubenv can be read, which is likely
a link from /boot/grub2/grubenv to ../efi/EFI/$vendor/grubenv.
Additionally this will make rpm verify not report that all the efi
binaries are missing, e.g.:
"missing": [
"/boot/efi/EFI",
"/boot/efi/EFI/BOOT",
"/boot/efi/EFI/BOOT/BOOTIA32.EFI",
"/boot/efi/EFI/BOOT/BOOTX64.EFI",
"/boot/efi/EFI/BOOT/fbia32.efi",
"/boot/efi/EFI/BOOT/fbx64.efi",
"/boot/efi/EFI/fedora",
"/boot/efi/EFI/fedora",
...
These tests (will) test more than just image-info: they'll take a
blueprint, verify that `osbuild-pipeline` generates the correct
pipeline, run osbuild with that pipeline and verify that the resulting
image has the expected image-info output.
This change only includes the latter half (i.e., only moves the already
existing tests).
Also drop python's unittest. It was hard to control output (important
for quickly spotting failures and to make travis happy). This introduces
test/run, which runs all test cases in test/cases or the ones given on
the command line.
When a failure occurs, it prints a diff of the actual and the expected
image info.
The main difference (according to image-info) is an additional package
containing a gpg key which was used to verify packages. The one
generated by lorax-composer doesn't have this, because it doesn't verify
signatures.
Also, lorax generates an image with a separate /boot partition. This is
not yet addressed here, because osbuild doesn't support it yet.
The main difference (according to `rpm -qa`) is an additional package
containing a gpg key which was used to verify packages. The one
generated by lorax-composer doesn't have this, because it doesn't verify
signatures.
The main difference (according to image-info) is an additional package
containing a gpg key which was used to verify packages. The one
generated by lorax-composer doesn't have this, because it doesn't verify
signatures.
Also, lorax generates an image with a separate /boot partition. This is
not yet addressed here, because osbuild doesn't support it yet.
The main difference (according to image-info) is an additional package
containing a gpg key which was used to verify packages. The one
generated by lorax-composer doesn't have this, because it doesn't verify
signatures.
This is the output of disk info ran against the images produced
by the specified pipelines.
Skip the actual test for now, because it is taking too long to run.
Signed-off-by: Tom Gundersen <teg@jklm.no>
A test case is now expressed purely by dropping in a json document in
`tools/test_image_info/pipelines`. It should contain the input compose
(output format and blueprint) as well as the expected pipeline. All the
existing tests are moved over to this format.
This shares the same infrastructure as the image tests, ideally we want
to run the blueprint tests and the image tests against the same pipelines.
For now, test cases are skipped from the blueprint tests if they do not
contain a 'compose' section, and from the image tests if they do not
cotain an 'expected' section. In the future we may want to make both
mandatory.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add two kinds of tests, with one case each:
1. Run image-info against an osbuild pipeline. Uses osbuild
from a submodule to make an image from the pipeline.
2. Run image-info against an existing image, fetched from the internet.
This shows the changes an image has relative to what its rpm database
thinks is installed. Output is:
"rpm-verify": {
"missing": [ <missing files> ],
"changed": { <map from filename to rpm attribute octet > }
}
Alas, this makes running image-info slower.
