Attempt to clarify the structure of our tests. Each test case is now
encapsulated in a script in `test/cases`. Each of these scripts should
be runnable on a pristine machine and be independent of each other. It
is up to the test-orchestractor to decide if they should be run
consequtively instance, or in parallel on separate instances. Each
script can execute several tests and call whatever helper binaries
is desired. However, each case should be assumed to always run as one.
No tests should be run directly from git, but should rather be installed
onto the test system using rpm and run from there. This moves towards
unifying our two types of test cases.
The new structure of is now:
`test/cmd`: the executors, one for each test-case. This is installed
into `/usr/libexec/test/osbuild-composer`.
`test/data`: data and config used by the tests. This is installed into
`/usr/share/tests/osbuild-composer`.
`schutzbot`: configuration of the actual test run. In particular, this
is where the distros and repositories to test against are
configured.
This is very much still work-in-progress, and is only the first step
towards simplifying schutzbot. Apart from moving files around, this
should be a noop.
Signed-off-by: Tom Gundersen <teg@jklm.no>
By labeling `cp` with `system_u:object_r:install_exec_t:s0` we allow it
to copy labels unknown to the host.
See also corresponding commit in osbuild:
e80130a830
Previously, cloud-init was installed on qcow and openstack images but was not
enabled, preventing a user from logging in to these images. Cloud-init, cloud-final,
cloud-init-local and cloud-config services are enabled to make cloud-init work
and to maintain consistency with official fedora 31 and fedora 32 images.
The image tests have been updated to reflect these changes.
Fixes#698
Although the vhdx format is smaller than raw, AWS often has some issues
booting instances from an imported vhdx snapshot. The raw format seems
to boot consistently at AWS.
Fixes#754.
Signed-off-by: Major Hayden <major@redhat.com>
Rather than getting a set of base packages from the ImageType, and then
appending the requested packages from the blueprint, pass the blueprint
into the new Packages() function, and return the full set of packages to
be depsolved.
This allows us to also append packages based on other customizations
too, and use that to append chrony when the timezone is set. This
matches the behavior anaconda had, and there was a TODO item to do this,
which had been overlooked.
Fixes#787.
Signed-off-by: Tom Gundersen <teg@jklm.no>
How exactly the final file system layout looks like is determined
by the installer (anaconda) and thus can not be known at commit
creation time. Thus creating an /etc/fstab file is unnecessary and
the information in it probably wrong. The file wont be used though
because it will be overwritten during the installation process.
Rather than passing an array of checksums, pass an array of objects
with package properties. One is the `checksum`, which is required, and
in addition to that we now pass the boolean check_gpg, which indicates
that that RPM should have its GPG signature verified.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The input options of qemu assembler do not require filesystem for every
partition. This patch makes it optional and omits the object in case it
is empty.
osbuild schema definition:
162797f206/assemblers/org.osbuild.qemu (L63)
Partition type, as defined in osbuild qemu assembler, is of type string
because it is either UUID or identifier. See here for details:
162797f206/assemblers/org.osbuild.qemu (L88)
Change type of the "Type" struct member to match the definition in
osbuild. This patch is needed for ppc64le support.
Rather than Manifest() returning an osbuild.Manifest object, introduce a
new distro.Manifest object which represents it as an opaque, JSON
serializable object. This new type has the following properties:
1) its serialization is compatible with the input to osbuild,
2) any valid osbuild input can be deserialized into it, and
3) marshalling and unmarshaling to and from JSON is lossless.
This means that even as we change the subset of valid osbulid manifests
that we support, we can still load any previous state from disk, and it
will continue to work just as before, even though we can no longer
deserialize it into our internal notion of osbuild.Manifest.
This fixes the underlying problem of which #685 was a symptom.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rename the `Fedora32` type to simply `distribution`, to avoid the
stutter. Move `New()` to the bottom of the file, which is the only
non-generic part. Also make the linter happy.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We do not properly test, and do not have properly defined use-cases for
the ext4-filesystem, partitioned-disk, nor tar image types. Drop them to
focus on delivering the things we car properly test.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When generating an osbuild manifest for an image type, we take a
customizations struct, which specifies the image-type-independent
customizations to apply. We also take the size argument, which is
specific to the image build and not part of the blueprint.
Introduce a new argument ImageOptions, which for now just wraps the size
argument. These options are specific to the image build/type, and
therefore does not belong with the other customizations.
For now this is a non-functional change, but follow-up commits will
introduce more types of image options.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Prior this commit the ami image type produced raw.xz images. This was bad for
two reasons:
- The upload was broken because AWS doesn't support tar.xz format
- XZ compression is terribly slow
This commit changes the format to vhdx, which is supported by AWS and also
quite quick. See https://github.com/osbuild/osbuild-composer/issues/257
why vhdx was chosen.
Fixes#257
We need to make partition uuids stable to be able to run image-info
tests on images using gpt. For example all aarch64 images use gpt.
Also change the type of the (part) UUID to string because that's what
the other UUIDs use and it is easier to work with.
There is no point in having the grub2 stage in pipelines for image types
that are not bootable. The current version is probably a result of
previous refactoring where the member variable was named `IncludeFSTab`.
Moving the grub2 stage into the conditional branch should also fix test
generation on aarch64.
Finally it is necessary to regenerate test cases for non-bootable image
types.
Dracut is unfortunately very host-dependant by default. The package
dracut-config-generic forces it use a generic configuration instead of a
configuration generated from the host environment.
This change should make the image generation more reproducible. For example
it was not possible to boot ami images built on Travis on AWS prior this
commit.
Also, the tests were re-generated in this commit.
In PR#395 we discussed the spelling of archs vs. arches and we agreed to
use arches. This patch only renames the public method `ListArchs`in the
`Distro` interface.
The main reason here is not to make sure we have the right set of build
packages, but rather to make sure the pointer hierarchy (distro->arch->
image type) works as expected.
This patch uses the same approach as fedora31 few commits ago. It moves
the arch pointer to the imageType struct and sets it while creating the
distro->arch->image type hierarchy in the setImageType function. The
rest of the patch is just renaming.
It uses the same approach as fedora31 few commits ago. The pointer to
distro is moved to "arch" and "arch" now contains a map of allowed
image types for each arch.
Also include setArches and setImageTypes helper functions to ease the
creation of pointers from the structures to a parent structure while
creating the distro->arches->image types hierarchy.
This information is now provided only when an architecture is specified,
so it is necessary to first obtain object implementing the Arch interface
then object implementing the ImageType interface and then you can get
the filename and mime type.
Tests are changed accordingly to the new API.
xz compression is used for the ami and tar output types, it should be in
the buildroot. This fixes Weldr integration test (/cmd/osbuild-tests) on
ARM architecture (aarch64).
The fact that it worked on x86_64 was just a happy coincidence because
we require grub2-pc which in turn requires dracut which requires xz. We
should not rely on these implicit dependencies because we need xz
unconditionaly, therefore adding it to build packages for all platforms.
Delete unused methods and make types and fields private where
possible. Some code is moved around, but apart from that there
is no change in behavior.
The naming of the distros were moved back into the distro
packages as the common types now only had one user, and this
allowed us to drop some redundant error checking.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Objects implementing these interfaces will represent the
architecture support for a given distro and the image type
support for a given architecture distro combination, respectively.
The idea is to always resolve to these objects early, and drop
the equilavent methods from the distro interface. This means that
we convert our input strings to real objects once, and then never
have to verify their correctness again.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Simplify the code by dropping the potential error return. The
constructor simply instantiates some maps, this cannot fail.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Mixing the way to build a distribution with where to get the source
packages from is wrong: it breaks pre-release repos, local mirrors, and
other use cases. To accommodate those, we introduced
`/etc/osbuild-composer/repositories`.
However, that doesn't work for the RCM API, which receives repository
URLs to use from outside requests. This API has been wrongly using the
`additionalRepos` parameter to inject those repos. That's broken,
because the resulting manifests contained both the installed repos and
the repos from the request.
To fix this, stop exposing repositories from the distros, but require
passing them on every call to `Manifest()`. This makes `additionalRepos`
redundant.
Fixes#341
This makes two changes simultaneously, to avoid too much churn:
- move accessors from being on the blueprint struct to the
customizations struct, and
- pass the customizations struct rather than the whole blueprint
as argumnet to distro.Manifest().
@larskarlitski pointed out in a previous review that it feels
redundant to pass the whole blueprint as well as the list of
packages to the Manifest funciton. Indeed it is, so this
simplifies things a bit.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Conceptually, we used to insert the high-level packages and package
groups into the pipeline together with the expected repository
metadata checksum.
osbuild, using the dnf stage, would then fetch the metadata, verify
that its checksum is correct, compute the dependencies, and install
the packages.
Among the problems this has is that it made it impossible to cache
and share the resolved metadata as well as the rpms. Moreover,
as the checksum was at the repository-level, rather than at the
package level, it meant that we would refuse to build a pipeline
as soon as there were any changes at all to the repository, as we
could no longer guarantee the installed packages would be the same.
As of this patch, all repository and metadata handling is done by
composer, rather than osbuild. This means that the resolved metadata
can be cached between runs, which and it means that we can now
pin individual packages, rather than the entire repository. Meaning,
that as long as the rpms are still available, we are able to build
a pipeline.
The downloading of rpms is now done by a source helper in osbuild,
which means that they can be cached and shared between runs too.
One consequence of this change is that we resolve the location of
each rpm in composer, and pass that to the worker. As the worker
may not be in the same location, we do not want to use metalinks
in composer for this, as it would pin the repository closest to
composer, rather than the runner. Instead, we now manually select
a baseurl for each repository, which should be generally the
most useful one. Fedora helpfully provides such baseurls, so
this should work ok.
The most important thing to verify when checking this commit, is
that the image info in our test-cases remains unchanged.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For now, this simply wraps Pipeline and Sources, and retruns the
resulting manifest object. In the future, Pipeline and Sources
may be dropped from the interface.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Return errors from all distro's New() functions instead of logging and
returning nil. Also, return errors instead of panicking from
NewRegistry() and NewDefaultRegistry().
These packages (and their tests) shouldn't access the distro package,
because that's cyclic.
Also, these packages should only test the objects they expose.
