Dracut is unfortunately very host-dependant by default. The package
dracut-config-generic forces it use a generic configuration instead of a
configuration generated from the host environment.
This change should make the image generation more reproducible. For example
it was not possible to boot ami images built on Travis on AWS prior this
commit.
Also, the tests were re-generated in this commit.
In PR#395 we discussed the spelling of archs vs. arches and we agreed to
use arches. This patch only renames the public method `ListArchs`in the
`Distro` interface.
The main reason here is not to make sure we have the right set of build
packages, but rather to make sure the pointer hierarchy (distro->arch->
image type) works as expected.
This patch uses the same approach as fedora31 few commits ago. It moves
the arch pointer to the imageType struct and sets it while creating the
distro->arch->image type hierarchy in the setImageType function. The
rest of the patch is just renaming.
It uses the same approach as fedora31 few commits ago. The pointer to
distro is moved to "arch" and "arch" now contains a map of allowed
image types for each arch.
Also include setArches and setImageTypes helper functions to ease the
creation of pointers from the structures to a parent structure while
creating the distro->arches->image types hierarchy.
This information is now provided only when an architecture is specified,
so it is necessary to first obtain object implementing the Arch interface
then object implementing the ImageType interface and then you can get
the filename and mime type.
Tests are changed accordingly to the new API.
xz compression is used for the ami and tar output types, it should be in
the buildroot. This fixes Weldr integration test (/cmd/osbuild-tests) on
ARM architecture (aarch64).
The fact that it worked on x86_64 was just a happy coincidence because
we require grub2-pc which in turn requires dracut which requires xz. We
should not rely on these implicit dependencies because we need xz
unconditionaly, therefore adding it to build packages for all platforms.
Delete unused methods and make types and fields private where
possible. Some code is moved around, but apart from that there
is no change in behavior.
The naming of the distros were moved back into the distro
packages as the common types now only had one user, and this
allowed us to drop some redundant error checking.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Objects implementing these interfaces will represent the
architecture support for a given distro and the image type
support for a given architecture distro combination, respectively.
The idea is to always resolve to these objects early, and drop
the equilavent methods from the distro interface. This means that
we convert our input strings to real objects once, and then never
have to verify their correctness again.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Simplify the code by dropping the potential error return. The
constructor simply instantiates some maps, this cannot fail.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Mixing the way to build a distribution with where to get the source
packages from is wrong: it breaks pre-release repos, local mirrors, and
other use cases. To accommodate those, we introduced
`/etc/osbuild-composer/repositories`.
However, that doesn't work for the RCM API, which receives repository
URLs to use from outside requests. This API has been wrongly using the
`additionalRepos` parameter to inject those repos. That's broken,
because the resulting manifests contained both the installed repos and
the repos from the request.
To fix this, stop exposing repositories from the distros, but require
passing them on every call to `Manifest()`. This makes `additionalRepos`
redundant.
Fixes#341
This makes two changes simultaneously, to avoid too much churn:
- move accessors from being on the blueprint struct to the
customizations struct, and
- pass the customizations struct rather than the whole blueprint
as argumnet to distro.Manifest().
@larskarlitski pointed out in a previous review that it feels
redundant to pass the whole blueprint as well as the list of
packages to the Manifest funciton. Indeed it is, so this
simplifies things a bit.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Conceptually, we used to insert the high-level packages and package
groups into the pipeline together with the expected repository
metadata checksum.
osbuild, using the dnf stage, would then fetch the metadata, verify
that its checksum is correct, compute the dependencies, and install
the packages.
Among the problems this has is that it made it impossible to cache
and share the resolved metadata as well as the rpms. Moreover,
as the checksum was at the repository-level, rather than at the
package level, it meant that we would refuse to build a pipeline
as soon as there were any changes at all to the repository, as we
could no longer guarantee the installed packages would be the same.
As of this patch, all repository and metadata handling is done by
composer, rather than osbuild. This means that the resolved metadata
can be cached between runs, which and it means that we can now
pin individual packages, rather than the entire repository. Meaning,
that as long as the rpms are still available, we are able to build
a pipeline.
The downloading of rpms is now done by a source helper in osbuild,
which means that they can be cached and shared between runs too.
One consequence of this change is that we resolve the location of
each rpm in composer, and pass that to the worker. As the worker
may not be in the same location, we do not want to use metalinks
in composer for this, as it would pin the repository closest to
composer, rather than the runner. Instead, we now manually select
a baseurl for each repository, which should be generally the
most useful one. Fedora helpfully provides such baseurls, so
this should work ok.
The most important thing to verify when checking this commit, is
that the image info in our test-cases remains unchanged.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For now, this simply wraps Pipeline and Sources, and retruns the
resulting manifest object. In the future, Pipeline and Sources
may be dropped from the interface.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Return errors from all distro's New() functions instead of logging and
returning nil. Also, return errors instead of panicking from
NewRegistry() and NewDefaultRegistry().
These packages (and their tests) shouldn't access the distro package,
because that's cyclic.
Also, these packages should only test the objects they expose.
WithSingleDistro() doesn't follow go's naming convention for creating
objects (New*). Rename it to NewRegistry() and rename the old
NewRegistry() to NewDefaultRegistry().
The idea is that NewRegistry() can be used to create full Registry
objects from outside the package. NewDefaultRegistry() is a convenience
function that creates a Registry with all known distros.
This outputs the sources needed for the pipeline generated for the
distro. At the moment no pipelines require sources, and so this
always returns the empty list.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is unused for now, but will allow us to generate pipelines with
the pre-depsolved NEVRAs, so osbuild does not need to depsolve again.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is needed for depsolving, so expose it from the distro package
so it can be passed to dnf-json (and not only to osbuild) as that does
depsolving too.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rename the package from `pipeline` to `osbuild` to reflect that it
will no longer be specific to pipelines, but rather covers all
osbuild datatypes.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When a user does not define the image size for a compose the default
image size of that image type is used. In order to properly store the
compose's image size even if the default is used the store calls the
distro function GetSizeForOutputType. This function accepts an output
format and image size. If the image size is 0 then the default
value for the output format will be returned. Also, for vhd images the
size must be rounded. This is now handled in the distro function instead
of the api.
When creating a pipeline the assembler includes an image size. This
image size can be set when creating the pipeline but if it is 0 then a
default image size will be used. The default is 2 GB except for ami
images which are 6 GB.
During development of a new distro, we need to test composer against
nightly or beta repositories, but we cannot ship composer itself
with the nightly repository information hardcoded in. At the same
time, we want to distinguish between the system repositories of the
host and the repositories we use to generate images (the host may not
use the same distro/version/architecture as the target, and it may
include custom repositories that the target should not).
We therefore ship per distro repository information that can be
overriden (typically in testing) by dropping files in /etc.
For now use the latest nightlies for RHEL-8.2, we may want to
replace these with the official mirrors for GA eventually.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Introduce a DistroRegister object. For now this does not introduce
any functional changes, as the object is always instantited to be
the same. However, in follow-up patches it will get options.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Right now, there is no way to get at the name from a Distro instance.
We will need this to include the distro's name in the job we pass to the
worker, for instance.
On architectures that require EFI, we must create the ESP partition
and use a GPT partition table. We must also install either the UEFI
or the legacy version of GRUB2 in the image.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Move to the new options format, allowing more flexible partition
tables. The pipeline changes, but the result should be the same.
This requires a yet-to-be-released version of osbuild.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Allow bootloader specific packages to be defined per architecture,
and allow repositories to depend on the architecture.
This does not altert he pipelines we produce, part from the ami
image now contains the grub2-pc package, rather than the grub2
package. This should make no difference.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The pipeline generation now takes the architecture as an argument.
Currently only x86_64 is supported. The architecture is detected
at start-up, and passed down to each pipeline translation.
For osbuild-pipeline we now requrie the architecture to be passed
in.
Signed-off-by: Tom Gundersen <teg@jklm.no>
