The paths for the kernel and the initrd in the BLS snippets are
meant to be relative to the root of the filesystem they are on.
The current code assumes that kernel and initrd are installed
under '/boot' and that '/boot' is on the root file system and
thus all paths get fixed up to start with '/boot/…'. But the
'/boot' directory can be on a separate partition and thus file
system, and then paths need to be relative to that and should
be fixed up with '/…'. Introduce a new option 'prefix' that
can be used to manually specify the prefix after the fixup,
defaulting to '/boot' for backwards compatibility.
NB: The canonical Boot Loader Specification[1] requires that
a separate partition is used boot related files and it will
be mounted at '/boot' (or '/efi').
[1] https://systemd.io/BOOT_LOADER_SPECIFICATION/
Up until now the grub config theoretically supported having different
values for 'root' (via grubenv's $GRUB2_ROOT_FS_UUID) and 'boot' (via
grubenv's $GRUB2_BOOT_FS_UUID). 'boot' is a leftover from the initial
implementation when grub was looking for BLS snippets also in the ESP,
and will be removed in the future.
In our and also the canonical Fedora/RHEL grub configurations the BLS
are in the very same location for UEFI and legacy and thus 'boot' is
always 'root'.
Therefore we get rid of the extra grubenv variable refering to 'boot'
and just set 'boot' to 'root' after that was discovered.
Add a new `boot_fs_uuid` option for when a separate partition is
being used for '/boot' with the indicated uuid. This will then be
used for the grub2 "root" and "boot" variables. Additionally, in
the redirect config, need to refer to files and directories paths
relative to the partition they are contained, i.e. /boot/grub2 if
/boot is on root ('/') or /grub2 if /boot is on a extra dedicated
partition.
The canonical way to set the kernel commandline, which is used by
the kernel post install scripts, see kernel-install(8), is the
file /etc/kernel/cmdline, or in the case this does not exist,
/proc/cmdline. The new stages offers a way to write this file in a
more "type-safe" way, by providing explicit options for certain
well known params (for now only `root_fs_uuid`). Additional params
are specified via `kernel_opts`. This follows the grub2 stage name
convention.
In the case that the image should support booting via EFI *and*
legacy grub, i.e. hybrid booting, the canonical grub config is
stored in /boot/grub2 just as for normal legacy booting. The
config file for efi grub is a very small one that will just look
for the partition containing the /boot/grub2/grub.cfg file and use
then set the prefix accordingly and load that file. In the hybrid
case grubenv file also will just be located in /boot/grub2 and
not in the ESP therefore the symlink that was created by the
package needs to be removed.
In the case that legacy is of type bool it is automatically converted
to the platform string ("i386-legacy"). This is mainly done to keep
backwards comparability as it was just a boolean before. But the auto
conversion did not take the actual *value* of the boolean into account
meaning "legacy: False" would be turned into "legacy: i386-pc" and
thus effectively changing the value from False to True.
dnf skips unavailable repositories by default, which only leads to
harder understand errors later. Configure it to fail when any of the
passed repositories cannot be reached.
Add support for dnf's sslcacert, sslclientcert, and sslclientkey
options. The latter two are passed as secrets (clientcert as well
because it might be a pem file that also includes the private key).
Sources run on the host, so their options may contain paths to the host
file system. Make use of that by accepting only paths in those options,
because it allows using tools to deal with certificate files.
Also make sure that the dnf source only returns options it knows about.
Change the `legacy` option (keeping compatibility) from a boolean
to a string, specifying the target platform the legacy modules
are being installed for.
Pipelines encode which source content they need in the form of
repository metadata checksums (or rpm checksums). In addition, they
encode where they fetch that source content from in the form of URLs.
This is overly specific and doesn't have to be in the pipeline's hash:
the checksum is enough to specify an image.
In practice, this precluded using alternative ways of getting at source
packages, such as local mirrors, which could speed up development.
Introduce a new osbuild API: sources. With it, a stage can query for a
way to fetch source content based on checksums.
The first such source is `org.osbuild.dnf`, which returns repository
configuration for a metadata checksum. Note that the dnf stage continues
to verify that the content it received matches the checksum it expects.
Sources are implemented as programs, living in a `sources` directory.
They are run on the host (i.e., uncontained) right now. Each source gets
passed options, which are taken from a new command line argument to
osbuild, and an array of checksums for which to return content.
This API is only available to stages right now.
This is an expected error case, so we should not assert, but log
and return failure. In the future we should probably also return
the error as structured data.
Signed-off-by: Tom Gundersen <teg@jklm.no>
A simple stage like 'noop' that will return with `returncode` or
255 if nothing is specified. Like 'noop' it might be useful for
testing, debugging, and wasting time.
Introduce two new configuration options: `legacy` and `uefi`. The
first one being a boolean (default: True) that controls if GRUB
modules, fonts and the configuration is installed in the right
locations to support legacy boot mode.
The `uefi` option (of type object with a single `vendor` property)
enables UEFI support by writing the configuration into the correct
EFI directory, "/boot/efi/EFI/<vendor>/grub.cfg", where vendor is
taken from said `vendor` property.
This commit adds semi-structured documentation to all osbuild stages and
assemblers. The variables added work like this:
* STAGE_DESC: Short description of the stage.
* STAGE_INFO: Longer documentation of the stage, including expected
behavior, required binaries, etc.
* STAGE_OPTS: A JSON Schema describing the stage's expected/allowed
options. (see https://json-schema.org/ for details)
It also has a little unittest to check stageinfo - specifically:
1. All (executable) stages in stages/* and assemblers/ must define strings named
STAGE_DESC, STAGE_INFO, and STAGE_OPTS
2. The contents of STAGE_OPTS must be valid JSON (if you put '{' '}'
around it)
3. STAGE_OPTS, if non-empty, should have a "properties" object
4. if STAGE_OPTS lists "required" properties, those need to be present
in the "properties" object.
The test is *not* included in .travis.yml because I'm not sure we want
to fail the build for this, but it's still helpful as a lint-style
check.
It turns out that rpm will happily check signatures on `--install`,
that's just not the default behavior, because of Historical Reasons.
This commit enables RPM's signature checking and drops our manual check,
which will probably speed up the RPM stage a little bit. Fun!
Oh, also there's two bonus code cleanups: one to use f-strings harder,
and one to make sure we ignore whitespace in package checksum strings.
As a general rule, using temporary files with predictable names is a
security risk. It probably isn't _actually_ a security risk inside
osbuild stages, since they're usually running in some kind of isolated
container environment, but it's still a better idea to use tempfiles.
This makes the dnf and yum stages put their temporary files into a
temporary directory that gets deleted after dnf/yum finishes.
lorax-composer supports modifying timeservers, this stage implements it.
I was concerned if I should name this stage timeservers or chrony, but
I've decided to go with chrony. If some day in future Fedora/RHEL
changes the ntp client, we can easily introduce new stage named after
the new ntp client. Additionally, this solution enables us to create
systemd-timesyncd stage, which can change timeservers when chrony is not
installed (in that case systemd-timesyncd takes over the ntp
synchronization).
Downloading the gpg key is fragile and kept causing our tests to fail.
In general, we want to limit the network access, so let's just embed
the gpg keys directly in the pipeline.
Fixes#133.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Background:
grub2 works in three stages:
- The first stage is found in the first 440 bytes of the master
boot record, and its only purpose is to load and execute the
second stage. This stage is static, and just copied from the rpm
without modification.
- The second stage is found in the gap between the MBR and the
first partition, and may be up to 31kB in size. This stage is
specific to the host and must contain the instructions for
finding the right file system and subdirectory for the grub2
config and modules on the host, as well as the modules needed
to do this.
- The third stage is found in the `normal` module, which loads
grub2.conf, which in turn may load more modules and perform
arbitrary instructions.
Problem:
grub2-install is responsible for installing all these stages on the
target image. This goes against our design, as modifications outside
the filesystem should happen in the assembler, but modifications to
the filesystem should happen in a stage. In particular, we don't
want the contents of the image to differ in any way from the output
tree that is stored in our content store (the output of our last
stage). This causes a practical problem at the moment, as our
selinux stage is ran before the assembler, and as such the grub
modules do not get selinux labels applied.
It turns out that we could split grub2-install in two as we want,
by passing `--no-bootsector` to it to install only the modules,
and copy/genereta the two first stages as files under /boot and
then run `grub2-bios-setup` to write the stages from /boot into
the image where they belong.
Regrettably, this does not work as both `grub2-install` and
`grub2-bios-setup` introspect the system and block devices they
are being run on to generate the right configuration. This is not
what we want, as we would like to specifcy the config explicitly
and run them independently of the target image. The specific bug
we get in both cases is that the canonical path containing our
object store cannot be found.
Before osbuild this was not a problem, as other installers would
instal and assemble everything directly in the target image as a
loopback device. Something we explicitly do not want to do.
Solution:
This patch essentially reimplements grub2-install, or rather the
parts of it that we need. One change in behavior from the upstream
tool is that we no longer write the level one and level two boot
loaders to /boot before moving them into place, but just write them
directly where they belong (so they do not end up on the
filesystem).
The parts that copy files into /boot are now in the grub2 installer
and the parts that write the level one/two bootloaders are in the
qemu assembler.
This achieves a few principles I think we should always adher to:
- never run tools from the target image (no chroot)
- don't read/copy files from the target image that was written
by other stages. We already try to avoid sharing state, and
by treating the image as write-only, we avoid accidentally
sharing state through the target tree.
Based-on-suggestions-from: Javier Martinez Canillas <javierm@redhat.com>
With-god-like-debugging-and-fixes-by: Lars Karlitski <lubreni@redhat.com>
Signed-off-by: Tom Gundersen <teg@jklm.no>
This allows given packages to be excluded from the transaction. This
is useful if you want to install a group with certain exceptions.
A common thing to do in kicktstart files is:
```
rm -f /boot/*-rescue*
```
By instead excluding the dracut-rescue-config package we end up
with:
```
"deleted_files": [
"/etc/kernel/postinst.d",
"/usr/lib/dracut/dracut.conf.d/02-rescue.conf",
"/usr/lib/kernel/install.d/51-dracut-rescue.install",
"/boot/initramfs-0-rescue-ffffffffffffffffffffffffffffffff.img",
"/boot/vmlinuz-0-rescue-ffffffffffffffffffffffffffffffff"
],
```
Signed-off-by: Tom Gundersen <teg@jklm.no>
Some dnf packages introduce random seed file. If we leave in the tree
it would mean all systems running from the created image would use
the same random seed. This can be potentially dangerous, therefore we
just remove the generated random seed from our images.
Normally, machine ID is generated randomly when dnf installs @Core
group. Unfortunately this isn't helping us with reproducibility
of images.
This commit introduces the concept of fake machine ID. Before dnf
command is run in dnf stage, we set the machine ID to a fake one.
This ensures all the scriptlets requiring machine ID have predictable
outputs.
For example GRUB uses machine-id in file names inside
/boot/loader/entries. With fixed machine ID the file names are always
the same and totally predictable.
Require "checksum" option for each repository, which contains the
checksum of the `repodata/repomd.xml` file. This file (indirectly)
contains checksums for all packages.
Verify that the metadata dnf downloaded to install packages matches that
checksum. This way, this stage will give an error when a reposiory
changed between putting together the pipeline and running it.
This is similar to the previous commit for the dnf stage.
Don't pass through arbitrary options. This means that pipeline repo
objects don't have the same options as yum repo files anymore:
1. Hard code repo name to repo id. The name has no influence on the
resulting image and should thus not appear in a pipeline.
2. Set gpgcheck=1 when gpgkey is given. It defaults to false, which
means that all sample and test pipelines didn't verify packages. It
would have failed anyway, because the container doesn't have the key
referenced in /etc. Change all gpgkeys to refer to the key id and import
them manually.
3. Don't allow lists for baseurl and gpgkey. We can add that if we need
it at some point.
Also be less verbose.
Don't pass through arbitrary options. This means that pipeline repo
objects don't have the same options as dnf repo files anymore:
1. Hard code repo name to repo id. The name has no influence on the
resulting image and should thus not appear in a pipeline.
2. Set gpgcheck=1 when gpgkey is given. It defaults to false, which
means that all sample and test pipelines didn't verify packages. It
would have failed anyway, because the container doesn't have the key
referenced in /etc. Change all gpgkeys to refer to the key id and import
them manually.
3. Don't allow lists for baseurl and gpgkey. We can add that if we need
it at some point.
We've been effectively using the basearch of the host, making the stage
non-reproducible: if the same pipeline was run on machines with
different architectures, it would produce different results. However,
pipelines producing different outputs must be different. Thus, this
patch includes the basearch in the pipeline.
In principle, this allows cross-arch builds. dnf should be the only
stage running binaries from the target tree. This is not yet tested.
