This stage has been replaced by the org.osbuild.rpm stage. The
latter does not need access to network due inside the container
due to its use of the osbuild sources API.
This stage runs a given command only on the first boot of the image,
useful for doing instantiation tasks that can only be done in the
target environment, or that should be done per-instance, rather
than per image.
Ideally we would use systemd's ConditionFirstBoot for this, but that
requires images to ship without an /etc/machine-id, and currently
we only support shipping images with an empty /etc/machine-id.
Changing this would mean dropping /etc/fstab in favor of mounting
the rootfs rw from the initrd. This is likely the right thing to
do regardless, but we would have to audit what other first-boot
services we would end up with pulling in in this case.
Instead we introduce our own flag file /etc/osbuild-first-boot,
and use ConditionPathExists.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The file `/etc/defaults/grub` sets the defaults that are used by
grub2-mkconfig to (re-)generate the grub config (grub.cfg). This
command is not run by any scripts but by the user directly. On
modern installations (without the grubby-deprecated package)
the kernel is configured via Bootloader Specification snippets
and thus the grub config should not need to be touched at all
under normal circumstances. In the new future the grub2-mkconfig
will be updated to not require GRUB_ENABLE_BLSCFG which should
make the existence `/etc/defaults/grub` even more superfluous.
Additionally, in the future, some images might not contain
the grub2 packages at all.
Add support for copying EFI data from the build root. If
`uefi.install` is set to `true`, `BOOT` and `uefi.vendor`
directories will be copied from the build root. This is
useful for example on OSTree based systems where boot/efi/EFI
is not being populated by an RPM package; but it can be used
also on other systems where it is not desirable to deliver
the EFI data via packages.
Deploying an OSTree commit leads to creation of hardlinks from
the repository to the check out. These will have the correct
SELinux labels, since the files in the repository will have the
correct SELinux labels[1]. But new files are generated in '/etc'
of the new deployment, due to the 3-way configuration merge[2].
Also a new kernel, initramfs and the corresponding Bootloader
Specification entries are created in the global '/boot' dir.
In theory, ostree will set the correct SELinux labels by loading
the SELinux config from the deployment and then setting the
correct security contexts via ostree_sepolicy_setfscreatecon().
But it does so conditionally on is_selinux_enabled(2)[3], which
in our container is FALSE Therefore we have to do the same dance
as ostree does, at least for now, and manually re-label the
affected paths.
[1] Assuming they had the correct label when the commit was made
[2] https://ostree.readthedocs.io/en/latest/manual/deployment/
[3] via ostree_sepolicy_setfscreatecon in ostree-sepolicy.c
line 640 of commit 2c1658538f8fde5813e95e7408d65662a489be91
Add the ability to specify one ore more remotes for the system
repository. The required options for a single remote are its
`name` and the `url`. Optionally one or more branch can be passed
via `branches`. GPG keys can be given via `gpgkeys`; if none are
specified, no gpg verification will be done.
The list of mount points is changed from a list of strings to a
list of objects containing `path` and an optional `mode` value.
The latter can be used to set the mode of the mount point that
will be created in the file system tree. It defaults to 0755,
or 493 in decimal, because JSON does not support octal values.
Instead of taking a raw string for the root file system kernel
option, convert the `rootfs` option to an object that must have
either `uuid` or `label` set. This will then be translated into
the proper kernel command line argument.
Add a new optional `ref` option. When set, a references for the
commit that was pulled with the value of `ref` will be created
in the system ostree repo. It will also be used when deploying
the commit and thus will be set as the origin for it. This is
necessary for updates to work.
Initializes the filesystem via ostree and then pulls a specified
commit and deploys that. Options are `commit`, which is the id
of the commit to pull and deploy, `osname`, which is the name
to be used for the operating system root. The `roofs` together
with `kernel_opts` options are used to build the kernel command
line for the deployment.
Additionally, a `mounts` parameter can be supplied that indicates
file system boundaries. This is needed because ostree uses a hard
link farm, which must not link across said file system boundaries.
A stage that uses rpm-ostree compose to do post-processing of the
tree. Among other things the main steps are:
- moves /etc to /usr/etc
- move /boot to /usr/lib/ostree-boot
- potentially moving /var/lib/rpm
- re-creates the initramfs (dracut)
- stores the treefile at /usr/share/rpm-ostree/treefile.json
- adds altfiles module to nss
- Recompiles SELinux policy (semodule -nB)
- Migrates /usr/etc/{passwd, group} to /usr/lib/
- Postprocess SELinux policy
- Convert /var to tmpfiles.d
- Prepares symlinks
- /usr/local -> /var/usrlocal
- /var/lib/alternatives -> /usr/lib/alternatives
- /var/lib/vagrant -> /usr/lib/vagrant
- copies the rpmdb
Based on commit 1cf0d557ae8059e689b1fed670022727e9842288 (rpm-ostree)
This has now been entirely replaced by the rpm stage in all its
users. The dnf stage does not fit very nicely into the osbuild
module, in particular it requires direct network access, which
we would like to avoid.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For the sake of backwards compatibility, legacy support was enabled
by default. Flip this around, so that leaving the parameter out
means disabling it.
This is more intuitive, and will pave the way for dropping support
for the value being a bool in the future.
`osbuild-composer` always passes the argumnet explicitly, though
still always as a boolean.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Enabling the fastestmirror plugin allows dnf to choose the fastest
(also usually the closest) mirror to the instance of osbuild. It
has no effect on builds that force the use of a specific server
or mirror.
Signed-off-by: Major Hayden <major@redhat.com>
The dnf stage was very verbose, probably overly so. Rather than
benig completely silent, let's print basic progress indication
so we have some idea what's going on.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Drop the rpm downloading and instead use the files source. This gives
us caching for free, and is the last missing step before we can
deprecate the dnf stage.
The main benefit of the rpm over the dnf stage is that we pin the package
versions rather than the repo metadata version. This will allow us to
support continuously changing repositories as individual packages are much
less likely to change than the repos iteself, and old packages are meant
to stay around for some time, unlike the repo metadata which is instantly
swapped out.
Depsolving is also slow on the first run, which we were always hitting as
the depsolving was always happening in a fresh container.
Based on a patch by Lars Karlitski.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The paths for the kernel and the initrd in the BLS snippets are
meant to be relative to the root of the filesystem they are on.
The current code assumes that kernel and initrd are installed
under '/boot' and that '/boot' is on the root file system and
thus all paths get fixed up to start with '/boot/…'. But the
'/boot' directory can be on a separate partition and thus file
system, and then paths need to be relative to that and should
be fixed up with '/…'. Introduce a new option 'prefix' that
can be used to manually specify the prefix after the fixup,
defaulting to '/boot' for backwards compatibility.
NB: The canonical Boot Loader Specification[1] requires that
a separate partition is used boot related files and it will
be mounted at '/boot' (or '/efi').
[1] https://systemd.io/BOOT_LOADER_SPECIFICATION/
Up until now the grub config theoretically supported having different
values for 'root' (via grubenv's $GRUB2_ROOT_FS_UUID) and 'boot' (via
grubenv's $GRUB2_BOOT_FS_UUID). 'boot' is a leftover from the initial
implementation when grub was looking for BLS snippets also in the ESP,
and will be removed in the future.
In our and also the canonical Fedora/RHEL grub configurations the BLS
are in the very same location for UEFI and legacy and thus 'boot' is
always 'root'.
Therefore we get rid of the extra grubenv variable refering to 'boot'
and just set 'boot' to 'root' after that was discovered.
Add a new `boot_fs_uuid` option for when a separate partition is
being used for '/boot' with the indicated uuid. This will then be
used for the grub2 "root" and "boot" variables. Additionally, in
the redirect config, need to refer to files and directories paths
relative to the partition they are contained, i.e. /boot/grub2 if
/boot is on root ('/') or /grub2 if /boot is on a extra dedicated
partition.
The canonical way to set the kernel commandline, which is used by
the kernel post install scripts, see kernel-install(8), is the
file /etc/kernel/cmdline, or in the case this does not exist,
/proc/cmdline. The new stages offers a way to write this file in a
more "type-safe" way, by providing explicit options for certain
well known params (for now only `root_fs_uuid`). Additional params
are specified via `kernel_opts`. This follows the grub2 stage name
convention.
In the case that the image should support booting via EFI *and*
legacy grub, i.e. hybrid booting, the canonical grub config is
stored in /boot/grub2 just as for normal legacy booting. The
config file for efi grub is a very small one that will just look
for the partition containing the /boot/grub2/grub.cfg file and use
then set the prefix accordingly and load that file. In the hybrid
case grubenv file also will just be located in /boot/grub2 and
not in the ESP therefore the symlink that was created by the
package needs to be removed.
In the case that legacy is of type bool it is automatically converted
to the platform string ("i386-legacy"). This is mainly done to keep
backwards comparability as it was just a boolean before. But the auto
conversion did not take the actual *value* of the boolean into account
meaning "legacy: False" would be turned into "legacy: i386-pc" and
thus effectively changing the value from False to True.
dnf skips unavailable repositories by default, which only leads to
harder understand errors later. Configure it to fail when any of the
passed repositories cannot be reached.
Add support for dnf's sslcacert, sslclientcert, and sslclientkey
options. The latter two are passed as secrets (clientcert as well
because it might be a pem file that also includes the private key).
Sources run on the host, so their options may contain paths to the host
file system. Make use of that by accepting only paths in those options,
because it allows using tools to deal with certificate files.
Also make sure that the dnf source only returns options it knows about.
Change the `legacy` option (keeping compatibility) from a boolean
to a string, specifying the target platform the legacy modules
are being installed for.
Pipelines encode which source content they need in the form of
repository metadata checksums (or rpm checksums). In addition, they
encode where they fetch that source content from in the form of URLs.
This is overly specific and doesn't have to be in the pipeline's hash:
the checksum is enough to specify an image.
In practice, this precluded using alternative ways of getting at source
packages, such as local mirrors, which could speed up development.
Introduce a new osbuild API: sources. With it, a stage can query for a
way to fetch source content based on checksums.
The first such source is `org.osbuild.dnf`, which returns repository
configuration for a metadata checksum. Note that the dnf stage continues
to verify that the content it received matches the checksum it expects.
Sources are implemented as programs, living in a `sources` directory.
They are run on the host (i.e., uncontained) right now. Each source gets
passed options, which are taken from a new command line argument to
osbuild, and an array of checksums for which to return content.
This API is only available to stages right now.
This is an expected error case, so we should not assert, but log
and return failure. In the future we should probably also return
the error as structured data.
Signed-off-by: Tom Gundersen <teg@jklm.no>
A simple stage like 'noop' that will return with `returncode` or
255 if nothing is specified. Like 'noop' it might be useful for
testing, debugging, and wasting time.
Introduce two new configuration options: `legacy` and `uefi`. The
first one being a boolean (default: True) that controls if GRUB
modules, fonts and the configuration is installed in the right
locations to support legacy boot mode.
The `uefi` option (of type object with a single `vendor` property)
enables UEFI support by writing the configuration into the correct
EFI directory, "/boot/efi/EFI/<vendor>/grub.cfg", where vendor is
taken from said `vendor` property.
This commit adds semi-structured documentation to all osbuild stages and
assemblers. The variables added work like this:
* STAGE_DESC: Short description of the stage.
* STAGE_INFO: Longer documentation of the stage, including expected
behavior, required binaries, etc.
* STAGE_OPTS: A JSON Schema describing the stage's expected/allowed
options. (see https://json-schema.org/ for details)
It also has a little unittest to check stageinfo - specifically:
1. All (executable) stages in stages/* and assemblers/ must define strings named
STAGE_DESC, STAGE_INFO, and STAGE_OPTS
2. The contents of STAGE_OPTS must be valid JSON (if you put '{' '}'
around it)
3. STAGE_OPTS, if non-empty, should have a "properties" object
4. if STAGE_OPTS lists "required" properties, those need to be present
in the "properties" object.
The test is *not* included in .travis.yml because I'm not sure we want
to fail the build for this, but it's still helpful as a lint-style
check.
It turns out that rpm will happily check signatures on `--install`,
that's just not the default behavior, because of Historical Reasons.
This commit enables RPM's signature checking and drops our manual check,
which will probably speed up the RPM stage a little bit. Fun!
Oh, also there's two bonus code cleanups: one to use f-strings harder,
and one to make sure we ignore whitespace in package checksum strings.
As a general rule, using temporary files with predictable names is a
security risk. It probably isn't _actually_ a security risk inside
osbuild stages, since they're usually running in some kind of isolated
container environment, but it's still a better idea to use tempfiles.
This makes the dnf and yum stages put their temporary files into a
temporary directory that gets deleted after dnf/yum finishes.
lorax-composer supports modifying timeservers, this stage implements it.
I was concerned if I should name this stage timeservers or chrony, but
I've decided to go with chrony. If some day in future Fedora/RHEL
changes the ntp client, we can easily introduce new stage named after
the new ntp client. Additionally, this solution enables us to create
systemd-timesyncd stage, which can change timeservers when chrony is not
installed (in that case systemd-timesyncd takes over the ntp
synchronization).
