Instead of reading the arguments from sys.stdin, which requires
that stdin is setup properly for that in the runner, use the new
api.arguments() method to directly fetch the arguments.
Also fix missing newlines between imports and methods to be more
PEP-8 complaint, where needed.
Generate and report metadata about all the packages that were
installed. This information will be needed by composer, especially
the 'sigmd5' bit, for integration with koji[1].
[1] https://docs.pagure.org/koji/content_generator_metadata/
The `script` and `test` stages should not be used in produciton, and
their use should be discouraged in general. They may make sense for
debugging, but should not be shipped.
The test stage is still used by the boot tests, so leave that for now,
and only drop the scripts stage.
Signed-off-by: Tom Gundersen <teg@jlkm.no>
The documentation states that the stage accepts any options and
dumps them but the schema was not allowing any. Fix that and also
change the sample to include a random option.
It should be possible to use the stage to only set the default
target or disable services. There is no need to always require
having `enabled_services`.
Add the ability to mask services, which is done e.g. when building
installers. See systemctl(1) for more information about masked
services.
Modify the existing stage test to include a test for it.
Add a new `labels` option that can contain `path`: `label` pairs
to overwrite the default labels for `path`.
NB: These manually set labels will not survive a relabeling and
are most useful to adjust policy in the buildroot, e.g. for `cp`
to be able to copy labels unknown to the host, by labeling it
as `system_u:object_r:install_exec_t:s0`.
This brings it in line with the JSON schema. Also fix a bug where the
wrong root was used for checking the signatures.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add a new stage option `initramfs-args`, that will in turn be set
on the Treefile that is then passed to rpm-ostree. This will pass
the options on to dracut when building the initramfs module. NB:
the `ostree` dracut module will also always be automatically
included by rpm-ostree.
This stage will create a file '/boot/ignition.firstboot' that will,
with the help of support in grub, trigger ignition on the first
boot. The `network` option can be used to overwrite the default
network configuration set in grub2.
Add support for ignition[1] via a new `ignition` stage option. If
enabled, a new section is added to the main grub.cfg that will
create a 'ignition_firstboot' variable meant to be included in the
kernel command line configuration.
The grub.cfg snippet was taken from 'src/grub.cfg' of Fedora CoreOS
Assembler[2] at ec05cde20d3449fab8e4c76493ffa1ebd9b0b626 but with
PR #1373 applied to not hard-code the dhcp options.
[1] https://github.com/coreos/ignition
[2] https://github.com/coreos/coreos-assembler/
When adding a new user with a uid that already exists, the "-o",
option needs to be added. Always do so when the uid is specified.
A use case for this is e.g. an installer iso with a `install`
user that has 0 (like root) for its uid (and gid).
Support setting uids, gids with values of `0` as well as passwords
and descriptions with the empty string, by explicitly checking the
value of each against `None`, because simple `if` conditionals are
false for those.
The content hash of each RPM is already verified, so verifying
signatures again is not necessary if the manifest generation is trusted,
and verifying signatures does not help if the manifest generation is
not.
Let us follow what DNF does and default to not verify signatures, but in
order to preserve features already in use we still allow opting in to
verifying signatures as before on a per RPM basis.
This will make it possible to install unsigned RPMs, or a mixed of
signed and unsigned RPMs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Extract the grub.cfg redirect config as GRUB_REDIRECT_TEMPLATE,
meant to be used via python's string.Template class. Document
its intended use and also the template options.
Extract the strings for grub.cfg into a GRUB_CFG_TEMPLATE multi-line
string and turn it into a template meant to be used via python's
string.Template class. Document it, especially the template options.
Instead of having two different places within conditionals where
the configuration is written, have only one common. In the case
of hybrid boot, in the end there will be two grub configuration
files: the canonical one in /boot/grub2/grub.cfg and a redirect
one, in the EFI directory that will redirect to the canonical.
In case of legacy only, only the canonical one in the default
location (/boot/grub2/grub.cfg) will be written.
For EFI only mode, only /boot/efi/EFI/<vendor>/grub.cfg will
be written with the main grub configuration data.
Thus the writing of the main grub configuration will now always
be written in exactly one place (now line 319).
Move the write_grub_cfg_redirect to the new GrubConfig object as
write_redirect. Add a `separate_boot` property to be used by the
new write_redirect. Remove the corresponding variable and also
the `grub_fs` variable since that is now all handled by the
GrubConfig object.
The reason behind this is to combine all the necessary state in the
object instead of passing it all to the write_grub_cfg function.
The idea is that as more things will get configurable, say the
timeout or ignition support, more things need to be passed to it
and thus it is better to an object where these config options can
be set and then combined when writing the config.
Fedora CoreOS[1] uses a dracut module[2] together with a systemd
generator[3] to mount the file system, including the root one.
Thus neither '/etc/fstab' nor a `root=` kernel command line
option is needed. Support that use case by making the 'rootfs'
option optional.
[1] https://github.com/coreos/fedora-coreos-config/tree/testing-devel/
[2] overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree
[3] overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator
The ignition-dracut module for Fedora CoreOS and anaconda both have
code to populate '/var' via systemd-tmpfiles. In images that where
said dracut module is not used, but '/var' needs to be populate, it
can no be done by setting the `populate_var` option.
Fix all occurrences of format-strings without any interpolation. pylint
warns about those (and for some reason did not do so for our modules).
A followup will fix the pylint tests, so make sure all the warnings are
resolved.
For all currently supported modules, i.e. stages and assemblers,
convert the STAGE_DESC and STAGE_INFO into a proper doc-string.
Rename the STAGE_OPTS into SCHEMA.
Refactor meta.ModuleInfo loading accordingly.
The script to be used for the conversion is:
--- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< ---
import os
import sys
import osbuild
import osbuild.meta
from osbuild.meta import ModuleInfo
def find_line(lines, start):
for i, l in enumerate(lines):
if l.startswith(start):
return i
return None
def del_block(lines, prefix):
start = find_line(lines, prefix)
end = find_line(lines[start:], '"""')
print(start, end)
del lines[start:start+end+1]
def main():
index = osbuild.meta.Index(os.curdir)
modules = []
for klass in ("Stage", "Assembler"):
mods = index.list_modules_for_class(klass)
modules += [(klass, module) for module in mods]
for m in modules:
print(m)
klass, name = m
info = ModuleInfo.load(os.curdir, klass, name)
module_path = ModuleInfo.module_class_to_directory(klass)
path = os.path.join(os.curdir, module_path, name)
with open(path, "r") as f:
data = list(f.readlines())
i = find_line(data, "STAGE_DESC")
print(i)
del data[i]
del_block(data, "STAGE_INFO")
i = find_line(data, "STAGE_OPTS")
data[i] = 'SCHEMA = """\n'
docstr = '"""\n' + info.desc + "\n" + info.info + '"""\n'
doclst = docstr.split("\n")
doclst = [l + "\n" for l in doclst]
data = [data[0]] + doclst + data[1:]
with open(path, "w") as f:
f.writelines(data)
if __name__ == "__main__":
main()
Copies files obtained via a `source` to the tree. Multiple files or
directories can be copied by specifying multiple entries in `paths`.
If no paths are specified the whole contents of `source` is copied.
The source and the target path for each individual path entry might
optionally be specified via `from` and `to`, respectively; if no
path is given for any of the two, the root `/` is assumed.
Currently only an 'archive' 'source' is supported that in turn uses
the existing 'org.osbuild.files' source to fetch an archive (tarball)
and extracts it to a temporary directory.
Change all the schemata to not allow additional properties. This
should help with misspelled properties as well as missing schema
information in the stage itself.
Done via a small python3 script:
--- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< ---
import os
import sys
def list_stages(base):
return [(base, f) for f in os.listdir(base) if f.startswith("org.osbuild")]
stages = list_stages("stages")
stages += list_stages("assemblers")
def find_line(lines, start):
for i, l in enumerate(lines):
if l.startswith(start):
return i
return None
NOADD = '"additionalProperties": false'
for stage in stages:
with open(f"{stage[0]}/{stage[1]}", "r") as f:
print(f"{stage[0]}/{stage[1]}", file=sys.stderr)
data = f.readlines()
i = find_line(data, 'STAGE_OPTS = """')
if i:
data.insert(i+1, NOADD + ",\n")
else:
i = find_line(data, 'STAGE_OPTS = ""')
if i:
data[i] = f'STAGE_OPTS = """\n'
data.insert(i+1, NOADD + "\n")
data.insert(i+2, '"""\n')
with open(f"{stage[0]}/{stage[1]}", "w") as f:
f.writelines(data)
Drop the `osbuild -> ../osbuild` symlink from all module directories.
We now properly initialize the PYTHONPATH to provide the imported
osbuild module from the host environment. Therefore, these links are no
longer needed.
The sources run from the host environment, so they should just pick them
up from the environment the same way osbuild itself does.
Two cleanups for the context-managers we use:
* Use `contextlib.AbstractContextManager` if possible. This class
simply provides a default `__enter__` implementation which just
returns `self`. So use it where applicable.
Additionally, it provides an abstract `__exit__` method and thus
allows static checks for an existance of `__exit__` in the dependent
class. We might use that everywhere, but this is a separate
decision, so not included here.
* Explicitly return `None` from `__exit__`. The python docs state:
If an exception is supplied, and the method wishes to suppress
the exception (i.e., prevent it from being propagated), it
should return a true value. Otherwise, the exception will be
processed normally upon exit from this method.
That is, unless we want exceptions to be suppressed, we should
never return a `truthy` value. The python contextlib suggest using
`None` as a default return value, so lets just do that.
In particular, the explicit `return exc_type is None` that we use
has no effect at all, since it only returns `True` if no exception
was raised.
This commit cleans this up and just follows what the `contextlib`
module does and returns None everywhere (well, it returns nothing
which apparently is the same as returning `None` in python). It is
unlikely that we ever want to suppress any exceptions, anyway.
For historical and occult reasons the grubenv file is, according
to its documentation[1] a 'preallocated 1024-byte file'. The
unused space in the file needs to be filled with '#' as padding,
which tools will count as "free space"[2] and there must not be a
trailing new-line.
Fix our code to do as they say to make grub2-editenv work and in
turn greenboot.
[1] https://www.gnu.org/software/grub/manual/grub/html_node/Environment-block.html
[2] grub-core/lib/envblk.c#L105 (commit 0f102b9844f852d48501d231d32a17e1cc24062d)
Allow file systems to be identified via there label in addition to
their uuid; i.e. either `uuid` or `label` must be specified, which
results in either `UUID=<uuid>` or `LABEL=<label>` to end up in the
"fs_spec" field. See also fstab(5).
In addition to support for identifying file-systems via their uuids,
they now can be identified via their label as well. Two new options
are introduce for this: `rootfs` and `bootfs` for the root and boot
file system. The latter is option in the case a separated partition
is used for /boot. Both options are an object that can either have
`uuid` or `label` set. The old uuid based options, `root_fs_uuid` &
`boot_fs_uuid` are still supported for now.
Additionally, remove the `GRUB2_ROOT_FS_UUID` option from the
grubenv file and directly write the root file system identifier into
the grub config file.
This stage has been replaced by the org.osbuild.rpm stage. The
latter does not need access to network due inside the container
due to its use of the osbuild sources API.
This stage runs a given command only on the first boot of the image,
useful for doing instantiation tasks that can only be done in the
target environment, or that should be done per-instance, rather
than per image.
Ideally we would use systemd's ConditionFirstBoot for this, but that
requires images to ship without an /etc/machine-id, and currently
we only support shipping images with an empty /etc/machine-id.
Changing this would mean dropping /etc/fstab in favor of mounting
the rootfs rw from the initrd. This is likely the right thing to
do regardless, but we would have to audit what other first-boot
services we would end up with pulling in in this case.
Instead we introduce our own flag file /etc/osbuild-first-boot,
and use ConditionPathExists.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The file `/etc/defaults/grub` sets the defaults that are used by
grub2-mkconfig to (re-)generate the grub config (grub.cfg). This
command is not run by any scripts but by the user directly. On
modern installations (without the grubby-deprecated package)
the kernel is configured via Bootloader Specification snippets
and thus the grub config should not need to be touched at all
under normal circumstances. In the new future the grub2-mkconfig
will be updated to not require GRUB_ENABLE_BLSCFG which should
make the existence `/etc/defaults/grub` even more superfluous.
Additionally, in the future, some images might not contain
the grub2 packages at all.
Add support for copying EFI data from the build root. If
`uefi.install` is set to `true`, `BOOT` and `uefi.vendor`
directories will be copied from the build root. This is
useful for example on OSTree based systems where boot/efi/EFI
is not being populated by an RPM package; but it can be used
also on other systems where it is not desirable to deliver
the EFI data via packages.
