Add the initramfs-args Treefile option that can be used to pass
arguments to drauct via rpm-ostree. NB: the ostree module will
always be automatically be included by rpm-ostree.
This stage will create a file '/boot/ignition.firstboot' that will,
with the help of support in grub, trigger ignition on the first
boot. The `network` option can be used to overwrite the default
network configuration set in grub2.
Add support for ignition[1] via a new `ignition` stage option. If
enabled, a new section is added to the main grub.cfg that will
create a 'ignition_firstboot' variable meant to be included in the
kernel command line configuration.
The grub.cfg snippet was taken from 'src/grub.cfg' of Fedora CoreOS
Assembler[2] at ec05cde20d3449fab8e4c76493ffa1ebd9b0b626 but with
PR #1373 applied to not hard-code the dhcp options.
[1] https://github.com/coreos/ignition
[2] https://github.com/coreos/coreos-assembler/
When adding a new user with a uid that already exists, the "-o",
option needs to be added. Always do so when the uid is specified.
A use case for this is e.g. an installer iso with a `install`
user that has 0 (like root) for its uid (and gid).
Support setting uids, gids with values of `0` as well as passwords
and descriptions with the empty string, by explicitly checking the
value of each against `None`, because simple `if` conditionals are
false for those.
The content hash of each RPM is already verified, so verifying
signatures again is not necessary if the manifest generation is trusted,
and verifying signatures does not help if the manifest generation is
not.
Let us follow what DNF does and default to not verify signatures, but in
order to preserve features already in use we still allow opting in to
verifying signatures as before on a per RPM basis.
This will make it possible to install unsigned RPMs, or a mixed of
signed and unsigned RPMs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add support for querying information about sources: add the mapping
from name to directory and accept "Source" as a module name. Adapt
the ModuleInfo schema property to handle the different styles for
stage-like schemata as well as sources now.
Extract the grub.cfg redirect config as GRUB_REDIRECT_TEMPLATE,
meant to be used via python's string.Template class. Document
its intended use and also the template options.
Extract the strings for grub.cfg into a GRUB_CFG_TEMPLATE multi-line
string and turn it into a template meant to be used via python's
string.Template class. Document it, especially the template options.
Instead of having two different places within conditionals where
the configuration is written, have only one common. In the case
of hybrid boot, in the end there will be two grub configuration
files: the canonical one in /boot/grub2/grub.cfg and a redirect
one, in the EFI directory that will redirect to the canonical.
In case of legacy only, only the canonical one in the default
location (/boot/grub2/grub.cfg) will be written.
For EFI only mode, only /boot/efi/EFI/<vendor>/grub.cfg will
be written with the main grub configuration data.
Thus the writing of the main grub configuration will now always
be written in exactly one place (now line 319).
Move the write_grub_cfg_redirect to the new GrubConfig object as
write_redirect. Add a `separate_boot` property to be used by the
new write_redirect. Remove the corresponding variable and also
the `grub_fs` variable since that is now all handled by the
GrubConfig object.
The reason behind this is to combine all the necessary state in the
object instead of passing it all to the write_grub_cfg function.
The idea is that as more things will get configurable, say the
timeout or ignition support, more things need to be passed to it
and thus it is better to an object where these config options can
be set and then combined when writing the config.
Fedora CoreOS[1] uses a dracut module[2] together with a systemd
generator[3] to mount the file system, including the root one.
Thus neither '/etc/fstab' nor a `root=` kernel command line
option is needed. Support that use case by making the 'rootfs'
option optional.
[1] https://github.com/coreos/fedora-coreos-config/tree/testing-devel/
[2] overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree
[3] overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator
The ignition-dracut module for Fedora CoreOS and anaconda both have
code to populate '/var' via systemd-tmpfiles. In images that where
said dracut module is not used, but '/var' needs to be populate, it
can no be done by setting the `populate_var` option.
We currently run pylint on all files we find in the checkout. This is
particularly annoying when we have osbuild-stores in the checkout, which
then contain lots of python files in their stored trees.
Change the pylint test to use `git ls-tree` to find all files in the
index and then only run pylint against the subset that we are interested
in.
Fix all occurrences of format-strings without any interpolation. pylint
warns about those (and for some reason did not do so for our modules).
A followup will fix the pylint tests, so make sure all the warnings are
resolved.
For all currently supported modules, i.e. stages and assemblers,
convert the STAGE_DESC and STAGE_INFO into a proper doc-string.
Rename the STAGE_OPTS into SCHEMA.
Refactor meta.ModuleInfo loading accordingly.
The script to be used for the conversion is:
--- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< ---
import os
import sys
import osbuild
import osbuild.meta
from osbuild.meta import ModuleInfo
def find_line(lines, start):
for i, l in enumerate(lines):
if l.startswith(start):
return i
return None
def del_block(lines, prefix):
start = find_line(lines, prefix)
end = find_line(lines[start:], '"""')
print(start, end)
del lines[start:start+end+1]
def main():
index = osbuild.meta.Index(os.curdir)
modules = []
for klass in ("Stage", "Assembler"):
mods = index.list_modules_for_class(klass)
modules += [(klass, module) for module in mods]
for m in modules:
print(m)
klass, name = m
info = ModuleInfo.load(os.curdir, klass, name)
module_path = ModuleInfo.module_class_to_directory(klass)
path = os.path.join(os.curdir, module_path, name)
with open(path, "r") as f:
data = list(f.readlines())
i = find_line(data, "STAGE_DESC")
print(i)
del data[i]
del_block(data, "STAGE_INFO")
i = find_line(data, "STAGE_OPTS")
data[i] = 'SCHEMA = """\n'
docstr = '"""\n' + info.desc + "\n" + info.info + '"""\n'
doclst = docstr.split("\n")
doclst = [l + "\n" for l in doclst]
data = [data[0]] + doclst + data[1:]
with open(path, "w") as f:
f.writelines(data)
if __name__ == "__main__":
main()
Convert our custom code to list modules to the new ModuleInfo
method list_modules_for_class that does the same thing. This
is then indeed also testing that new function.
Make the mapping of module class to the corresponding directory
a method of the ModuleInfo class. This is so it can be re-used
by others in the future.
This was superseded by test_osbuild's test_moduleinfo. It also
seems to be non-functional do to assuming `properties` in all
the STAGE_OPTS. Removing this.
The are converging on a nomenclature where the sum of Stages,
Assemblers, Sources (and future entities like those) together
are called 'Modules'.
Thus rename StageInfo to ModuleInfo and the corresponding
variables and methods.
Change the assembler-commit to be conditional on checkpoints, just like
we already do for stages. This means, assembler output is not
automatically committed, but only if you requested so via a checkpoint.
With this in place we can start sharing caches in osbuild-composer. The
only thing in the cache will be sources as well as checkpointed stages.
We can start checkpointing known pipelines and thus make use of the
cache. Furthermore, we can cache sources, as long as we do not fetch an
unbound set of RPMs. However, our RPM set is currently static, so this
should not be an issue. Nevertheless, it is up to Composer to decide
when to enable the cache.
We no longer need the `tree_id` in the osbuild output. All callers have
been converted to use other means. Drop the ID from the output and
avoid exposing our internals.
Now that no caller requires the "output_id" anymore, drop it from our
results-dictionary. Instead, pass the output-directory through and copy
outputs where we produce / fetch them.
This still uses `objectstore.resolve_ref()`, since we do not have the
outputs pinned at the places where we want to copy. This needs a little
bit more rework, but we might just delay that until we have the cache
rework landed.
This already simplifies the output-directory path and drops the slight
hack which checked very late for produced outputs.
Note that we must be careful not to copy things too early, because we
do not want remnants in the output-directory if we return failure.
Hence, keep the copy-operation close to the commit-operation on the
store.
All callsites of `Pipeline.assemble()` already check early whether the
output-object exists in the store and then return it. Checking again in
`assemble()` will never catch anything (unless another stage would
happen to produce the same ID as the assembler as a side-effect).
It does seem useful to keep the shortcuts in `assemble()`, so other
callers would get the shortcut as well. However, this does not really
work well right now, since you want to skip the stage-compilation as
well, and `assemble()` is really just the last step of the job. Hence,
it really is the job of the pipeline-executor to check early.
With that in mind, lets drop this fast-path which has no effect in the
current setup.
We used to have a try-except block to catch URL requests that are not in
`urls`. This block has since then grown way bigger than it should be. We
may now accidentally catch KeyError exceptions from lots of other
places.
This commit extracts the accessor of `urls[checksum]` and saves the
result in a local variable and makes the remainder use that variable.
Using `[]` as default value for arguments makes `pylint` complain. The
reason is that it creates an array statically at the time the function
is parsed, rather than dynamically on invocation of the function. This
means, when you append to this array, you change the global instance and
every further invocation of that function works on this modified array.
While our use-cases are safe, this is indeed a common pitfall. Lets
avoid using this and resort to `None` instead.
This silences a lot of warnings from pylint about "dangerous use of []".
We use comments in all other tests, rather than doc-strings. Convert the
os-release test to do the same. If we wanted doc-strings, we can convert
all tests over. This commit just tries to keep the tests in-sync.
Note that doc-strings cause `unittest` to print the doc-strings to
stdout during test-execution, making it overly verbose (especially for
multiline docs). By converting it to comments, this behavior is
suppressed.
Copies files obtained via a `source` to the tree. Multiple files or
directories can be copied by specifying multiple entries in `paths`.
If no paths are specified the whole contents of `source` is copied.
The source and the target path for each individual path entry might
optionally be specified via `from` and `to`, respectively; if no
path is given for any of the two, the root `/` is assumed.
Currently only an 'archive' 'source' is supported that in turn uses
the existing 'org.osbuild.files' source to fetch an archive (tarball)
and extracts it to a temporary directory.
