Introduce support for ppc64le (Open Firmware). The main difference
to x86 legacy, i.e. non-efi, is that no stage 1 is required because
the core image is stored on a special 'PReP' partition, which must
be marked as bootable. The firmware then looks for that partition
and directly loads the core from there and executes it.
Introduce a `platform` parameter for the grub installer code which
controls various platform depended aspects, including a) the path
for the modules, b) what modules are compiled into the core, c) if
the boot image is written to the MBR and 4) where to write the core
image, i.e. mbr-gap or PReP partition.
Extract the function that writes the grub2 core to the image file.
The only supported location currently is the MBR gap, which is the
gap between the Master Boot Record and the first partition, which
for historical and performance reasons was aligned to a certain
sector (used to be 64 but now is even larger with 2048). In the
future other locations for the grub2 core will be supported such
as the PReP partition (ppc64le) or bios-boot (GPT hybrid booting).
Make the bootloader selection explicit by introducing a new option
called `bootloader`, which is an object, containing the `type` and
options belonging to the bootloader. For now only boot-loader that
is supported is "grub2".
Instead of hard-coding "msdos1", determine this partition id
dynamically based on the partition table type and the index
of the partition that contains /boot/grub2, which normally is
either a separate boot partition or the root partition. In
order to be able to do so, set the index of each Partition
when the partition information is read back via `sfdisk`.
NB: partition indexes start at 1 for grub2.
The filesystem module that grub2 needs to have in the core image
is the filesystem containing the grub modules, specifically the
"normal.mod", as well as the grub configuration. In the standard
case, which is also what osbuild uses, this is /boot/grub2; thus
we actually do want the filesystem containing that directory and
its type not the root filesystem.
Explain the concept and reason behind the grub2 core as well as the
details behind the selection of the core modules that get included.
Also elaborate a bit on the MBR gap. For more details about this see
https://en.wikipedia.org/wiki/GNU_GRUB#Version_2_(GRUB_2)
NB: This commit also changes the order of the grub modules, which in
turn changes the layout of the core.img and thus the hash value used
in the test; adapt those value to reflect the changed core.img.
Change the `legacy` option (keeping compatibility) from a boolean
to a string, specifying the target platform the legacy modules
are being installed for.
Pipelines encode which source content they need in the form of
repository metadata checksums (or rpm checksums). In addition, they
encode where they fetch that source content from in the form of URLs.
This is overly specific and doesn't have to be in the pipeline's hash:
the checksum is enough to specify an image.
In practice, this precluded using alternative ways of getting at source
packages, such as local mirrors, which could speed up development.
Introduce a new osbuild API: sources. With it, a stage can query for a
way to fetch source content based on checksums.
The first such source is `org.osbuild.dnf`, which returns repository
configuration for a metadata checksum. Note that the dnf stage continues
to verify that the content it received matches the checksum it expects.
Sources are implemented as programs, living in a `sources` directory.
They are run on the host (i.e., uncontained) right now. Each source gets
passed options, which are taken from a new command line argument to
osbuild, and an array of checksums for which to return content.
This API is only available to stages right now.
This is an expected error case, so we should not assert, but log
and return failure. In the future we should probably also return
the error as structured data.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The GPT (GUID Partition Table) standard for partition layout supports
giving partition a name in the Partition object as well as in the
option for the qemu stage when specifying the partition layout.
Introduce a method on the PartitionTable that returns the partition
containing the root filesystem. NB: this does not have to be the
first partition (which could be the EFI partition, or something
else), so we have to iterate through the partitions until we find
it.
Instead of having dictionaries representing the partition table,
partitions and filesystems together with some functions operating
on them, have proper python objects with methods. In the future
these objects could be extract and properly tested as well.
A simple stage like 'noop' that will return with `returncode` or
255 if nothing is specified. Like 'noop' it might be useful for
testing, debugging, and wasting time.
The recent changes removed the {Assembler,Stage}Failed exceptions,
which includes them being thrown from Stage.run and Assembler.run.
Instead result dictionaries are returned even on errors. But the
object store, used as a context manager, relies on exceptions to
detect the error case and thus needs them to cleanup the temporary
objects. Without those exceptions the temporary objects end up in
the store even when the sage or assembler failed.
Restore the old behavior by throwing a generic BuildError exception
from the Stage and Assembler, which will be caught directly in the
pipeline and converted to a result dict.
The nbd device might not be ready after `qemu-nbd --connect` returns,
leading to access errors such as this further down:
sfdisk: cannot open /dev/nbd12: Inappropriate ioctl for device
Fix this by polling the device with `nbd-client --check <device>`.
Also, the nbd device might not be released after `qemu-nbd --disconnect`
returns. Fix this by using `nbd-client --disconnect`, which waits.
This introduces a new test dependency on nbd-client (in the ndb package
on Fedora).
The socket that the osbuild and loop apis should talk on are passed into
their `__init__` function. The caller should be responsible for closing
those sockets.
This already happens in all current callers.
This fixes a non-fatal error on RHEL's python 3.6, because it was
calling `socket.close` on an already-closed socket:
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/base_events.py", line 529, in __del__
self.close()
File "/usr/lib64/python3.6/asyncio/unix_events.py", line 63, in close
super().close()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 99, in close
self._close_self_pipe()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 109, in _close_self_pipe
self._remove_reader(self._ssock.fileno())
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 268, in _remove_reader
key = self._selector.get_key(fd)
File "/usr/lib64/python3.6/selectors.py", line 189, in get_key
return mapping[fileobj]
File "/usr/lib64/python3.6/selectors.py", line 70, in __getitem__
fd = self._selector._fileobj_lookup(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 224, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 41, in _fileobj_to_fd
raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1
Commit 82a2be53d introduced a new return type from `Pipeline.run()`. It
changed the caller in `__main__.py`, but missed that the build pipeline
uses the same function.
A pipeline run only returned logs in the `StageFailed` and
`AssemblerFailed` exceptions. Remove those and always return structured
data instead.
It only returns data for stages that actually ran (i.e., didn't come
from the cache). This is similar to the output in interactive mode.
Also change osbuildtest to be able to deal with output that is larger
than the pipe buffer by using subprocess.communicate().
Based on the UEFI sample (f30-base-uefi.json). NB: the inclusion
of the dracut-config-generic is needed to disable "host-only" for
dracut so the initramfs will include the virtio_blk block device
driver that is needed to mount the root file system when running
the image in qemu.
Add mkfs_vfat and hook it up into the generic mkfs_for_type()
dispatcher function. Install grub2 to the MBR only if the partition
table is of type "MBR".
Introduce two new assembler options `pttype` and `partitions` to
allow fine grained control over how the partition table is created.
The first one controls the partition type, either `mbr` (default,
when the key is missing) or `gpt`; if specified the `partitions`
key must contain a list of objects describing the individual
partitions (`start`, `size`, `type`) together with a `filesystem`
object describing the filesystem (`type`, `uuid`, `mountpoint`) to
be created on that partition.
In the case the `pttype` option is missing, the legacy mode is used
where `root_fs_uuid` and `root_fs_type` need to be specified.
Use the newly available partition information in the install_grub2
method: detect which module to use for the root filesystem and
assert the second stage fits between the MBR and the first partition.
Introduce a generic mkfs_for_type() function that will dispatch
to the correct mkfs function depending on the type. Additionally
refactor the partition creation and mounting code to handle more
than one partition.
Part of the refactoring to support uefi/gpt: the method that creates
the partition table now returns an array of dictionaries corresponding
to the individual partitions that have been created together with the
information for the filesystem that this partition should end up with.
Prepare the stage for uefi/gpt support by extracting the code that
installs GRUB and creates the partitions into its own functions.
Should not have any effect on the actual data written to the image.
Introduce two new configuration options: `legacy` and `uefi`. The
first one being a boolean (default: True) that controls if GRUB
modules, fonts and the configuration is installed in the right
locations to support legacy boot mode.
The `uefi` option (of type object with a single `vendor` property)
enables UEFI support by writing the configuration into the correct
EFI directory, "/boot/efi/EFI/<vendor>/grub.cfg", where vendor is
taken from said `vendor` property.
The workaround of manually linking /lib64 -> /usr/lib64 inside the
container that is needed on s390 is also required on ppc64 because
here the dynamic linker is set to /lib64/ld64.so.2 and the /lib64
link is not created.
Work around a combination of systemd not creating the link from
/lib64 -> /usr/lib64 (see systemd issue #14311) and the dynamic
linker is being set to (/lib/ld64.so.1 -> /lib64/ld64.so.1)
Therefore we manually create the link before calling nspawn
Commit 283281f broke compression by appending the argument last to the
tar command line. It needs to appear before the file.
Fix that and add a test.
[teg: add minor fix]
osbuild currently throws an error when not passing a build environment
on the command line, because the runner is unset. This is annoying on
hosts which only need a runner set, but no build pipeline.
To simplify running osbuild in this common case, introduce
`org.osbuild.host`, which is a runner that is defined to work on the
host that osbuild is installed on. Use this runner by default and
include a symlink to the right runner in the Fedora and RHEL packages.
Also add `runners/org.osbuild.host` to `.gitignore`, so that developers
can set the symlink when running osbuild from the source directory.
Fixes#171
