As noted in earlier commits the grub2 boot image needs to be patched
to contain the position of the grub2 core. By default, the location
in the boot image is hard-coded to be the mbr gap (sector 1) but for
GPT partition schemes a separate BIOS boot partition is used that is
located at a "random" location. Refactor the code to generalize the
boot image patching, where the default mbr gap location is just a
special case of the general.
The GRUB2 bootloader in legacy mode, i.e. non-EFI mode, consists of
several stages. The fist one place in the in the Master Boot Record
of the disk will load and execute the next, second stage, consisting
of core modules and the grub kernel. The first bit is also known as
'boot' and the second as 'core'. When the 'MBR' partition layout is
being used, there is a gap between the Master Boot Record (MBR) and
the first partition (for historical and performance reasons). The
core image normally is placed into this gap (call the MBR gap).
When the partition layout is 'gpt' there is no standard gap that can
be used, instead a special partition ("BIOS boot" [1]) needs to be
created that can store the grub2 core image. Additionally, the 'boot'
image need to modified to point the sector of that partition. The
core image itself also needs to be modified with the information of
the location its own second sector. The location of the pointers
were taken from the grub2 source ([2] at commit [3]). For the 'boot'
image it is 'GRUB_BOOT_MACHINE_KERNEL_SECTOR' (0x5c) from 'pc/boot.h'
and for the core image "0x200 - GRUB_BOOT_MACHINE_LIST_SIZE (12)" to
be found in 'pc/diskboot.S'.
[1] https://en.wikipedia.org/wiki/BIOS_boot_partition
[2] https://github.com/rhboot/grub2
[3] 2a2e10c1b39672de3d5da037a50d5c371f49b40d
Extract the small piece of code that writes the grub2's boot image,
i.e. the first stage of the bootloader that will in turn jump to
the second stage. Currently the position of the core is hard-coded
to be the MBR gap, i.e. the gap between the MBR and the start of
the first partition. This is not a necessity, e.g. when using a
dedicated BIOS boot partition on GPT partition layouts. This re-
factoring should make it easier to add code dealing with such
situations.
This might (hopefully) fix a race in destructing the asyncio.EventLoop
that's used in all API classes, which leads to warnings about unhandled
exceptions on CI.
This also puts their creation closer to where the client-side sockets
are created.
Include a example of a ppc64le (open firmware) pipeline based on
f30-s390x. It includes changes to the grub2 stage (selecting the
platform), the partition table contains the PReP partition needed
to store the grub2 core image.
Introduce support for ppc64le (Open Firmware). The main difference
to x86 legacy, i.e. non-efi, is that no stage 1 is required because
the core image is stored on a special 'PReP' partition, which must
be marked as bootable. The firmware then looks for that partition
and directly loads the core from there and executes it.
Introduce a `platform` parameter for the grub installer code which
controls various platform depended aspects, including a) the path
for the modules, b) what modules are compiled into the core, c) if
the boot image is written to the MBR and 4) where to write the core
image, i.e. mbr-gap or PReP partition.
Extract the function that writes the grub2 core to the image file.
The only supported location currently is the MBR gap, which is the
gap between the Master Boot Record and the first partition, which
for historical and performance reasons was aligned to a certain
sector (used to be 64 but now is even larger with 2048). In the
future other locations for the grub2 core will be supported such
as the PReP partition (ppc64le) or bios-boot (GPT hybrid booting).
Make the bootloader selection explicit by introducing a new option
called `bootloader`, which is an object, containing the `type` and
options belonging to the bootloader. For now only boot-loader that
is supported is "grub2".
Instead of hard-coding "msdos1", determine this partition id
dynamically based on the partition table type and the index
of the partition that contains /boot/grub2, which normally is
either a separate boot partition or the root partition. In
order to be able to do so, set the index of each Partition
when the partition information is read back via `sfdisk`.
NB: partition indexes start at 1 for grub2.
The filesystem module that grub2 needs to have in the core image
is the filesystem containing the grub modules, specifically the
"normal.mod", as well as the grub configuration. In the standard
case, which is also what osbuild uses, this is /boot/grub2; thus
we actually do want the filesystem containing that directory and
its type not the root filesystem.
Explain the concept and reason behind the grub2 core as well as the
details behind the selection of the core modules that get included.
Also elaborate a bit on the MBR gap. For more details about this see
https://en.wikipedia.org/wiki/GNU_GRUB#Version_2_(GRUB_2)
NB: This commit also changes the order of the grub modules, which in
turn changes the layout of the core.img and thus the hash value used
in the test; adapt those value to reflect the changed core.img.
Change the `legacy` option (keeping compatibility) from a boolean
to a string, specifying the target platform the legacy modules
are being installed for.
Pipelines encode which source content they need in the form of
repository metadata checksums (or rpm checksums). In addition, they
encode where they fetch that source content from in the form of URLs.
This is overly specific and doesn't have to be in the pipeline's hash:
the checksum is enough to specify an image.
In practice, this precluded using alternative ways of getting at source
packages, such as local mirrors, which could speed up development.
Introduce a new osbuild API: sources. With it, a stage can query for a
way to fetch source content based on checksums.
The first such source is `org.osbuild.dnf`, which returns repository
configuration for a metadata checksum. Note that the dnf stage continues
to verify that the content it received matches the checksum it expects.
Sources are implemented as programs, living in a `sources` directory.
They are run on the host (i.e., uncontained) right now. Each source gets
passed options, which are taken from a new command line argument to
osbuild, and an array of checksums for which to return content.
This API is only available to stages right now.
This is an expected error case, so we should not assert, but log
and return failure. In the future we should probably also return
the error as structured data.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The GPT (GUID Partition Table) standard for partition layout supports
giving partition a name in the Partition object as well as in the
option for the qemu stage when specifying the partition layout.
Introduce a method on the PartitionTable that returns the partition
containing the root filesystem. NB: this does not have to be the
first partition (which could be the EFI partition, or something
else), so we have to iterate through the partitions until we find
it.
Instead of having dictionaries representing the partition table,
partitions and filesystems together with some functions operating
on them, have proper python objects with methods. In the future
these objects could be extract and properly tested as well.
A simple stage like 'noop' that will return with `returncode` or
255 if nothing is specified. Like 'noop' it might be useful for
testing, debugging, and wasting time.
The recent changes removed the {Assembler,Stage}Failed exceptions,
which includes them being thrown from Stage.run and Assembler.run.
Instead result dictionaries are returned even on errors. But the
object store, used as a context manager, relies on exceptions to
detect the error case and thus needs them to cleanup the temporary
objects. Without those exceptions the temporary objects end up in
the store even when the sage or assembler failed.
Restore the old behavior by throwing a generic BuildError exception
from the Stage and Assembler, which will be caught directly in the
pipeline and converted to a result dict.
The nbd device might not be ready after `qemu-nbd --connect` returns,
leading to access errors such as this further down:
sfdisk: cannot open /dev/nbd12: Inappropriate ioctl for device
Fix this by polling the device with `nbd-client --check <device>`.
Also, the nbd device might not be released after `qemu-nbd --disconnect`
returns. Fix this by using `nbd-client --disconnect`, which waits.
This introduces a new test dependency on nbd-client (in the ndb package
on Fedora).
The socket that the osbuild and loop apis should talk on are passed into
their `__init__` function. The caller should be responsible for closing
those sockets.
This already happens in all current callers.
This fixes a non-fatal error on RHEL's python 3.6, because it was
calling `socket.close` on an already-closed socket:
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/base_events.py", line 529, in __del__
self.close()
File "/usr/lib64/python3.6/asyncio/unix_events.py", line 63, in close
super().close()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 99, in close
self._close_self_pipe()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 109, in _close_self_pipe
self._remove_reader(self._ssock.fileno())
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 268, in _remove_reader
key = self._selector.get_key(fd)
File "/usr/lib64/python3.6/selectors.py", line 189, in get_key
return mapping[fileobj]
File "/usr/lib64/python3.6/selectors.py", line 70, in __getitem__
fd = self._selector._fileobj_lookup(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 224, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 41, in _fileobj_to_fd
raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1
Commit 82a2be53d introduced a new return type from `Pipeline.run()`. It
changed the caller in `__main__.py`, but missed that the build pipeline
uses the same function.
A pipeline run only returned logs in the `StageFailed` and
`AssemblerFailed` exceptions. Remove those and always return structured
data instead.
It only returns data for stages that actually ran (i.e., didn't come
from the cache). This is similar to the output in interactive mode.
Also change osbuildtest to be able to deal with output that is larger
than the pipe buffer by using subprocess.communicate().
Based on the UEFI sample (f30-base-uefi.json). NB: the inclusion
of the dracut-config-generic is needed to disable "host-only" for
dracut so the initramfs will include the virtio_blk block device
driver that is needed to mount the root file system when running
the image in qemu.
Add mkfs_vfat and hook it up into the generic mkfs_for_type()
dispatcher function. Install grub2 to the MBR only if the partition
table is of type "MBR".
Introduce two new assembler options `pttype` and `partitions` to
allow fine grained control over how the partition table is created.
The first one controls the partition type, either `mbr` (default,
when the key is missing) or `gpt`; if specified the `partitions`
key must contain a list of objects describing the individual
partitions (`start`, `size`, `type`) together with a `filesystem`
object describing the filesystem (`type`, `uuid`, `mountpoint`) to
be created on that partition.
In the case the `pttype` option is missing, the legacy mode is used
where `root_fs_uuid` and `root_fs_type` need to be specified.
Use the newly available partition information in the install_grub2
method: detect which module to use for the root filesystem and
assert the second stage fits between the MBR and the first partition.
Introduce a generic mkfs_for_type() function that will dispatch
to the correct mkfs function depending on the type. Additionally
refactor the partition creation and mounting code to handle more
than one partition.
Part of the refactoring to support uefi/gpt: the method that creates
the partition table now returns an array of dictionaries corresponding
to the individual partitions that have been created together with the
information for the filesystem that this partition should end up with.
Prepare the stage for uefi/gpt support by extracting the code that
installs GRUB and creates the partitions into its own functions.
Should not have any effect on the actual data written to the image.
Introduce two new configuration options: `legacy` and `uefi`. The
first one being a boolean (default: True) that controls if GRUB
modules, fonts and the configuration is installed in the right
locations to support legacy boot mode.
The `uefi` option (of type object with a single `vendor` property)
enables UEFI support by writing the configuration into the correct
EFI directory, "/boot/efi/EFI/<vendor>/grub.cfg", where vendor is
taken from said `vendor` property.
The workaround of manually linking /lib64 -> /usr/lib64 inside the
container that is needed on s390 is also required on ppc64 because
here the dynamic linker is set to /lib64/ld64.so.2 and the /lib64
link is not created.
Work around a combination of systemd not creating the link from
/lib64 -> /usr/lib64 (see systemd issue #14311) and the dynamic
linker is being set to (/lib/ld64.so.1 -> /lib64/ld64.so.1)
Therefore we manually create the link before calling nspawn
Commit 283281f broke compression by appending the argument last to the
tar command line. It needs to appear before the file.
Fix that and add a test.
[teg: add minor fix]
