72c3157162
Background: grub2 works in three stages: - The first stage is found in the first 440 bytes of the master boot record, and its only purpose is to load and execute the second stage. This stage is static, and just copied from the rpm without modification. - The second stage is found in the gap between the MBR and the first partition, and may be up to 31kB in size. This stage is specific to the host and must contain the instructions for finding the right file system and subdirectory for the grub2 config and modules on the host, as well as the modules needed to do this. - The third stage is found in the `normal` module, which loads grub2.conf, which in turn may load more modules and perform arbitrary instructions. Problem: grub2-install is responsible for installing all these stages on the target image. This goes against our design, as modifications outside the filesystem should happen in the assembler, but modifications to the filesystem should happen in a stage. In particular, we don't want the contents of the image to differ in any way from the output tree that is stored in our content store (the output of our last stage). This causes a practical problem at the moment, as our selinux stage is ran before the assembler, and as such the grub modules do not get selinux labels applied. It turns out that we could split grub2-install in two as we want, by passing `--no-bootsector` to it to install only the modules, and copy/genereta the two first stages as files under /boot and then run `grub2-bios-setup` to write the stages from /boot into the image where they belong. Regrettably, this does not work as both `grub2-install` and `grub2-bios-setup` introspect the system and block devices they are being run on to generate the right configuration. This is not what we want, as we would like to specifcy the config explicitly and run them independently of the target image. The specific bug we get in both cases is that the canonical path containing our object store cannot be found. Before osbuild this was not a problem, as other installers would instal and assemble everything directly in the target image as a loopback device. Something we explicitly do not want to do. Solution: This patch essentially reimplements grub2-install, or rather the parts of it that we need. One change in behavior from the upstream tool is that we no longer write the level one and level two boot loaders to /boot before moving them into place, but just write them directly where they belong (so they do not end up on the filesystem). The parts that copy files into /boot are now in the grub2 installer and the parts that write the level one/two bootloaders are in the qemu assembler. This achieves a few principles I think we should always adher to: - never run tools from the target image (no chroot) - don't read/copy files from the target image that was written by other stages. We already try to avoid sharing state, and by treating the image as write-only, we avoid accidentally sharing state through the target tree. Based-on-suggestions-from: Javier Martinez Canillas <javierm@redhat.com> With-god-like-debugging-and-fixes-by: Lars Karlitski <lubreni@redhat.com> Signed-off-by: Tom Gundersen <teg@jklm.no> |
||
|---|---|---|
| assemblers | ||
| osbuild | ||
| samples | ||
| stages | ||
| test | ||
| .gitignore | ||
| .packit.yaml | ||
| .pylintrc | ||
| .travis.yml | ||
| bump-version.sh | ||
| LICENSE | ||
| Makefile | ||
| MANIFEST.in | ||
| osbuild-run | ||
| osbuild.spec | ||
| README.md | ||
| setup.py | ||
| tree-diff | ||
osbuild
A build system for operating system images, working towards an image build pipeline that's more comprehensible, reproducible, and extendable.
Pipelines
The build process for an image is described by a pipeline. Each stage in a pipeline is a program that, given some configuration, modifies a file system tree. Finally, an assembler takes a filesystem tree, and assembles it into an image. Pipelines are defined as JSON files like this one:
{
"name": "Example Image",
"stages": [
{
"name": "org.osbuild.dnf",
"options": {
"releasever": "30",
"basearch": "x86_64",
"repos": [
{
"metalink": "https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch",
"gpgkey": "F1D8 EC98 F241 AAF2 0DF6 9420 EF3C 111F CFC6 59B9",
"checksum": "sha256:9f596e18f585bee30ac41c11fb11a83ed6b11d5b341c1cb56ca4015d7717cb97"
}
],
"packages": [ "@Core", "grub2-pc", "httpd" ]
}
},
{
"name": "org.osbuild.systemd",
"options": {
"enabled_services": [ "httpd" ]
}
},
{
"name": "org.osbuild.grub2",
"options": {
"root_fs_uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac"
}
}
],
"assembler": {
"name": "org.osbuild.qemu",
"options": {
"format": "qcow2",
"filename": "example.qcow2",
"ptuuid": "0x7e83a7ba",
"root_fs_uuid": "76a22bf4-f153-4541-b6c7-0332c0dfaeac",
"size": 3221225472
}
}
}
osbuild runs each of the stages in turn, isolating them from the host and
from each other, with the exception that they all operate on the same
filesystem-tree. The assembler is similarly isolated, and given the same
tree, in read-only mode and assembles it into an image without altering
its contents.
The filesystem tree produced by the final stage of a pipeline, is named and optionally saved to be reused as the base for future pipelines.
Each stage is passed the (appended) options object as JSON over stdin.
The above pipeline has no base and produces a qcow2 image.
Running
usage: python3 -m osbuild [-h] [--build-pipeline PIPELINE] [--store DIRECTORY]
[-l DIRECTORY]
PIPELINE
Build operating system images
positional arguments:
PIPELINE json file containing the pipeline that should be built
optional arguments:
-h, --help show this help message and exit
--build-pipeline PIPELINE
json file containing the pipeline to create a build
environment
--store DIRECTORY the directory where intermediary os trees are stored
-l DIRECTORY, --libdir DIRECTORY
the directory containing stages, assemblers, and the
osbuild library
Running example
You can build basic qcow2 image of Fedora 30 by running a following command:
sudo python3 -m osbuild --libdir . samples/base-qcow2.json
-
Root rights are required because osbuild heavily relies on creating systemd containers and bind mounting.
It shouldn't interfere with host OS but please be careful! It's still under development!
-
--libdirargument is required becauseosbuildexpects itself to be installed in directories under/usr. Using this argument you can change the expected path. -
You don't need to use any kind of virtual environment, modern version of Python 3 is enough.
osbuilduses only standard library and linux commands.