This commit adds semi-structured documentation to all osbuild stages and
assemblers. The variables added work like this:
* STAGE_DESC: Short description of the stage.
* STAGE_INFO: Longer documentation of the stage, including expected
behavior, required binaries, etc.
* STAGE_OPTS: A JSON Schema describing the stage's expected/allowed
options. (see https://json-schema.org/ for details)
It also has a little unittest to check stageinfo - specifically:
1. All (executable) stages in stages/* and assemblers/ must define strings named
STAGE_DESC, STAGE_INFO, and STAGE_OPTS
2. The contents of STAGE_OPTS must be valid JSON (if you put '{' '}'
around it)
3. STAGE_OPTS, if non-empty, should have a "properties" object
4. if STAGE_OPTS lists "required" properties, those need to be present
in the "properties" object.
The test is *not* included in .travis.yml because I'm not sure we want
to fail the build for this, but it's still helpful as a lint-style
check.
Rather than relying on the offset parameter, simply run mkfs on the
loopback device which is anyway being set up. This also allows us
not to specify the size explicitly.
Before this patch mkfs would complain (uneccesarily) about the
backing file containing a partition table. This is a false positive
as the partition table is in the region of the file before the
passed offset.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We know the root partition we want, as we are setting it up. There
is no need to search for it by filesystem UUID. This simplifies the
setup and means the level 1.5 bootloader is always the same, and
not dependent on an embedded UUID.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Similar to the existing test, but uses qemu-nbd to mount the generated
image.
Using unittest.TestCase.subTest() for now, which means that the tests
aren't very independent. I think this is fine in this case, because
we're testing images independently from each other, reusing the base
tree in the store.
Closing the socket is the responsibility of whoever opened it.
Fix this in the only user (qemu assembler) by using socket() in a `with`
block, which closes the socket on exit.
Background:
grub2 works in three stages:
- The first stage is found in the first 440 bytes of the master
boot record, and its only purpose is to load and execute the
second stage. This stage is static, and just copied from the rpm
without modification.
- The second stage is found in the gap between the MBR and the
first partition, and may be up to 31kB in size. This stage is
specific to the host and must contain the instructions for
finding the right file system and subdirectory for the grub2
config and modules on the host, as well as the modules needed
to do this.
- The third stage is found in the `normal` module, which loads
grub2.conf, which in turn may load more modules and perform
arbitrary instructions.
Problem:
grub2-install is responsible for installing all these stages on the
target image. This goes against our design, as modifications outside
the filesystem should happen in the assembler, but modifications to
the filesystem should happen in a stage. In particular, we don't
want the contents of the image to differ in any way from the output
tree that is stored in our content store (the output of our last
stage). This causes a practical problem at the moment, as our
selinux stage is ran before the assembler, and as such the grub
modules do not get selinux labels applied.
It turns out that we could split grub2-install in two as we want,
by passing `--no-bootsector` to it to install only the modules,
and copy/genereta the two first stages as files under /boot and
then run `grub2-bios-setup` to write the stages from /boot into
the image where they belong.
Regrettably, this does not work as both `grub2-install` and
`grub2-bios-setup` introspect the system and block devices they
are being run on to generate the right configuration. This is not
what we want, as we would like to specifcy the config explicitly
and run them independently of the target image. The specific bug
we get in both cases is that the canonical path containing our
object store cannot be found.
Before osbuild this was not a problem, as other installers would
instal and assemble everything directly in the target image as a
loopback device. Something we explicitly do not want to do.
Solution:
This patch essentially reimplements grub2-install, or rather the
parts of it that we need. One change in behavior from the upstream
tool is that we no longer write the level one and level two boot
loaders to /boot before moving them into place, but just write them
directly where they belong (so they do not end up on the
filesystem).
The parts that copy files into /boot are now in the grub2 installer
and the parts that write the level one/two bootloaders are in the
qemu assembler.
This achieves a few principles I think we should always adher to:
- never run tools from the target image (no chroot)
- don't read/copy files from the target image that was written
by other stages. We already try to avoid sharing state, and
by treating the image as write-only, we avoid accidentally
sharing state through the target tree.
Based-on-suggestions-from: Javier Martinez Canillas <javierm@redhat.com>
With-god-like-debugging-and-fixes-by: Lars Karlitski <lubreni@redhat.com>
Signed-off-by: Tom Gundersen <teg@jklm.no>
Otherwise, sfdik would pick one at random. We want our images to be
reproducible to the extent possible, so we must move all randomness
out of the assemblers when we can.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Opt in to supporting the most common ones, if we want to support more
we can add support as the need arises.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In tests we often use tar assembler as final stage. This means we
compress the image tree and decompress it right away. For this purposes
it is nice to have option to not have any compression. Actually,
this could very drastically improve CI running time.
A better option would be not to use tar at all and instead let osbuild
just dump the resulting tree. However, we felt this behaviour needs
more discussion and we need a fix asap.
Directory /tmp is hosted on tmpfs. Therefore the image size could be
limited by memory size. By moving the image to /var/tmp we assure that
the file is hosted on disk allowing us to build bigger images or build
images on memory-constrained machines (e.g. CI).
Don't try to guess how much room the filesystem will take up. In
practice, most people will want to specify a size anyway, depending on
their use case.
As is typical for osbuild, there are no convenience features for the
pipeline (it's not meant to be written manually). `size` must be given
in bytes and it must be a multiple of 512.
Assemblers are always run in their own, clean environment and can be
sure that there's only one instance of themselves running. Remove the
extra layer of temporary directory and use static names.
We used to let mkfs.ext4 initialize the filesystem for us, but it
turns out that the metadata attributes of the root directory were
not being initialized from the source tree. In particular, this
meant that the SELinu labels were left as unconfined_t, rather
than root_t, which would not allow us to boot in enforcing mode.
An alternative approach might be to fixup the root inode manually,
while still doing the rest using mkfs.ext4, but let's leave that
for the future if it turns out to be worth it.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This way the assemblers/stages are valid in isolation, even without
osbulid installed system-wide. This would be needed to have this work
when --libdir is not the system-wide one, as the library would
otherwise not be in sys.path.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Create a loopback device for the raw partiton, rather than relying on
the partition devices the kernel puts in /dev. This requires us to
specify the part_msdos module directly as grub2-install now seems
unable to detect the partition table type.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Create and instantiate the ext4 filesystem directly on the device, without
ever mounting it. This means that only grub2 now requires loopback devices
to function properly.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This removes the possibility of passing in arbitrary input data. We
now restrict ourselves to explicitly specified files/directories or
a base tree given by its pipeline id.
This drops the tar/tree stages/assemblers, as the tree/untree ones
are implicit in osbuild, and if we wish to also support compressed
trees, then we should add that to osbuild core as an option.
Signed-off-by: Tom Gundersen <teg@jklm.no>
By default the whole image file is attechd to the loopback device, but
we want to only attach a slice, namely individual partitions.
This is a noop, just adds the parameters to the helpers.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We want to copy the contents of a directory to another, the correct
syntax for that is `cp -a src/. dst`. I was not aware of this beauty,
so the previous patch simulated the functionality in python code.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In the simplest case we don't need to make an image, a filesystem tree
will do. Also support using such a tree as input for further pipelines
through the untree stage.
Signed-off-by: Tom Gundersen <teg@jklm.no>
