Rather than relying on the offset parameter, simply run mkfs on the
loopback device which is anyway being set up. This also allows us
not to specify the size explicitly.
Before this patch mkfs would complain (uneccesarily) about the
backing file containing a partition table. This is a false positive
as the partition table is in the region of the file before the
passed offset.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We know the root partition we want, as we are setting it up. There
is no need to search for it by filesystem UUID. This simplifies the
setup and means the level 1.5 bootloader is always the same, and
not dependent on an embedded UUID.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The tests from the integration_tests directory, were superseded
by the new stage tests.
The Vagrant integration seems not to have been working since
ea68bb0c26, as a test-setup.py was
dropped there, which it relies on. Remove it for now. If we want
that back, we should consider that in a separate PR.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Downloading the gpg key is fragile and kept causing our tests to fail.
In general, we want to limit the network access, so let's just embed
the gpg keys directly in the pipeline.
Fixes#133.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When the test runner receives SIGINT, osbuild's mounts stay around.
osbuild handles SIGING correctly, but it doesn't have time before being
killed because it's parent went away.
Fix this by waiting on it explicitly in the test runner.
Fixes#119
Similar to the existing test, but uses qemu-nbd to mount the generated
image.
Using unittest.TestCase.subTest() for now, which means that the tests
aren't very independent. I think this is fine in this case, because
we're testing images independently from each other, reusing the base
tree in the store.
For stages testing it is too slow to rebuild the a image containing
@Core packages every time. Let's just reuse the a image for all tests.
This should speed up the test running time a LOT.
The stage testing is based on an output from the tree-diff tool. During
one test two pipelines are run and their outputs are compared using
tree-diff. The diff is then compared with expected diff included in
the repository.
We need to know the exact difference of modified files in both trees.
Outputting the whole files into a diff might make a huge diff file,
therefore only their hashes are written.
It is similar to the official Fedora cloud base image except for few
minor differences. The reason for this divergence is that we don't want
to include all hacks that are currently present in the official
kickstart file. You can see it here as a reference:
https://pagure.io/fedora-kickstarts/blob/master/f/fedora-cloud-base.ks#_149
If dir_fd wasn't passed, create_device() openend it to `/dev` and forgot
about closing it. To fix this, it would have to gain logic to only close
the fd if it wasn't passed in.
Side-step the problem by removing dir_fd, since nothing is using it
right now. We can add it back if something needs it.
Closing the socket is the responsibility of whoever opened it.
Fix this in the only user (qemu assembler) by using socket() in a `with`
block, which closes the socket on exit.
build-from-yum.json is the one that's being used for testing on Ubuntu.
Remove base-from-yum.json, because it's confusing to have two similarly
named pipelines like this.
The testosbuild.TestCase class creates a fresh store for each test,
because tests should run independent of each other.
This can lead to long waiting times while developing a new test case.
Allow overriding the store used with OSBUILD_TEST_STORE. This should
never be used where tests are actually run. It is a development-only
feature.
_libdir is platform dependant, but that is not what we want because we
would need additional runtime logic to handle platforms. this patch
overrides the defautl location
Background:
grub2 works in three stages:
- The first stage is found in the first 440 bytes of the master
boot record, and its only purpose is to load and execute the
second stage. This stage is static, and just copied from the rpm
without modification.
- The second stage is found in the gap between the MBR and the
first partition, and may be up to 31kB in size. This stage is
specific to the host and must contain the instructions for
finding the right file system and subdirectory for the grub2
config and modules on the host, as well as the modules needed
to do this.
- The third stage is found in the `normal` module, which loads
grub2.conf, which in turn may load more modules and perform
arbitrary instructions.
Problem:
grub2-install is responsible for installing all these stages on the
target image. This goes against our design, as modifications outside
the filesystem should happen in the assembler, but modifications to
the filesystem should happen in a stage. In particular, we don't
want the contents of the image to differ in any way from the output
tree that is stored in our content store (the output of our last
stage). This causes a practical problem at the moment, as our
selinux stage is ran before the assembler, and as such the grub
modules do not get selinux labels applied.
It turns out that we could split grub2-install in two as we want,
by passing `--no-bootsector` to it to install only the modules,
and copy/genereta the two first stages as files under /boot and
then run `grub2-bios-setup` to write the stages from /boot into
the image where they belong.
Regrettably, this does not work as both `grub2-install` and
`grub2-bios-setup` introspect the system and block devices they
are being run on to generate the right configuration. This is not
what we want, as we would like to specifcy the config explicitly
and run them independently of the target image. The specific bug
we get in both cases is that the canonical path containing our
object store cannot be found.
Before osbuild this was not a problem, as other installers would
instal and assemble everything directly in the target image as a
loopback device. Something we explicitly do not want to do.
Solution:
This patch essentially reimplements grub2-install, or rather the
parts of it that we need. One change in behavior from the upstream
tool is that we no longer write the level one and level two boot
loaders to /boot before moving them into place, but just write them
directly where they belong (so they do not end up on the
filesystem).
The parts that copy files into /boot are now in the grub2 installer
and the parts that write the level one/two bootloaders are in the
qemu assembler.
This achieves a few principles I think we should always adher to:
- never run tools from the target image (no chroot)
- don't read/copy files from the target image that was written
by other stages. We already try to avoid sharing state, and
by treating the image as write-only, we avoid accidentally
sharing state through the target tree.
Based-on-suggestions-from: Javier Martinez Canillas <javierm@redhat.com>
With-god-like-debugging-and-fixes-by: Lars Karlitski <lubreni@redhat.com>
Signed-off-by: Tom Gundersen <teg@jklm.no>
Otherwise, sfdik would pick one at random. We want our images to be
reproducible to the extent possible, so we must move all randomness
out of the assemblers when we can.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This allows given packages to be excluded from the transaction. This
is useful if you want to install a group with certain exceptions.
A common thing to do in kicktstart files is:
```
rm -f /boot/*-rescue*
```
By instead excluding the dracut-rescue-config package we end up
with:
```
"deleted_files": [
"/etc/kernel/postinst.d",
"/usr/lib/dracut/dracut.conf.d/02-rescue.conf",
"/usr/lib/kernel/install.d/51-dracut-rescue.install",
"/boot/initramfs-0-rescue-ffffffffffffffffffffffffffffffff.img",
"/boot/vmlinuz-0-rescue-ffffffffffffffffffffffffffffffff"
],
```
Signed-off-by: Tom Gundersen <teg@jklm.no>
Storytime! I tried to run multiple osbuilds at once. It failed when
unmounting the buildtree. Weird. It turned out the buildtree was not
there anymore when osbuild tried to unmount it. But who unmounted it?
We need to deep dive into mount-types.
Nowadays, the / directory is shared-mounted by systemd. See:
https://serverfault.com/questions/868682/implications-of-mount-make-private
This has interesting implications, see the following example:
we start osbuild1 with /var/tmp/os1 as its store
osbuild1 creates /var/tmp/os1/tmp
osbuild1 bind-mounts / onto /var/tmp/os1/tmp
we start osbuild2 with /var/tmp/os2 as its store
osbuild2 creates /var/tmp/os2/tmp
osbuild2 bind-mounts / onto /var/tmp/os2/tmp
Now, the shared-mounting goes into effect:
The second mount-event gets propagated into the first mount, where it
creates another mount, so we get something like this:
/var/tmp/os1/tmp/var/tmp/os2/tmp
But this is just a start! Imagine running three osbuilds at once.
The event would get propagated to those 3 mounts created by two
osbuilds, creating 3 extra mounts, 7 in total.
It turns out this mounting strategy creates an *exponential number* of
mounts. Crazy, right?
This commit mounts the root inside build root using private bind, which
doesn't propagate bind-events. This solves the problem with the
exponential growth.
But the original problem was different, mount points were disappearing.
So how does this fix solve the problem?
Honestly, I don't know. Something with mount-event propagation is
probably responsible, but I cannot imagine how it is actually affecting
the unbinding.
Opt in to supporting the most common ones, if we want to support more
we can add support as the need arises.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This key carries no information and is never used anywhere. The json
files are not meant to be human readable, so simply drop this.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Some dnf packages introduce random seed file. If we leave in the tree
it would mean all systems running from the created image would use
the same random seed. This can be potentially dangerous, therefore we
just remove the generated random seed from our images.
Normally, machine ID is generated randomly when dnf installs @Core
group. Unfortunately this isn't helping us with reproducibility
of images.
This commit introduces the concept of fake machine ID. Before dnf
command is run in dnf stage, we set the machine ID to a fake one.
This ensures all the scriptlets requiring machine ID have predictable
outputs.
For example GRUB uses machine-id in file names inside
/boot/loader/entries. With fixed machine ID the file names are always
the same and totally predictable.
