osbuild/pipeline: Fix crashes when running multiple builds at once

Storytime! I tried to run multiple osbuilds at once. It failed when
unmounting the buildtree. Weird. It turned out the buildtree was not
there anymore when osbuild tried to unmount it. But who unmounted it?

We need to deep dive into mount-types.
Nowadays, the / directory is shared-mounted by systemd. See:
https://serverfault.com/questions/868682/implications-of-mount-make-private
This has interesting implications, see the following example:

we start osbuild1 with /var/tmp/os1 as its store
osbuild1 creates /var/tmp/os1/tmp
osbuild1 bind-mounts / onto /var/tmp/os1/tmp

we start osbuild2 with /var/tmp/os2 as its store
osbuild2 creates /var/tmp/os2/tmp
osbuild2 bind-mounts / onto /var/tmp/os2/tmp

Now, the shared-mounting goes into effect:
The second mount-event gets propagated into the first mount, where it
creates another mount, so we get something like this:
/var/tmp/os1/tmp/var/tmp/os2/tmp

But this is just a start! Imagine running three osbuilds at once.
The event would get propagated to those 3 mounts created by two
osbuilds, creating 3 extra mounts, 7 in total.

It turns out this mounting strategy creates an *exponential number* of
mounts. Crazy, right?

This commit mounts the root inside build root using private bind, which
doesn't propagate bind-events. This solves the problem with the
exponential growth.

But the original problem was different, mount points were disappearing.
So how does this fix solve the problem?

Honestly, I don't know. Something with mount-event propagation is
probably responsible, but I cannot imagine how it is actually affecting
the unbinding.
This commit is contained in:
Ondřej Budai 2019-10-01 08:46:01 +02:00
parent 6ed426773f
commit adf5989de2

View file

@ -189,7 +189,7 @@ class Pipeline:
yield tree
else:
with tempfile.TemporaryDirectory(dir=object_store.store) as tmp:
subprocess.run(["mount", "-o", "bind,ro,mode=0755", "/", tmp], check=True)
subprocess.run(["mount", "--make-private", "-o", "bind,ro,mode=0755", "/", tmp], check=True)
try:
yield tmp
finally: