particle-os/debian-forge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

osbuild: no auto commit of the last stage

Browse source 
Do not automatically commit the last stage of the pipeline to the
store. The last stage is most likely not what should be cached,
because it will contain all the individual customization and thus
be very likely different for different users. Instead, the dnf or
rpm stages have a higher chance of being the same and thus are
better candidates for caching.
Technically this change is done via two big changes that build
upon new features introduces in the previous commits, most notably
the copy on write semantics of Object and that input/output is
being done via `objectstore.Object` instead of plain paths. The
first of the two big changes is  to create one new `Object` at
the beginning of `pipeline.run` and use that, in write mode via
`Object.write` across invocations of `stage.run` calls, with
checkpoints being created after each stage on demand.
The very same `Object` is then used in read mode via `Object.read`
as the input tree for the Assembler. After the assembler is done
the resulting image/tree is manually committed to the store.
The other big change is to remove the `ObjectStore.commit` call
from the `ObjectStore.new` method and thus the automatic commit
after the last stage is gone.
NB: since the build tree is being retrieved in `get_buildtree`
from the store, a checkpoint for the last stage of the build
pipeline is forced for now. Future commits will refactor will
do away with that forced commit as well.
Change osbuildtest.TestCase to always create a checkpoint at
the final tree (the last stage of the pipeline), since tests
need it to check the tree contents.
This commit is contained in:
Christian Kellner 2020-02-18 18:19:04 +01:00 • committed by Tom Gundersen
parent 7720b5508d
commit 42a365d12f
4 changed files with 66 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

15
test/test_objectstore.py
View file

@ -24,10 +24,11 @@ class TestObjectStore(unittest.TestCase):
# always use a temporary store so item counting works
with tempfile.TemporaryDirectory(dir="/var/tmp") as tmp:
object_store = objectstore.ObjectStore(tmp)
with object_store.new("a") as tree:
with object_store.new() as tree:
path = tree.write()
p = Path(f"{path}/A")
p.touch()
object_store.commit(tree, "a")
assert os.path.exists(f"{object_store.refs}/a")
assert os.path.exists(f"{object_store.refs}/a/A")
@ -35,12 +36,13 @@ class TestObjectStore(unittest.TestCase):
assert len(os.listdir(object_store.objects)) == 1
assert len(os.listdir(f"{object_store.refs}/a/")) == 1
with object_store.new("b") as tree:
with object_store.new() as tree:
path = tree.write()
p = Path(f"{path}/A")
p.touch()
p = Path(f"{path}/B")
p.touch()
object_store.commit(tree, "b")
assert os.path.exists(f"{object_store.refs}/b")
assert os.path.exists(f"{object_store.refs}/b/B")
@ -52,15 +54,17 @@ class TestObjectStore(unittest.TestCase):
def test_duplicate(self):
with tempfile.TemporaryDirectory(dir="/var/tmp") as tmp:
object_store = objectstore.ObjectStore(tmp)
with object_store.new("a") as tree:
with object_store.new() as tree:
path = tree.write()
p = Path(f"{path}/A")
p.touch()
object_store.commit(tree, "a")
with object_store.new("b") as tree:
with object_store.new() as tree:
path = tree.write()
shutil.copy2(f"{object_store.refs}/a/A",
f"{path}/A")
object_store.commit(tree, "b")
assert os.path.exists(f"{object_store.refs}/a")
assert os.path.exists(f"{object_store.refs}/a/A")
@ -73,7 +77,7 @@ class TestObjectStore(unittest.TestCase):
def test_snapshot(self):
object_store = objectstore.ObjectStore(self.store)
with object_store.new("b") as tree:
with object_store.new() as tree:
path = tree.write()
p = Path(f"{path}/A")
p.touch()
@ -81,6 +85,7 @@ class TestObjectStore(unittest.TestCase):
path = tree.write()
p = Path(f"{path}/B")
p.touch()
object_store.commit(tree, "b")
# check the references exist
assert os.path.exists(f"{object_store.refs}/a")