particle-os/debian-forge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

osbuid: integrate FsCache into ObjectStore

Browse source 
Integrate the recently added file system cache `FsCache` into our
object store `ObjectStore`. NB: This changes the semantics of it:
previously a call to `ObjectStore.commit` resulted in the object
being in the cache (i/o errors aside). But `FsCache.store`, which
is now the backing store for objects, will only commit objects if
there is enough space left. Thus we cannot rely that objects are
present for reading after a call to `FsCache.store`. To cope with
this we now always copy the object into the cache, even for cases
where we previously moved it: for the case where commit is called
with `object_id` matching `Object.id`, which is the case for when
`commit` is called for last stage in the pipeline. We could keep
this optimization but then we would have to special case it and
not call `commit` for these cases but only after we exported all
objects; or in other words, after we are sure we will never read
from any committed object again. The extra complexity seems not
worth it for the little gain of the optimization.
Convert all the tests for the new semantic and also remove a lot
of them that make no sense under this new paradigm.

Add a new command line option `--cache-max-size` which will set
the maximum size of the cache, if specified.
This commit is contained in:
Christian Kellner 2022-12-06 18:41:52 +01:00
parent 1e0e1fa2c2
commit ae0680da11
6 changed files with 158 additions and 247 deletions
Download patch file Download diff file Expand all files Collapse all files

6
osbuild/main_cli.py
View file

@ -15,6 +15,7 @@ import osbuild
import osbuild.meta
import osbuild.monitor
from osbuild.objectstore import ObjectStore
from osbuild.util.parsing import parse_size
from osbuild.util.term import fmt as vt
@ -66,6 +67,8 @@ def parse_arguments(sys_argv):
help="directory where intermediary os trees are stored")
parser.add_argument("-l", "--libdir", metavar="DIRECTORY", type=os.path.abspath, default="/usr/lib/osbuild",
help="directory containing stages, assemblers, and the osbuild library")
parser.add_argument("--cache-max-size", metavar="SIZE", type=parse_size, default=None,
help="maximum size of the cache (bytes) or 'unlimited' for no restriction")
parser.add_argument("--checkpoint", metavar="ID", action="append", type=str, default=None,
help="stage to commit to the object store during build (can be passed multiple times)")
parser.add_argument("--export", metavar="ID", action="append", type=str, default=[],
@ -150,6 +153,9 @@ def osbuild_cli():
try:
with ObjectStore(args.store) as object_store:
if args.cache_max_size is not None:
object_store.maximum_size = args.cache_max_size
stage_timeout = args.stage_timeout
pipelines = manifest.depsolve(object_store, exports)