particle-os/debian-forge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

sources: generalizing download method

Browse source 
Before, the download method was defined in the inherited class of each
program. With the same kind of workflow redefined every time. This
contribution aims at making the workflow more clear and to generalize
what can be in the SourceService class.

The download worklow is as follow:
Setup -> Filter -> Prepare -> Download

The setup mainly step sets up caches. Where the download data will be
stored in the end.

The filter step is used to discard some of the items to download based
on some criterion. By default, it is used to verify if an item is
already in the cache using the item's checksum.

The Prepare step goes from each element and let the overloading step the
ability to alter each item before downloading it. This is used mainly
for the curl command which for rhel must generate the subscriptions.

Then the download step will call fetch_one for each item. Here the
download can be performed sequentially or in parallel depending on the
number of workers selected.
This commit is contained in:
Thomas Lavocat 2022-04-12 15:37:21 +02:00 • committed by Thomas Lavocat
parent 0953cf64e0
commit 1de74ce2c9
5 changed files with 129 additions and 141 deletions
Download patch file Download diff file Expand all files Collapse all files

26
sources/org.osbuild.inline
View file

@ -15,9 +15,6 @@ import base64
import contextlib
import os
import sys
import tempfile
from typing import Dict
from osbuild import sources
from osbuild.util.checksum import verify_file
@ -56,15 +53,18 @@ SCHEMA = """
"""
def process(items: Dict, cache: str, tmpdir):
for checksum, item in items.items():
target = os.path.join(cache, checksum)
floating = os.path.join(tmpdir, checksum)
class InlineSource(sources.SourceService):
content_type = "org.osbuild.files"
def fetch_one(self, checksum, desc):
target = os.path.join(self.cache, checksum)
floating = os.path.join(self.tmpdir, checksum)
if os.path.isfile(target):
return
data = base64.b64decode(item["data"])
data = base64.b64decode(desc["data"])
# Write the bits to disk and then verify the checksum
# This ensures that 1) the data is ok and that 2) we
@ -73,20 +73,12 @@ def process(items: Dict, cache: str, tmpdir):
f.write(data)
if not verify_file(floating, checksum):
raise RuntimeError("Checksum mismatch for {}".format(checksum))
raise RuntimeError(f"Checksum mismatch for {format(checksum)}")
with contextlib.suppress(FileExistsError):
os.rename(floating, target)
class InlineSource(sources.SourceService):
content_type = "org.osbuild.files"
def download(self, items):
process(items, self.cache, self.tmpdir)
def main():
service = InlineSource.from_args(sys.argv[1:])
service.main()