sources: add new _fetch_all_new_curl() helper

When using a modern curl we can download download multiple urls
in parallel which avoids connection setup overhead and is generally
more efficient. Use when it's detected.

TODO: ensure both old and new curl are tested automatically via
the testsuite.
This commit is contained in:
Michael Vogt 2024-04-29 13:35:06 +02:00 committed by Simon de Vlieger
parent 974c8adff9
commit 0d3a153c78
2 changed files with 113 additions and 11 deletions

View file

@ -34,6 +34,14 @@ class SilentHTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
def log_message(self, *args, **kwargs):
pass
def do_GET(self):
# silence errors when the other side "hangs up" unexpectedly
# (our tests will do that when downloading in parallel)
try:
super().do_GET()
except (ConnectionResetError, BrokenPipeError):
pass
class DirHTTPServer(ThreadingHTTPServer):
def __init__(self, *args, directory=None, simulate_failures=0, **kwargs):