builder: add retries to composer API calls

The status calls are sometimes failing on:
upstream connect error or disconnect/reset before headers. reset reason:
connection termination

Since all requests are going through the company proxy, I think that the
networking isn't working 100% reliably. This commit adds a retry mechanism
provided by the urllib3 library. It will retry on all networking issues and
also on some 5xx errors that makes sense to retry (like gateway failures).

A test is added that runs the compose waiting code against a mock server
that fails every second request. This is imho sufficient to mimick a flaky
networking.

plugins/builder/osbuild.py

@@ -29,6 +29,7 @@ from typing import Dict, List, Optional, Union
 import requests
 import koji
 
+from requests.adapters import HTTPAdapter, Retry
 from koji.daemon import fast_incremental_upload
 from koji.tasks import BaseTaskHandler
 
@@ -336,6 +337,14 @@ class Client:
         self.url = urllib.parse.urljoin(url, API_BASE)
         self.http = requests.Session()
 
+        retries = Retry(total=5,
+                        backoff_factor=0.1,
+                        status_forcelist=[500, 502, 503, 504],
+                        raise_on_status=False
+                        )
+
+        self.http.mount(self.server, HTTPAdapter(max_retries=retries))
+
     @staticmethod
     def parse_certs(string):
         certs = [s.strip() for s in string.split(',')]