particle-os/debian-forge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

sources/curl: Use our own User-Agent

Browse source 
Currently, osbuild downloads are identified as coming from `curl`. This
is unfortunate because some RPM mirrors block requests from curl. Let's
"fix" that by introducing our own user-agent. While this can certainly
be seen as "circumventing" a policy, I think that this change is
actually helpful: Now, the mirror maintainers can actually distinguish
osbuild requests from regular curl calls. If they want to block osbuild,
they certainly can, we have no power there, but at least this allows
more fine-grained filtering. Also, our new user-agent contains our
domain name, so if there's a problem, they can contact us.
This commit is contained in:
Ondřej Budai 2024-04-29 17:41:49 +02:00 • committed by Simon de Vlieger
parent d50857e5aa
commit af0e849081
2 changed files with 20 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

3
sources/org.osbuild.curl
View file

@ -20,6 +20,7 @@ up the download.
import concurrent.futures
import os
import platform
import subprocess
import sys
import tempfile
@ -153,6 +154,7 @@ class CurlSource(sources.SourceService):
# some mirrors are sometimes broken. retry manually, because we could be
# redirected to a different, working, one on retry.
return_code = 0
arch = platform.machine()
for _ in range(10):
curl_command = [
"curl",
@ -161,6 +163,7 @@ class CurlSource(sources.SourceService):
"--connect-timeout", "30",
"--fail",
"--location",
"--header", f"User-Agent: osbuild (Linux.{arch}; https://osbuild.org/)",
"--output", checksum,
]
if proxy:

17
sources/test/test_curl_source.py
View file

@ -208,3 +208,20 @@ def test_curl_download_proxy(mocked_run, tmp_path, monkeypatch, sources_service,
assert args[0][idx:idx + 2] == ["--proxy", "http://my-proxy"]
else:
assert "--proxy" not in args[0]
@patch("subprocess.run")
def test_curl_user_agent(mocked_run, tmp_path, sources_service):
test_sources = make_test_sources(tmp_path, 80, 2,)
fake_curl_downloader = FakeCurlDownloader(test_sources)
mocked_run.side_effect = fake_curl_downloader.faked_run
sources_service.cache = tmp_path / "curl-cache"
sources_service.cache.mkdir()
sources_service.fetch_all(test_sources)
for call_args in mocked_run.call_args_list:
args, _kwargs = call_args
idx = args[0].index("--header")
assert "User-Agent: osbuild" in args[0][idx + 1]
assert "https://osbuild.org/" in args[0][idx + 1]