An error message has changed in skopeo [1] (or one of it's underlying
libraries). This new version is now in our CI so let's fix our tests.
[1]: f423f01d1b
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
This commit adds a small unit test that the inline source
works as expected by creating some test_data and then
checking that it ends up in the cache.
Note that this is also already tested in the tests in
osbuild/test/run/test_sources.py but there its a lot more
indirect and in the spirit of tests-as-documentation having
an explicit test here seems useful (its also quicker to
run and easier to discover).
Small followup for https://github.com/osbuild/osbuild/pull/2090
where this was discussed originally.
Extend the inline source to support lzma compressed and base64 encoded
'data'. This will allow us to reduce the potential manifest size when
embedding big files. The aim is specifically at eventually embedding
SBOMs of the image.
An example single SBOM can be a JSON file with size of about 1.9 MiB.
The lzma+base64 combination reduces the 'data' to embed to "only" around
250 KiB.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit relaxes the "mirror" schema regex from only allow
hashes to the repo ID regex as described in
https://dnf.readthedocs.io/en/latest/conf_ref.html#description
This is needed because in e.g. bootc-image-builder we use
"SetRootDir()" to get the repository information from an existing
bootc image and there the repo_id (that is then used as the mirror_id)
is written in a human friendly way instead of hex.
Thanks to Tomáš Hozza for the suggestions and background info, see
https://github.com/osbuild/osbuild/pull/1974#issuecomment-2579325272
This commit adds support for the `org.osbuild.mtls` sources and
adds a test.
Note that it reuses the `OSBUILD_SOURCES_CURL_` - this may or
may not be desired, it is certainly simpler this way and librepo
is using libcurl as well internally but it feels a bit messy so
maybe we want to rename the env?
This commit fixes the pylint errors and as a drive-by tweaks some naming
in `fetch_all()`. Maybe they became a bit verbose along the way,
idk.
Thanks also to Tomáš Hozza for the dict.setdefault() suggestion.
This commit adds unit a basic test for librepo and also updates
the code to the latest requirements to provide "fetch_all".
It also fixes a bug in the handling of org.osbuild.rhsm secrets.
Using a metalink or mirrorlist along with the package paths and
checksums allows them to be reliably downloaded even when mirrors are
not all in sync. It will retry with a new mirror until it succeeds, or
has tried all of the mirrors.
Not all distros ship `/var/empty` so just create an empty dir
on demand as needed.
This also tweaks `test_ostree_source_exists()` into calling
`make_repo()` instead of duplicating that code.
This commit keeps track of individual errors as curl will only report
the last download operation success/failure via it's exit code.
There is "--fail-early" but the downside of that is that abort all in
progress downloads too.
curl keeps a global parser state. This means that if there are
multiple "cacert =" values they are just overriden and the last
one wins. This is why the `test_curl_download_many_mixed_certs`
test did not work - the second `cacert = ` overwrites the previous
one.
To fix this we need to use `--next` when we need to change options
on a per url (like `cacert`) basis. With `--next` curl starts a
new parser state for the next url (but keeps the options for the
previous ones set). This commit does that in a slightly naive
way by just repeating our options for each url. Technically
we could sort the sources so that we have less repetition but
other then slightly smaller auto-generated files it has no
advantage.
With this commit the `test_curl_download_many_mixed_certs` test
works.
When investigating https://github.com/osbuild/osbuild-composer/pull/4247
we found that it would fail when a download required two sets of
`--cacert` keys. This commits adds a test for this that fails on
the centos9 7.76.1 version.
storage_conf was never not None, so the loading was called every time.
This never crashed because conf was always being set, but it wasn't
working properly regardless.
Add a new util module called host which is used for functions that are
meant for interactions with the host. These functions should not be
used in stages.
The containers.get_host_storage() function is renamed to
host.get_container_storage() for clarity, since it is no longer
namespaced under containers.
When using `--write-out` we are not using %{json} because older curl
(7.76) will write {"http_connect":000} which python cannot parse.
So we had a custom `--write-out` with `\1xc` as "record" separators
between the fields. This is a bit old-school and not very extensible
so Achilleas had the idea to still use json but "define" our own
subset via the variables that curl provides. This commit does that.
As part of the investigation of the CI failure in
https://github.com/osbuild/osbuild-composer/pull/4247
we noticed that curl can return a return_code of `0` even
when it did not downloaded all the urls in a `--config` provided
file. This seems to be curl version dependent, I had a hard
time writing a test-case with the real curl (8.6.0) that
reproduces this so I went with mocking it. We definietly saw
this failure with the centos 9 version (7.76).
Our current code is buggy and assumes that the exit status
of curl is always non-zero if any download fails but that is
only the case when `--fail-early` is used.
The extra paranoia will not hurt even when relying on the
exit code of curl is fixed.
Disable `curl --parallel` by default until the failure in
https://github.com/osbuild/osbuild-composer/pull/4247
is fully understood. It can be enabled via the environment:
```
OSBUILD_SOURCES_CURL_USE_PARALLEL=1
```
in the osbuild-composer test.
When using a modern curl we can download download multiple urls
in parallel which avoids connection setup overhead and is generally
more efficient. Use when it's detected.
TODO: ensure both old and new curl are tested automatically via
the testsuite.
Modern curl (7.68+) has a --parallel option that will download
multiple sources in parallel. This commit adds detection for this
feature as it is only available after RHEL 8.
In addition we need some more feature to properly support --parallel,
i.e. `--write-out` with json and exitcode options. This bumps the
requirements to 7.75+ which is still fine, centos9/RHEL9 have
7.76.
Instead of passing the url and options on the commandline this
commit moves it into a config file. This is not useful just yet
but it will be once we download multiple urls per curl instance.
Setting the user-agent using `--header` is broken in combination with
`--location`, `--proxy`, and an https endpoint which redirects. The
user-agent sent to the proxy changes after the client is redirected,
tripping up proxies.
For more information see https://issues.redhat.com/browse/RHEL-45364
Currently, osbuild downloads are identified as coming from `curl`. This
is unfortunate because some RPM mirrors block requests from curl. Let's
"fix" that by introducing our own user-agent. While this can certainly
be seen as "circumventing" a policy, I think that this change is
actually helpful: Now, the mirror maintainers can actually distinguish
osbuild requests from regular curl calls. If they want to block osbuild,
they certainly can, we have no power there, but at least this allows
more fine-grained filtering. Also, our new user-agent contains our
domain name, so if there's a problem, they can contact us.
Only py3.7+ has ThreadingHTTPServer and SimpleHTTPRequestHandler
that can take a directory argument. We could reimplement this
on py36 (easy for threading, harder for missing directory) but
instead this commit just skips tests that try to use a
ThreadingHTTPServer.
Remove once we no longer support py3.6.
Tiny tweak to remove some boilerplate related to tmpfile handling.
The pytest `tmp_path` fixture gives us the tmpdir without having
to worry about cleanup etc (and in a slightly more concise way).
The `amend_secrets()` does not work with real files so there is
no need to mock cachedirs or create fake input files. This commit
just removes those.
It also changes the checksum to `"1"*64` to make it very clear
that the checksum has no significance in this test.
There was a regression with the secrets adding of rhsm for the
curl source. This was my mistake (sorry!). Here is a regression
test that would have prevented this (if we have had it earlier).
Similar to the previous commit to include a `inputs_service` fixture
this does the same for `source.SourcesService` imports.
Note that we cannot easily share the helpers so we have to life with
a bit of very similar but duplicated code. To fix this we would have
to have a shared confftest.py that pytest can find. Which would mean
that we need to put the tests under a common dir that is reachable
via __init__.py files (which we currently not have because stages,
inputs etc do not have a __init__.py so python does not considers
them modules).
The curl source is the only source left that uses "transform". And
here the name is very generic but in fact we only do a single thing:
we add secrets for subscriptions for for mtls to the download.
So rename to make it clear what this is all about.
