The time it takes to depsolve a blueprint varies widely depending on
where the job is running and which mirrors are randomly chosen based on
the data returned in the metalink XML.
Use dnf's fastestmirror plugin to choose the fastest mirror for
downloading metadata. This returns consistent results in PSI + AWS and
every depsolve completed in under 60 seconds after 25 tests in each
cloud.
Fixes#845.
Signed-off-by: Major Hayden <major@redhat.com>
The issue was introduced in 0d3c8329c0.
The patch correctly changed the base exception class, but it didn't
change the unfortunate use of hardcoded type name. This patch uses
Python's internal `__name__` attribute to get the type (exception) name.
If a repo passed to dnf-json contains an sslcacert, sslclientkey, or
sslclientcert then dnf-json will include those values in that repo in
the dnf base.
We were using dnf's default of 48h, but that does not work for
updates repositories, as they depend on an expiration time of 6h.
Allow the metadata_expire value to be configured per repository.
If the value is unset, then never expire the metadata. Set the
value to 6h for all the fedora testing repos.
This fixes issue #476.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Even though `dnf.exceptions.RepoError` is documented as the base error,
`dnf.exceptions.Error` is actually the base error (and also documented
as such).
This was never actually used anywhere, as passing it to dnf-json
was a noop.
We may want to reconsider the concept of a source/repo name and
how it differs from an ID, but for now drop the name.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This will eventually replace the remote_location property. The latter
pins a specific location (a specific mirror), but the two former
can together be used to re-resolve to a more suitable mirror at the
time/place the package will actually be downloaded.
Rather than pinning mirrors in the osbuild manifests, we want to be
able to include the metalink and relative locations so each worker
can use mirrors closer to them.
This would be particularly important when pipelines are rebuilt in
the future, and the best mirrors may have changed.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When we used the dnf-based pipelines, we were relying on the fact
that the metadata was unlikely to have changed between we generated
the pipeline and called osbuild. We achieved this by always updating
to the most recent metadata on every call to rpmmd.Depsolve that
would end up in a pipelin.
Refreshing the metadata is time-consuming, and something we want
to avoid if at all possible. Now that our pipelines no longer
rely on this property, we can drop the flushing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We want depsolving via dnf-json, followed by rpm installation to be
the same as installing directly with dnf. However, the `install_set()`
helper we used inserts the list of packgaes into a set internally
before returning it to us to iterate. Set order iteration is not
a FIFO in python, and because the order of package installation
in rpm is only a partial order, we ended up with different images
depending on whether we installed through dnf or dircetly via rpm.
To avoid the indirection via a set, open-code `install_set()` without
the intermediate allocation.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Without passing in a cachedir, dnf would create a random one for every
invocation. This meant that caches were never reused, nor cleaned up
properly.
Let systemd create a cache directory for us in /var/cache/ and use
that via the environment variable systemd sets for us.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We must avoid depending on the host's state in any way. This achieves
isolation in the following ways:
- rather than the default config file /dev/null is used
- rather than sharing the host persistent state dir a temporary one
is used and thrown away for each call
- the module_platform_id is set explicitly per supported distro, rather
than taken from /etc/os-release.
Optionally, the cache directory can be configured, as we may want to keep
this separate from the host, if for no other reason than accounting.
However, the cache appears to be well-behaved, so we can keep sharing
it between calls (or even with the host). This speeds up things
considerably, so this is definitely what we want.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In our base distro definitions we exclude packages in addition to
including them. Extend dnf-json to support this, so we can depsolve
the base package set as well as the packages added in blueprints.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In adition to the NEVRA, include the location and hash over the rpm
file. This allows us to separately fetch and verify that refernces
to RPMs are correct, as the NEVRA alone is not sufficient for fetching
nor verifying.
This is a prerequisite for using the rpm rather than the dnf stage
in our osbuild pipelines.
Signed-off-by: Tom Gundersen <teg@jklm.no>
dnf-json relies on dnf's ability to cache repository metadata. This is
important, because the API calls it quite often to serve requests for
package lists and depsolves.
However, osbuild's dnf stage always fetches new metadata, because it
doesn't have access to the host's cache. Since metadata is valid for
some time, even after a repository changed, the checksum we put in
the pipeline might be old.
Force a new metadata download when producing the pipeline. This is still
not perfect, but greatly reduces the probability of putting stale
metadata into the pipeline.
Instead of having a static repository checksum, set it dynamically from
the metadata that osbuild-composer last saw. This is implemented in
dnf-json, which returns the checksums for each repository on every call.
This enables the use of repositories that change over time, such as
fedora-updates. Note that the osbuild pipeline will break when such a
repository changes. This is intentional: pipelines have to be
reproducible.
Make each command accept a `repos` key containing repository
descriptions.
Make weldr API pass the repository like this. Nothing should change,
because the repos were the same (Fedora 30).
When osbuild-composer is run as systemd service, we don't want to write
anything into working directory. Currently, we write dnf cache into it.
Instead, let's just use the default dnf cache directory.
