Add a new `rpmmdImpl` method `chainDepsolve`, which is able to
depsolve multiple chained package sets as separate DNF transactions
layered on top of each other.
This new method allows to depsolve the `blueprint` package set on top of
the base image package set (usually called `packages`).
Introduce a helper function `chainPackageSets` for constructing
arguments to the `chainDepsolve` method based on the provided arguments:
- slice of package set names to chain as transactions
- map of package sets
- slice of system repositories used by all package sets
- map of package-set-specific repositories
Extend `dnf-json` with a new command `chain-depsolve` allowing to
depsolve multiple transaction in a row, layered on top of each other.
Add unit tests where appropriate.
By default `timeout` is 30 seconds, but we had it set to 5. Drop
the override and use the default.
This has two effects: it increases the time before we give up on
connecting (as it says on the tin), and it also increases the time
download has to be slow for before we give up.
Internally, we were seing failures in downlaoding metadata from ODCS
and similar issues have occurred in CI too.
The potential downside to this is in case of having several mirrors
this means it takes longer before giving up on a bad one and trying
a better one. But slow is better than broken, so for now rever to
the default behavior.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Repo name defaults to the repo ID if the name is not set. `dnf-json`
should not rely on the `reponame` being set to the ID and intsead
return the actual `repoid`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Whitespace around operators and after commas.
No whitespace after opening and before closing brackets.
Two blank lines between top-level functions and classes.
One blank line between class methods.
Indentation fixes.
CacheState.load_cache_state_from_disk() is long and redundant.
CacheState.store_on_disk() is fine (and load_from_disk() would also be
fine) but in the absence of any other store/load sources, the
from_disk() part is also unnecessary.
CacheState.store() and CacheState.load() should be enough.
The dump function will be used to search packages and as we are implementing third party repositories, adding the repo_id of the package will allow us to identify the package repository used.
Detect folders that are not used since some timeout and delete them.
The cache folder must be empty when dnf-json is started in order to
avoid the situation where some folders can never be cleaned up (dnf-json
does not look at the cache directory content but uses information from
the requests to deduce which folders to keep and to delete).
Solves #2020
To avoid dnf leaking memory, dnf-json as a service calls fork() on each
request. This allow memory to be freed automatically when the process
handling the request is destroyed.
Prepare the multi-cache architecture by doing some refactoring.
Mainly this commit adds a solver class that embeds all the logic around
dnf. Responsibilities of communicating on the socket and depsolving are
separated.
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
Never expiring metadata by default leads to surprising behavior
especially for our long-running services. The overhead of expiration
is small but noticeable, attempt some compromise.
This should all be revisited to make dnf-json handle caches better
and be more performant.
acf91a4 enabled fastestmirror but also calls `base.init_plugins()` to
initialize dnf plugins. This is not necessary and not what we want
conceptually.
Not necessary, because `fastestmirror` is a dnf built-in (it was a
plugin during yum-times [1]). The same patch sets the `fastestmirror`
option as well. Thus, this patch does not revert functionality.
Not what we want, because we're using dnf more as a library, explicitly
passing all options. Plugins depend on additional host configuration,
which we'd like to avoid pulling in. In particular, the
subscription-manager plugin tries reading certificates in `/etc/pki`,
which are not readable by the `osbuild-composer` user. This leads to
these errors in the journal:
[ERROR] dnf-json:54297:MainThread @logutil.py:194 -
[Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' -
Further logging output will be written to stderr
[ERROR] dnf-json:54297:MainThread @identity.py:156 -
Reload of consumer identity cert /etc/pki/consumer/cert.pem
raised an exception with msg:
[Errno 13] Permission denied: '/etc/pki/consumer/key.pem'
These errors are not fatal, but could confuse people when inspecting
logs to find unrelated problems. This patch makes them disappear.
[1] https://fedoraproject.org/wiki/Yum_to_DNF_Cheatsheet
The time it takes to depsolve a blueprint varies widely depending on
where the job is running and which mirrors are randomly chosen based on
the data returned in the metalink XML.
Use dnf's fastestmirror plugin to choose the fastest mirror for
downloading metadata. This returns consistent results in PSI + AWS and
every depsolve completed in under 60 seconds after 25 tests in each
cloud.
Fixes#845.
Signed-off-by: Major Hayden <major@redhat.com>
The issue was introduced in 0d3c8329c0.
The patch correctly changed the base exception class, but it didn't
change the unfortunate use of hardcoded type name. This patch uses
Python's internal `__name__` attribute to get the type (exception) name.
If a repo passed to dnf-json contains an sslcacert, sslclientkey, or
sslclientcert then dnf-json will include those values in that repo in
the dnf base.
We were using dnf's default of 48h, but that does not work for
updates repositories, as they depend on an expiration time of 6h.
Allow the metadata_expire value to be configured per repository.
If the value is unset, then never expire the metadata. Set the
value to 6h for all the fedora testing repos.
This fixes issue #476.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Even though `dnf.exceptions.RepoError` is documented as the base error,
`dnf.exceptions.Error` is actually the base error (and also documented
as such).
This was never actually used anywhere, as passing it to dnf-json
was a noop.
We may want to reconsider the concept of a source/repo name and
how it differs from an ID, but for now drop the name.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This will eventually replace the remote_location property. The latter
pins a specific location (a specific mirror), but the two former
can together be used to re-resolve to a more suitable mirror at the
time/place the package will actually be downloaded.
Rather than pinning mirrors in the osbuild manifests, we want to be
able to include the metalink and relative locations so each worker
can use mirrors closer to them.
This would be particularly important when pipelines are rebuilt in
the future, and the best mirrors may have changed.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When we used the dnf-based pipelines, we were relying on the fact
that the metadata was unlikely to have changed between we generated
the pipeline and called osbuild. We achieved this by always updating
to the most recent metadata on every call to rpmmd.Depsolve that
would end up in a pipelin.
Refreshing the metadata is time-consuming, and something we want
to avoid if at all possible. Now that our pipelines no longer
rely on this property, we can drop the flushing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We want depsolving via dnf-json, followed by rpm installation to be
the same as installing directly with dnf. However, the `install_set()`
helper we used inserts the list of packgaes into a set internally
before returning it to us to iterate. Set order iteration is not
a FIFO in python, and because the order of package installation
in rpm is only a partial order, we ended up with different images
depending on whether we installed through dnf or dircetly via rpm.
To avoid the indirection via a set, open-code `install_set()` without
the intermediate allocation.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Without passing in a cachedir, dnf would create a random one for every
invocation. This meant that caches were never reused, nor cleaned up
properly.
Let systemd create a cache directory for us in /var/cache/ and use
that via the environment variable systemd sets for us.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We must avoid depending on the host's state in any way. This achieves
isolation in the following ways:
- rather than the default config file /dev/null is used
- rather than sharing the host persistent state dir a temporary one
is used and thrown away for each call
- the module_platform_id is set explicitly per supported distro, rather
than taken from /etc/os-release.
Optionally, the cache directory can be configured, as we may want to keep
this separate from the host, if for no other reason than accounting.
However, the cache appears to be well-behaved, so we can keep sharing
it between calls (or even with the host). This speeds up things
considerably, so this is definitely what we want.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In our base distro definitions we exclude packages in addition to
including them. Extend dnf-json to support this, so we can depsolve
the base package set as well as the packages added in blueprints.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In adition to the NEVRA, include the location and hash over the rpm
file. This allows us to separately fetch and verify that refernces
to RPMs are correct, as the NEVRA alone is not sufficient for fetching
nor verifying.
This is a prerequisite for using the rpm rather than the dnf stage
in our osbuild pipelines.
Signed-off-by: Tom Gundersen <teg@jklm.no>
dnf-json relies on dnf's ability to cache repository metadata. This is
important, because the API calls it quite often to serve requests for
package lists and depsolves.
However, osbuild's dnf stage always fetches new metadata, because it
doesn't have access to the host's cache. Since metadata is valid for
some time, even after a repository changed, the checksum we put in
the pipeline might be old.
Force a new metadata download when producing the pipeline. This is still
not perfect, but greatly reduces the probability of putting stale
metadata into the pipeline.
Instead of having a static repository checksum, set it dynamically from
the metadata that osbuild-composer last saw. This is implemented in
dnf-json, which returns the checksums for each repository on every call.
This enables the use of repositories that change over time, such as
fedora-updates. Note that the osbuild pipeline will break when such a
repository changes. This is intentional: pipelines have to be
reproducible.
