The _dnfrepo is full of branches, turn off warning.
The keyfile doesn't use 'with' because the file needs to remain
available when the function exits. Cleanup of persistdir will clean up
the temporary file used for the key.
This allows verification of repository metadata signatures.
The gpgkeys field is a list of key urls, or the gpg key itself, starting
with '-----BEGIN PGP PUBLIC KEY BLOCK-----'. These will be written to a
temporary file, and that file:// url will be passed to dnf.
Use the DNF query API
(https://dnf.readthedocs.io/en/latest/api_queries.html) to quickly
return results matching a glob pattern. Multiple package glob results
are combined into a single response.
This adds a search dict to the arguments. 'packages' is a list of package
names or globs to search for.
An optional 'latest' boolean will return only the latest NEVRA instead
of all matching builds in the metadata.
eg.
"search": {
"latest": false,
"packages": ["tmux", "vim*", "*ssh*"]
},
When a DepsolveError exception occurs, the error message would print the
packages in the request. When the request arguments changed, the error
message handling wasn't updated and would fail to produce the correct
error message.
Compile a list of packages from all transactions and print them in the
error message as a comma-separated list.
The repository checksums in the response from dnf-json aren't used
anywhere. Since we're making changes to dnf-json and depsolving, now is
a good opportunity to drop them completely.
Defined a Hash() method on rpmmd.RepoConfig that calculates a SHA-256 ID
for a repository based on its configuration. Identical configurations
should produce the same ID. The Name and ImageTypeTags of a repository
aren't taken into account. These attributes affect a repository's
functional configuration.
This ID lets us change the way we handle repository configurations in a
few places:
- Preparing the depsolve job arguments is simpler since we have
predictable IDs for the repository configurations. We don't need to
rely on the index of a RepoConfig in a list to identify or access it,
which prevented us from building a list of all repository
configurations, since we needed them to be placed in the list in a
certain order.
- Associating packages from the depsolve result with the repository
configuration (in depsToRPMMD) no longer relies on an ID string
converted from and back to an integer index. Repositories define
their own IDs.
- Tests are a bit messier now but the changes simplify the main code, so
it's an acceptable trade-off.
- Fixtures need to change based on the repository configuration for
the test.
- We need to calculate the ID for the repository configuration for
the temporary file server URL.
Remove the single Depsolve function from the dnfjson package and the
depsolve command from the dnf-json tool. The new ChainDepsolve
functions and chain-depsolve command can handle single depsolves in the
same way so there's no need to keep (and have to maintain) two versions
of very similar code.
The ChainDepsolve function (in Go) and chain-depsolve command (in
Python) have been renamed to plain Depsolve and depsolve respectively,
since they are now general purpose depsolve functions.
- Removed server class and handlers
The dnf-json Python script will no longer run as a service. In the
future, we will create a service in the Go package that will handle
receiving requests and calling the script accordingly.
- Removed CacheState class
- Added standalone functions for setting up cache and running the
depsolve
- Validate the input before reading
- Print all messages (status and error) to stderr and print only the
machine-readable results to stdout (including structured error)
The status messages on stderr are useful for troubleshooting. When
called from the service they will appear in the log/journal.
- Catch RepoError exceptions
This can occur when dnf fails to load the repository configuration.
- Support multiple depsolve jobs per request
The structure is changed to support making multiple depsolve
requests but reuse the dnf.Base object to make chained (incremental)
dependency resolution requests.
Before:
{
"command": "depsolve",
"arguments": {
"package-specs": [...],
"exclude-specs": [...],
"repos": [{...}],
"cachedir": "...",
"module_platform_id": "...",
"arch": "..."
}
}
After:
{
"command": "depsolve",
"cachedir": "...",
"module_platform_id": "...",
"arch": "...",
"arguments": {
"repos": [{...}],
"transactions": [
{
"package-specs": [...],
"exclude-specs": [...],
"repo-ids": [...]
}
]
}
}
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Add a new `rpmmdImpl` method `chainDepsolve`, which is able to
depsolve multiple chained package sets as separate DNF transactions
layered on top of each other.
This new method allows to depsolve the `blueprint` package set on top of
the base image package set (usually called `packages`).
Introduce a helper function `chainPackageSets` for constructing
arguments to the `chainDepsolve` method based on the provided arguments:
- slice of package set names to chain as transactions
- map of package sets
- slice of system repositories used by all package sets
- map of package-set-specific repositories
Extend `dnf-json` with a new command `chain-depsolve` allowing to
depsolve multiple transaction in a row, layered on top of each other.
Add unit tests where appropriate.
By default `timeout` is 30 seconds, but we had it set to 5. Drop
the override and use the default.
This has two effects: it increases the time before we give up on
connecting (as it says on the tin), and it also increases the time
download has to be slow for before we give up.
Internally, we were seing failures in downlaoding metadata from ODCS
and similar issues have occurred in CI too.
The potential downside to this is in case of having several mirrors
this means it takes longer before giving up on a bad one and trying
a better one. But slow is better than broken, so for now rever to
the default behavior.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Repo name defaults to the repo ID if the name is not set. `dnf-json`
should not rely on the `reponame` being set to the ID and intsead
return the actual `repoid`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Whitespace around operators and after commas.
No whitespace after opening and before closing brackets.
Two blank lines between top-level functions and classes.
One blank line between class methods.
Indentation fixes.
CacheState.load_cache_state_from_disk() is long and redundant.
CacheState.store_on_disk() is fine (and load_from_disk() would also be
fine) but in the absence of any other store/load sources, the
from_disk() part is also unnecessary.
CacheState.store() and CacheState.load() should be enough.
The dump function will be used to search packages and as we are implementing third party repositories, adding the repo_id of the package will allow us to identify the package repository used.
Detect folders that are not used since some timeout and delete them.
The cache folder must be empty when dnf-json is started in order to
avoid the situation where some folders can never be cleaned up (dnf-json
does not look at the cache directory content but uses information from
the requests to deduce which folders to keep and to delete).
Solves #2020
To avoid dnf leaking memory, dnf-json as a service calls fork() on each
request. This allow memory to be freed automatically when the process
handling the request is destroyed.
Prepare the multi-cache architecture by doing some refactoring.
Mainly this commit adds a solver class that embeds all the logic around
dnf. Responsibilities of communicating on the socket and depsolving are
separated.
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
Never expiring metadata by default leads to surprising behavior
especially for our long-running services. The overhead of expiration
is small but noticeable, attempt some compromise.
This should all be revisited to make dnf-json handle caches better
and be more performant.
acf91a4 enabled fastestmirror but also calls `base.init_plugins()` to
initialize dnf plugins. This is not necessary and not what we want
conceptually.
Not necessary, because `fastestmirror` is a dnf built-in (it was a
plugin during yum-times [1]). The same patch sets the `fastestmirror`
option as well. Thus, this patch does not revert functionality.
Not what we want, because we're using dnf more as a library, explicitly
passing all options. Plugins depend on additional host configuration,
which we'd like to avoid pulling in. In particular, the
subscription-manager plugin tries reading certificates in `/etc/pki`,
which are not readable by the `osbuild-composer` user. This leads to
these errors in the journal:
[ERROR] dnf-json:54297:MainThread @logutil.py:194 -
[Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' -
Further logging output will be written to stderr
[ERROR] dnf-json:54297:MainThread @identity.py:156 -
Reload of consumer identity cert /etc/pki/consumer/cert.pem
raised an exception with msg:
[Errno 13] Permission denied: '/etc/pki/consumer/key.pem'
These errors are not fatal, but could confuse people when inspecting
logs to find unrelated problems. This patch makes them disappear.
[1] https://fedoraproject.org/wiki/Yum_to_DNF_Cheatsheet
The time it takes to depsolve a blueprint varies widely depending on
where the job is running and which mirrors are randomly chosen based on
the data returned in the metalink XML.
Use dnf's fastestmirror plugin to choose the fastest mirror for
downloading metadata. This returns consistent results in PSI + AWS and
every depsolve completed in under 60 seconds after 25 tests in each
cloud.
Fixes#845.
Signed-off-by: Major Hayden <major@redhat.com>
The issue was introduced in 0d3c8329c0.
The patch correctly changed the base exception class, but it didn't
change the unfortunate use of hardcoded type name. This patch uses
Python's internal `__name__` attribute to get the type (exception) name.
If a repo passed to dnf-json contains an sslcacert, sslclientkey, or
sslclientcert then dnf-json will include those values in that repo in
the dnf base.
We were using dnf's default of 48h, but that does not work for
updates repositories, as they depend on an expiration time of 6h.
Allow the metadata_expire value to be configured per repository.
If the value is unset, then never expire the metadata. Set the
value to 6h for all the fedora testing repos.
This fixes issue #476.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Even though `dnf.exceptions.RepoError` is documented as the base error,
`dnf.exceptions.Error` is actually the base error (and also documented
as such).
This was never actually used anywhere, as passing it to dnf-json
was a noop.
We may want to reconsider the concept of a source/repo name and
how it differs from an ID, but for now drop the name.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This will eventually replace the remote_location property. The latter
pins a specific location (a specific mirror), but the two former
can together be used to re-resolve to a more suitable mirror at the
time/place the package will actually be downloaded.
Rather than pinning mirrors in the osbuild manifests, we want to be
able to include the metalink and relative locations so each worker
can use mirrors closer to them.
This would be particularly important when pipelines are rebuilt in
the future, and the best mirrors may have changed.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When we used the dnf-based pipelines, we were relying on the fact
that the metadata was unlikely to have changed between we generated
the pipeline and called osbuild. We achieved this by always updating
to the most recent metadata on every call to rpmmd.Depsolve that
would end up in a pipelin.
Refreshing the metadata is time-consuming, and something we want
to avoid if at all possible. Now that our pipelines no longer
rely on this property, we can drop the flushing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We want depsolving via dnf-json, followed by rpm installation to be
the same as installing directly with dnf. However, the `install_set()`
helper we used inserts the list of packgaes into a set internally
before returning it to us to iterate. Set order iteration is not
a FIFO in python, and because the order of package installation
in rpm is only a partial order, we ended up with different images
depending on whether we installed through dnf or dircetly via rpm.
To avoid the indirection via a set, open-code `install_set()` without
the intermediate allocation.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Without passing in a cachedir, dnf would create a random one for every
invocation. This meant that caches were never reused, nor cleaned up
properly.
Let systemd create a cache directory for us in /var/cache/ and use
that via the environment variable systemd sets for us.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We must avoid depending on the host's state in any way. This achieves
isolation in the following ways:
- rather than the default config file /dev/null is used
- rather than sharing the host persistent state dir a temporary one
is used and thrown away for each call
- the module_platform_id is set explicitly per supported distro, rather
than taken from /etc/os-release.
Optionally, the cache directory can be configured, as we may want to keep
this separate from the host, if for no other reason than accounting.
However, the cache appears to be well-behaved, so we can keep sharing
it between calls (or even with the host). This speeds up things
considerably, so this is definitely what we want.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In our base distro definitions we exclude packages in addition to
including them. Extend dnf-json to support this, so we can depsolve
the base package set as well as the packages added in blueprints.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In adition to the NEVRA, include the location and hash over the rpm
file. This allows us to separately fetch and verify that refernces
to RPMs are correct, as the NEVRA alone is not sufficient for fetching
nor verifying.
This is a prerequisite for using the rpm rather than the dnf stage
in our osbuild pipelines.
Signed-off-by: Tom Gundersen <teg@jklm.no>
