Move `OSBuildMetadataToRPMs()` and `PackageMetadataToSignature()`
functions from the `rpmmd` package to `osbuild2` package to prevent
import cycles while de-duplicating `rpmStageInputs()` function from
`stage_inputs.go` of distro definitions.
Rename `PackageMetadataToSignature()` to
`RPMPackageMetadataToSignature()`, since it takes specifically
`RPMPackageMetadata` type as an argument.
Adjust affected parts of code (unit tests, cloudapi, worker).
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the `kernelVerStr()` function duplicated in many
distro definitions to the `rpmmd` package as
`GetVerStrFromPackageSpecListPanic()`.
I could not come up with a better name, sorry.
This will prevent creating another copy of the code in rhel-84 for
the `gce` image.
This change initially exposed a bug in the original implementation of
`kernelVerStr()`. Since on the first line, we allocate an empty structure
into `kernelPkg` variable, it can never be `nil` and the function never
panicked even if there was no `kernel` package in the PackageSpec list.
Fix all unit tests to provide valid arguments when calling `Manifest()`
method of image types.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
kernelVerStr fixup
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
Function renamed to better fit the argument element type
(StageMetadata).
Argument is a map to fit the pipeline metadata in the result object.
Signature function is made public to be reused in the cloud API
conversion.
Metadata test raw value updated to v2 result format.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
This is a confusing part of the startup process. We have changed the naming
and we also added the cross distro building feature. It is unclear which
files are loaded and from where (if /etc or /usr).
Log the files that are loaded so every user can clearly see what
configuration osbuild-composer uses. This complements the log of loaded
configuration.
It can happen that the system is not subscribed and the user requests a
source with rhsm set to "true". Return useful error message in such case
informing the user what to do about it.
The problem: osbuild-composer used to have a rather uncomplete logic for
selecting client certificates and keys while fetching data from
repositories that use the "subscription model". In this scenario, every
repo requires the user to use a client-side TLS certificate. The problem
is that every repo can use its own CA and require a different pair of
a certificate and a key. This case wasn't handled at all in composer.
Furthermore, osbuild-composer can use remote workers which complicates
things even more.
Assumptions: The problem outlined above is hard to solve in the general
case, but Red Hat Subscription Manager places certain limitations on how
subscriptions might be used. For example, a subscription must be tight to
a host system, so there is no way to use such a repository in osbuild-composer
without it being available on the host system as well.
Also, if a user wishes to use a certain repository in osbuild-composer it
must be available on both hosts: the composer and the worker. It will come
with different pair of a client certificate and a key but otherwise, its
configuration remains the same.
The solution: Expect all the subscriptions to be registered in the
/etc/yum.repos.d/redhat.repo file. Read the mapping of URLs to certificates
and keys from there and use it. Don't change the manifest format and let
osbuild guess the appropriate subscription to use.
Add package sets to the top level distribution and the
architectures. Merge package sets when requested through
`imageType.PackageSets()`.
This allows us to have "cascading" package sets:
distro > arch > image type
Previously this was only supported for the build packages. The idea is
generalised based on the idea of arbitrary package sets.
Certain package sets have special meaning (for the distro):
- packages: blueprint packages and special option packages are merged
into this one.
- build: is not expected to be defined on the image type so it is merged
from the distro and arch explicitly.
Move the OSBuildStagesToRPMs function, associated test, and RPM type
from the worker into the rpmmd subpackge. We will use this function in
the cloud API to compile the NEVRAs for the new metadata endpoint.
Extend the RepoConfig structure to contain new field ImageTypeTags.
Extend also other structures and functions as needed, to support loading
repository definitions, which use this new field. The idea is that a
repository should be used for building all image types, unless it has
some ImageTypeTags defined. In such case, it should be used only for
building the specific image types, which names are specified in the new
field.
Add RepoRegistry as a higher-level API to load and manage repository
definitions for each distribution. Currently it provides one method,
which returns a set of repositories needed to build a given image
type. The RepoRegistry uses the new ImageTypeTags field in the RepoConfig
structure and returns all the needed repositories for the image type.
Modify rpmmd unit tests and add unit tests for RepoRegistry.
Add News entry describing the change done to RepoConfig and its JSON
representation.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Separate the loading of repo definitions from JSON file from
`LoadRepositories()` to a standalone function
`loadRepositoriesFromFile()`, to make it easy to reuse it in the future.
Add unit tests for `LoadRepositories()` function.
Exclude github.com/osbuild/osbuild-composer/internal/rpmmd/test package
from test coverage. Package with just tests and no other code makes `go
test` to fail. This should be fixed in go 1.17.
See https://github.com/golang/go/issues/27333
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This replaces Packages() and BuildPackages() by returning a map of
package sets, the semantics of which is up to the distro to define.
They are meant to be depsolved and the result returned back as a
map to Manifest(), with the same keys.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Currently each repo is annotated with this information, use the mapping
we have from packages to their originating repos to annotate each
PackageSpec with the same information.
This information is not serialized, and not yet used. It may be used to
expose this information to osbuild's org.osbuild.rpm stage.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use the new dnfPackageSpec only for communicating with dnf-json. This
allows us to drop the RepoID from the public type, as that is only meant
to be used internally in `Depsolve()`.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The sources weldr API already supports this, so hook it up to be
represented on disk and in our internal state tracking too.
This does not yet hook this up to be respected by osbuild, which
currently takes this to be unconditionally set to true.
Signed-off-by: Tom Gundersen <teg@jklm.no>
RHEL doesn't have python3 in PATH by default. Instead it has /usr/libexec/
platform-python. When building the RPM, RHEL correctly mangles dnf-json's
shebang to the platform python. This commit switches osbuild-composer
to use this shebang because it should always respect target platform's
default python binary.
Fixes#745
RPMMD had hardcoded path to dnf-json helper. This required all executables
using RPMMD to be run in the directory where dnf-json was located. This commit
makes RPMMD take the path to dnf-json as an argument. This allows its
consumers to specify whichever path they want.
Not a functional change
This is currently only used in tests. We broke backwards compatibility
when we split the repo name and the repo id. The tests expect this to be
indexed by name, not index.
This should probably be reworked or dropped, for now revert to the old
behavior.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The 'id' field in the json files is exposed as 'name' in the weldr API,
and that is now its only use. Rename it to match.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is how it is used in the rest of the code, as a name to represent
the repository in the weldr API. Rename to match its use, and avoid
confusion with the ID passed to dnf-json, which is not the same.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We don't have to go via a map, and also don't have to rely on the ID's
in the passed in repos are the same, by instead relying on RepoID being
the offset into the passed in repos array, as set in the previous patch.
Signed-off-by: Tom Gundersen <teg@jklm.no>
rpmmd now contains three repo config structs. RepoConfig is for use
throughout osbuild-composer. repository is for reading from our
internal json repository files before creating a RepoConfig.
dnfRepoConfig is is for use within rpmmd and it matches what dnf-json
expects from a repo.
Repos now contain support for rhsm. In order to connect to rhel's
cdn, dnf must pass an sslcacert, sslcakey, and sslclientcert. When a
repo is used for fetching metadata or depsolving it is checked for
rhsm secrets. If secrets are needed they are retrieved from the host
system. Packages requiring rhsm are marked as requiring
"org.osbuild.rhsm" secrets.
We were using dnf's default of 48h, but that does not work for
updates repositories, as they depend on an expiration time of 6h.
Allow the metadata_expire value to be configured per repository.
If the value is unset, then never expire the metadata. Set the
value to 6h for all the fedora testing repos.
This fixes issue #476.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The format of the BuildTime returned by /projects/list and /modules/list
does not include the 'Z' at the end. This fixed the format and adjusts
the tests.
The following commit will introduce support for forced architecture in
dnf-json. The APIs already have this kind of information, so we can
simply pass it to the Depsolve and FetchMetadata functions.
This was never actually used anywhere, as passing it to dnf-json
was a noop.
We may want to reconsider the concept of a source/repo name and
how it differs from an ID, but for now drop the name.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This will eventually replace the remote_location property. The latter
pins a specific location (a specific mirror), but the two former
can together be used to re-resolve to a more suitable mirror at the
time/place the package will actually be downloaded.
Rather than pinning mirrors in the osbuild manifests, we want to be
able to include the metalink and relative locations so each worker
can use mirrors closer to them.
This would be particularly important when pipelines are rebuilt in
the future, and the best mirrors may have changed.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When we used the dnf-based pipelines, we were relying on the fact
that the metadata was unlikely to have changed between we generated
the pipeline and called osbuild. We achieved this by always updating
to the most recent metadata on every call to rpmmd.Depsolve that
would end up in a pipelin.
Refreshing the metadata is time-consuming, and something we want
to avoid if at all possible. Now that our pipelines no longer
rely on this property, we can drop the flushing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
rpmmd looked at the CACHE_DIRECTORY environment variable to set a path
for the dnf repository cache. Aside from being a smelly thing to do
from a library, this breaks osbuild-pipeline and osbuild-dnf-json-tests,
which don't run as systemd services and thus don't have CACHE_DIRECTORY
set.
Explicitly pass the cache directory to rpmmd. Keep using a path based on
CACHE_DIRECTORY for osbuild-composer. Use the user's `.cache` directory
for osbuild-pipeline and a temporary directory for the tests.
This can happen when CacheDirectory= is missing from the service file.
That's unlikely to happen, but it's hard to figure out what caused the
failure when it does. Be explicit and panic.
The LoadRepositories function interates over a list of paths and expects
to find a distro configuration in one of them. The case when no path
with valid configuration is found was not handled. This patch introduces
the check.
Without passing in a cachedir, dnf would create a random one for every
invocation. This meant that caches were never reused, nor cleaned up
properly.
Let systemd create a cache directory for us in /var/cache/ and use
that via the environment variable systemd sets for us.
Signed-off-by: Tom Gundersen <teg@jklm.no>
the name was misleading because the function could do more than just
download package list. In PushComposeRequest it is also used to fetch
checksums for the repositories, therefore I decided to rename it to
reflect this usage.
We must avoid depending on the host's state in any way. This achieves
isolation in the following ways:
- rather than the default config file /dev/null is used
- rather than sharing the host persistent state dir a temporary one
is used and thrown away for each call
- the module_platform_id is set explicitly per supported distro, rather
than taken from /etc/os-release.
Optionally, the cache directory can be configured, as we may want to keep
this separate from the host, if for no other reason than accounting.
However, the cache appears to be well-behaved, so we can keep sharing
it between calls (or even with the host). This speeds up things
considerably, so this is definitely what we want.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In our base distro definitions we exclude packages in addition to
including them. Extend dnf-json to support this, so we can depsolve
the base package set as well as the packages added in blueprints.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In adition to the NEVRA, include the location and hash over the rpm
file. This allows us to separately fetch and verify that refernces
to RPMs are correct, as the NEVRA alone is not sufficient for fetching
nor verifying.
This is a prerequisite for using the rpm rather than the dnf stage
in our osbuild pipelines.
Signed-off-by: Tom Gundersen <teg@jklm.no>
During development of a new distro, we need to test composer against
nightly or beta repositories, but we cannot ship composer itself
with the nightly repository information hardcoded in. At the same
time, we want to distinguish between the system repositories of the
host and the repositories we use to generate images (the host may not
use the same distro/version/architecture as the target, and it may
include custom repositories that the target should not).
We therefore ship per distro repository information that can be
overriden (typically in testing) by dropping files in /etc.
For now use the latest nightlies for RHEL-8.2, we may want to
replace these with the official mirrors for GA eventually.
Signed-off-by: Tom Gundersen <teg@jklm.no>
dnf-json relies on dnf's ability to cache repository metadata. This is
important, because the API calls it quite often to serve requests for
package lists and depsolves.
However, osbuild's dnf stage always fetches new metadata, because it
doesn't have access to the host's cache. Since metadata is valid for
some time, even after a repository changed, the checksum we put in
the pipeline might be old.
Force a new metadata download when producing the pipeline. This is still
not perfect, but greatly reduces the probability of putting stale
metadata into the pipeline.
Instead of having a static repository checksum, set it dynamically from
the metadata that osbuild-composer last saw. This is implemented in
dnf-json, which returns the checksums for each repository on every call.
This enables the use of repositories that change over time, such as
fedora-updates. Note that the osbuild pipeline will break when such a
repository changes. This is intentional: pipelines have to be
reproducible.
We should always specify a checksum when describing a repository to pull
content from. For now, fedora-30 duplicated the checksum. The idea is
that we can have it in one place at some point.
