Similar to `stages` and `sources` we need some basic infrastructure
so that we can use a `mounts_module` fixture for the coming tests
to the mount modules.
The "main" branch is failing right now in tests. The reason is
that we do not have a merge queue and when
https://github.com/osbuild/osbuild/pull/1715
was merged we had no test for `osbuild-depsolve-dnf` yet.
We have one now (THANK YOU achilleas-k) and it shows an issue :)
This commit fixes the issue.
Depsolver test that starts a temporary file server and queries it using
osbuild-depsolve-dnf.
Generates all combinations of repositories configured through the
depsolve-dnf request or the repositories directory and runs the test
cases. The results should be the same regardless of combination.
Test repos are defined with a fake gpg key on the request or repo config
and check if it is read correctly and attached to the repo configs in
the response. The name of the repo is appended to each repo's gpg key
so we can make sure that repo option values don't get swapped.
Add two test rpm metadata directories that can be served as RPM repos.
One was copied from osbuild/images and contains the repository metadata
for CentOS Stream 9 BaseOS.
The second was created by building a simple spec file into an RPM and
creating the metadata using createrepo.
Some of the repository properties in the request were named differently
than the equivalent properties in the dnf repository configuration.
This can introduce bugs and confusion.
One such issue already existed with osbuild/images using 'gpgcheck' in
the request, osbuild-depsolve-dnf5 checking for 'check_gpg', and the dnf
repository configuration property being 'gpgcheck'. This didn't cause
any bad behaviour because osbuild/images reused the original (internal)
configuration to set the property in stages and depsolving isn't
affected by the value of this property.
Change the request properties to match the dnf repository configuration
to avoid confusion: gpgcheck, repo_gpgcheck, and sslverify. Users of
osbuild-depsolve-dnf5 should use property names that match dnf. Use
the same names in the response.
To maintain the same behaviour for SSL verification, a missing sslverify
default to True. The previous property had the opposite meaning,
ignore_ssl, and defaulted to False.
Add the full gpg keys to the repository configs in the response.
On each repository object from dnf, the gpg keys are URLs, either
file:// or http(s)://. We need to resolve these and return them with
in the response.
When the URL is a file:// path, and it comes from a .repo config file,
we assume that the path is relative to the root_dir, so we prepend it to
the path in the file. This is so that repo configs in OS root trees can
be used unmodified. However, when a key is defined in the request, we
should assume that the path is valid, either because it was defined by
the caller as a URL, or because it was defined in-line in the request
and osbuild-depsolve-dnf5 wrote it to the persistdir itself.
A new exception is defined to identify errors during this process.
Support loading repositories from a root tree instead of supplying them
with the request. The repositories should be in the standard yum repo
format. Both repository sources can be defined simultaneously, but at
least one is required.
The root_dir is expected to contain files necessary for depsolving in
the standard paths.
These files are:
- Repository (.repo) configurations in <root_dir>/etc/yum.repos.d/
- GPG key files in <root_dir>/etc/pki/rpm-gpg/
- This will be used to resolve gpg key paths specified in the .repo
files that are relative to the root_dir.
- (Optional) Custom dnf config variables in <root_dir>/etc/dnf/vars or
<root_dir>/usr/share/dnf5/vars.d.
- This is used by CentOS Stream to set the value of $stream.
Custom repository configurations in arbitrary (non-root) paths will have
to follow this directory structure.
A new variable is added to the request, `releasever`, which is mandatory
when using `root_dir`. This variable is used in repository URLs and GPG
key paths. In the default case, dnf reads this variable by inspecting
the rpm database. We will override it in the Solver the same way we
override the arch and basearch for variable substitution. In the
future, we will make this variable mandatory in all cases, which will
make the variable available for repo configs defined in the request as
well.
The root_dir is used in three ways:
- Set the base.conf.installroot
- Set the base.conf.varsdir to <root_dir>/usr/share/dnf5/vars.d and
<root_dir>/etc/dnf/vars to read resolve custom variables when loading
repositories.
- Call create_repos_from_dir() with <root_dir>/etc/yum.repos.d.
base.setup() should be called before loading repositories otherwise
substitutions might not work.
See https://github.com/rpm-software-management/dnf5/issues/1374#issuecomment-2038995031
Some of the repository properties in the request were named differently
than the equivalent properties in the dnf repository configuration.
This can introduce bugs and confusion.
One such issue already existed with osbuild/images using 'gpgcheck' in
the request, osbuild-depsolve-dnf checking for 'check_gpg', and the dnf
repository configuration property being 'gpgcheck'. This didn't cause
any bad behaviour because osbuild/images reused the original (internal)
configuration to set the property in stages and depsolving isn't
affected by the value of this property.
Change the request properties to match the dnf repository configuration
to avoid confusion: gpgcheck, repo_gpgcheck, and sslverify. Users of
osbuild-depsolve-dnf (osbuild/images) should use property names that
match dnf. Use the same names in the response.
To maintain the same behaviour for SSL verification, a missing sslverify
default to True. The previous property had the opposite meaning,
ignore_ssl, and defaulted to False.
Add the full gpg keys to the repository configs in the response.
On each repository object from dnf, the gpg keys are URLs, either
file:// or http(s)://. We need to resolve these and return them with
in the response.
When the URL is a file:// path, and it comes from a .repo config file,
we assume that the path is relative to the root_dir, so we prepend it to
the path in the file. This is so that repo configs in OS root trees can
be used unmodified. However, when a key is defined in the request, we
should assume that the path is valid, either because it was defined by
the caller as a URL, or because it was defined in-line in the request
and osbuild-depsolve-dnf wrote it to the persistdir itself.
A new exception is defined to identify errors during this process.
When generating package sources and rpm stage metadata for a manifest
from a list of packages, we need to associate repository configuration
options to each package [1]. Previously, a caller had all the
repository configurations because they were part of the request, so
packages could be associated with all the repository options by the
repository ID. Now, osbuild-depsolve-dnf will use repositories loaded
from a directory that the caller shouldn't have to read, so returning
all repository configurations in the response makes it possible to
get all package metadata from the response.
This changes the whole structure of the response to a depsolve request.
Previously, we returned an array of packages. Now we return an object
with two keys:
- packages: the array of packages as before
- repositories: an object mapping repository IDs to repository
configurations.
Each package contains the repository ID it comes from (as before), under
`repo_id`. This can be used to get repository configurations and
determine gpg keys and SSL certs for each package.
The new structure avoids duplicating values across all the (sometimes
hundreds) of packages.
[1] 92497c7b1f/pkg/dnfjson/dnfjson.go (L499-L507)
Support loading repositories from a root tree instead of supplying them
with the request. The repositories should be in the standard yum repo
format. Both repository sources can be defined simultaneously, but at
least one is required.
The root_dir is expected to contain files necessary for depsolving in
the standard paths.
These files are:
- Repository (.repo) configurations in <root_dir>/etc/yum.repos.d/
- GPG key files in <root_dir>/etc/pki/rpm-gpg/
- This will be used to resolve gpg key paths specified in the .repo
files that are relative to the root_dir.
- (Optional) Custom dnf config variables in <root_dir>/etc/dnf/vars or
<root_dir>/etc/yum/vars.
- This is used by CentOS Stream to set the value of $stream.
Custom repository configurations in arbitrary (non-root) paths will have
to follow this directory structure.
A new variable is added to the request, `releasever`, which is mandatory
when using `root_dir`. This variable is used in repository URLs and GPG
key paths. In the default case, dnf reads this variable by inspecting
the rpm database. We will override it in the Solver the same way we
override the arch and basearch for variable substitution. In the
future, we will make this variable mandatory in all cases, which will
make the variable available for repo configs defined in the request as
well.
The root_dir is used in two ways:
- Set the base.conf.reposdir to <root_dir>/etc/yum.repos.d.
- Call update_from_etc() with root_dir to read custom variables in
<root_dir>/etc/yum/vars and <root_dir>/etc/dnf/vars.
This allows to combine `org.osbuild.mkdir` with the `osbuild.deployment`
mount and with the upcoming `org.osbuild.bind` mount. The use case is
that we need to create the dir `/var/home` so that `useradd` from inside
a ostree root works (there /home is a symlink and useradd will not
follow the symlink and create a dir in the target by itself).
This allows to write:
```json
{
"type": "org.osbuild.mkdir",
"options": {
"paths": [
{
"path": "/var/home"
}
]
},
"devices": {
"disk": {
"type": "org.osbuild.loopback",
"options": {
"filename": "disk.raw",
"partscan": true
}
}
},
"mounts": [
{
"name": "part4",
"type": "org.osbuild.ext4",
"source": "disk",
"target": "/",
"partition": 4
},
{
"name": "part3",
"type": "org.osbuild.ext4",
"source": "disk",
"target": "/boot",
"partition": 3
},
{
"name": "part2",
"type": "org.osbuild.fat",
"source": "disk",
"target": "/boot/efi",
"partition": 2
},
{
"name": "ostree.deployment",
"type": "org.osbuild.ostree.deployment",
"options": {
"source": "mount",
"deployment": {
"default": true
}
}
},
{
"name": "bind",
"type": "org.osbuild.bind",
"target": "tree://",
"options": {
"source": "mount://"
}
}
]
},
```
to fix this.
To test the curl sources it is very useful to have a small httpd
server that can serve an arbitrary directory. This helper will
ensure that via:
```python
with with osbuild.testutil.net.http_serve_directory(fake_httpd_root) as httpd:
port = httpd.server_port
# download from http://localhost:{port}/<any-path-under-httpd-root>
```
When moving to `bootc install to-filesystem` we will need support
for mounting the deployed disk and writing to the deployment root
this requires that we teach the users and selinux stages to
have them available. This is a first step towards this.
It also adds tests to ensure the options can be passed.
This is a preparation to allow adding mounts/devices to the users
stage so that we can eventually support bootc install to-filesystem.
It also adds some smoke tests for the schema to ensure it's still
valid.
In CoreOS Assembler, some hyperv artifact we `zip` for compression. This
new stage is modeled after the `org.osbuild.tar` stage with necessary
modifications.
Add an org.osbuild.systemd.unit stage using the new format for the
Environment option with two instances to the test manifest.
The contents of the new dropin file at
tree/usr/lib/systemd/system/boltd.service.d/30-boltd-debug.conf are:
[Service]
Environment="G_MESSAGES_DEBUG=all"
Environment="G_MESSAGES_TRACE=none"
Update the org.osbuild.systemd.unit stage to also support multiple
Environment options where each is an object with {key: value}. Enable
the allow_no_value option in configparser so we can add the multiple
entries.
Add the new options to the b.json test and update the diff.
The new file has the following contents:
[Unit]
Description=Create directory
DefaultDependencies=False
ConditionPathExists=|!/etc/myfile
ConditionPathIsDirectory=|!/etc/mydir
[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=mkdir -p /etc/mydir
ExecStart=touch /etc/myfile
Environment="DEBUG=1"
EnvironmentFile=/etc/example.env
[Install]
WantedBy=local-fs.target
RequiredBy=multi-user.target
Add an empty file to the location where the service file will be
created in the b.json version of the test. This way, we will get a
content hash of the created file which is a slightly better test than
just knowing that it was created.
Note that, in the diff, the "before" checksum is the empty file hash:
echo -n '' | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -
In 2d2cdd8097 the file was replaced by
the generated json and it went unnoticed in the PR. Reverted and
updated the options to match the generated json file.
Support the Environment and EnvironmentFile options in the Service
section of the unit file.
The Environment option is set as an object with keys "key" and "value"
and the key is validated with a pattern.
Updated the stage to special-case the Environment option.
Tiny tweak to remove some boilerplate related to tmpfile handling.
The pytest `tmp_path` fixture gives us the tmpdir without having
to worry about cleanup etc (and in a slightly more concise way).
The `amend_secrets()` does not work with real files so there is
no need to mock cachedirs or create fake input files. This commit
just removes those.
It also changes the checksum to `"1"*64` to make it very clear
that the checksum has no significance in this test.
There was a regression with the secrets adding of rhsm for the
curl source. This was my mistake (sorry!). Here is a regression
test that would have prevented this (if we have had it earlier).
