We run tests using tox to set up environments for different python
versions to test against. The problem is that when a test shells out to
a python script, like osbuild-depsolve-dnf5, it's not run inside the
environment but in the system environment. The `has_dnf5()` check
returns False because it fails to import dnf5 in the tox environment,
even though the script can be run and the test will succeed.
Use `python3 -c "import libdnf5"` to decide if the script is runnable
instead of using `importlib`. This doesn't solve the problem of our
tests running python scripts in a different environment than the one (we
think) we are testing, but it will enable tests of osbuild-depsolve-dnf5
for now.
The JSON output by go will use 'null' for nil slices, so we need to
use a [] when the field is missing, or when it is set to null.
Previously this was handled by checking the value before iterating but
when the code moved for the directory handling it was changed.
This implements the same behavior in a slightly cleaner way.
The go json serialize process takes a nil slice and converts that to a
'null' instead of an '[]' or leaving off the field. This means that the
json generated by python is not representative of what is output by
images/dnfjson and we have not been properly testing
osbuild-depsolve-dnf5
Testing dnf5 depsolving *also* requires that libdnf5 be present in the
environment, which is a separate problem that still needs to be fixed.
Cache files will split the extension, this means that all pyc cache
files looks like we get many clashing `org.osbuild.cpython-py311.pyc
files. Moreover, the cache bytecode invalidation is based on the
timestamp (which is the same after git checkout) and the file size
(which may be the same for two different files). This means that we
can't rely on the cache files.
This issue has been found after the previous commit made the
`org.osbuild.systemd` and `org.osbuild.selinux` stages to have exactly
the same size, which caused the interpreter to reuse the bytecode for
the selinux stage when running unit tests for the systemd stage. This
resulted in consistent and weird failures when the systemd stage
options were passed to the selinux stage code.
The credit for this fix goes to Michael Vogt, who found the cause and
fix. Also thanks to Simon de Vlieger for his help with debugging the
problem.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the SELinux stage would not force full contexts reset when
forcing auto-relabel on first boot. As a result, all files remained
`unconfined_u` after the auto-relabeling on first boot and only the type
part was reset.
We really need to mimic the behavior of `fixfiles -F onboot` command,
which creates the `/.autorelabel` file with "-F" in it.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Given that the tests `test_depsolve{,_dnf5}` are identical except
the `command` extract them into a common test that is just parameterized.
This will also help to avoid drift between the two resolvers.
This commit allows use to append kernel commandline options via
the kickstart file. This is useful for e.g. the bootc installer
where we support customizing this via blueprints.
Verifying the systemd unit also checks if any referred systemd units
(Wants, Requires, After) exist and if all commands in Exec exist and are
executable. Without '--root', the systemd-analyze verify command is
testing this against files in the build root, which isn't valid.
Units and binaries might not exist in the build root when referenced in
the image root tree, making the unit fail when when it's valid.
Conversely, the verification can succeed by finding executables in the
build root that don't exist in the image root tree when it should be
failing.
When verifying user units, systemd expects runtime directories.
All of this makes it quite difficult to verify systemd units properly
when building an image. The call is useful for making sure the unit is
structured properly, but the user unit verification setup is difficult
to accomplish in a general way while building.
Remove the systemd-analyze verify step from the stage. Move it to the
unit test so that we have some assurance that our unit file structure is
correct and things work as expected. Create referenced unit files and
commands to make the unit valid.
Add test values for Wants, Requires, and After.
Adding multiple values to test that arrays work and made sure they're
all different.
The units need to be valid, real unit names otherwise the
'systemd-analyze verify' check will fail.
The current `make test-all` will only run tests under `test/`. This
is no longer the only place we have tests so update the code to
just run `pytest` to collect all tests.
We currently do not set the `--target-imgref` and do not allow to
override it. This means that on a fresh deploy it is set to an
incorrect value. This commit allows to set it via the
org.osbuild.bootc.install-to-filesystem stage.
Only py3.7+ has ThreadingHTTPServer and SimpleHTTPRequestHandler
that can take a directory argument. We could reimplement this
on py36 (easy for threading, harder for missing directory) but
instead this commit just skips tests that try to use a
ThreadingHTTPServer.
Remove once we no longer support py3.6.
Run only two jobs in the GH runner for the unittest. The `test_stage.py`
because it takes a very long time and needs to run in parallel and
all the other tests.
This split avoid that we forget to add new unittests to the matrix
as we did before (see e.g. https://github.com/osbuild/osbuild/pull/1731)
and it will also enable the tests in:
- sources/test
- inputs/tests
- mounts/test
to run.
This will reduce the "granularity" of the test output a bit, in the
GH runner we only see two unit test matrix jobs now. However that
should not be too bad because the non-stage tests are really quick
to run.
Using an environ for passing the tests is a bit of a headache when
it comes to quoting which is important when trying to write something
like: `-k "not test_stages.py"`.
I (personally) also find it slightly nicer/more intuitive to be able to do:
```
$ tox -e py36 -- ./test/mod
```
compared to
```
$ TEST_CATEGORY="./test/mod" tox -e py36
```
With the new `bootc install to-filesystem` support many stages
will need a devices/mount setup to bind mount the deployment root
from the bootc deployment root of the generated image. To make
this globally available just allow "devices/mounts" for all stages
in the schema validation.
Note that `mounts` is already globally allowed so this just adds
devices (this was added in `7e776a076` with ostree as the use-case).
Nothing will change for the filesystem stages that already define
"devices" in a more specialized way.
str.removeprefix() is not available in Python 3.6, which we need to
support EL8.
Instead of removing the prefix from the original path string, take the
path property of the parsed URL. Since we're dealing with file paths,
there will be no params, query, or fragment strings so the path should
be all we need.
So that we see any error output during the tests in "realtime". With
subprocess check=True and capture_output=True on exit_code != 0 no
stderr as part of the exception by default so this change helps
seeing issues from depsolve-dnf more easily.
Add a new line to after a successful build and before the final output
is printed to the terminal. Since the final build output and the
"manifest finished successfully" line were being printed to the same
line.
Originally, I made releasever required only when root_dir was set. This
was initially done to maintain backwards compatibility but we broke that
already and osbuild/images will always include releasever in the
request.
Extract make_dnf_scafolding as a helper, mostly so that the config_combos()
function is easier to read. It seems one core concept here is the iteration
of "combo[0]" and "combo[1]" so having them symetrical at the same indent
level feel easier to read to me.
This adds a new `org.osbuild.bind` mount feature to the osbuild
mount modules. This allows to (r)bind mount parts of another mount
into the tree (or replace the default tree for a stage entirely).
The use case is the `bootc install to-filesystem` where we get
a populated disk and need to do customizations directly there
without going through an intermediate tree.
Note that right now only "--rbind" is supported and used but
we could trivially change that to become an option in either
direction. Given that the main use-case right now is to be
paried with `org.osbuild.ostree.deployment` and here the
`rbind` is crucial I would leave that the default.
Here is an example what this looks like:
```json
{
"type": "org.osbuild.users",
"options": {
"users": {
"alice": {
"home": "/home/alice",
"groups": [
"wheel"
],
"password": "$6$NV3P7UzUqP3xb1ML$3qnHpWs037VRTaOc.kirQ4.RwNz4gu9dkhAhpBYVCkHw8CMhpBKnegyyqw0QfURowarZnRnQi.jo4JEzIOvPO/",
"key": "ssh-rsa AAA ... user@email.com"
}
}
},
"devices": {
"disk": {
"type": "org.osbuild.loopback",
"options": {
"filename": "disk.raw",
"partscan": true
}
}
},
"mounts": [
{
"name": "part4",
"type": "org.osbuild.ext4",
"source": "disk",
"target": "/",
"partition": 4
},
...
{
"name": "ostree.deployment",
"type": "org.osbuild.ostree.deployment",
"options": {
"source": "mount",
"deployment": {
"default": true
}
}
},
{
"name": "bind",
"type": "org.osbuild.bind",
"target": "tree://",
"options": {
"source": "mount://"
}
}
]
},
```
Similar to `stages` and `sources` we need some basic infrastructure
so that we can use a `mounts_module` fixture for the coming tests
to the mount modules.
The "main" branch is failing right now in tests. The reason is
that we do not have a merge queue and when
https://github.com/osbuild/osbuild/pull/1715
was merged we had no test for `osbuild-depsolve-dnf` yet.
We have one now (THANK YOU achilleas-k) and it shows an issue :)
This commit fixes the issue.
Depsolver test that starts a temporary file server and queries it using
osbuild-depsolve-dnf.
Generates all combinations of repositories configured through the
depsolve-dnf request or the repositories directory and runs the test
cases. The results should be the same regardless of combination.
Test repos are defined with a fake gpg key on the request or repo config
and check if it is read correctly and attached to the repo configs in
the response. The name of the repo is appended to each repo's gpg key
so we can make sure that repo option values don't get swapped.
Add two test rpm metadata directories that can be served as RPM repos.
One was copied from osbuild/images and contains the repository metadata
for CentOS Stream 9 BaseOS.
The second was created by building a simple spec file into an RPM and
creating the metadata using createrepo.
Some of the repository properties in the request were named differently
than the equivalent properties in the dnf repository configuration.
This can introduce bugs and confusion.
One such issue already existed with osbuild/images using 'gpgcheck' in
the request, osbuild-depsolve-dnf5 checking for 'check_gpg', and the dnf
repository configuration property being 'gpgcheck'. This didn't cause
any bad behaviour because osbuild/images reused the original (internal)
configuration to set the property in stages and depsolving isn't
affected by the value of this property.
Change the request properties to match the dnf repository configuration
to avoid confusion: gpgcheck, repo_gpgcheck, and sslverify. Users of
osbuild-depsolve-dnf5 should use property names that match dnf. Use
the same names in the response.
To maintain the same behaviour for SSL verification, a missing sslverify
default to True. The previous property had the opposite meaning,
ignore_ssl, and defaulted to False.
Add the full gpg keys to the repository configs in the response.
On each repository object from dnf, the gpg keys are URLs, either
file:// or http(s)://. We need to resolve these and return them with
in the response.
When the URL is a file:// path, and it comes from a .repo config file,
we assume that the path is relative to the root_dir, so we prepend it to
the path in the file. This is so that repo configs in OS root trees can
be used unmodified. However, when a key is defined in the request, we
should assume that the path is valid, either because it was defined by
the caller as a URL, or because it was defined in-line in the request
and osbuild-depsolve-dnf5 wrote it to the persistdir itself.
A new exception is defined to identify errors during this process.
Support loading repositories from a root tree instead of supplying them
with the request. The repositories should be in the standard yum repo
format. Both repository sources can be defined simultaneously, but at
least one is required.
The root_dir is expected to contain files necessary for depsolving in
the standard paths.
These files are:
- Repository (.repo) configurations in <root_dir>/etc/yum.repos.d/
- GPG key files in <root_dir>/etc/pki/rpm-gpg/
- This will be used to resolve gpg key paths specified in the .repo
files that are relative to the root_dir.
- (Optional) Custom dnf config variables in <root_dir>/etc/dnf/vars or
<root_dir>/usr/share/dnf5/vars.d.
- This is used by CentOS Stream to set the value of $stream.
Custom repository configurations in arbitrary (non-root) paths will have
to follow this directory structure.
A new variable is added to the request, `releasever`, which is mandatory
when using `root_dir`. This variable is used in repository URLs and GPG
key paths. In the default case, dnf reads this variable by inspecting
the rpm database. We will override it in the Solver the same way we
override the arch and basearch for variable substitution. In the
future, we will make this variable mandatory in all cases, which will
make the variable available for repo configs defined in the request as
well.
The root_dir is used in three ways:
- Set the base.conf.installroot
- Set the base.conf.varsdir to <root_dir>/usr/share/dnf5/vars.d and
<root_dir>/etc/dnf/vars to read resolve custom variables when loading
repositories.
- Call create_repos_from_dir() with <root_dir>/etc/yum.repos.d.
base.setup() should be called before loading repositories otherwise
substitutions might not work.
See https://github.com/rpm-software-management/dnf5/issues/1374#issuecomment-2038995031
Some of the repository properties in the request were named differently
than the equivalent properties in the dnf repository configuration.
This can introduce bugs and confusion.
One such issue already existed with osbuild/images using 'gpgcheck' in
the request, osbuild-depsolve-dnf checking for 'check_gpg', and the dnf
repository configuration property being 'gpgcheck'. This didn't cause
any bad behaviour because osbuild/images reused the original (internal)
configuration to set the property in stages and depsolving isn't
affected by the value of this property.
Change the request properties to match the dnf repository configuration
to avoid confusion: gpgcheck, repo_gpgcheck, and sslverify. Users of
osbuild-depsolve-dnf (osbuild/images) should use property names that
match dnf. Use the same names in the response.
To maintain the same behaviour for SSL verification, a missing sslverify
default to True. The previous property had the opposite meaning,
ignore_ssl, and defaulted to False.
Add the full gpg keys to the repository configs in the response.
On each repository object from dnf, the gpg keys are URLs, either
file:// or http(s)://. We need to resolve these and return them with
in the response.
When the URL is a file:// path, and it comes from a .repo config file,
we assume that the path is relative to the root_dir, so we prepend it to
the path in the file. This is so that repo configs in OS root trees can
be used unmodified. However, when a key is defined in the request, we
should assume that the path is valid, either because it was defined by
the caller as a URL, or because it was defined in-line in the request
and osbuild-depsolve-dnf wrote it to the persistdir itself.
A new exception is defined to identify errors during this process.
When generating package sources and rpm stage metadata for a manifest
from a list of packages, we need to associate repository configuration
options to each package [1]. Previously, a caller had all the
repository configurations because they were part of the request, so
packages could be associated with all the repository options by the
repository ID. Now, osbuild-depsolve-dnf will use repositories loaded
from a directory that the caller shouldn't have to read, so returning
all repository configurations in the response makes it possible to
get all package metadata from the response.
This changes the whole structure of the response to a depsolve request.
Previously, we returned an array of packages. Now we return an object
with two keys:
- packages: the array of packages as before
- repositories: an object mapping repository IDs to repository
configurations.
Each package contains the repository ID it comes from (as before), under
`repo_id`. This can be used to get repository configurations and
determine gpg keys and SSL certs for each package.
The new structure avoids duplicating values across all the (sometimes
hundreds) of packages.
[1] 92497c7b1f/pkg/dnfjson/dnfjson.go (L499-L507)
Support loading repositories from a root tree instead of supplying them
with the request. The repositories should be in the standard yum repo
format. Both repository sources can be defined simultaneously, but at
least one is required.
The root_dir is expected to contain files necessary for depsolving in
the standard paths.
These files are:
- Repository (.repo) configurations in <root_dir>/etc/yum.repos.d/
- GPG key files in <root_dir>/etc/pki/rpm-gpg/
- This will be used to resolve gpg key paths specified in the .repo
files that are relative to the root_dir.
- (Optional) Custom dnf config variables in <root_dir>/etc/dnf/vars or
<root_dir>/etc/yum/vars.
- This is used by CentOS Stream to set the value of $stream.
Custom repository configurations in arbitrary (non-root) paths will have
to follow this directory structure.
A new variable is added to the request, `releasever`, which is mandatory
when using `root_dir`. This variable is used in repository URLs and GPG
key paths. In the default case, dnf reads this variable by inspecting
the rpm database. We will override it in the Solver the same way we
override the arch and basearch for variable substitution. In the
future, we will make this variable mandatory in all cases, which will
make the variable available for repo configs defined in the request as
well.
The root_dir is used in two ways:
- Set the base.conf.reposdir to <root_dir>/etc/yum.repos.d.
- Call update_from_etc() with root_dir to read custom variables in
<root_dir>/etc/yum/vars and <root_dir>/etc/dnf/vars.
