Add the force_password_reset field to the users option in the
org.osbuild.users stage. This option forces the user to change their
password on first login.
This reverts commit bc04bfc366.
The `remove_tmpfiles()` helper is nice but it is also problematic
because it creates extra output after the command was run and
created output. E.g. a test failure on centos stream9 [0]
```
r = root.run(["stat", "--format=%a", "/var/tmp"], monitor)
assert r.returncode == 0
> assert r.stdout.strip().split("\n")[-1] == "1777"
E AssertionError: assert '/usr/lib/tmp... such process' == '1777'
E
E - 1777
E + /usr/lib/tmpfiles.d/rpcbind.conf:2: Failed to resolve user 'rpc': No such process
```
Here the output from "stat" is not the last output because the
rempve_tmpfiles runs `systemd-tmpfiles --clean --remove` which
produces some noisy output after stat was run.
This was found by @thozza (thanks!) and discussed in osbuild PR#1785.
There are various ways to fix this, the one is to use the
`--graceful` option of systemd-tmpfiles. However that only got added in
systemd v256 and centos-stream9 has v252 so that is sadly not an option.
Plus even when avaialble it will produce some informational output like
```
All rules containing unresolvable specifiers will be skipped.
```
Another way would be to sent the output from systemd-tmpfiles cleanup
to /dev/null. Not really great as we will not know about real problems
or warnings that we should care about.
None of the option above is good. So I started looking at the tmpfiles.d
rules and the cleanup and why we are doing it. It was added relatively
recently in https://github.com/osbuild/osbuild/pull/1458 and after
some medidiation not having it seems to do no harm (details below). The
tl;dr is that the buildroot is created inside bubblewrap and the
dirs that `--clean` and `--remove` touch are already tmpdirs created
just for the buildroot so the cleanup in the runner is redundant
(and because the cleanup is now run for each buidlroot.run() command
there *might* be unintended conequences but the current rules seem
to not have any).
In detail, the tmpfiles_cleanup() does two things:
1. `--clean`
It will remove files that are older then the given age
in tmpfiles.d. The tmpfiles in centos9 give me the following ages:
```
$ systemd-tmpfiles --cat-config|grep -E '[0-9]+d$'
d /var/lib/systemd/pstore 0755 root root 14d
d /var/lib/systemd/coredump 0755 root root 3d
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
```
Given that we run our commands inside a bubblewrap environment and
give it a fresh /run, /tmp, /var [1] there really should be no long
lived things and even if there are they are cleaned up from the
buildroot itself
2. `--remove`
It will remove files marked for removal in tmpdfiles.d. Running
it on a centos9 env it yields for me:
```
$ systemd-tmpfiles --cat-config|grep -E '^[rRD]'
R /var/tmp/dnf*/locks/*
r /var/cache/dnf/download_lock.pid
r /var/cache/dnf/metadata_lock.pid
r /var/lib/dnf/rpmdb_lock.pid
r /var/log/log_lock.pid
r! /forcefsck
r! /fastboot
r! /forcequotacheck
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
R! /var/tmp/container_images*
D /run/rpcbind 0700 rpc rpc - -
D /run/sudo/ts 0700 root root
R! /tmp/systemd-private-*
R! /var/tmp/systemd-private-*
r! /var/lib/systemd/coredump/.#*
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
r! /tmp/.X[0-9]*-lock
```
which is also covered by the bwrap cleanup.
[0] https://artifacts.dev.testing-farm.io/2d07b8f3-5f52-4e61-b1fa-5328a0ff1058/#artifacts-/plans/unit-tests
[1] https://github.com/osbuild/osbuild/blob/main/osbuild/buildroot.py#L218
Based on the filename, different sections of the unit configuration are
valid. Service (.service) units require a [Service] section and mount
(.mount) units requires a [Mount] section.
It might be possible to validate this in the schema but it would
probably require a major rewrite and could get complicated. For now,
validate it in the stage code itself.
This test ensures that the inputs of devices/mounts we generate for
bootc are actually considered valid by the schema. This is a more
blackbox style test compared to `test_get_schema_automatically_added`
which just checks that we get the expected schema but not that the
expected schema actually parses our inputs.
With PR#1727 merged there is no need to test for individual stages
if they support the bootc style mounting in their schema. All
stages now support devices/mounts now. So let's remove this
redundant code.
This will include the latest osbuild-ci container changes from
https://github.com/osbuild/containers/pull/73
This includes a bunch of binaries that were missing and lead
to skpped tests.
On Fedora 40 we switched to dnf5 too early. There is going to be a new
version in Fedora 41 soon, with breaking API changes, so in order to
reduce the number of things we need to support, switch back to using
libdnf on Fedora 40 (and 41 until dnf5 version 5.2.0 is available,
see #1769)
Currently, osbuild downloads are identified as coming from `curl`. This
is unfortunate because some RPM mirrors block requests from curl. Let's
"fix" that by introducing our own user-agent. While this can certainly
be seen as "circumventing" a policy, I think that this change is
actually helpful: Now, the mirror maintainers can actually distinguish
osbuild requests from regular curl calls. If they want to block osbuild,
they certainly can, we have no power there, but at least this allows
more fine-grained filtering. Also, our new user-agent contains our
domain name, so if there's a problem, they can contact us.
Also fix the test, it wasn't updated to use sslverify instead of
ignoressl. This now correctly passes on -dnf and fails on -dnf5 which
will be fixed in the next commit.
During the work on PR#1752 Florian discovered that make_containers()
is broken for nested containers like:
```
with make_container(tmp_path, {"file1": "file1 from base"}) as base_tag:
with make_container(tmp_path, {"file1": "file1 from final layer"}, base_tag) as cont_tag:
```
It errors with:
```
Error: 5b947de461ee21b858dd5b4224e80442b2f65b6410189147f2445884d9e4e3d8: image not known
```
The reason is that we work with hashes for the image and then call
`podman image rm` which by default will also remove all dangling
references. Those are defined by not having a tag and not referenced
anymore. So the inner container cleanup also removes the outter.
There are many ways to fix this, I went with re-adding tags to the
test containers because it also makes it easy for the user to see if
we left any containers (accidently) around.
In PR#1752 Florian noticed that `make_containers` is buggy [0] in
one specific testcase. However the GH actions did not trigger and
uppon inspection the test is skipped:
```
stages/test/test_container_deploy.py::test_container_deploy_integration SKIPPED [ 7%]
```
This commit adds `-rs` to get a summary why the tests are skipped.
This is useful in general, I suspect the test container might miss
podman but this commit will bring clarity.
[0] https://github.com/osbuild/osbuild/pull/1752#discussion_r1578294358
Create a unit using an inline file called -.mount with the following
content:
[Unit]
Before=local-fs.target
After=blockdev@dev-disk-by\x2duuid-af34257d\x2d3e14\x2d4a51\x2db91d\x2dc430a956dcba.target
[Mount]
What=/dev/disk/by-uuid/af34257d-3e14-4a51-b91d-c430a956dcba
Where=/
Type=ext4
Options=rw,noatime
[Install]
RequiredBy=local-fs.target
and enable it in the systemd stage to test that we can enable units with
a - prefix.
Units can start with a -, for example, -.mount. Trying to enable,
disable, or mask a service like that fails with
systemctl: invalid option -- '.'
Hint: to specify units starting with a dash, use "--":
systemctl [OPTIONS...] COMMAND -- -.mount ...
Adding -- to all 'systemctl' calls lets us specify unit files that start
with -.
We run tests using tox to set up environments for different python
versions to test against. The problem is that when a test shells out to
a python script, like osbuild-depsolve-dnf5, it's not run inside the
environment but in the system environment. The `has_dnf5()` check
returns False because it fails to import dnf5 in the tox environment,
even though the script can be run and the test will succeed.
Use `python3 -c "import libdnf5"` to decide if the script is runnable
instead of using `importlib`. This doesn't solve the problem of our
tests running python scripts in a different environment than the one (we
think) we are testing, but it will enable tests of osbuild-depsolve-dnf5
for now.
The JSON output by go will use 'null' for nil slices, so we need to
use a [] when the field is missing, or when it is set to null.
Previously this was handled by checking the value before iterating but
when the code moved for the directory handling it was changed.
This implements the same behavior in a slightly cleaner way.
The go json serialize process takes a nil slice and converts that to a
'null' instead of an '[]' or leaving off the field. This means that the
json generated by python is not representative of what is output by
images/dnfjson and we have not been properly testing
osbuild-depsolve-dnf5
Testing dnf5 depsolving *also* requires that libdnf5 be present in the
environment, which is a separate problem that still needs to be fixed.
Cache files will split the extension, this means that all pyc cache
files looks like we get many clashing `org.osbuild.cpython-py311.pyc
files. Moreover, the cache bytecode invalidation is based on the
timestamp (which is the same after git checkout) and the file size
(which may be the same for two different files). This means that we
can't rely on the cache files.
This issue has been found after the previous commit made the
`org.osbuild.systemd` and `org.osbuild.selinux` stages to have exactly
the same size, which caused the interpreter to reuse the bytecode for
the selinux stage when running unit tests for the systemd stage. This
resulted in consistent and weird failures when the systemd stage
options were passed to the selinux stage code.
The credit for this fix goes to Michael Vogt, who found the cause and
fix. Also thanks to Simon de Vlieger for his help with debugging the
problem.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the SELinux stage would not force full contexts reset when
forcing auto-relabel on first boot. As a result, all files remained
`unconfined_u` after the auto-relabeling on first boot and only the type
part was reset.
We really need to mimic the behavior of `fixfiles -F onboot` command,
which creates the `/.autorelabel` file with "-F" in it.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
