- Move functions to the 'util' to centralize common
functionality, reducing code duplication and improving
maintainability across the codebase.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
The other tests have concurrency issues at the moment.
It seems in https://github.com/osbuild/osbuild/pull/1655 we lost
parallel running. The issue seems to be that `env` does not
actually define a shell environment but only a github environment.
This sets commit sets the shell env explicitly to unbreak us
again.
When moving to `bootc install to-filesystem` we need more information
for bootc from /etc/selinux than our current /etc/selinux/targeted/contexts
policy.
This commit makes all of /etc/selinux available which unblocks
the bootc install.
This is needed because on a mounted `bootc` container `setfiles`
without excluding `/sysroot` will create many warnings like:
```
setfiles: conflicting specifications for /run/osbuild/tree/sysroot/ostree/repo/objects/00/0ef9ada2ee87792e8ba21afd65aa00d79a1253018832652b8694862fb80e84.file and /run/osbuild/tree/usr/lib/firmware/cirrus/cs35l41-dsp1-spk-prot-103c8b8f-r1.bin.xz, using system_u:object_r:lib_t:s0.
```
but simply excluding this dir fixes them.
There are recent GH failures because the github cache action
complains:
```
Input required and not supplied: key
```
This is slightly odd as we did not provide a cache key before and
it was fine but *shrug*. We also don't really need a cache key,
we always get the same cache, osbuild is smart enough to figure
it out.
Instead of requiring only one of the properties to be present require at
least one of them being present; some stages specify both schema
versions (`org.osbuild.rpm`)
The curl source is the only source left that uses "transform". And
here the name is very generic but in fact we only do a single thing:
we add secrets for subscriptions for for mtls to the download.
So rename to make it clear what this is all about.
The issue with mocking subprocess.call() is that we will not
catch if arguments are passed as eg. integers. So instead use
the `mock_command()` helper so that a real binary is called.
Instead of just mocking the binary also write a log of the way
it got called so that tests can use this to check if the right
options are passed.
Note that the API should be improved here, instead of returning
a "naked" path to the calllog file there should be a class wrapping
it. And of course there should be tests.
The current `make_container()` helper is a bit silly (which is
entirely my fault). It requires a container tag as input but all
tests end up creating a random number for this input. So instead
just remove the input and return the container_id from the podman
build in the contextmanager and use that.
To start using the `bootc.instal-to-filesystem` stage we need
to be able to test the generated images. This requires a login
on the disk. Traditionally we did that via the `users` stage.
But on a bootc system we do not want to modify the disk after
bootc did the install to avoid messing with things like the
selinux labels or (future) fsverity setups.
So for now we will use the `--root-ssh-authorized-keys` feature
to inject support for login.
So this commit adds a new option to the stage called
`root-ssh-authorized-keys`.
Based on the feedback from Tomáš in [0] this commit adds tests
that ensure consistent behavior between the python and the json
loader.
It's not 100% because the python is extremly leaniant and does
not even check if the required pieces of the json are there.
I.e. it will load a module without a SCHEMA or SCHEMA_2 variable
and the json loader code will warn about the issue but not
raise an error.
Fwiw, I have no strong opinion here but I do lean slightly towards
staying close to the original code (but both approaches of failing
with an exectption and continue with a warning have good arguments).
[0] https://github.com/osbuild/osbuild/pull/1618#discussion_r1521141148
Instead of always parsing the python stage to load meta information
allow the user of a new `{stage}-meta.json` file. This is a first
step towards allowing modules to be written in a different language
than python. It also has some practical advantages:
- slightly faster as it avoids calling python to output the schemas
- easier to write schemas as this can be done in a real json editor
now
- more extensible in a future where stages maybe binaries with
shlib dependencies that are only satisfied in the buildroot
but not on the host
This reverts commit 158acaac78.
With https://github.com/osbuild/bootc-image-builder/pull/238 the
original reason to call mknod goes away so we can just revert
it. osbuild now requires not only the loop device but also uses
`losetup --partscan` quite a lot now so the mknod approach becomes
impractical and the consumers of osbuild in a container should
just setup devtmpfs.
Since we support python3.6 we cannot assume that dicts are ordered
in any way. To ensure the `id` is still always valid we pass
sort_keys=True to json.dump().
Thanks to Simon!
When an alternative monitor like JSONSeqMonitor is used there is
still non json output printed to stdout. This was a TODO but
this commit removes it because it's okay, there is the
"--monitor-fd" that should be used when using the json-seq monitor.
This commit is somewhat poor, sorry for that. It mostly adds
workaround so that the osbuild sources can emit some progress
reporting as well. Without that the user experience is rather poor
and there is a long delay before any sort of progress can be
reported (even before the normal stages run).
With it the user experience is still not good but slightly better,
i.e. the progress monitor will report that the sources have
started downloading and curl will generated some log output. No
real progress unfortunately (sources subprogress will jump from
zero to 100%).
Add an example render to test/demo how the json-seq based progress
works. It needs the python `tqdm` package for the actual rendering.
See the output with:
```
$ sudo OSBUILD_TEST_STORE=/var/tmp/osbuild-test-store \
python3 -m osbuild --libdir=. --monitor=JSONSeqMonitor --export image \
--output-dir=/tmp/output-dir ./test/data/manifests/fedora-boot.json | ./tools/osbuild-json-seq-progress-example-renderer
```
Generate log messages with origin "org.osbuild.main" when
pipelines/stages start and finish. This way a higher level
frontend can display high level progress coming from this
origin and filter out e.g. stages based log messages (that
are usually quite technical as they are just stdout/stderr
from the stages).
