The curl source is the only source left that uses "transform". And
here the name is very generic but in fact we only do a single thing:
we add secrets for subscriptions for for mtls to the download.
So rename to make it clear what this is all about.
The issue with mocking subprocess.call() is that we will not
catch if arguments are passed as eg. integers. So instead use
the `mock_command()` helper so that a real binary is called.
Instead of just mocking the binary also write a log of the way
it got called so that tests can use this to check if the right
options are passed.
Note that the API should be improved here, instead of returning
a "naked" path to the calllog file there should be a class wrapping
it. And of course there should be tests.
The current `make_container()` helper is a bit silly (which is
entirely my fault). It requires a container tag as input but all
tests end up creating a random number for this input. So instead
just remove the input and return the container_id from the podman
build in the contextmanager and use that.
To start using the `bootc.instal-to-filesystem` stage we need
to be able to test the generated images. This requires a login
on the disk. Traditionally we did that via the `users` stage.
But on a bootc system we do not want to modify the disk after
bootc did the install to avoid messing with things like the
selinux labels or (future) fsverity setups.
So for now we will use the `--root-ssh-authorized-keys` feature
to inject support for login.
So this commit adds a new option to the stage called
`root-ssh-authorized-keys`.
Based on the feedback from Tomáš in [0] this commit adds tests
that ensure consistent behavior between the python and the json
loader.
It's not 100% because the python is extremly leaniant and does
not even check if the required pieces of the json are there.
I.e. it will load a module without a SCHEMA or SCHEMA_2 variable
and the json loader code will warn about the issue but not
raise an error.
Fwiw, I have no strong opinion here but I do lean slightly towards
staying close to the original code (but both approaches of failing
with an exectption and continue with a warning have good arguments).
[0] https://github.com/osbuild/osbuild/pull/1618#discussion_r1521141148
Instead of always parsing the python stage to load meta information
allow the user of a new `{stage}-meta.json` file. This is a first
step towards allowing modules to be written in a different language
than python. It also has some practical advantages:
- slightly faster as it avoids calling python to output the schemas
- easier to write schemas as this can be done in a real json editor
now
- more extensible in a future where stages maybe binaries with
shlib dependencies that are only satisfied in the buildroot
but not on the host
This reverts commit 158acaac78.
With https://github.com/osbuild/bootc-image-builder/pull/238 the
original reason to call mknod goes away so we can just revert
it. osbuild now requires not only the loop device but also uses
`losetup --partscan` quite a lot now so the mknod approach becomes
impractical and the consumers of osbuild in a container should
just setup devtmpfs.
Since we support python3.6 we cannot assume that dicts are ordered
in any way. To ensure the `id` is still always valid we pass
sort_keys=True to json.dump().
Thanks to Simon!
When an alternative monitor like JSONSeqMonitor is used there is
still non json output printed to stdout. This was a TODO but
this commit removes it because it's okay, there is the
"--monitor-fd" that should be used when using the json-seq monitor.
This commit is somewhat poor, sorry for that. It mostly adds
workaround so that the osbuild sources can emit some progress
reporting as well. Without that the user experience is rather poor
and there is a long delay before any sort of progress can be
reported (even before the normal stages run).
With it the user experience is still not good but slightly better,
i.e. the progress monitor will report that the sources have
started downloading and curl will generated some log output. No
real progress unfortunately (sources subprogress will jump from
zero to 100%).
Add an example render to test/demo how the json-seq based progress
works. It needs the python `tqdm` package for the actual rendering.
See the output with:
```
$ sudo OSBUILD_TEST_STORE=/var/tmp/osbuild-test-store \
python3 -m osbuild --libdir=. --monitor=JSONSeqMonitor --export image \
--output-dir=/tmp/output-dir ./test/data/manifests/fedora-boot.json | ./tools/osbuild-json-seq-progress-example-renderer
```
Generate log messages with origin "org.osbuild.main" when
pipelines/stages start and finish. This way a higher level
frontend can display high level progress coming from this
origin and filter out e.g. stages based log messages (that
are usually quite technical as they are just stdout/stderr
from the stages).
The existing JSONSeqMonitor was saving/restoring the "origin"
when generating a new log-entry. This allows logging from
different origins (e.g. "org.osbuild.main") in a kind of
"out-of-band" fashion.
But this save/restore feels slightly inelegant because
JSONSeqMonitor feels like the wrong layer to deal with this.
This is why a new `with_origin()` helper is introduced that
will either reuse the existing context or create a new one
with the requested origin.
Tweak the Progress class to be simpler. Given that progress does
not need to support arbitrary depth but only has a single level
the class now just exposes "sub_progress" to the caller.
When the main progress is advanced the sub_progress is now fully
deleted instead of just reset. The rational is that when the main
progress is done and advances a step it is very likely that a
new sub_progress is required and it's most likely an error if
the same sub_progress will get re-used.
This means that `reset()` can be removed as it's not used anymore
(and YAGNI). We can add it back when we have a use-case.
It also change the code so that "total" starts with 0 instead
of `None` (principle of least surprise). This means that now
`progress.incr()` is called in the JSONSeqMonitor() for
`finish()` and `result()` to indicate that the pipeline/stage
is finished.
This commit tweaks Context a bit so that any write will automatically
reset the `_id`. This ensures that we do not forget to reset `_id`
when the code changes.
It also tweaks the naming a bit, before there was a "setter" for
origin and functions to set "pipeline" and "stage". They are all
functions now with a "set_" prefix for symetry mostly.
The class LogLine() is purely used as a dataclass with no state
and the only function on it is `as_dict()`. This got refactored
into a new function `log_entry()` because there is no need for
this to be a class. The function that takes the same inputs.
This changes the way monitors are initialized to always include
a `manifest` and now they will always log a `start` message.
This makes creation of a monitor symetric accross all monitors
again and no special cases for JSONSeqMonitor are required.
It means one needs to run the json-seq monitor with:
```
$ python3 -m osbuild --monitor=JSONSeqMonitor
```
Fwiw, I'm not 100% confident this is a win but it feels slightly
more right then the special cases in `main_cli.py` that is
replaces.
Create an origin string of the form 'stages/<stagename>' from the argv
argument and add it to the log prints for messages coming from the
stage.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
When specified with --json controls the mode of the json output:
- batch (default) is the old mode which prints the result
of the entire build after it's finished (NullMonitor)
- progress is the new mode that prints one json line per message
(JSONProgressMonitor)
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
New monitor type that emits a JSON object for each log message.
Unlike other monitors:
- The constructor takes a build manifest as argument to initialise the
Pipeline and Stage counts for the Progress part of the report.
- It doesn't print a 'result' at the end of the build.
- The log() method that prints a log message supports specifying an
origin to override the default that's set by the constructor.
Although the Logline supports reporting errors separately, this isn't
used yet.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Foundation for new monitor type that will emit a JSON object for each
log message. The following classes are defined:
- LogLine: The top-level object that can be serialised into a single
object containing a message and associated metadata.
- Context: Contextual information for a log line message. Describes the
origin of the message and the current pipeline and stage.
Automatically deduplicates this information using a hash/ID: keeps a
history of IDs and omits the context when the context is not new.
- Progress: Information on the progress of the build. The object is
recursive: can contain a sub-progress for nested progress reporting
(pipelines > stages > ...).
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Add the expiredate field to the users option in the org.osbuild.users
stage. This option maps to the --expiredate option of useradd/usermod,
which can be useful when creating users whose password must be changed
upon first login.
