The existing JSONSeqMonitor was saving/restoring the "origin"
when generating a new log-entry. This allows logging from
different origins (e.g. "org.osbuild.main") in a kind of
"out-of-band" fashion.
But this save/restore feels slightly inelegant because
JSONSeqMonitor feels like the wrong layer to deal with this.
This is why a new `with_origin()` helper is introduced that
will either reuse the existing context or create a new one
with the requested origin.
Tweak the Progress class to be simpler. Given that progress does
not need to support arbitrary depth but only has a single level
the class now just exposes "sub_progress" to the caller.
When the main progress is advanced the sub_progress is now fully
deleted instead of just reset. The rational is that when the main
progress is done and advances a step it is very likely that a
new sub_progress is required and it's most likely an error if
the same sub_progress will get re-used.
This means that `reset()` can be removed as it's not used anymore
(and YAGNI). We can add it back when we have a use-case.
It also change the code so that "total" starts with 0 instead
of `None` (principle of least surprise). This means that now
`progress.incr()` is called in the JSONSeqMonitor() for
`finish()` and `result()` to indicate that the pipeline/stage
is finished.
This commit tweaks Context a bit so that any write will automatically
reset the `_id`. This ensures that we do not forget to reset `_id`
when the code changes.
It also tweaks the naming a bit, before there was a "setter" for
origin and functions to set "pipeline" and "stage". They are all
functions now with a "set_" prefix for symetry mostly.
The class LogLine() is purely used as a dataclass with no state
and the only function on it is `as_dict()`. This got refactored
into a new function `log_entry()` because there is no need for
this to be a class. The function that takes the same inputs.
This changes the way monitors are initialized to always include
a `manifest` and now they will always log a `start` message.
This makes creation of a monitor symetric accross all monitors
again and no special cases for JSONSeqMonitor are required.
It means one needs to run the json-seq monitor with:
```
$ python3 -m osbuild --monitor=JSONSeqMonitor
```
Fwiw, I'm not 100% confident this is a win but it feels slightly
more right then the special cases in `main_cli.py` that is
replaces.
Create an origin string of the form 'stages/<stagename>' from the argv
argument and add it to the log prints for messages coming from the
stage.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
When specified with --json controls the mode of the json output:
- batch (default) is the old mode which prints the result
of the entire build after it's finished (NullMonitor)
- progress is the new mode that prints one json line per message
(JSONProgressMonitor)
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
New monitor type that emits a JSON object for each log message.
Unlike other monitors:
- The constructor takes a build manifest as argument to initialise the
Pipeline and Stage counts for the Progress part of the report.
- It doesn't print a 'result' at the end of the build.
- The log() method that prints a log message supports specifying an
origin to override the default that's set by the constructor.
Although the Logline supports reporting errors separately, this isn't
used yet.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Foundation for new monitor type that will emit a JSON object for each
log message. The following classes are defined:
- LogLine: The top-level object that can be serialised into a single
object containing a message and associated metadata.
- Context: Contextual information for a log line message. Describes the
origin of the message and the current pipeline and stage.
Automatically deduplicates this information using a hash/ID: keeps a
history of IDs and omits the context when the context is not new.
- Progress: Information on the progress of the build. The object is
recursive: can contain a sub-progress for nested progress reporting
(pipelines > stages > ...).
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Change the contianers store from `/containers/storage` to
`/var/tmp/containers/storage` since technically `/containers/storage`
isn't ostree compatible.
The LoopServer._create_device() duplicates the code from
`Loop.loop_for_fd()` right now. Given that this is relatively
subtle code it's better to not duplicate it and instead just
reuse the loop_for_fd() implementation in LoopServer.
This is a follow up to #1550 where we enabled a `rw` permissions mode,
which is not ideal since it would theoretically be possible to set both
`ro` and `rw` modes at the same time. This commit fixes the issue by only
allowing one option at a time.
Fixes#1588
The BLS specification [0] says the `options` field is optional and
can also appear multiple times. This commit tweaks the code to
deal with these corner cases and also adds tests that ensure that
this works correctly.
It also tweaks the file handling to be atomic.
[0] https://uapi-group.org/specifications/specs/boot_loader_specification/
- Add functions for appending kernel parameters to the
Boot Loader Specification (BLS) as needed.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
When parsing a container input, add the checksum to the data as well.
Usually with other inputs, the stage only needs to know the filepath
where it can find the source content. In most (all, so far) cases, this
is a checksum appended to the content type.
In this case, the filepath is the location of the storage bind mount and
the checksum is needed to retrieve the container. The name might only
be a destination name (a name to use for storing the container in the
image), so we can't rely on it being valid in the source.
The system-wide location for the containers storage.conf is
/usr/share/containers. The existence of a file in /etc/containers
completely overrides this (see containers-storage.conf(5)).
If no file is found at /etc/containers/storage.conf then fall back to
reading the config from /usr/share/containers/storage.conf.
If neither file exists, this is an error since the default config should
be packaged with any tool that requires it (skopeo, podman, etc).
Containers with the `containers-storage` are bind mounted to
the osbuild store. This helper function bind mounts the
containers-storage from the store into the `/containers/storage`
so that stages that require containers can then make use of these
containers.
Under certain conditions a bind mount without a specified `rw` or `ro`
option gets mounted read-only. We need a way to be explicit about
needing a rw mount. We might want to change this in the future to be a
single option (mode optional?) with valid values "rw", "ro".
It's not entirely clear what the conditions are but it occurs when bind
mounting the containers storage into the osbuild store, which we will
need for the next few commits.
The new `testutil.mock_command` context manager can be used to
mock commands in PATH and replace them with arbitrary shell
scripts. This is useful in testing to e.g. simulate exact error
conditions that would be hard to trigger otherwise or to replace
long running commands with faked results.
Example:
```
fake_cmd = textwrap.dedent("""\
do-something
""")
with mock_command("some-cmd", fake_cmd):
your_code
```
This adds a `default: true` option for all cases where OSTree
information is specified in schemas and allows for the information
to be picked up from the filesystem.
This is a safe operation because when building disk images there is
no known case where having two deployments makes sense. In the case
there ever were a case then the osname, ref, and serial options still
exist and can be used.
Co-authored-by: Luke Yang <luyang@redhat.com>
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
This commit adds code that will remove the least recently used
entries when a store() operation does not succeeds because the
cache is full. To be more efficient it will try to free
twice the requested size (this can be configured in the code).
Instead of operating directly on the tree for a stage we can operate
on a mount too. This is useful in the case where operating on the
directory tree of files isn't sufficient and the modifications need
to be made directly to the filesystems on the disk image that we are
creating.
One such example of this is we are having a problem right now where
the immutable bit being set on an OSTree deployment root doesn't
survive the `cp -a --reflink=auto` in the org.osbuild.copy stage when
being copied from the directory tree into the mounted XFS filesystem
we created on the disk image. Thus we have to workaround this loss
of attribute by applying the attribute directly on the mounted
filesystem from the disk.
In this change here we also add a check in osbuild/mounts.py to not
attempt a umount of the root of the mounts directory if that path
is no longer a mountpoint, which can happen when the umount -R
from the mounts/org.osbuild.ostree.deployment also removes the
overmount.
Here is an example of how this would be used:
```
- type: org.osbuild.chattr
options:
immutable: true
path: mount://root/
devices:
disk:
type: org.osbuild.loopback
options:
filename: disk.img
partscan: true
mounts:
- name: root
type: org.osbuild.xfs
source: disk
partition:
mpp-format-int: '{image.layout[''root''].partnum}'
target: /
- name: ostree.deployment
type: org.osbuild.ostree.deployment
options:
source: mount
deployment:
osname: fedora-coreos
ref: ostree/1/1/0
```
The initial mount on `/` is the filesystem from the root partition
on the disk. The second mount (of type org.osbuild.ostree.deployment)
then reconfigures things similar to how an OSTree system is set up.
