Create a new api endpoint called exception, that communicates
exception backtraces separately back to osbuild, as opposed to
dumping them into the normal log. Additionally, add a corresponding
test to check that a call to api.exception correctly sets
API.exception.
Access metadata.api only after `api` has exited the context and
thus the event loop has stopped and all incoming messages, like
the one setting the metadata, have been processed.
See commit 803433fb62 for a lecture
about the internals and all the details involved.
Now that the `org.osbuild.linux` runner does not use `api.setup_stdio`
anymore, the output of the binary run from the BuildRoot must end up
in `BuildRoot.output`. Check for that.
Now that the BuildRoot is capable of capturing the output of the
runner and modules (stages, assemblers), there is no need for
using `api.setup_stdio`. Therefore, drop it from all runners and
replace `api.output` with `BuildRoot.output`, which will contain
the output if `api.setup_stdio` is not called from the runners.
Create a new CompletedBuild object that wraps and is very similar
to the subprocess.CompletedProcess, i.e. it has a process member
but also has shortcuts for returncode. Additionally, the output
of the process is not only forwarded to the monitor, but also
captured and then handed to CompletedBuild, so its output member
will actually contain the full build output. To be compatible
with the previously returned CompletedProcess, `stderr`, `stdout`
members exist on CompletedBuild that also return `output`.
Instead of reading the arguments from sys.stdin, which requires
that stdin is setup properly for that in the runner, use the new
api.arguments() method to directly fetch the arguments.
Also fix missing newlines between imports and methods to be more
PEP-8 complaint, where needed.
In case that bubblewrap fails to, e.g. because it fails to execute
the runner, it will print an error message to stderr. Currently,
this output is not capture and thus not logged. To fix that, the
`BuildRoot.run` method now takes a monitor object and will stream
stdout/stderr to the log via the monitor.
Instead of reading the arguments from sys.stdin, which requires
that stdin is setup properly for that in the runner, use the new
api.arguments() method to directly fetch the arguments.
Also fix missing newlines between imports and methods to be more
PEP-8 complaint, where needed.
Instead of reading the arguments from sys.stdin, which requires
that stdin is setup properly for that in the runner, use the new
api.arguments() method to directly fetch the arguments.
Also fix missing newlines between imports and methods to be more
PEP-8 complaint, where needed.
Simple check for the new server side method, `get-arguments`, and
client side counterpart, `api.arguments`, that compares that using
the later we get the supplied input (arguments) to API.
Add a new `get-arguments` API call to fetch the input/arguments.
To avoid running into any limitings on maximum package size on
the socket, the actual data is written to a temp file and a fd
to that passed to the client - very much as in `setup_stdio`.
Additionally, new `arguments` method is provided as a client
counterpart for the new API call.
Change the API endpoint to prevent retrieving monitor-output from a
running instance. Instead, we require the caller to exit the API context
before querying the monitor-output. This guarantees that the api-thread
was synchronously taken down and scheduled any outstanding events.
This fixes an issue where a side-channel notifies us of a buildroot
exit, but the api-thread has not yet returned from epoll, and thus might
not have dispatched pending I/O events, yet. If we instead wait for the
thread to exit, we have a synchronous shutdown and know that all
*ordered* kernel events must have been handled.
In particular, imagine a build-root program running (like `echo` in the
test_monitor unittest) which writes data to the stdout-pipe and then
immediately exits. The syscall-order guarantees that the data is written
to the pipe before the SIGCHLD is sent (or wait(2) returns). However, we
retrieve the SIGCHLD from our main-thread usually (p.join() in our test,
and BuildRoot() in our main code), while the pipe-reading is done from
an API thread. Therefore, we might end up handling the SIGCHLD first
(just imagine a single-threaded CPU that schedules the main task before
the thread). To avoid this race, we can simply synchronize with the
api-thread. Since we already have this synchronization as part of the
api-thread takedown, it is as simple as stopping the api-thread before
continuing with operations.
Lastly, if a write operation to a pipe was issued, we are guaranteed
that a SIGCHLD synchronization across processes is ordered correctly.
Furthermore, the python event-loop also guarantees that stopping an
event-loop will necessarily dispatch all outstanding events. A read is
guaranteed to be outstanding in our race-scenario, so the read will be
dispatched. The only possible problem is `_output_ready()` only
dispatching a maximum of 4096 bytes. This might need to be fixed
separately. A comment is left in place.
If the stage test folder contains a `metadata.json` file, it will
contain a dictionary where the keys are stage ids and the values
are dictionaries containing the metadata to verify. For each of
those the stage will be looked up in the pipeline result of 'b'
and verified that the metadata matches.
Generate and report metadata about all the packages that were
installed. This information will be needed by composer, especially
the 'sigmd5' bit, for integration with koji[1].
[1] https://docs.pagure.org/koji/content_generator_metadata/
Add support for setting metadata via `osbuild.API`. It is meant
to be used by modules (stages, assemblers) to pass additional data
that belong to the result back to osbuild. For this, a new api
method `set-metadata` can be used to set and update a metadata
dictionary on the `osbuild.API` class. A client side method
`metadata` is provided to do so.
This is a crucial pre-condition for the org.osbuild.selinux stage
to work properly, especially that it can set labels that are not
present in the policy on the host. If /sys/fs/selinux is writable,
setfiles will try to verify the labels via /sys/fs/selinux/context
and fail for unknown labels.
Add a new helper script to check if a mount / file-system was
mounted with specific flags. Currently only "ro", "nosuid",
"nodev" and "noexec" flags are supported. This script is in
test/data since it will be used from other tests and is itself
not a test per se.
Make sure "/sys/fs/selinux" is read-only, otherwise libselinux and
tools will assume that SELinux is available and active and in turn
use /sys/fs/selinux to e.g. verify the file systems labels; this
will then prevent setting unknown labels via `setfiles`.
Add a new `os_version` option that will result in the `version`
metadata being set as commit metadata. This will then be shown
in the `rpm-ostree status` output.
Allow osbuild_t to no_new_privs (nnp) and nosuid domain transition
into setfiles_mac_t and install_t. nnp is a inheritable per-thread
flag (PR_SET_NO_NEW_PRIVS, see prctl(2)), whereby a promise is made
by execve(2) to not grant any new privileges that could not have
been done without the execv call. This is on contrast to what can
be done via SELinux rules, i.e. in our case `setfiles_mac_t` and
`install_t` can set arbitrary SELinux labels, but `osbuild_t`
itself can not; but `osbuild_t` enables the transitioning of
`setfiles_mac_t` for the `setfiles` binary via execve(2) from a
process with `osbuild_t`. Related, the nosuid mount flag, prevents
the suid, sgid bits to be interpreted and thus are in the same
spirit as nnp, i.e. no new privs during execve(2).
Thus SELinux domain transitions stand in contrast with nnp and
nosuid transitions, and have therefore been de-coupled. See also
the corresponding kernel patch at [1] for more information.
bubblewrap (bwrap) in contrast to `systemd-nspawn` always sets the
nnp flag, as well as the nosuid option for all bind-mounts. Since
we no use bwrap to contain processes we need to allow the nnp and
nosuid transitions from `osbuild_t` to `setfiles_mac_t` and
`install_t`.
[1] https://patchwork.kernel.org/patch/9841441/
Make the output_directory argument in Pipeline.assemble
and Assembler.run required. The qemu assembler assumes
it is passed in args and will crash without it. Making
it mandatory prevents this.
Allow a user to see the duration for each step in the osbuild pipeline.
This allows a user to optimize the build system for the best performance
and identify performance bottlenecks.
Signed-off-by: Major Hayden <major@redhat.com>
