The problem is that some deployments might not have the redhat.repo
file, yet they might have the key and certificate to access Red Hat CDN.
If that was the case, the new approach would cause a regression compared
to the previous behavior.
This patch uses the previous method if the redhat.repo file is not
found or does not contain any matching URL.
The order of entries in a dictionary is not specified by the JSON
standard and hard to control when marshalling dictionaries in Go.
Since the order of mounts is important and the wrong order leads
to wrong mount trees change the `mounts` field to an array. This
breaks existing manifests but after careful deliberation it was
concluded that the original schema with mounts as dictionaries
is not something we want to support. Apologies to everyone.
Adjust the schema of the copy and zipl stage accordingly.
Validate source references while loading manifests so that a bad
reference would result in a meaningful error message instead of a
hard-to-understand Python exception.
Add a simple helper method that returns the path for a deployment,
given the sysroot, the osname, the reference or commit and the
deployment serial. Path might not exist.
`
When bind-mounting the tree for i/o, use recursive bind mounts.
This could be needed in the case that `/usr` is not one single
mount but assembled from different ones. Normally this should
not be the case but we want to support in, just in case.
Conversely, when unmounting, do so recursively too.
NB: This should not make any differences for trees that we have
built ourselves since they don't contain any mounts.
Catch the BrokenPipeError exception when sending a reply. This will
happen when the other side closes their side of the connection/pipe
so in that case we just break out of the serve loop.
Now that arguments are transmitted via a mapped, i.e. bind-mounted,
file instead of using the jsoncomm RPC mechanism, all the methods
related to the latter can be removed from API.
Instead of using the jsoncomm API to transmit stagge arguments,
write them out to a file that is then mapped into the container.
The `api.arguments` function is re-written just read that file
from within the container.
This should, in theory, not be necessary because the bubblewrap
process and its children should be stopped already and umount
should just block until it is finished. But, if the store is on
a filesystem, like the one used by docker machine, unmounting
frequently produces errors like:
`umount: .../tmp9nlyzwdu-writer: target is busy.`
Syncing the filesystem before that seems to help in some cases
and it surely does not hurt.
Checks if one path is a child of a second one. Useful for checking if
paths defined in a manifest exist inside the tree.
Optionally checks if the target path exists.
This will prevent any modification of anything in `/sys`. It will
also prevent `udevadm tigger` to run, which needs /sys writeable.
This is a desired effect, since uevents are not delivered to the
contained environment, so `udevadm trigger` might hang.
In the object store, temporary bind mounts are used when accessing the
content, i.e. the individual trees. Their unmount is currently done
with the `--lazy` flag. The use of this flag goes way back to commit
da121beda1, which sadly does not mention
why the flag was introduced. Since the tree and files in the tree will
be used by consequent stages it seems reasonable to do the un-mounting
eagerly and thus this reverts back to that behavior.
Add the ability to only read a sub-tree of a tree via `Object.read_at`.
Expose the functionality via the `Store{Server,Client}.read_tree_at`.
Extend the tests to check this new functionality.
Allows stages to access file systems provided by devices.
This makes mount handling transparent to the stages, i.e.
the individual stages do not need any code for different
file system types and the underlying devices.
A new host service that provides device functionality to stages.
Since stages run in a container and are restricted from creating
device nodes, all device handling is done in the main osbuild
process. Currently this is done with the help of APIs and RPC,
e.g. `LoopServer`. Device host services on the other hand allow
declaring devices in the manifest itself and then osbuild will
prepare all devices before running the stage. One desired effect
is that it makes device handling transparent to the stages, e.g.
they don't have to know about loopback devices, LVM or LUKS.
Another result is that specific device handling is now modular
like Inputs and Source are and thus moved out of osbuild itself.
Create a `InputService` class with an abstract method called `map`,
meant to be implemented by all inputs. An `unmap` method may be
optionally overridden by inputs to cleanup resources.
Instantiate a `host.ServiceManager` in the `Stage.run` section and
pass the to the host side input code so it can be used to spawn the
input services.
Convert all existing inputs to the new service framework.
Instead of bind-mounting each individual input into the container,
create a temporary directory that is used by all inputs and bind-
mount this to the well known location ("/run/osbuild/inputs"). The
temporary directory is then passed to the input so that it can
make the requested resources available relative to that directory.
This is enforced by the common input handling code.
Additionally, pass the well known input path via a new "paths" key
to the arguments dictionary passed to the stage.
This helper property is misleading since it is not the name of the
input in the context of the manifest, but actually "type". Name is
a left-over from the nomenclature of format v1, where the type of
stages and inputs was called `name`.
Resolve relative paths for items the `api.arguments` call: Since paths
are different on the host and in the container, they can be transmitted
relative. Resolve the items for all groups that have paths registered.
Instead if using `check=True` for `subprocess.run`, which turns
a process failure (i.e. non-zero return codes) into generic a
`CalledProcessError` exception, use `check=False` and explicitly
handle mount errors, translating them into a `RuntimeError` with
a better error message.
Implement a new `read_at` method that will bind mount the tree of the
object to a specified location, instead of a temporary directory as
it done in the `read` method. Implement the latter via `read_at`.
Implement the corresponding methods for `Store{Client,Server}`. Since
the `ObjectStore.read_at` method will fail if the target directory
does not exist (or is of the wrong type), catch any exceptions in
the `StoreServer` and send those to the `StoreClient` via an `error`
entry.
This one is for David: also fix a missing blank line.
This disables buffering for the standard output stream for python
executables spawn within the build root. This should help with
the ordering of text output in stages: when stdout is buffered,
debug messages via `print` will be end up in that buffer. When
executables are run in the stage, via `subprocess.run` their
stdout has its own buffering, which will be flushed at the end
of the run. If stdout was not manually flushed before invoking
the executable, the output of the tool will be emitted before
anything in the buffer. For example:
print("stage")
subprocess.run(["echo", "tool"])
Will lead have the following ordering:
"tool"
"stage"
To avoid this, without having to manually flush the stdout
buffer before every `subprocess.run`, disable buffering for
python binaries run inside the build root.
Since low-level primitives (os.read) are used to read from the stdout
pipe, manual text decoding was necessary there anyway. The `encoding`
argument meant that we could forgo the manual decoding for the call
to `communicate`. But this meant that text handling is not uniform.
Therefore, remove the `encoding` argument from the `Popen` call and
manual decode all the text.
Host services are a way to provide functionality to stages that is
restricted to the host and not directly available in the container,
such as providing input to stages, devices access and mounting.
This commit introduces a `ServiceManager` class that can be used to
start and (automatically) stop host service, as well as a `Service`
base class together with a `ServiceClient` class that be used to
implement host services and communicate with them. Refer to the doc
string of the module for more information.
Often, a message is being sent and followed by a call to `recv`
to wait for a reply. Create a simple helper `send_and_recv` that
does both in one method.
Add a simple check for that helper to the tests.
Add a new constructor method that allows creating a `Socket` from
an existing file-descriptor of a socket. This might be need when
the socket was passed to a child process.
Add a simple test for the new constructor method.
Add a new constructor method, `Socket.new_pair`, to create a pair
of connected sockets (via `socketpair`) and wrap both sides via
`jsoncomm.Socket`.
Add a simple test to check it.
For schema version 2 of modules, the `definitions` node, as defined in
the module itself, won't be at the `options` level but at the level of
the `properties` node. Look for a `definitions` at that `properties`
level and move it to the top, if found.
When parsing the module file, parse the JSON directly from the AST
node, because the AST node contains the line number of the schema
in the module and thus we can resolve the correct line number for
errors within the JSON. Convert the `JSONDecodeError` to a
`SyntaxError` which results in an overall better exception message:
Before:
Traceback (most recent call last):
File "/workspaces/osbuild/osbuild/meta.py", line 331, in get_schema
opts = self._make_options(version)
[...]
File "/usr/lib64/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in
double quotes: line 2 column 1 (char 14)
After:
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
[...]
raise SyntaxError(msg, detail) from None
File "stages/org.osbuild.ostree.init-fs", line 31
additionalProperties: False
^
SyntaxError: Invalid schema: Expecting property name enclosed in ...
Define the mapping of modules and their paths at the `ModuleInfo` class
level instead of having it inline in a function. This makes it possible
to use it from other places in the code.
The previous version covered too few use cases, more specifically a
single subscription. That is of course not the case for many hosts, so
osbuild needs to understand subscriptions.
When running org.osbuild.curl source, read the
/etc/yum.repos.d/redhat.repo file and load the system subscriptions from
there. While processing each url, guess which subscription is tied to
the url and use the CA certificate, client certificate, and client key
associated with this subscription. It must be done this way because the
depsolving and fetching of RPMs may be performed on different hosts and
the subscription credentials are different in such case.
More detailed description of why this approach was chosen is available
in osbuild-composer git: https://github.com/osbuild/osbuild-composer/pull/1405
The usage of the `sources.SourcesServer` and `sources.get` have
been removed from `Stage.run`, which was the only usage throughout
osbuild and thus it is not needed anymore and can be removed.
All sources are now pre-fetched before any pipeline and thus any
stage is being built. Additionally, in the version 1 foramt, all
stages that were using source are converted to use inputs when
the manifest is loaded. Thus, nothing should use `source.get`
and thus the sources API (`SourcesServer`) anymore.
If a pipeline has an assembler and that assembler failed, the
overall status of the build also needs to be marked as failed.
This used to be the case, but a bug got introduced when the
format abstraction code was added.
A new module that can parse and execute Lorax script templates,
which are mako template based files that support a limited set
of commands, like "install", "remove" and such.
The module provides helper functions to parse such templates
and execute them by providing a re-implementation of a subset
of the commands. All commands needed for running the post
installationtemplates were implemented.
