The current implementation of `rmtree` will try to fix permissions
when it encounters permission errors during its operation. This is
done by opening the target via `os.open` and then adjusting the
immutable flag and the permission bits. This is a problem when the
target is a broken symlink since open will fail with `ENOENT`. A
simple reproducer of this scenario is:
$ mkdir subdir
$ ln -s foo subdir/broken
$ chmod a-w subdir/
$ python3 -c 'import osbuild; osbuild.util.rmrf.rmtree("subdir")'
Since subdir is not writable, removing `subdir/broken` will fail
with `EPERM` and the `on_error` callback will try to fix it by
invoking `fixperms` on `subdir/broken` which will in `open` since
the target does not exist (broken symlink).
This is fixed by using `O_NOFOLLOW` to open so we will never open
the target. Instead `open` will fail with `ELOOP`; we ignore that
error and in fact we ignore now all errors from `open` since it
does not matter: if fixing the permissions didn't work `unlink`
will just fail (again) with `EPERM` and for symlinks it actually
doesn't matter since "on Linux the permissions of an ordinary
symbolic link are not used in an operations", see symlinks(7).
When `get_fallback_rhsm_secrets` was used, `Subscriptions.repositories`
was None, and `get_secrets` never returned the fallback secrets.
So check if `repositories` is None before
iterating over it, otherwise return the fallback secrets.
This stage takes /usr/lib/passwd and /usr/etc/passwd from an OSTree
checkout, merges them into one file, and store it as /etc/passwd in the
buildroot.
It does the same for /etc/group.
The reason for doing this is that there is an issue with unstable UIDs
and GIDs when creating OSTree commits from scratch. When there is a
package that creates a system user or a system group, it can change the
UID and GID of users and groups that are created later.
This is not a problem in traditional deployments because already created
users and groups never change their UIDs and GIDs, but with OSTree we
recreate the files from scratch and then replace the previous one so it
can actually change.
By copying the files to the build root before doing any other
operations, we can make sure that the UIDs and GIDs of already existing
users and groups won't change.
Co-author: Christian Kellner <christian@kellner.me>
Add a helper method to call `ioctl(fd, BLK_IOC_FLUSH_BUFFER, 0)`
from python. NB: the ioctl number 0x1261 is wrong on at least
alpha and sparc. A later test will use this call so we should
catch the usage of it on those platforms.
This module provides a `Disk` class that can be used
to read in LVM images and explore and manipulate its
metadata directly, i.e. it reads and writes the data
and headers directly. This allows one to rename an
volume group without having to involve the kernel,
which does not like to have two active LVM volume
groups with the same name.
The problem is that some deployments might not have the redhat.repo
file, yet they might have the key and certificate to access Red Hat CDN.
If that was the case, the new approach would cause a regression compared
to the previous behavior.
This patch uses the previous method if the redhat.repo file is not
found or does not contain any matching URL.
Add a simple helper method that returns the path for a deployment,
given the sysroot, the osname, the reference or commit and the
deployment serial. Path might not exist.
`
Checks if one path is a child of a second one. Useful for checking if
paths defined in a manifest exist inside the tree.
Optionally checks if the target path exists.
Often, a message is being sent and followed by a call to `recv`
to wait for a reply. Create a simple helper `send_and_recv` that
does both in one method.
Add a simple check for that helper to the tests.
Add a new constructor method that allows creating a `Socket` from
an existing file-descriptor of a socket. This might be need when
the socket was passed to a child process.
Add a simple test for the new constructor method.
Add a new constructor method, `Socket.new_pair`, to create a pair
of connected sockets (via `socketpair`) and wrap both sides via
`jsoncomm.Socket`.
Add a simple test to check it.
The previous version covered too few use cases, more specifically a
single subscription. That is of course not the case for many hosts, so
osbuild needs to understand subscriptions.
When running org.osbuild.curl source, read the
/etc/yum.repos.d/redhat.repo file and load the system subscriptions from
there. While processing each url, guess which subscription is tied to
the url and use the CA certificate, client certificate, and client key
associated with this subscription. It must be done this way because the
depsolving and fetching of RPMs may be performed on different hosts and
the subscription credentials are different in such case.
More detailed description of why this approach was chosen is available
in osbuild-composer git: https://github.com/osbuild/osbuild-composer/pull/1405
A new module that can parse and execute Lorax script templates,
which are mako template based files that support a limited set
of commands, like "install", "remove" and such.
The module provides helper functions to parse such templates
and execute them by providing a re-implementation of a subset
of the commands. All commands needed for running the post
installationtemplates were implemented.
Explicit re-raise the BufferError exception in recv from the orignal
JSONDecodeError, so the latter gets recorded as the underlying cause.
Uncovered by pylint 2.6.0: W0707: "Not using raise from makes the
traceback inaccurate, because the message implies there is a bug in
the exception-handling code itself, which is a separate situation
than wrapping an exception."
Now that jsoncomm.Socket is using a connection-oriented socket,
the destination in `socket.sendmsg` is ignored and thus can and
should be dropped from the `jsoncomm.Socket.send` method.
Adjust the tests accordingly.
Switch to use a connection oriented datagram based protocol, i.e.
`SOCK_SEQPACKET`, instead of `SOCK_DGRAM`. It sill preserves
message boundaries, but since it is connection oriented the client
nor the server do not need to specify the destination addresses
of the peer in sendmsg/recvmesg. Moreover, the host will be able
to send messages to the client, even if the latter is sandboxed
with a separate network namespace. In the `SOCK_DRAM` case the
auto-bound address of the client would not be visible to the host
and thus sending messages would to it would fail.
Adapt the jsoncomm tests as well as `BaseAPI`.
Implement `accept` and `listen`, that call the equivalent methods
on the underlying socket; this prepares the move to a connection
oriented socket, i.e. `SOCK_SEQPACKET`.
Add a new `blocking` property to get and set the blocking state
of the underlying socket. In Python this is tied to the timeout
setting of the `socket.Socket`, i.e. non-blocking means having
any timeout specified, including "0" for not waiting at all.
Blocking means having a timeout value of `None`.
The getter is emulating the logic of `Socket.getblocking`, which
was added in 3.7, and we need to stay compatible with 3.6.
The logic is implemented in `Modules/socketmodule.c` in Python.
FdSet does derive directly and only from `object`. Not specifying
any base classes is the same as specifying an empty list of base
classes; therefore get rid of the empty list.
Add support for `util.types.PathLike` paths for socket addresses,
instead of just plain strings. Test it by using pathlib.Path to
create the address in the corresponding test.
Add a simple new module meant to define types that are commonly
used throughout the code-base. For starters, define `PathLike`
meant to represent file system paths, i.e. strings, bytes, or
anything that provides the `os.PathLike` protocol, i.e. that
can be used with `os.fspath`.
In python 3.6 the value of `__origin__` for typing.List[str] is
typing.List. This then changed to the actual `list` type in later
versions. Accept both versions.
Add the initramfs-args Treefile option that can be used to pass
arguments to drauct via rpm-ostree. NB: the ostree module will
always be automatically be included by rpm-ostree.
Using `[]` as default value for arguments makes `pylint` complain. The
reason is that it creates an array statically at the time the function
is parsed, rather than dynamically on invocation of the function. This
means, when you append to this array, you change the global instance and
every further invocation of that function works on this modified array.
While our use-cases are safe, this is indeed a common pitfall. Lets
avoid using this and resort to `None` instead.
This silences a lot of warnings from pylint about "dangerous use of []".
Extract the `suppress_oserror()` function from the ObjectManager and
make it available as utility for other code as well.
This also adds a bunch of tests that verify it works as expected.
Make use of the new immutable-flag ioctl helpers. While at it, move the
`chmod` to `fchmod` and re-use the open file-descriptor. Document the
behavior and move the `fchmod` into its own try-block for the same
reasons as the `ioctl` call: We rely on the following unlink() to catch
any errors. Errors in the fixperms() step are non-consequential.
The FS_IOC_{GET,SET}FLAGS ioctl numbers are not stable across different
architectures. Most of them use the asm-generic versions, but ALPHA and
SPARC in particular use completely different IOC number setups (see the
definition of _IOC, _IOR, _IOW, etc. in the kernel).
This commit moves the helpers for `FS_IMMUTABLE_FL` into
`osbuild/util/` and adds explicit tests. This will make sure that we
catch any ioctl mismatches as soon as possible when we run the osbuild
test-suite on other architectures. Until then, we will have to live with
this mismatch.
Move remove_tree() into its own module in `osbuild.util.rmrf`. This way
we can use it in other modules as well, without cross-referencing
internal helpers.
Add a new module that implements a simple JSON communication channel.
This is meant to replace all our hard-coded SOCK_DGRAM code that is
copied all over the place.
This is intentionally left simple. It only supports synchronous
operations, trivial JSON encoding and decoding, and uses a message-based
transport mode.
Add a small wrapper around the setfiles(8) utility that can be used
to set the security context fields on one or multiple provided paths,
given a specification. The root of the file system tree can be given
via `root` and all elements of `paths` will be interpreted as
relative to that root.
Add a helper, `parse_config`, to parse a selinux configuration file,
see selinux(8), and return a dictionary containing the configuration
data in key, value pairs. This, in turn, can be fed into the other
helper method, `config_get_policy`, to get the effective policy or
`None` if SELinux is disabled or the policy type is not configured.
Add a new test suite that checks the basic functionality of the
helpers above.
When using rpm-ostree compose, a Treefile[1] controls various
aspects of its behaviour. Since rpm-ostree will, at least in
the beginning, be used to post-process and committing the tree
add a helper class to ease the creation of correct Treefiles.
The docstring of the Treefile contains the information in which
phases ('install', 'postprocess', 'commit') the option is used,
as of rpm-ostree commit 1cf0d557ae8059e689b1fed670022727e9842288
Add basic checks for the ostree.Treefile helper. Some of the
tests require rpm-ostree to be installed.
[1] https://rpm-ostree.readthedocs.io/en/stable/manual/treefile/
