A new helper for the util.linux module which exposes the linux boot-id.
For security reasons, the boot-id is never exposed directly, but
instead only exposed through an application-id combined with the boot-id
via HMAC-SHA256.
Note that a raw kernel boot-id is always considered confidential, since
we never want an outside entity to deduce any information when they see
a boot-id used in protocol A and one in protocol B. It should not be
possible to tell whether both are from the same user and boot or not.
Hence, both should use their own boot-id namespace.
This adds a new accessor-function for the file-locking operations
through `fcntl(2)`. In particular, it adds the new function
`fcntl_flock()`, which wraps the `F_OFD_SETLK` command on `fcntl(2)`.
There were a few design considerations:
* The name `fcntl_flock` comes from the `struct flock` structure that
is the argument type of all file-locking syscalls. Furthermore, it
mirrors what the `fcntl` module already provides as a wrapper for
the classic file-locking syscall.
* The wrapper only exposes very limited access to the file-locking
commands. There already is `fcntl.fcntl()` and `fcntl.fcntl_flock()`
in the standard library, which expose the classic file-locks.
However, those are implemented in C, which gives much more freedom
and access to architecture dependent types and functions.
We do not have that freedom (see the in-code comments for the
things to consider when exposing more fcntl-locking features).
Hence, this only exposes a very limited set of functionality,
exactly the parts we need in the objectstore rework.
* We cannot use `fcntl.fcntl_flock()` from the standard library,
because we really want the `OFD` version. OFD stands for
`open-file-description`. These locks were introduced in 2014 to the
linux kernel and mirror what the non-OFD locks do, but bind the
locks to the file-description, rather than to a process. Therefore,
closing a file-description will release all held locks on that
file-description.
This is so much more convenient to work with, and much less
error-prone than the old-style locks. Hence, we really want these,
even if it means that we have to introduce this new helper.
* There is an open bug to add this to the python standard library:
https://bugs.python.org/issue22367
This is unresolved since 2014.
The implementation of the `fcntl_flock()` helper is straighforward and
should be easy to understand. However, the reasoning behind the design
decisions are not. Hence, the code contains a rather elaborate comment
explaining why it is done this way.
Lastly, this adds a small, but I think sufficient unit-test suite which
makes sure the API works as expected. It does not test for full
functionality of the underlying locking features, but that is not the
job of a wrapping layer, I think. But more tests can always be added.
This modification will allow a user to ask to mount the system as read
only for instance. Which would be super useful for image-info who is
progressively using more of OSbuild internals to mount partitions.
A bug afflicts image-info on these distributions. We need to perform
modifications to the way image-info mounts the image to do its analyzis.
OSBuild needs some changes for that to happen:
- see https://github.com/osbuild/osbuild/pull/1182.
Since it will not be possible to land anything on OSBuild until
image-info is fixed, let's temporarily deactivate these two archs.
The `Object.{read,write}` methods were introduced to implement
copy on write support. Calling `write` would trigger the copy,
if the object had a `base`. Additionally, a level of indirection
was introduced via bind mounts, which allowed to hide the actual
path of the object in the store and make sure that `read` really
returned a read-only path.
Support for copy-on-write was recently removed[1], and thus the
need for the `read` and `write` methods. We lose the benefits
of the indirection, but they are not really needed: the path to
the object is not really hidden since one can always use the
`resolve_ref` method to obtain the actual store object path.
The read only property of build trees is ensured via read only
bind mounts in the build root.
Instead of using `read` and `write`, `Object` now gained a new
`tree` property that is the path to the objects tree and also
is implementing `__fspath__` and so behaves like an `os.PathLike`
object and can thus transparently be used in many places, like
e.g. `os.path.join` or `pathlib.Path`.
[1] 5346025031
Create a private mount point for the tree, so that later we can
move the `root` mount point. This is needed since "moving a mount
residing under a shared mount is invalid and unsupported.", see
`mount(8)`. Currently the `tree` is mounted via a private mount-
point since reading the tree is done via bind-mounts, but this
will change in subsequent commits; this prepares for it.
When creating the JSON data, call `os.fspath` on all paths, like
`root` and `devices.tree` to ensure they are strings; this allows
for tree to be an object that conforms to `os.PathLike`.
Include the new journald config stage to configure journald to
persist the journal. This is needed since we don't create the
`/var/log/journal` directory that journald uses to switch the
default to persistent storage. But instead of creating that
directory, we explicitly configure journald via the new stage.
This is also what Fedora CoreOS does.
A recent commit (8a7b6d3) fixed the ostree config stage and thus
we actually properly set the readonly flag for the deployment.
This broke the image since we did not specify the `rw` kernel flag
and as a consequence the ostree mounting code in the initrd broke.
If the object's id does not match with the one supplied for the
commit, we create a clone. Otherwise we store the tree.
The code path is arranged in a way that we always go through
`Object.store_tree` so we always call `Object.finalize` as a
prepration for the future, where we might actually do something
meaningful in the finalizer, like reset the *times or count the
tree size.
Remove copy-on-write support from `objectstore.Object`. The main
reason for introducing copy-on-write was to save an additional
copy in the non DAG-pipeline model[1]. With the introduction of
the latter and the explicit `--export` option, we can achieve the
same result without the complexity of copy-on-write semantics.
[1] See commit 39213b7, part of 3b7c87d5..42a365d1 changeset.
There is little use in sharing the store between test, quite to
opposite: all tests expect a clean store and some currently set
that up themselves. Create a fresh store for each test.
When committing an object to the store, clone it if the current
stage is not the latests stage, i.e. `todo` has still entries.
This is the second step of the removal of copy-on-write support
in `Object`.
Add a new `clone` parameter to the `commit` method on `ObjectStore`
that when used will clone the object to the store instead of using
the `store_tree` method which moves the object and resets it. This
is the first step of removing copy-on-write support from `Object`.
In newer version of the container storage package the config file
moved from `/etc/containers` to `/usr/containers/`. The later is
not marked as config, so we don't want to change it.
The current containers code[1] will read _either_ a file in `usr`
or in `etc` depending on the existence of the latter. This we can
not just write the keys we want into a file in `/etc/containers`
without losing all other defaults set in the config file.
A new option `filebase` is therefore added, that when given will
be read and form the bases of the configuration data. Then data
from the target file (given via `filename`) will be merged into
and finally the actual configuration will be applied on top.
[1] 232bf398bd/types/options.go (L85)
If the existing config does not have a section that is being set
by the stage it currently would crash since we assume that it
exists. Default to an empty dict in that case.
Create a squashfs image from the root file system and append it
to the existing initram fs. Passing `root=live:/rootfs.img`
is enough to use the existing dmsquash machiner in dracut.
Add an new module with utility functions to inspect PE32+ files,
mainly listing the sections and their addresses and sizes.
Include a simple test to check that we can successfully parse the
EFI stub contained in systemd (systemd-udev package).
add the ability to configure the grub timeout. Today it defealts to 1
minute and we can't configure it from osbuild-composer either (the
simplified-installer is mainly using this afaict and 1 min is just too
much).
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
According to the OCI Image Index specification the mediaType field is not mandatory
Assume that it is a list if mediaType is not set while manifests field is
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
This code was clearly influenced/copied from rpm-ostree,
since it's now duplicated let's cross-link to help ensure that
if someone wants to change this they hopefully consider changing
rpm-ostree too.
