The BLS specification [0] says the `options` field is optional and
can also appear multiple times. This commit tweaks the code to
deal with these corner cases and also adds tests that ensure that
this works correctly.
It also tweaks the file handling to be atomic.
[0] https://uapi-group.org/specifications/specs/boot_loader_specification/
- Add functions for appending kernel parameters to the
Boot Loader Specification (BLS) as needed.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
When parsing a container input, add the checksum to the data as well.
Usually with other inputs, the stage only needs to know the filepath
where it can find the source content. In most (all, so far) cases, this
is a checksum appended to the content type.
In this case, the filepath is the location of the storage bind mount and
the checksum is needed to retrieve the container. The name might only
be a destination name (a name to use for storing the container in the
image), so we can't rely on it being valid in the source.
The system-wide location for the containers storage.conf is
/usr/share/containers. The existence of a file in /etc/containers
completely overrides this (see containers-storage.conf(5)).
If no file is found at /etc/containers/storage.conf then fall back to
reading the config from /usr/share/containers/storage.conf.
If neither file exists, this is an error since the default config should
be packaged with any tool that requires it (skopeo, podman, etc).
Containers with the `containers-storage` are bind mounted to
the osbuild store. This helper function bind mounts the
containers-storage from the store into the `/containers/storage`
so that stages that require containers can then make use of these
containers.
Under certain conditions a bind mount without a specified `rw` or `ro`
option gets mounted read-only. We need a way to be explicit about
needing a rw mount. We might want to change this in the future to be a
single option (mode optional?) with valid values "rw", "ro".
It's not entirely clear what the conditions are but it occurs when bind
mounting the containers storage into the osbuild store, which we will
need for the next few commits.
The new `testutil.mock_command` context manager can be used to
mock commands in PATH and replace them with arbitrary shell
scripts. This is useful in testing to e.g. simulate exact error
conditions that would be hard to trigger otherwise or to replace
long running commands with faked results.
Example:
```
fake_cmd = textwrap.dedent("""\
do-something
""")
with mock_command("some-cmd", fake_cmd):
your_code
```
This adds a `default: true` option for all cases where OSTree
information is specified in schemas and allows for the information
to be picked up from the filesystem.
This is a safe operation because when building disk images there is
no known case where having two deployments makes sense. In the case
there ever were a case then the osname, ref, and serial options still
exist and can be used.
Co-authored-by: Luke Yang <luyang@redhat.com>
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
This commit adds code that will remove the least recently used
entries when a store() operation does not succeeds because the
cache is full. To be more efficient it will try to free
twice the requested size (this can be configured in the code).
Instead of operating directly on the tree for a stage we can operate
on a mount too. This is useful in the case where operating on the
directory tree of files isn't sufficient and the modifications need
to be made directly to the filesystems on the disk image that we are
creating.
One such example of this is we are having a problem right now where
the immutable bit being set on an OSTree deployment root doesn't
survive the `cp -a --reflink=auto` in the org.osbuild.copy stage when
being copied from the directory tree into the mounted XFS filesystem
we created on the disk image. Thus we have to workaround this loss
of attribute by applying the attribute directly on the mounted
filesystem from the disk.
In this change here we also add a check in osbuild/mounts.py to not
attempt a umount of the root of the mounts directory if that path
is no longer a mountpoint, which can happen when the umount -R
from the mounts/org.osbuild.ostree.deployment also removes the
overmount.
Here is an example of how this would be used:
```
- type: org.osbuild.chattr
options:
immutable: true
path: mount://root/
devices:
disk:
type: org.osbuild.loopback
options:
filename: disk.img
partscan: true
mounts:
- name: root
type: org.osbuild.xfs
source: disk
partition:
mpp-format-int: '{image.layout[''root''].partnum}'
target: /
- name: ostree.deployment
type: org.osbuild.ostree.deployment
options:
source: mount
deployment:
osname: fedora-coreos
ref: ostree/1/1/0
```
The initial mount on `/` is the filesystem from the root partition
on the disk. The second mount (of type org.osbuild.ostree.deployment)
then reconfigures things similar to how an OSTree system is set up.
This is a drive-by change after spending some quality time with the
mount code. The `id` field of `Mount` is calculated only once and
only when creating a `Mount`. This seems slightly dangerous as
any change to an attribute after creation will not update the
id. This means two options:
1. dynamically update the `id` on changes
2. forbid changes after the `id` is calculcated
I went with (2) but happy to discuss of course but it seems more
the spirit of the class.
It also does the same change for "devices.Device"
Similar to rd.break for dracut this allows a user to specify:
- --break or --break=*
- to get a shell before each stage is run
- --break=stage.name
- to get a shell each time the stage with that name is run
- example: --break=org.osbuild.copy
- --break=stage.id
- to get a shell each time the stage with that ID is run
- get the ID for the stages for your manifest by running
osbuild on the manifest with --inspect
- example: --break=dc6e3a66fef3ebe7c815eb24d348215b9e5e2ed0cd808c15ebbe85fc73181a86
and get a bash shell where they can inspect the environment to debug
and develop OSBuild stages.
This commit reworks the `org.osbuild.container-deploy` stage to
not use a tmp storage when mounting the container image. This
is needed because of [0] but it should generally be fine because
inside the stages the real /var is a tmpfs (which is why we
triggered the bug in the first place).
[0] https://github.com/containers/storage/issues/1779
Extract a new helper `make_fake_tree()` that generalizes the existing
helper `make_fake_input_tree()`. The later will always create the
content under `{basedir}/tree` which is convinient for input tree
based tests but too specialized when using it in different contexts.
The existing `make_fake_input_tree()` is preserved unchanged and
becomes just a tiny wrapper.
To workaround the issue that inside macOS containers the ownership
cannot be preserved we introduce a new environment that can be
used to forcefully relax the use of `cp -a`.
I did it via an environment instead of a commandline option mostly
because `github.com/osbuild/images/osbuild:RunOBuild()` already has
`extraEnv` option.
This commit allows to exclude preserving ownership from an object
export. This is required to fix the issue that on macOS the an
podman based workflow cannot export objects with preserving
ownerships.
Originally this was a `no_preserve: Optional[List[str]] = None)`
to be super flexible in what we pass to `cp` but then I felt like
YAGNI - if we need more we can trivially change this (internal)
API again :)
Similar to the aleph file created for builds of FCOS based on ostree
commit inputs, this adds an aleph file that contains information about
the initial deployment of data when the disk image was built
A new stage is preferred here as both the org.osbuild.ostree.deploy
and org.osbuild.ostree.deploy.container stages need an aleph file and
use of the aleph file may depend on the project/product. For example,
right now CoreOS is the only project that uses an aleph file, but others
may want it in the future.
And also set stdout=subprocess.PIPE. This will allow for callers to
parse and use the output of the command, but has the side effect of
meaning less gets printed to the screen during run.
Co-authored-by: Luke Yang <luyang@redhat.com>
Adjust the source path to just have /dev in front (i.e. /dev/loop0)
and not be a path to the temporary dev directory that was prepared.
We do this because some tools (like grub2-install) consult
mountinfo to try to canonicalize paths for mounts.
Fixes https://github.com/osbuild/osbuild/issues/1492
This helper can be used to implement a strategy to find the oldest
cache entries and evict them when the cache is full.
The implementation uses the `atime` of the per object `cache.lock`
file and ensures in `load()` that it's actually updated.
This commit adds mount output to the error raised by
FileSystemMountService.mount(). This is useful when running into
mount failures during osbuild runs.
The issue was discovered while debugging a mount failure for
osbuild-composer PR#3820. Initially osbuild PR#1490 was meant
to fix it but it turned out there is a third mount helper in
the code that was originally overlooked (sorry for that!).
This is a convenient way for tests to assert that some nested dicts
(like a parsed json) has a particular key/value somewhere in it.
For example:
assert_dict_has(config, "toplevel.subitem.key", True)
