- Move functions to the 'util' to centralize common
functionality, reducing code duplication and improving
maintainability across the codebase.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
This is needed because on a mounted `bootc` container `setfiles`
without excluding `/sysroot` will create many warnings like:
```
setfiles: conflicting specifications for /run/osbuild/tree/sysroot/ostree/repo/objects/00/0ef9ada2ee87792e8ba21afd65aa00d79a1253018832652b8694862fb80e84.file and /run/osbuild/tree/usr/lib/firmware/cirrus/cs35l41-dsp1-spk-prot-103c8b8f-r1.bin.xz, using system_u:object_r:lib_t:s0.
```
but simply excluding this dir fixes them.
Change the contianers store from `/containers/storage` to
`/var/tmp/containers/storage` since technically `/containers/storage`
isn't ostree compatible.
This is a follow up to #1550 where we enabled a `rw` permissions mode,
which is not ideal since it would theoretically be possible to set both
`ro` and `rw` modes at the same time. This commit fixes the issue by only
allowing one option at a time.
Fixes#1588
The BLS specification [0] says the `options` field is optional and
can also appear multiple times. This commit tweaks the code to
deal with these corner cases and also adds tests that ensure that
this works correctly.
It also tweaks the file handling to be atomic.
[0] https://uapi-group.org/specifications/specs/boot_loader_specification/
- Add functions for appending kernel parameters to the
Boot Loader Specification (BLS) as needed.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
When parsing a container input, add the checksum to the data as well.
Usually with other inputs, the stage only needs to know the filepath
where it can find the source content. In most (all, so far) cases, this
is a checksum appended to the content type.
In this case, the filepath is the location of the storage bind mount and
the checksum is needed to retrieve the container. The name might only
be a destination name (a name to use for storing the container in the
image), so we can't rely on it being valid in the source.
The system-wide location for the containers storage.conf is
/usr/share/containers. The existence of a file in /etc/containers
completely overrides this (see containers-storage.conf(5)).
If no file is found at /etc/containers/storage.conf then fall back to
reading the config from /usr/share/containers/storage.conf.
If neither file exists, this is an error since the default config should
be packaged with any tool that requires it (skopeo, podman, etc).
Containers with the `containers-storage` are bind mounted to
the osbuild store. This helper function bind mounts the
containers-storage from the store into the `/containers/storage`
so that stages that require containers can then make use of these
containers.
Under certain conditions a bind mount without a specified `rw` or `ro`
option gets mounted read-only. We need a way to be explicit about
needing a rw mount. We might want to change this in the future to be a
single option (mode optional?) with valid values "rw", "ro".
It's not entirely clear what the conditions are but it occurs when bind
mounting the containers storage into the osbuild store, which we will
need for the next few commits.
This adds a `default: true` option for all cases where OSTree
information is specified in schemas and allows for the information
to be picked up from the filesystem.
This is a safe operation because when building disk images there is
no known case where having two deployments makes sense. In the case
there ever were a case then the osname, ref, and serial options still
exist and can be used.
Co-authored-by: Luke Yang <luyang@redhat.com>
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
This commit adds code that will remove the least recently used
entries when a store() operation does not succeeds because the
cache is full. To be more efficient it will try to free
twice the requested size (this can be configured in the code).
Similar to the aleph file created for builds of FCOS based on ostree
commit inputs, this adds an aleph file that contains information about
the initial deployment of data when the disk image was built
A new stage is preferred here as both the org.osbuild.ostree.deploy
and org.osbuild.ostree.deploy.container stages need an aleph file and
use of the aleph file may depend on the project/product. For example,
right now CoreOS is the only project that uses an aleph file, but others
may want it in the future.
And also set stdout=subprocess.PIPE. This will allow for callers to
parse and use the output of the command, but has the side effect of
meaning less gets printed to the screen during run.
Co-authored-by: Luke Yang <luyang@redhat.com>
This helper can be used to implement a strategy to find the oldest
cache entries and evict them when the cache is full.
The implementation uses the `atime` of the per object `cache.lock`
file and ensures in `load()` that it's actually updated.
While debugging a failure of osbuild-composer [0] on fc39 it was
noticed that a mount failure does not include the output of
the mount command:
```
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 78, in mount
path = client.call("mount", args)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 348, in call
ret, _ = self.call_with_fds(method, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 384, in call_with_fds
raise error
osbuild.host.RemoteError: CalledProcessError: Command '['mount', '-t', 'xfs', '-o', 'ro,norecovery', '--source', '/dev/rootvg/applv', '--target', '/tmp/tmpjtfmth56/app']' returned non-zero exit status 32.
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 268, in serve
reply, reply_fds = self._handle_message(msg, fds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 301, in _handle_message
ret, fds = self.dispatch(name, args, fds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 111, in dispatch
r = self.mount(args)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 160, in mount
subprocess.run(
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
```
which makes diagnostic errors harder of course. This commit adds
a test that ensures that mount output is visbile and also changes
the code to include it.
[0] https://github.com/osbuild/osbuild-composer/pull/3820
In OSBuild we'll often be operating on sparse files. Let's make the
tabulation of the size of files on disk used when determining cache
size for pruning consider the actual size of the file usage on disk
rather than the size the file reports to be.
This means using os.lstat().st_blocks * 512 versus os.lstat().st_size.
See https://stackoverflow.com/a/55203604
The `shutil.rmtree(onerror=...)` kwarg got deprecated with py3.12.
We still need to support older version of python all the way
back to 3.6 so just ignore this pylint error for a while.
This is a feature that was added in rpm-ostree 2023.10 and is needed
for the new transient /etc feature to work. What it does is change the
labeling of /usr/etc to match those of /etc, so that /usr/etc can be used
directly as a bind-mount or an overlay mount when mounted on /etc.
See https://github.com/coreos/rpm-ostree/pull/4640 for details.
This will hoist even more code into util out of the skopeo stage.
Now a caller can call:
with containers.container_source(image) as (image_name, image_source):
print(f"{image_name}, {image_source}")
to process containers inputs.
This hoists container handling code from the skopeo stage into
util/containers. It is prep for adding another stage that accepts
containers as an input. The code is common so we should share it
amongst all stages that use containers as input.
`tox` is a standard testing tool for Python projects, this allows you to
test locally with all your installed Python version with the following
command:
`tox -m test -p all`
To run the tests in parallel for all supported Python versions.
To run linters or type analysis:
```
tox -m lint -p all
tox -m type -p all
```
This commit *also* disables the `import-error` warning from `pylint`,
not all Python versions have the system-installed Python libraries
available and they can't be fetched from PyPI.
Some linters have been added and the general order linters run in has
been changed. This allows for quicker test failure when running
`tox -m lint`. As a consequence the `test_pylint` test has been removed
as it's role can now be fulfilled by `tox`.
Other assorted linter fixes due to newer versions:
- use a str.join method (`consider-using-join`)
- fix various (newer) mypy and pylint issues
- comments starting with `#` and no space due to `autopep8`
This also changes our CI to use the new `tox` setup and on top of that
pins the versions of linters used. This might move into separate
requirements.txt files later on to allow for easier updating of those
dependencies.
Fix the following errors:
```
osbuild/util/lvm2.py:117: error: Only instance methods can be decorated with @property
osbuild/api.py:50: error: Only instance methods can be decorated with @property
osbuild/sources.py:85: error: Only instance methods can be decorated with @property
```
Chaining of `@classmethod` and `@property` has been deprecated since
Python 3.11 with a note that chaining didn't work correctly in some
cases.
Relevant links:
https://github.com/python/mypy/issues/13746https://docs.python.org/3.11/whatsnew/3.11.html#language-builtins
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The cachedir-tag specification defines how to mark directories as
cache-directories. This allows tools like `tar` to ignore those
directories if desired (e.g., see `tar --ignore-caches`). This is very
useful to avoid huge cache-directories in backups and remote
synchronizations.
The spec simply defines a file called `CACHEDIR.TAG` with the first 43
bytes to be: "Signature: 8a477f597d28d172789f06886806bc55" (which
happens to be the MD5-checksum of ".IsCacheDirectory". Further content
is to be ignored. Any such files marks the directory in question as a
cache-directory.
The cachedir-tag has been successfully deployed in tools like `cargo`
and `VLC`, and is currently discussed to be implemented in Firefox. More
information is available here: https://bford.info/cachedir/
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Add trace-hooks to the FsCache._atomic_open() helper, including a
primitive trace-infrastructure. They allow interrupting cache operation
and running arbitrary code.
The trace-hooks will be used by the test-suite to trigger the races we
want to protect against. During runtime, the traces should not be used
and thus will always be `None`.
This is a very primitive way to hook into the runtime execution and test
the atomicity of the operations. However, it is simple enough for our
tests and avoids pulling in huge tracing suites.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
On NFS, we need to be careful with cached metadata. To make sure our
_atomic_open() can correctly catch races during open+lock, we must be
careful to catch `ESTALE` and `ENOENT` from `stat()` calls. Otherwise,
the lock-acquisition guarantees that data is coherent, even on NFS.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
We used to commit cache-entries with a rename+RENAME_NOREPLACE. This,
however, is not available on NFS. Change the code to use `os.rename()`
and rely on the _documented_ kernel behavior that non-empty target
directories cannot be replaced.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
The `RENAME_NOREPLACE` option is not available on NFS. Avoid using it
in _atomic_file() to allow NFS backed storage.
If the caller allows replacing the destination entry, we simply use the
original `os.rename()` system call. This will unconditionally replace
the destination on all file-systems.
If the caller requests `no-replace`, we cannot use `os.rename()`.
Instead, we use `os.link()` to create a new hard-link on the
destination. This will always fail if the destination already exists.
We then rely on the cleanup-path to unlink the original temporary
entry.
This will require adjustments in future maintenance tasks on the cache,
since they need to be aware that entries can be hardlinked temporarily.
However, we already consider `uuid-*` entries in the object-store to be
temporary and unaccounted for similar reasons, so this doesn't even
break our cache-maintenance ideas.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
