This is a feature that was added in rpm-ostree 2023.10 and is needed
for the new transient /etc feature to work. What it does is change the
labeling of /usr/etc to match those of /etc, so that /usr/etc can be used
directly as a bind-mount or an overlay mount when mounted on /etc.
See https://github.com/coreos/rpm-ostree/pull/4640 for details.
In the case we are not using a buildroot (i.e. we are using
the host as the buildroot) let's also mount in /etc/containers
into the environment. There are sometimes where software running
from /usr can't operate without configuration in /etc and this
will allow it to work.
An example of software hitting this problem is skopeo. With a
simple config like:
```
version: '2'
mpp-vars:
release: 38
pipelines:
- name: skopeo-tree
# build: name:build
source-epoch: 1659397331
stages:
- type: org.osbuild.skopeo
inputs:
images:
type: org.osbuild.containers
origin: org.osbuild.source
mpp-resolve-images:
images:
- source: quay.io/fedora/fedora-coreos
tag: stable
name: localhost/fcos
options:
destination:
type: containers-storage
storage-path: /usr/share/containers/storage
```
We end up hitting an error like this:
```
time="2023-10-24T18:27:14Z" level=fatal msg="Error loading trust policy: open /etc/containers/policy.json: no such file or directory"
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.skopeo", line 90, in <module>
r = main(args["inputs"], args["tree"], args["options"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/osbuild/bin/org.osbuild.skopeo", line 73, in main
subprocess.run(["skopeo", "copy", image_source, dest], check=True)
File "/usr/lib64/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['skopeo', 'copy', 'dir:/tmp/tmp5_qcng99/image', 'containers-storage:[overlay@/run/osbuild/tree/usr/share/containers/storage+/run/containers/storage]localhost/fcos']' returned non-zero exit status 1.
```
This PR adds in a mount for /etc/containers from the host so that
/etc/containers/policy.json can be accessed.
This will hoist even more code into util out of the skopeo stage.
Now a caller can call:
with containers.container_source(image) as (image_name, image_source):
print(f"{image_name}, {image_source}")
to process containers inputs.
This hoists container handling code from the skopeo stage into
util/containers. It is prep for adding another stage that accepts
containers as an input. The code is common so we should share it
amongst all stages that use containers as input.
When developing or rebuilding manifests a lot it is common to want to
checkpoint everything to the store. It seems we all have small shell
scripts hanging around for this.
Let `--checkpoint` take a shell-like glob such as `--checkpoint="*"` to
checkpoint everything.
Note that there's a behavioral change here; previously `osbuild
--checkpoint=a` would error if that specific checkpoint wasn't found.
Now `osbuild` will only error if nothing was selected by the passed
globs.
`tox` is a standard testing tool for Python projects, this allows you to
test locally with all your installed Python version with the following
command:
`tox -m test -p all`
To run the tests in parallel for all supported Python versions.
To run linters or type analysis:
```
tox -m lint -p all
tox -m type -p all
```
This commit *also* disables the `import-error` warning from `pylint`,
not all Python versions have the system-installed Python libraries
available and they can't be fetched from PyPI.
Some linters have been added and the general order linters run in has
been changed. This allows for quicker test failure when running
`tox -m lint`. As a consequence the `test_pylint` test has been removed
as it's role can now be fulfilled by `tox`.
Other assorted linter fixes due to newer versions:
- use a str.join method (`consider-using-join`)
- fix various (newer) mypy and pylint issues
- comments starting with `#` and no space due to `autopep8`
This also changes our CI to use the new `tox` setup and on top of that
pins the versions of linters used. This might move into separate
requirements.txt files later on to allow for easier updating of those
dependencies.
Prior this commit, the arguments for the input service were passed inline.
However, jsoncomm uses the SOCK_SEQPACKET socket type underneath that has
a fixed maximum packet size. On my system, it's 212960 bytes. Unfortunately,
that's not enough for big inputs (e.g. when building packages with a lot
of rpms).
This commit moves all arguments to a temporary file. Then, just a file
descriptor is sent. Thus, we are now able to send arbitrarily sized args
for inputs, making osbuild work even for large image builds.
LOOP_CONFIGURE allows to atomically configure the decive when opening
it. This avoid the possibility of a race condition where between set_fd
and set_status some operations are already accepted by the loopback
device. See https://lwn.net/Articles/820408/
This feature was included in the linux kernel 5.8 however it is safe to
not include any kind of fallback to the previous method as @obudai
points out that:
LOOP_CONFIGURE was backported into RHEL 8 kernel in RHEL 8.4 as a part
of https://bugzilla.redhat.com/show_bug.cgi?id=1881760 (block layer:
update to upstream v5.8).
Since RHEL 8.4 is currently the oldest supported release that we support
running osbuild on, it might be just fine implementing this without the
fallback.
From a centos stream 8 container:
kernel-4.18.0-448.el8.x86_64
- loop: Fix missing discard support when using LOOP_CONFIGURE (Ming Lei) [1997338]
- [block] loop: Set correct device size when using LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: unset GENHD_FL_NO_PART_SCAN on LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: Add LOOP_CONFIGURE ioctl (Ming Lei) [1881760]
Fix the following errors:
```
osbuild/util/lvm2.py:117: error: Only instance methods can be decorated with @property
osbuild/api.py:50: error: Only instance methods can be decorated with @property
osbuild/sources.py:85: error: Only instance methods can be decorated with @property
```
Chaining of `@classmethod` and `@property` has been deprecated since
Python 3.11 with a note that chaining didn't work correctly in some
cases.
Relevant links:
https://github.com/python/mypy/issues/13746https://docs.python.org/3.11/whatsnew/3.11.html#language-builtins
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
To avoid kernel panics if the kernel attempts to recover the filesystem
when it's mounted as readonly. Offer the possiblity to use the
norecovery option for journaling file systems (Xfs, Ext4, Btrfs).
Before we could only ask OSBuild to mount a device as readonly. But
devices can have more mount options than this. Supporting more options
is necessary for the new version of image-info that is using OSBuild's
internals in order to mount the image it wants to work on. Otherwise,
for instance, some umasks aren't applied properly and we can get
differences in rpm-verify results, thus corrupting the DB.
Mount is now accepting:
* readonly
* uid
* gid
* umask
* shortname
The cachedir-tag specification defines how to mark directories as
cache-directories. This allows tools like `tar` to ignore those
directories if desired (e.g., see `tar --ignore-caches`). This is very
useful to avoid huge cache-directories in backups and remote
synchronizations.
The spec simply defines a file called `CACHEDIR.TAG` with the first 43
bytes to be: "Signature: 8a477f597d28d172789f06886806bc55" (which
happens to be the MD5-checksum of ".IsCacheDirectory". Further content
is to be ignored. Any such files marks the directory in question as a
cache-directory.
The cachedir-tag has been successfully deployed in tools like `cargo`
and `VLC`, and is currently discussed to be implemented in Firefox. More
information is available here: https://bford.info/cachedir/
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Add trace-hooks to the FsCache._atomic_open() helper, including a
primitive trace-infrastructure. They allow interrupting cache operation
and running arbitrary code.
The trace-hooks will be used by the test-suite to trigger the races we
want to protect against. During runtime, the traces should not be used
and thus will always be `None`.
This is a very primitive way to hook into the runtime execution and test
the atomicity of the operations. However, it is simple enough for our
tests and avoids pulling in huge tracing suites.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
On NFS, we need to be careful with cached metadata. To make sure our
_atomic_open() can correctly catch races during open+lock, we must be
careful to catch `ESTALE` and `ENOENT` from `stat()` calls. Otherwise,
the lock-acquisition guarantees that data is coherent, even on NFS.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
We used to commit cache-entries with a rename+RENAME_NOREPLACE. This,
however, is not available on NFS. Change the code to use `os.rename()`
and rely on the _documented_ kernel behavior that non-empty target
directories cannot be replaced.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
