We've temporarily disabled the merge queue because our tests often
require retries to go all-green and this isn't possible to do on the
queue, meaning that it's close to impossible to get a PR merged.
Run tests on push so that they run in main when a PR is merged.
This reverts commit 63feab7d86.
We can now add an entire device and then get the partitions added
to our environment for use, rather than to have to map each partition
in to a separate loopback device.
This is a prep patch for https://github.com/osbuild/osbuild/issues/1495
Partitions are usually indexed starting a 1 so an index starting
at 0 is confusing (i.e. you never say mount the filesystem on
partition 0). Let's add a partnum field that can be used in
definitions instead.
To workaround the issue that inside macOS containers the ownership
cannot be preserved we introduce a new environment that can be
used to forcefully relax the use of `cp -a`.
I did it via an environment instead of a commandline option mostly
because `github.com/osbuild/images/osbuild:RunOBuild()` already has
`extraEnv` option.
This commit allows to exclude preserving ownership from an object
export. This is required to fix the issue that on macOS the an
podman based workflow cannot export objects with preserving
ownerships.
Originally this was a `no_preserve: Optional[List[str]] = None)`
to be super flexible in what we pass to `cp` but then I felt like
YAGNI - if we need more we can trivially change this (internal)
API again :)
Similar to the aleph file created for builds of FCOS based on ostree
commit inputs, this adds an aleph file that contains information about
the initial deployment of data when the disk image was built
A new stage is preferred here as both the org.osbuild.ostree.deploy
and org.osbuild.ostree.deploy.container stages need an aleph file and
use of the aleph file may depend on the project/product. For example,
right now CoreOS is the only project that uses an aleph file, but others
may want it in the future.
Diffs for stage tests have changed after the sources were updated.
Update them to match expected behaviour. This was mostly done with
some form of:
```
foo=update-crypto-policies;
sudo tools/gen-stage-test-diff --libdir . --store /var/osbuild/store/ test/data/stages/$foo > test/data/stages/$foo/diff.json
```
For the dracut one I had to figure out what new kernel was used
and the new modules and update the vanilla.json file to get the
test to pass.
For the rpm one I had to also update the metadata.json with something
like:
```
sudo python3 -m osbuild --libdir . --store /var/osbuild/store/ --export tree \
--output-directory /var/osbuild/out/ test/data/stages/rpm/b.json --json \
| jq .metadata >test/data/stages/rpm/metadata.json
```
And also set stdout=subprocess.PIPE. This will allow for callers to
parse and use the output of the command, but has the side effect of
meaning less gets printed to the screen during run.
Co-authored-by: Luke Yang <luyang@redhat.com>
This adds a new key masked_generators, similar to masked_services,
which masks systemd generators from running at boot, by creating
symlinks to /dev/null in /etc/systemd/systemd-generators, as
described in:
https://www.freedesktop.org/software/systemd/man/latest/systemd.generator.html#Description
This will be useful for the automotive project, as it allows disabling
of unsupported things like sysv or rc.local legacy support, while
improving boot performance.
Adjust the source path to just have /dev in front (i.e. /dev/loop0)
and not be a path to the temporary dev directory that was prepared.
We do this because some tools (like grub2-install) consult
mountinfo to try to canonicalize paths for mounts.
Fixes https://github.com/osbuild/osbuild/issues/1492
Update the skopeo sources stage to check other container-transports [1]
for downloading container images. This commit adds a transport for
`containers-storage` in addition to the existing `docker://`
transport.
Jira: https://issues.redhat.com/browse/HMS-3235
[1] CONTAINERS-TRANSPORTS(5)
The skopeo stage is updated to accommodate two types of
destinations: 'containers-storage' and 'oci'. Now, it can
copy a container image to either a container store or
a directory as per 'Open Container Image Layout
Specification'.
Pytoml is no longer being maintained: https://github.com/avakar/pytoml
The author suggest switching to toml.
We already use the
```
try:
import toml
except ModuleNotFoundError:
import pytoml as toml
```
pattern in stages/org.osbuild.containers.storage.conf so use it in the tests too to prefer "toml" instead of pytoml.
This helper can be used to implement a strategy to find the oldest
cache entries and evict them when the cache is full.
The implementation uses the `atime` of the per object `cache.lock`
file and ensures in `load()` that it's actually updated.
This commit adds mount output to the error raised by
FileSystemMountService.mount(). This is useful when running into
mount failures during osbuild runs.
The issue was discovered while debugging a mount failure for
osbuild-composer PR#3820. Initially osbuild PR#1490 was meant
to fix it but it turned out there is a third mount helper in
the code that was originally overlooked (sorry for that!).
This is a convenient way for tests to assert that some nested dicts
(like a parsed json) has a particular key/value somewhere in it.
For example:
assert_dict_has(config, "toplevel.subitem.key", True)
While debugging a failure of osbuild-composer [0] on fc39 it was
noticed that a mount failure does not include the output of
the mount command:
```
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 78, in mount
path = client.call("mount", args)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 348, in call
ret, _ = self.call_with_fds(method, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 384, in call_with_fds
raise error
osbuild.host.RemoteError: CalledProcessError: Command '['mount', '-t', 'xfs', '-o', 'ro,norecovery', '--source', '/dev/rootvg/applv', '--target', '/tmp/tmpjtfmth56/app']' returned non-zero exit status 32.
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 268, in serve
reply, reply_fds = self._handle_message(msg, fds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 301, in _handle_message
ret, fds = self.dispatch(name, args, fds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 111, in dispatch
r = self.mount(args)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 160, in mount
subprocess.run(
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
```
which makes diagnostic errors harder of course. This commit adds
a test that ensures that mount output is visbile and also changes
the code to include it.
[0] https://github.com/osbuild/osbuild-composer/pull/3820
