This unwinds part of a25ae2b. The way the code ended up both
self.tree and self.mountpoint ended up pointing to the exactly
same path and so we'd end up doing two `umount -R` operations
on the same path. This ended up being a duplicate unmount.
On Fedora 39 this yields an error like:
```
mount/ostree.deployment (org.osbuild.ostree.deployment): umount: /var/osbuild/store/stage/uuid-efaac9370d25455d9e8df6d847ecb5b3/data/tree: not mounted
mount/ostree.deployment (org.osbuild.ostree.deployment): Traceback (most recent call last):
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/mounts/org.osbuild.ostree.deployment", line 136, in <module>
mount/ostree.deployment (org.osbuild.ostree.deployment): main()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/mounts/org.osbuild.ostree.deployment", line 132, in main
mount/ostree.deployment (org.osbuild.ostree.deployment): service.main()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/osbuild/host.py", line 252, in main
mount/ostree.deployment (org.osbuild.ostree.deployment): self.stop()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/osbuild/mounts.py", line 126, in stop
mount/ostree.deployment (org.osbuild.ostree.deployment): self.umount()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/mounts/org.osbuild.ostree.deployment", line 125, in umount
mount/ostree.deployment (org.osbuild.ostree.deployment): subprocess.run(["umount", "-R", self.tree],
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/usr/lib64/python3.12/subprocess.py", line 571, in run
mount/ostree.deployment (org.osbuild.ostree.deployment): raise CalledProcessError(retcode, process.args,
mount/ostree.deployment (org.osbuild.ostree.deployment): subprocess.CalledProcessError: Command '['umount', '-R', '/var/osbuild/store/stage/uuid-efaac9370d25455d9e8df6d847ecb5b3/data/tree']
' returned non-zero exit status 1.
⏱ Duration: 103s
```
I think this was necessary because of a bug in util-linux that
mean some of the accounting information got out of date when
doing a `mount --move` operation, which we use here. I think this
bug (or bugs) is now fixed [1][2] in util-linux v2.39 (in Fedora 39),
which is now causing the above pasted error on F39.
Let's just add code here that mentions the problem and workaround
it with a loop to keep unmounting (essentially what the umount -R
should have done to overmounted filesystems if the mountinfo/utab
was correct) and also mention when we should be able to drop this
workaround.
[1] a04149fbb7
[2] 8cf6c50757
Introduce a 'stage_schema' fixture, which will load the stage schema
by the stage name defined in the STAGE_NAME defined in the test module
and optionally provided schema version and return it. If no schema
version is specified, version "2" is assumed. Modify all stage unit
tests to use this fixture, instead of loading the stage schema on their
own.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Introduce a 'stage_module' fixture, which will load the stage module by
its name defined in the test module STAGE_NAME variable. Modify all
stage unit tests to use this fixture, instead of loading the stage
module on their own.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a simple unit test, which ensures that the `datasource_list` key
is dumped in the configuration file on a single line if defined in
the stage options.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The stage dumps invalid cloud-init configuration, in case the
`datasource_list` key has a value assigned. The value is supposed to be
a list, but cloud-init documentation mandates that the value is always a
single line, with no newlines. This was not true in the past.
Fix#1554
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
With pytest 8.0.0 the `-k` option seem to have changed it's
behavior. Drop `-k` therefore.
Tests started to fail recently and it looks like this is because
pytest 8.0.0 changes the semantic of the `-k` option. We used
to pass `-k stages/test` but that seems to no longer work. So
pin pytest to the last good version until this is better
understood.
This commit adds a new `exclude` option to the container-deploy
stage. This is needed when we deploy `bootc` containers that are
used for the buildroot. Here the `/sysroot` dir needs to be
excluded because it has conflicting selinux definitions for
files there and in the normal "root" dir.
See also https://github.com/osbuild/bootc-image-builder/pull/138
It appears to work with the `docker://` prefix as well as the
`registry:` prefix.
```
$ ostree container image list --repo=/ostree/repo
docker://quay.io/fedora/fedora-coreos:rawhide
$
$ ostree container image metadata --repo=/ostree/repo docker://quay.io/fedora/fedora-coreos:rawhide | md5sum
aaf7fa84896358730f0a69c330db31a6 -
$ ostree container image metadata --repo=/ostree/repo registry:quay.io/fedora/fedora-coreos:rawhide | md5sum
aaf7fa84896358730f0a69c330db31a6 -
```
Let's just keep the name unmodified. This also fixes it so that
`registry:` doesn't get prepended to refs that aren't from a registry
like:
```
$ cat /ostree/deploy/rhcos/deploy/3824ff6c279d0f4ef043e80f448eeb0d24269d50e452224a5614f915ee73fb7e.0.origin
[origin]
container-image-reference=ostree-unverified-image:oci-archive:/rhcos-416.92.202401191512-0-ostree.x86_64.ociarchive
$ ostree container image metadata --repo=/ostree/repo oci-archive:/rhcos-416.92.202401191512-0-ostree.x86_64.ociarchive | wc -c
16292
```
This is a drive-by change after spending some quality time with the
mount code. The `id` field of `Mount` is calculated only once and
only when creating a `Mount`. This seems slightly dangerous as
any change to an attribute after creation will not update the
id. This means two options:
1. dynamically update the `id` on changes
2. forbid changes after the `id` is calculcated
I went with (2) but happy to discuss of course but it seems more
the spirit of the class.
It also does the same change for "devices.Device"
The test_libc_futimes_works() is failing under RHEL/Centos right
now. To make it more robust a tiny sleep and rounding of the
timestamps is introduced to ensure that we are not run into
floating point comaparison funnines.
The second part of the fix is to open the stamp_file in read-only
mode to ensure that the mtime is not modified by the open itself
which is what lead to the actual test failure.
Older versions of RHEL/Centos do not have `mkfs.fat -g` yet so
this test will fail. Detect this and skip the test if mkfs.fat
is too old (see 7af2f1a for the original commit).
When constructing a minimal environment for osbuild the exported
dirs lacked "schemas" so the test_exports.py test failed on RHEL8.
This commit adds it (and also "assemblers" for good measure). With
that the test will pass.
The test starts failing because a new version of jsonschema (4.21.0)
changed the error messages for `minProperties: 1`.
To fix this we just use a regex and check for both possible values.
As a drive-by the commit also improves the error output in case
the match is not found.
Older version of `lsblk --json` will not have the plural `mountpoints`
but only a singular `mountpoint`. But newer version lost the
singular `mountpoint` in the json. Adjust the test accordingly.
The new `create_image_with_partitions()` helper fails under rhel8
currently. The reason is that `mkfs.ext4 -E offset=` will warn
in older versions about a partition table and require user input.
This got fixed `e2fsprogs` 1.46.3 in Jul 2021 but RHEL8 still
has 1.45.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989612
I mistakenly thought the .strip('.0') would strip off just '.0` but
in reality it strips off all instances of `.` or `0` on the end so
'4075520.0'.strip('.0') => '407552', which is clearly wrong.
This is a fixup for 5207e92.
Similar to rd.break for dracut this allows a user to specify:
- --break or --break=*
- to get a shell before each stage is run
- --break=stage.name
- to get a shell each time the stage with that name is run
- example: --break=org.osbuild.copy
- --break=stage.id
- to get a shell each time the stage with that ID is run
- get the ID for the stages for your manifest by running
osbuild on the manifest with --inspect
- example: --break=dc6e3a66fef3ebe7c815eb24d348215b9e5e2ed0cd808c15ebbe85fc73181a86
and get a bash shell where they can inspect the environment to debug
and develop OSBuild stages.
For ppc64le we need to pass in a partition (i.e. /dev/loop0p1) rather
than the root device (/dev/loop0) to the --device argument of bootupctl.
Let's add a partition field and find the device node based on the user
specified partition.
On ppc64le this would look something like:
```
- type: org.osbuild.bootupd
options:
bios:
device: disk
partition:
mpp-format-int: '{image.layout[''POWERPC-PREP-BOOT''].partnum}'
static-configs: true
deployment:
osname: fedora-coreos
ref: ostree/1/1/0
devices:
disk:
type: org.osbuild.loopback
options:
filename: disk.img
partscan: true
mounts:
- name: root
type: org.osbuild.xfs
source: disk
partition:
mpp-format-int: '{image.layout[''root''].partnum}'
target: /
- name: boot
type: org.osbuild.ext4
source: disk
partition:
mpp-format-int: '{image.layout[''boot''].partnum}'
target: /boot
```
The stable stream currently doesn't have a new enough bootupd to pass
the tests for the bootupd stage. Let's update to `:testing` for now
and we'll switch back to `:stable` later.
Add the bootupd stage to install GRUB on both BIOS and UEFI systems,
ensuring that your bootloader stays up-to-date.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
The EFI binaries are currently pulled from a hardcoded path in the
buildroot. When moving to containers as buildroots this will no
longer work as they have an alternative layout. This is an easy
"fix" - make the location of the `EFI/` directory configurable.
This allows us set `efi_src_dir` to `/usr/lib/bootupd/updates/EFI/`
and keep our existing `bootc-image-builder` workflow.
Note that this may actually not be the desired solution and instead
we want the new `bootupd`: https://github.com/osbuild/osbuild/pull/1519
This commit ensures that `/var/tmp` is available. Skopeo expects
this dir but the bwrap environment starts with a very minimal
`/var` so `/var/tmp` may not be available.
So we can use something like mpp-format-string inside of mpp-embed. An
example would be below where we want to substitute the ociarchive var
in using `mpp-format-string: 'file://{ociarchive}'`.
```
version: '2'
mpp-vars:
ociarchive: /path/to/fedora-coreos-39.20240104.dev.2-ostree.x86_64.ociarchive
pipelines:
- name: oci-archive
stages:
- type: org.osbuild.copy
inputs:
inlinefile:
type: org.osbuild.files
origin: org.osbuild.source
mpp-embed:
id: fcos.ociarchive
url:
mpp-format-string: 'file://{ociarchive}'
options:
paths:
- from:
mpp-format-string: input://inlinefile/{embedded['fcos.ociarchive']}
to: tree:///fcos.ociarchive
```
