This commit adds example manifests for a bootc.install-to-filesystem
system. It does not do more with them because running a full test
requires a working podman which is difficult to use inside our
GH runners that are already running inside docker.
This adds a `default: true` option for all cases where OSTree
information is specified in schemas and allows for the information
to be picked up from the filesystem.
This is a safe operation because when building disk images there is
no known case where having two deployments makes sense. In the case
there ever were a case then the osname, ref, and serial options still
exist and can be used.
Co-authored-by: Luke Yang <luyang@redhat.com>
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
It seems like an artifical limitation to prevent copying from a mount
or from one location in the tree to another. It just so happens we need
this functionality when building CoreOS images because we want to take
a file embedded in the OSTree at a location and copy it to another
location in the tree. The particular example here is we want to copy
/usr/share/coreos-assembler/platforms.json -> /boot/coreos/platforms.json
See https://github.com/coreos/coreos-assembler/pull/3709
Allowing to copy from/to the tree we can now do something like:
```
- type: org.osbuild.copy
options:
paths:
- from: tree:///usr/share/coreos-assembler/platforms.json
to: tree:///boot/coreos/platforms.json
mounts:
- name: ostree.deployment
type: org.osbuild.ostree.deployment
options:
deployment:
ref: ostree/1/1/0
osname:
fedora-coreos
```
In some cases it could useful to add additional kernel options without
modifying BLS config, so any subsequent call to `zipl` would ignore them.
We are going to use this to append `ignition.firstboot` to kernel cmdline
for CoreOS on s390x.
This stage was failing for me in bib, with this change I now
get more useful information from podman's stderr, e.g.:
```
RuntimeError: Failed to mount image (125): time="2024-02-06T14:23:06Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/06456126e7c06cf1b21de024e08e64eddead2b8d03779be213e63aeeea9dec94/merged: invalid argument"
Error: creating overlay mount (...snip...)
fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
```
Since the stage depends on quite a specific tree state (ostree prepped
tree with boot files), we can't really unit test it any simpler than
generating a tree with and without running the stage and diffing the
tree.
Runs bootupctl generate-update-metadata in the tree to transform
/usr/lib/ostree-boot into a bootupd-compatible update payload.
This stage should be run on the sysroot of an ostree deployment or
ostree-converted tree.
We only support `gpt` here so it would seem this option doesn't
make much sense to add, but it will make it so that the mpp-define-images
from osbuild-mpp can be passed in to `org.osbuild.sgdisk` just as it
can be passed in today to `org.osbuild.sfdisk`.
Partitions by default are indexed starting at 1, but in
some cases, such as CoreOS for IBM Z, it may be usefull
to set the 'partnum' for GPT disks explicitly, without
creating dummy partitions.
Now user can define an image:
```
mpp-define-images:
- id: image
size: 10737418240
table:
uuid: 00000000-0000-4000-a000-000000000001
label: gpt
partitions:
- name: boot
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 3
size: 786432
- name: root
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 4
size: 4194304
```
So target disk would look like:
```
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001
Device Start End Sectors Size Type
/dev/loop0p3 2048 788479 786432 384M Linux filesystem
/dev/loop0p4 788480 4982783 4194304 2G Linux filesystem
```
This patch updates the osbuild-mpp tool and the sgdisk and sfdisk
stages to support this.
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
This commit adds code that will remove the least recently used
entries when a store() operation does not succeeds because the
cache is full. To be more efficient it will try to free
twice the requested size (this can be configured in the code).
This adds an optional `paths` option to the tar stage. If specified, the
paths will be included in the tarball in order.
The OVA image needs to have its contents in a specific order in the
archive.
Add or remove the immutable bit to the specified mount directory.
The need we have for this right now is for the CoreOS builds where
the immutable bit being set on an OSTree deployment root doesn't
survive the `cp -a --reflink=auto` in the org.osbuild.copy stage when
being copied from the directory tree into the mounted XFS filesystem
we created on the disk image. Thus we have to workaround this loss
of attribute by applying the attribute directly on the mounted
filesystem from the disk.
Instead of operating directly on the tree for a stage we can operate
on a mount too. This is useful in the case where operating on the
directory tree of files isn't sufficient and the modifications need
to be made directly to the filesystems on the disk image that we are
creating.
One such example of this is we are having a problem right now where
the immutable bit being set on an OSTree deployment root doesn't
survive the `cp -a --reflink=auto` in the org.osbuild.copy stage when
being copied from the directory tree into the mounted XFS filesystem
we created on the disk image. Thus we have to workaround this loss
of attribute by applying the attribute directly on the mounted
filesystem from the disk.
In this change here we also add a check in osbuild/mounts.py to not
attempt a umount of the root of the mounts directory if that path
is no longer a mountpoint, which can happen when the umount -R
from the mounts/org.osbuild.ostree.deployment also removes the
overmount.
Here is an example of how this would be used:
```
- type: org.osbuild.chattr
options:
immutable: true
path: mount://root/
devices:
disk:
type: org.osbuild.loopback
options:
filename: disk.img
partscan: true
mounts:
- name: root
type: org.osbuild.xfs
source: disk
partition:
mpp-format-int: '{image.layout[''root''].partnum}'
target: /
- name: ostree.deployment
type: org.osbuild.ostree.deployment
options:
source: mount
deployment:
osname: fedora-coreos
ref: ostree/1/1/0
```
The initial mount on `/` is the filesystem from the root partition
on the disk. The second mount (of type org.osbuild.ostree.deployment)
then reconfigures things similar to how an OSTree system is set up.
We still target the tree here, but we open ourselves up to be able
to target something other than the tree in the future. This mostly
exchanges the `tree` variable for `target`.
We also update the comment to try to enhance clarity.
This unwinds part of a25ae2b. The way the code ended up both
self.tree and self.mountpoint ended up pointing to the exactly
same path and so we'd end up doing two `umount -R` operations
on the same path. This ended up being a duplicate unmount.
On Fedora 39 this yields an error like:
```
mount/ostree.deployment (org.osbuild.ostree.deployment): umount: /var/osbuild/store/stage/uuid-efaac9370d25455d9e8df6d847ecb5b3/data/tree: not mounted
mount/ostree.deployment (org.osbuild.ostree.deployment): Traceback (most recent call last):
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/mounts/org.osbuild.ostree.deployment", line 136, in <module>
mount/ostree.deployment (org.osbuild.ostree.deployment): main()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/mounts/org.osbuild.ostree.deployment", line 132, in main
mount/ostree.deployment (org.osbuild.ostree.deployment): service.main()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/osbuild/host.py", line 252, in main
mount/ostree.deployment (org.osbuild.ostree.deployment): self.stop()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/osbuild/mounts.py", line 126, in stop
mount/ostree.deployment (org.osbuild.ostree.deployment): self.umount()
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/var/b/shared/code/github.com/osbuild/osbuild/mounts/org.osbuild.ostree.deployment", line 125, in umount
mount/ostree.deployment (org.osbuild.ostree.deployment): subprocess.run(["umount", "-R", self.tree],
mount/ostree.deployment (org.osbuild.ostree.deployment): File "/usr/lib64/python3.12/subprocess.py", line 571, in run
mount/ostree.deployment (org.osbuild.ostree.deployment): raise CalledProcessError(retcode, process.args,
mount/ostree.deployment (org.osbuild.ostree.deployment): subprocess.CalledProcessError: Command '['umount', '-R', '/var/osbuild/store/stage/uuid-efaac9370d25455d9e8df6d847ecb5b3/data/tree']
' returned non-zero exit status 1.
⏱ Duration: 103s
```
I think this was necessary because of a bug in util-linux that
mean some of the accounting information got out of date when
doing a `mount --move` operation, which we use here. I think this
bug (or bugs) is now fixed [1][2] in util-linux v2.39 (in Fedora 39),
which is now causing the above pasted error on F39.
Let's just add code here that mentions the problem and workaround
it with a loop to keep unmounting (essentially what the umount -R
should have done to overmounted filesystems if the mountinfo/utab
was correct) and also mention when we should be able to drop this
workaround.
[1] a04149fbb7
[2] 8cf6c50757
Introduce a 'stage_schema' fixture, which will load the stage schema
by the stage name defined in the STAGE_NAME defined in the test module
and optionally provided schema version and return it. If no schema
version is specified, version "2" is assumed. Modify all stage unit
tests to use this fixture, instead of loading the stage schema on their
own.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Introduce a 'stage_module' fixture, which will load the stage module by
its name defined in the test module STAGE_NAME variable. Modify all
stage unit tests to use this fixture, instead of loading the stage
module on their own.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a simple unit test, which ensures that the `datasource_list` key
is dumped in the configuration file on a single line if defined in
the stage options.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The stage dumps invalid cloud-init configuration, in case the
`datasource_list` key has a value assigned. The value is supposed to be
a list, but cloud-init documentation mandates that the value is always a
single line, with no newlines. This was not true in the past.
Fix#1554
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
With pytest 8.0.0 the `-k` option seem to have changed it's
behavior. Drop `-k` therefore.
Tests started to fail recently and it looks like this is because
pytest 8.0.0 changes the semantic of the `-k` option. We used
to pass `-k stages/test` but that seems to no longer work. So
pin pytest to the last good version until this is better
understood.
This commit adds a new `exclude` option to the container-deploy
stage. This is needed when we deploy `bootc` containers that are
used for the buildroot. Here the `/sysroot` dir needs to be
excluded because it has conflicting selinux definitions for
files there and in the normal "root" dir.
See also https://github.com/osbuild/bootc-image-builder/pull/138
It appears to work with the `docker://` prefix as well as the
`registry:` prefix.
```
$ ostree container image list --repo=/ostree/repo
docker://quay.io/fedora/fedora-coreos:rawhide
$
$ ostree container image metadata --repo=/ostree/repo docker://quay.io/fedora/fedora-coreos:rawhide | md5sum
aaf7fa84896358730f0a69c330db31a6 -
$ ostree container image metadata --repo=/ostree/repo registry:quay.io/fedora/fedora-coreos:rawhide | md5sum
aaf7fa84896358730f0a69c330db31a6 -
```
Let's just keep the name unmodified. This also fixes it so that
`registry:` doesn't get prepended to refs that aren't from a registry
like:
```
$ cat /ostree/deploy/rhcos/deploy/3824ff6c279d0f4ef043e80f448eeb0d24269d50e452224a5614f915ee73fb7e.0.origin
[origin]
container-image-reference=ostree-unverified-image:oci-archive:/rhcos-416.92.202401191512-0-ostree.x86_64.ociarchive
$ ostree container image metadata --repo=/ostree/repo oci-archive:/rhcos-416.92.202401191512-0-ostree.x86_64.ociarchive | wc -c
16292
```
