Add critical grub2-mkconfig composefs compatibility fix

- Add comprehensive fix for grub2-mkconfig composefs incompatibility
- Document the critical issue from Red Hat Bugzilla #2308594
- Provide complete patch, automated script, and Containerfile integration
- Add troubleshooting documentation with verification steps
- Update COMPATIBILITY.md with critical issue warning
- Update README.md to highlight this critical fix

This fix is ESSENTIAL for all composefs-based bootc images to boot properly.
Without this fix, grub2-mkconfig will generate incorrect configurations
and cause boot failures.

Based on:
- Red Hat Bugzilla #2308594 (reported by Colin Walters)
- ostree issue #3198
- Affects all bootc/ostree systems using composefs

COMPATIBILITY.md

@@ -120,6 +120,37 @@ sudo apt install -y ostree podman systemd
 
 ## Known Issues
 
+### Critical: grub2-mkconfig Composefs Incompatibility
+
+**⚠️ CRITICAL ISSUE**: `grub2-mkconfig` is incompatible with composefs and will cause boot failures.
+
+**Problem**: 
+- `grub2-mkconfig` doesn't know how to read `/sysroot` where the physical root filesystem lives
+- This affects all bootc/ostree systems using composefs
+- Reported in [Red Hat Bugzilla #2308594](https://bugzilla.redhat.com/show_bug.cgi?id=2308594)
+
+**Solution**: Apply this patch to `/usr/sbin/grub2-mkconfig`:
+
+```bash
+# Add this code after line 138 in grub2-mkconfig
+# Path to the real rootfs; we may be in a container or using bootc/ostree
+GRUB_PHYSICAL_FILESYSTEM=/
+GRUB_FS="$(stat -f -c %T / || echo unknown)"
+if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then
+  GRUB_PHYSICAL_FILESYSTEM=/sysroot
+fi
+
+# Then change this line:
+# GRUB_DEVICE="`${grub_probe} --target=device /`"
+# To:
+GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"
+```
+
+**Workaround**: 
+- Apply the patch before running `grub2-mkconfig`
+- Or use alternative bootloader configuration methods
+- This issue affects all composefs-based bootc images
+
 ### Debian-Specific Problems
 1. **bootc install reliability**: May fail on Debian due to Fedora-centric development
 2. **Missing dependencies**: Some bootc dependencies may not be available

README.md

@@ -126,6 +126,9 @@ This repository contains comprehensive technical documentation for the `bootc` p
   - `base-images-wo-bootc.md` - Creating bootc images without bootc binary (Debian-specific)
   - `initramfs-integration.md` - Essential initramfs files for composefs support
 
+#### Troubleshooting
+- `troubleshooting/grub2-composefs-fix.md` - **CRITICAL**: Fix for grub2-mkconfig composefs incompatibility
+
 #### Installation
 - **`installation.md`** - Installation instructions (corrected for source compilation)
 
@@ -253,6 +256,7 @@ podman build -f examples/nginx/Containerfile -t debian-bootc-nginx:latest .
 ### Key Files for Debian Users
 - `building/base-images-wo-bootc.md` - Complete guide for creating bootc images without bootc binary
 - `building/base-images.md` - Standard methods for creating base images
+- `troubleshooting/grub2-composefs-fix.md` - **CRITICAL**: Fix for grub2-mkconfig composefs incompatibility
 - All command documentation includes Debian-specific adaptations
 
 ## Contributing

troubleshooting/grub2-composefs-fix.md

@@ -0,0 +1,201 @@
+# grub2-mkconfig Composefs Compatibility Fix
+
+This document provides a critical fix for the incompatibility between `grub2-mkconfig` and composefs that affects all bootc images.
+
+## Problem Description
+
+**Issue**: `grub2-mkconfig` is incompatible with composefs and will cause boot failures in bootc images.
+
+**Root Cause**: 
+- `grub2-mkconfig` doesn't know how to read `/sysroot` where the physical root filesystem lives
+- It tries to probe the overlay filesystem instead of the actual root filesystem
+- This results in incorrect grub configuration and boot failures
+
+**Reference**: [Red Hat Bugzilla #2308594](https://bugzilla.redhat.com/show_bug.cgi?id=2308594) - Reported by Colin Walters (ostree maintainer)
+
+## The Fix
+
+### Step 1: Backup Original File
+
+```bash
+sudo cp /usr/sbin/grub2-mkconfig /usr/sbin/grub2-mkconfig.backup
+```
+
+### Step 2: Apply the Patch
+
+Edit `/usr/sbin/grub2-mkconfig` and add this code after line 138:
+
+```bash
+# Path to the real rootfs; we may be in a container or using bootc/ostree
+GRUB_PHYSICAL_FILESYSTEM=/
+GRUB_FS="$(stat -f -c %T / || echo unknown)"
+if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then
+  GRUB_PHYSICAL_FILESYSTEM=/sysroot
+fi
+```
+
+### Step 3: Update Device Detection
+
+Find this line (around line 145):
+```bash
+GRUB_DEVICE="`${grub_probe} --target=device /`"
+```
+
+Replace it with:
+```bash
+GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"
+```
+
+### Step 4: Test the Fix
+
+```bash
+# Test grub2-mkconfig
+sudo grub2-mkconfig -o /boot/grub2/grub.cfg
+
+# Verify it detects the correct filesystem
+echo "Physical filesystem: ${GRUB_PHYSICAL_FILESYSTEM}"
+```
+
+## Complete Patch
+
+Here's the complete patch to apply:
+
+```diff
+--- /usr/sbin/grub2-mkconfig.orig	2024-08-29 20:29:13.483212450 +0000
++++ /usr/sbin/grub2-mkconfig	2024-08-29 20:28:36.605582110 +0000
+@@ -138,8 +138,15 @@
+     exit 1
+ fi
+ 
++# Path to the real rootfs; we may be in a container or using bootc/ostree
++GRUB_PHYSICAL_FILESYSTEM=/
++GRUB_FS="$(stat -f -c %T / || echo unknown)"
++if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then
++  GRUB_PHYSICAL_FILESYSTEM=/sysroot
++fi
++
+ # Device containing our userland.  Typically used for root= parameter.
+-GRUB_DEVICE="`${grub_probe} --target=device /`"
++GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"
+ GRUB_DEVICE_UUID="`${grub_probe} --device ${GRUB_DEVICE} --target=fs_uuid 2> /dev/null`" || true
+ GRUB_DEVICE_PARTUUID="`${grub_probe} --device ${GRUB_DEVICE} --target=partuuid 2> /dev/null`" || true
+```
+
+## Automated Fix Script
+
+Create a script to apply the fix automatically:
+
+```bash
+#!/bin/bash
+# grub2-composefs-fix.sh
+
+set -euo pipefail
+
+GRUB_MKCONFIG="/usr/sbin/grub2-mkconfig"
+BACKUP_FILE="${GRUB_MKCONFIG}.backup"
+
+echo "🔧 Applying grub2-mkconfig composefs compatibility fix..."
+
+# Backup original file
+if [ ! -f "$BACKUP_FILE" ]; then
+    echo "📋 Creating backup: $BACKUP_FILE"
+    sudo cp "$GRUB_MKCONFIG" "$BACKUP_FILE"
+fi
+
+# Create temporary file with fix
+TEMP_FILE=$(mktemp)
+sudo cp "$GRUB_MKCONFIG" "$TEMP_FILE"
+
+# Apply the fix
+sed -i '138a\
+# Path to the real rootfs; we may be in a container or using bootc/ostree\
+GRUB_PHYSICAL_FILESYSTEM=/\
+GRUB_FS="$(stat -f -c %T / || echo unknown)"\
+if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then\
+  GRUB_PHYSICAL_FILESYSTEM=/sysroot\
+fi\
+' "$TEMP_FILE"
+
+# Update device detection line
+sed -i 's|GRUB_DEVICE="`${grub_probe} --target=device /`"|GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"|' "$TEMP_FILE"
+
+# Replace original file
+sudo mv "$TEMP_FILE" "$GRUB_MKCONFIG"
+sudo chmod +x "$GRUB_MKCONFIG"
+
+echo "✅ Fix applied successfully!"
+echo "🧪 Testing grub2-mkconfig..."
+
+# Test the fix
+if sudo grub2-mkconfig -o /tmp/grub-test.cfg 2>/dev/null; then
+    echo "✅ grub2-mkconfig test passed!"
+    rm -f /tmp/grub-test.cfg
+else
+    echo "❌ grub2-mkconfig test failed!"
+    echo "🔄 Restoring backup..."
+    sudo mv "$BACKUP_FILE" "$GRUB_MKCONFIG"
+    exit 1
+fi
+
+echo "🎉 grub2-mkconfig composefs fix completed successfully!"
+```
+
+## Integration in Base Images
+
+Add this fix to your base image Containerfile:
+
+```dockerfile
+# Apply grub2-mkconfig composefs fix
+RUN sed -i '138a\
+# Path to the real rootfs; we may be in a container or using bootc/ostree\
+GRUB_PHYSICAL_FILESYSTEM=/\
+GRUB_FS="$(stat -f -c %T / || echo unknown)"\
+if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then\
+  GRUB_PHYSICAL_FILESYSTEM=/sysroot\
+fi\
+' /usr/sbin/grub2-mkconfig
+
+RUN sed -i 's|GRUB_DEVICE="`${grub_probe} --target=device /`"|GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"|' /usr/sbin/grub2-mkconfig
+```
+
+## Verification
+
+After applying the fix, verify it works:
+
+```bash
+# Check if the fix is applied
+grep -A 5 "GRUB_PHYSICAL_FILESYSTEM" /usr/sbin/grub2-mkconfig
+
+# Test grub2-mkconfig
+sudo grub2-mkconfig -o /boot/grub2/grub.cfg
+
+# Check for errors
+echo $?
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Permission denied**: Run with `sudo`
+2. **File not found**: Check if `grub2-mkconfig` exists at `/usr/sbin/grub2-mkconfig`
+3. **Syntax error**: Verify the sed commands are applied correctly
+4. **Test fails**: Check if `/sysroot` exists and is accessible
+
+### Rollback
+
+If the fix causes issues, restore the backup:
+
+```bash
+sudo mv /usr/sbin/grub2-mkconfig.backup /usr/sbin/grub2-mkconfig
+```
+
+## Impact
+
+This fix is **essential** for:
+- All composefs-based bootc images
+- Any system using ostree with composefs
+- Proper grub configuration generation
+- Successful boot process
+
+Without this fix, bootc images will fail to boot due to incorrect grub configuration.