bootc-docs/troubleshooting/grub2-composefs-fix.md

# grub2-mkconfig Composefs Compatibility Fix

This document provides a critical fix for the incompatibility between `grub2-mkconfig` and composefs that affects all bootc images.

## Problem Description

**Issue**: `grub2-mkconfig` is incompatible with composefs and will cause boot failures in bootc images.

**Root Cause**: 
- `grub2-mkconfig` doesn't know how to read `/sysroot` where the physical root filesystem lives
- It tries to probe the overlay filesystem instead of the actual root filesystem
- This results in incorrect grub configuration and boot failures

**Reference**: [Red Hat Bugzilla #2308594](https://bugzilla.redhat.com/show_bug.cgi?id=2308594) - Reported by Colin Walters (ostree maintainer)

## The Fix

### Step 1: Backup Original File

```bash
sudo cp /usr/sbin/grub2-mkconfig /usr/sbin/grub2-mkconfig.backup
```

### Step 2: Apply the Patch

Edit `/usr/sbin/grub2-mkconfig` and add this code after line 138:

```bash
# Path to the real rootfs; we may be in a container or using bootc/ostree
GRUB_PHYSICAL_FILESYSTEM=/
GRUB_FS="$(stat -f -c %T / || echo unknown)"
if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then
  GRUB_PHYSICAL_FILESYSTEM=/sysroot
fi
```

### Step 3: Update Device Detection

Find this line (around line 145):
```bash
GRUB_DEVICE="`${grub_probe} --target=device /`"
```

Replace it with:
```bash
GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"
```

### Step 4: Test the Fix

```bash
# Test grub2-mkconfig
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Verify it detects the correct filesystem
echo "Physical filesystem: ${GRUB_PHYSICAL_FILESYSTEM}"
```

## Complete Patch

Here's the complete patch to apply:

```diff
--- /usr/sbin/grub2-mkconfig.orig	2024-08-29 20:29:13.483212450 +0000
+++ /usr/sbin/grub2-mkconfig	2024-08-29 20:28:36.605582110 +0000
@@ -138,8 +138,15 @@
     exit 1
 fi
 
+# Path to the real rootfs; we may be in a container or using bootc/ostree
+GRUB_PHYSICAL_FILESYSTEM=/
+GRUB_FS="$(stat -f -c %T / || echo unknown)"
+if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then
+  GRUB_PHYSICAL_FILESYSTEM=/sysroot
+fi
+
 # Device containing our userland.  Typically used for root= parameter.
-GRUB_DEVICE="`${grub_probe} --target=device /`"
+GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"
 GRUB_DEVICE_UUID="`${grub_probe} --device ${GRUB_DEVICE} --target=fs_uuid 2> /dev/null`" || true
 GRUB_DEVICE_PARTUUID="`${grub_probe} --device ${GRUB_DEVICE} --target=partuuid 2> /dev/null`" || true
```

## Automated Fix Script

Create a script to apply the fix automatically:

```bash
#!/bin/bash
# grub2-composefs-fix.sh

set -euo pipefail

GRUB_MKCONFIG="/usr/sbin/grub2-mkconfig"
BACKUP_FILE="${GRUB_MKCONFIG}.backup"

echo "🔧 Applying grub2-mkconfig composefs compatibility fix..."

# Backup original file
if [ ! -f "$BACKUP_FILE" ]; then
    echo "📋 Creating backup: $BACKUP_FILE"
    sudo cp "$GRUB_MKCONFIG" "$BACKUP_FILE"
fi

# Create temporary file with fix
TEMP_FILE=$(mktemp)
sudo cp "$GRUB_MKCONFIG" "$TEMP_FILE"

# Apply the fix
sed -i '138a\
# Path to the real rootfs; we may be in a container or using bootc/ostree\
GRUB_PHYSICAL_FILESYSTEM=/\
GRUB_FS="$(stat -f -c %T / || echo unknown)"\
if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then\
  GRUB_PHYSICAL_FILESYSTEM=/sysroot\
fi\
' "$TEMP_FILE"

# Update device detection line
sed -i 's|GRUB_DEVICE="`${grub_probe} --target=device /`"|GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"|' "$TEMP_FILE"

# Replace original file
sudo mv "$TEMP_FILE" "$GRUB_MKCONFIG"
sudo chmod +x "$GRUB_MKCONFIG"

echo "✅ Fix applied successfully!"
echo "🧪 Testing grub2-mkconfig..."

# Test the fix
if sudo grub2-mkconfig -o /tmp/grub-test.cfg 2>/dev/null; then
    echo "✅ grub2-mkconfig test passed!"
    rm -f /tmp/grub-test.cfg
else
    echo "❌ grub2-mkconfig test failed!"
    echo "🔄 Restoring backup..."
    sudo mv "$BACKUP_FILE" "$GRUB_MKCONFIG"
    exit 1
fi

echo "🎉 grub2-mkconfig composefs fix completed successfully!"
```

## Integration in Base Images

Add this fix to your base image Containerfile:

```dockerfile
# Apply grub2-mkconfig composefs fix
RUN sed -i '138a\
# Path to the real rootfs; we may be in a container or using bootc/ostree\
GRUB_PHYSICAL_FILESYSTEM=/\
GRUB_FS="$(stat -f -c %T / || echo unknown)"\
if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then\
  GRUB_PHYSICAL_FILESYSTEM=/sysroot\
fi\
' /usr/sbin/grub2-mkconfig

RUN sed -i 's|GRUB_DEVICE="`${grub_probe} --target=device /`"|GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"|' /usr/sbin/grub2-mkconfig
```

## Verification

After applying the fix, verify it works:

```bash
# Check if the fix is applied
grep -A 5 "GRUB_PHYSICAL_FILESYSTEM" /usr/sbin/grub2-mkconfig

# Test grub2-mkconfig
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Check for errors
echo $?
```

## Troubleshooting

### Common Issues

1. **Permission denied**: Run with `sudo`
2. **File not found**: Check if `grub2-mkconfig` exists at `/usr/sbin/grub2-mkconfig`
3. **Syntax error**: Verify the sed commands are applied correctly
4. **Test fails**: Check if `/sysroot` exists and is accessible

### Rollback

If the fix causes issues, restore the backup:

```bash
sudo mv /usr/sbin/grub2-mkconfig.backup /usr/sbin/grub2-mkconfig
```

## Impact

This fix is **essential** for:
- All composefs-based bootc images
- Any system using ostree with composefs
- Proper grub configuration generation
- Successful boot process

Without this fix, bootc images will fail to boot due to incorrect grub configuration.