bootc-docs/troubleshooting/grub2-composefs-fix.md

grub2-mkconfig Composefs Compatibility Fix

This document provides a critical fix for the incompatibility between grub2-mkconfig and composefs that affects all bootc images.

Problem Description

Issue: grub2-mkconfig is incompatible with composefs and will cause boot failures in bootc images.

Root Cause:

• grub2-mkconfig doesn't know how to read /sysroot where the physical root filesystem lives
• It tries to probe the overlay filesystem instead of the actual root filesystem
• This results in incorrect grub configuration and boot failures

Reference: Red Hat Bugzilla #2308594 - Reported by Colin Walters (ostree maintainer)

The Fix

Step 1: Backup Original File

sudo cp /usr/sbin/grub2-mkconfig /usr/sbin/grub2-mkconfig.backup

Step 2: Apply the Patch

Edit /usr/sbin/grub2-mkconfig and add this code after line 138:

# Path to the real rootfs; we may be in a container or using bootc/ostree
GRUB_PHYSICAL_FILESYSTEM=/
GRUB_FS="$(stat -f -c %T / || echo unknown)"
if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then
  GRUB_PHYSICAL_FILESYSTEM=/sysroot
fi

Step 3: Update Device Detection

Find this line (around line 145):

GRUB_DEVICE="`${grub_probe} --target=device /`"

Replace it with:

GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"

Step 4: Test the Fix

# Test grub2-mkconfig
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Verify it detects the correct filesystem
echo "Physical filesystem: ${GRUB_PHYSICAL_FILESYSTEM}"

Complete Patch

Here's the complete patch to apply:

--- /usr/sbin/grub2-mkconfig.orig	2024-08-29 20:29:13.483212450 +0000
+++ /usr/sbin/grub2-mkconfig	2024-08-29 20:28:36.605582110 +0000
@@ -138,8 +138,15 @@
     exit 1
 fi
 
+# Path to the real rootfs; we may be in a container or using bootc/ostree
+GRUB_PHYSICAL_FILESYSTEM=/
+GRUB_FS="$(stat -f -c %T / || echo unknown)"
+if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then
+  GRUB_PHYSICAL_FILESYSTEM=/sysroot
+fi
+
 # Device containing our userland.  Typically used for root= parameter.
-GRUB_DEVICE="`${grub_probe} --target=device /`"
+GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"
 GRUB_DEVICE_UUID="`${grub_probe} --device ${GRUB_DEVICE} --target=fs_uuid 2> /dev/null`" || true
 GRUB_DEVICE_PARTUUID="`${grub_probe} --device ${GRUB_DEVICE} --target=partuuid 2> /dev/null`" || true

Automated Fix Script

Create a script to apply the fix automatically:

#!/bin/bash
# grub2-composefs-fix.sh

set -euo pipefail

GRUB_MKCONFIG="/usr/sbin/grub2-mkconfig"
BACKUP_FILE="${GRUB_MKCONFIG}.backup"

echo "🔧 Applying grub2-mkconfig composefs compatibility fix..."

# Backup original file
if [ ! -f "$BACKUP_FILE" ]; then
    echo "📋 Creating backup: $BACKUP_FILE"
    sudo cp "$GRUB_MKCONFIG" "$BACKUP_FILE"
fi

# Create temporary file with fix
TEMP_FILE=$(mktemp)
sudo cp "$GRUB_MKCONFIG" "$TEMP_FILE"

# Apply the fix
sed -i '138a\
# Path to the real rootfs; we may be in a container or using bootc/ostree\
GRUB_PHYSICAL_FILESYSTEM=/\
GRUB_FS="$(stat -f -c %T / || echo unknown)"\
if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then\
  GRUB_PHYSICAL_FILESYSTEM=/sysroot\
fi\
' "$TEMP_FILE"

# Update device detection line
sed -i 's|GRUB_DEVICE="`${grub_probe} --target=device /`"|GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"|' "$TEMP_FILE"

# Replace original file
sudo mv "$TEMP_FILE" "$GRUB_MKCONFIG"
sudo chmod +x "$GRUB_MKCONFIG"

echo "✅ Fix applied successfully!"
echo "🧪 Testing grub2-mkconfig..."

# Test the fix
if sudo grub2-mkconfig -o /tmp/grub-test.cfg 2>/dev/null; then
    echo "✅ grub2-mkconfig test passed!"
    rm -f /tmp/grub-test.cfg
else
    echo "❌ grub2-mkconfig test failed!"
    echo "🔄 Restoring backup..."
    sudo mv "$BACKUP_FILE" "$GRUB_MKCONFIG"
    exit 1
fi

echo "🎉 grub2-mkconfig composefs fix completed successfully!"

Integration in Base Images

Add this fix to your base image Containerfile:

# Apply grub2-mkconfig composefs fix
RUN sed -i '138a\
# Path to the real rootfs; we may be in a container or using bootc/ostree\
GRUB_PHYSICAL_FILESYSTEM=/\
GRUB_FS="$(stat -f -c %T / || echo unknown)"\
if test "x${GRUB_FS}" = "xoverlay" && test -d /sysroot; then\
  GRUB_PHYSICAL_FILESYSTEM=/sysroot\
fi\
' /usr/sbin/grub2-mkconfig

RUN sed -i 's|GRUB_DEVICE="`${grub_probe} --target=device /`"|GRUB_DEVICE="`${grub_probe} --target=device ${GRUB_PHYSICAL_FILESYSTEM}`"|' /usr/sbin/grub2-mkconfig

Verification

After applying the fix, verify it works:

# Check if the fix is applied
grep -A 5 "GRUB_PHYSICAL_FILESYSTEM" /usr/sbin/grub2-mkconfig

# Test grub2-mkconfig
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Check for errors
echo $?

Troubleshooting

Common Issues

1. Permission denied: Run with sudo
2. File not found: Check if grub2-mkconfig exists at /usr/sbin/grub2-mkconfig
3. Syntax error: Verify the sed commands are applied correctly
4. Test fails: Check if /sysroot exists and is accessible

Rollback

If the fix causes issues, restore the backup:

sudo mv /usr/sbin/grub2-mkconfig.backup /usr/sbin/grub2-mkconfig

Impact

This fix is essential for:

• All composefs-based bootc images
• Any system using ostree with composefs
• Proper grub configuration generation
• Successful boot process

Without this fix, bootc images will fail to boot due to incorrect grub configuration.