deb-bootc-image-builder/docs/bootloader.md

# Bootloader Implementation Guide

## Overview

This document provides comprehensive guidance on implementing bootloaders in the deb-bootc-image-builder project. It covers the integration of traditional GRUB bootloaders with modern bootupd management, drawing from both debos and osbuild methodologies.

## Table of Contents

1. [Bootloader Fundamentals](#bootloader-fundamentals)
2. [Integration Approaches](#integration-approaches)
3. [debos Methodology](#debos-methodology)
4. [osbuild Integration](#osbuild-integration)
5. [Hybrid Implementation Strategy](#hybrid-implementation-strategy)
6. [Technical Implementation](#technical-implementation)
7. [Testing and Validation](#testing-and-validation)
8. [Troubleshooting](#troubleshooting)
9. [Future Considerations](#future-considerations)

## Bootloader Fundamentals

### What is a Bootloader?

A bootloader is the first software program that runs when a computer starts up. It's responsible for:
- Loading the operating system kernel
- Providing boot configuration options
- Handling different boot modes (UEFI vs BIOS)
- Managing boot parameters and kernel command lines

### Types of Bootloaders

#### 1. Traditional GRUB2
- **Purpose**: Classic bootloader for Linux systems
- **Features**: Menu-based boot selection, kernel parameter editing
- **Limitations**: Manual updates, distribution-specific configurations
- **Use Case**: Standard Linux installations, legacy systems

#### 2. Modern bootupd
- **Purpose**: Distribution-independent bootloader updater
- **Features**: Automatic updates, UEFI/BIOS compatibility, OSTree support
- **Advantages**: Cross-distribution, automated management
- **Use Case**: Container-based systems, modern Linux distributions

### Boot Process Flow

```
Firmware (UEFI/BIOS) → Bootloader → Kernel → Init System → User Space
     ↓                    ↓         ↓         ↓           ↓
  Hardware           GRUB/bootupd  Linux   systemd     Applications
  Detection          Selection     Load    Startup     Execution
```

## Integration Approaches

### 1. Pure debos Approach

**Philosophy**: "Build from scratch" with full control
**Method**: Manual bootloader installation via `run` actions
**Advantages**: Complete control, customizable, distribution-agnostic
**Disadvantages**: Complex, error-prone, requires extensive scripting

### 2. Pure osbuild Approach

**Philosophy**: "Declarative pipeline" with structured stages
**Method**: Dedicated bootloader stages (`org.osbuild.grub2`, `org.osbuild.bootupd`)
**Advantages**: Structured, reliable, well-tested
**Disadvantages**: Less flexible, Red Hat/Fedora centric

### 3. Hybrid Approach (Recommended)

**Philosophy**: "Best of both worlds" - debos flexibility with osbuild patterns
**Method**: debos `run` actions implementing osbuild-style bootloader management
**Advantages**: Flexible, reliable, modern, distribution-independent
**Disadvantages**: Requires careful implementation, testing overhead

## debos Methodology

### Core Principles

debos doesn't have built-in bootloader support, but provides the infrastructure to implement it:

```yaml
# debos recipe structure for bootloader integration
actions:
  - action: debootstrap
    # Create base system
  
  - action: apt
    # Install bootloader packages
  
  - action: run
    chroot: true
    script: install-bootloader.sh
  
  - action: run
    chroot: true
    script: configure-bootloader.sh
```

### Key Components

#### 1. Package Installation
```bash
# install-bootloader.sh
apt-get update
apt-get install -y grub-efi-amd64 efibootmgr bootupd
```

#### 2. GRUB Configuration
```bash
# configure-grub.sh
grub-install --target=x86_64-efi --efi-directory=/boot/efi --boot-directory=/boot /dev/sda
update-grub
```

#### 3. bootupd Integration
```bash
# run-bootupd.sh
bootupctl backend install --device=/dev/sda
```

### Advantages of debos Approach

- **Full Control**: Complete customization of bootloader setup
- **Distribution Independence**: Works with any Debian-based system
- **Flexibility**: Can implement complex boot scenarios
- **Debugging**: Easy to troubleshoot and modify

### Limitations

- **Complexity**: Requires extensive shell scripting
- **Error Handling**: Manual error checking and recovery
- **Maintenance**: Scripts need ongoing maintenance
- **Testing**: Each change requires full pipeline testing

## osbuild Integration

### Architecture Overview

osbuild uses a stage-based pipeline with dedicated bootloader stages:

```json
{
  "type": "org.osbuild.grub2",
  "options": {
    "root_fs_uuid": "6e4ff95f-f662-45ee-a82a-bdf44a2d0b75",
    "boot_fs_uuid": "0194fdc2-fa2f-4cc0-81d3-ff12045b73c8",
    "kernel_opts": "ro no_timer_check console=ttyS0,115200n8",
    "uefi": {
      "vendor": "fedora",
      "unified": true
    }
  }
}
```

### Key Stages

#### 1. GRUB2 Stage (`org.osbuild.grub2`)
- **Purpose**: Install and configure GRUB bootloader
- **Features**: UEFI/BIOS support, kernel parameter management
- **Configuration**: Declarative options for boot settings

#### 2. bootupd Stage (`org.osbuild.bootupd`)
- **Purpose**: Modern bootloader management and updates
- **Features**: Cross-distribution compatibility, automatic updates
- **Integration**: Works with existing GRUB installations

### Implementation Details

```python
# From osbuild bootupd stage
def main(args, options):
    deployment = options.get("deployment", None)
    static_configs = options.get("static-configs", False)
    bios = options.get("bios", {})
    device = bios.get("device", "")
    
    # Run bootupctl backend install
    cmd = ['chroot', root, '/usr/bin/bootupctl', 'backend', 'install']
    cmd.extend(bootupd_args)
    cmd.append(mounts)
    subprocess.run(cmd, check=True)
```

### Advantages of osbuild Approach

- **Structured**: Well-defined stages and options
- **Reliable**: Extensive testing and validation
- **Modern**: Built-in support for contemporary bootloaders
- **Maintainable**: Clear separation of concerns

### Limitations

- **Red Hat Centric**: Primarily designed for Fedora/RHEL
- **Less Flexible**: Limited customization options
- **Dependency**: Tied to osbuild ecosystem
- **Complexity**: Requires understanding of osbuild internals

## Hybrid Implementation Strategy

### Design Philosophy

Combine the flexibility of debos with the reliability patterns of osbuild:

1. **Use debos infrastructure** for system building and customization
2. **Implement osbuild patterns** for bootloader management
3. **Create reusable components** that can be shared across recipes
4. **Maintain distribution independence** while leveraging proven approaches

### Implementation Structure

```yaml
# Hybrid debos recipe with osbuild patterns
actions:
  - action: debootstrap
    suite: bookworm
    variant: minbase
  
  - action: apt
    packages:
      - grub-efi-amd64
      - efibootmgr
      - bootupd
  
  - action: run
    chroot: true
    script: scripts/bootloader/install-grub.sh
  
  - action: run
    chroot: true
    script: scripts/bootloader/configure-grub.sh
  
  - action: run
    chroot: true
    script: scripts/bootloader/install-bootupd.sh
  
  - action: run
    chroot: true
    script: scripts/bootloader/run-bootupd.sh
```

### Component Organization

```
scripts/
├── bootloader/
│   ├── install-grub.sh      # GRUB package installation
│   ├── configure-grub.sh    # GRUB configuration and installation
│   ├── install-bootupd.sh   # bootupd installation
│   ├── run-bootupd.sh       # bootupd backend configuration
│   └── common.sh            # Shared functions and variables
├── system/
│   ├── setup-filesystem.sh  # Filesystem preparation
│   └── configure-system.sh  # System configuration
└── validation/
    ├── test-bootloader.sh   # Bootloader validation
    └── test-boot.sh         # Boot testing
```

## Technical Implementation

### 1. GRUB Installation

#### Package Requirements
```bash
# Essential GRUB packages
grub-efi-amd64      # UEFI GRUB bootloader
efibootmgr          # UEFI boot manager
grub-common         # GRUB configuration files
```

#### Installation Process
```bash
#!/bin/bash
# install-grub.sh

set -e

echo "Installing GRUB bootloader..."

# Install packages
apt-get update
apt-get install -y grub-efi-amd64 efibootmgr grub-common

# Verify installation
grub-install --version
efibootmgr --version

echo "GRUB installation completed successfully"
```

### 2. GRUB Configuration

#### Basic Configuration
```bash
#!/bin/bash
# configure-grub.sh

set -e

echo "Configuring GRUB bootloader..."

# Create GRUB configuration directory
mkdir -p /boot/grub

# Set GRUB defaults
cat > /etc/default/grub << 'EOF'
GRUB_TIMEOUT=5
GRUB_DEFAULT=0
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=console
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
EOF

# Install GRUB to disk
grub-install --target=x86_64-efi \
    --efi-directory=/boot/efi \
    --boot-directory=/boot \
    /dev/sda

# Generate GRUB configuration
update-grub

echo "GRUB configuration completed successfully"
```

#### Advanced Configuration
```bash
# Custom kernel parameters
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 ro no_timer_check"

# UEFI-specific settings
GRUB_CMDLINE_LINUX_UEFI="efi=novamap"

# Boot menu customization
GRUB_THEME="/boot/grub/themes/debian/theme.txt"
GRUB_GFXMODE=1024x768
```

### 3. bootupd Integration

#### Installation
```bash
#!/bin/bash
# install-bootupd.sh

set -e

echo "Installing bootupd..."

# Check if bootupd is available
if command -v bootupctl >/dev/null 2>&1; then
    echo "bootupd already installed"
    exit 0
fi

# Try to install from package repositories
if apt-get install -y bootupd; then
    echo "bootupd installed from package repository"
else
    echo "Package installation failed, attempting manual installation"
    
    # Clone and build bootupd
    git clone https://github.com/coreos/bootupd.git /tmp/bootupd
    cd /tmp/bootupd
    
    # Build requirements
    apt-get install -y cargo build-essential
    
    # Build and install
    cargo build --release
    cp target/release/bootupd /usr/bin/
    cp target/release/bootupctl /usr/bin/
    
    # Cleanup
    cd /
    rm -rf /tmp/bootupd
fi

echo "bootupd installation completed"
```

#### Configuration
```bash
#!/bin/bash
# run-bootupd.sh

set -e

echo "Configuring bootupd backend..."

# Verify bootupd is available
if ! command -v bootupctl >/dev/null 2>&1; then
    echo "ERROR: bootupctl not found"
    exit 1
fi

# Get device information
DEVICE=${1:-/dev/sda}
if [ ! -b "$DEVICE" ]; then
    echo "ERROR: Device $DEVICE not found or not a block device"
    exit 1
fi

# Run bootupd backend install
echo "Installing bootupd backend on $DEVICE..."
bootupctl backend install --device="$DEVICE"

# Verify installation
bootupctl status

echo "bootupd backend configuration completed"
```

### 4. Filesystem Preparation

#### Partition Layout
```bash
#!/bin/bash
# setup-filesystem.sh

set -e

echo "Setting up filesystem for bootloader..."

# Create GPT partition table
parted /dev/sda mklabel gpt

# Create EFI system partition (ESP)
parted /dev/sda mkpart primary fat32 1MiB 257MiB
parted /dev/sda set 1 boot on
parted /dev/sda set 1 esp on

# Create root filesystem partition
parted /dev/sda mkpart primary ext4 257MiB 100%

# Format partitions
mkfs.fat -F32 /dev/sda1
mkfs.ext4 /dev/sda2

# Mount partitions
mkdir -p /mnt/boot/efi
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot/efi

echo "Filesystem setup completed"
```

#### Mount Points
```bash
# /etc/fstab configuration
cat > /mnt/etc/fstab << 'EOF'
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID=<root-uuid> /               ext4    defaults        0       1
UUID=<efi-uuid>  /boot/efi       vfat    umask=0077     0       2
proc            /proc           proc    defaults        0       0
sysfs           /sys            sysfs   defaults        0       0
tmpfs           /tmp            tmpfs   defaults        0       0
EOF
```

## Testing and Validation

### 1. Bootloader Validation

#### GRUB Verification
```bash
#!/bin/bash
# test-bootloader.sh

set -e

echo "Validating bootloader installation..."

# Check GRUB installation
if [ -f /boot/grub/grub.cfg ]; then
    echo "✅ GRUB configuration file exists"
else
    echo "❌ GRUB configuration file missing"
    exit 1
fi

# Check EFI boot entries
if efibootmgr | grep -q "debian"; then
    echo "✅ EFI boot entry found"
else
    echo "❌ EFI boot entry missing"
    exit 1
fi

# Check bootupd status
if command -v bootupctl >/dev/null 2>&1; then
    if bootupctl status | grep -q "ready"; then
        echo "✅ bootupd backend ready"
    else
        echo "⚠️  bootupd backend not ready"
    fi
fi

echo "Bootloader validation completed"
```

#### Boot Testing
```bash
#!/bin/bash
# test-boot.sh

set -e

echo "Testing boot process..."

# Create test kernel
mkdir -p /boot
cp /boot/vmlinuz-$(uname -r) /boot/vmlinuz-test
cp /boot/initrd.img-$(uname -r) /boot/initrd-test

# Test GRUB boot
if timeout 30s grub-bios-setup /dev/sda; then
    echo "✅ GRUB boot setup successful"
else
    echo "❌ GRUB boot setup failed"
    exit 1
fi

# Test UEFI boot
if [ -d /sys/firmware/efi ]; then
    if efibootmgr | grep -q "debian"; then
        echo "✅ UEFI boot entry configured"
    else
        echo "❌ UEFI boot entry missing"
        exit 1
    fi
fi

echo "Boot testing completed"
```

### 2. QEMU Testing

#### Basic Boot Test
```bash
#!/bin/bash
# qemu-test.sh

set -e

echo "Testing boot in QEMU..."

# Create QCOW2 image
qemu-img convert -f raw -O qcow2 debian-bootable.img debian-bootable.qcow2

# Test UEFI boot
echo "Testing UEFI boot..."
timeout 30s qemu-system-x86_64 \
    -m 2G \
    -drive file=debian-bootable.qcow2,format=qcow2 \
    -bios /usr/share/ovmf/OVMF.fd \
    -nographic \
    -serial mon:stdio

# Test BIOS boot
echo "Testing BIOS boot..."
timeout 30s qemu-system-x86_64 \
    -m 2G \
    -drive file=debian-bootable.qcow2,format=qcow2 \
    -nographic \
    -serial mon:stdio

echo "QEMU testing completed"
```

## Troubleshooting

### Common Issues

#### 1. GRUB Installation Failures

**Problem**: `grub-install` fails with "EFI variables are not supported"
**Cause**: Running in non-UEFI environment
**Solution**: Use `--target=i386-pc` for BIOS or ensure UEFI environment

```bash
# BIOS installation
grub-install --target=i386-pc /dev/sda

# UEFI installation
grub-install --target=x86_64-efi --efi-directory=/boot/efi --boot-directory=/boot /dev/sda
```

#### 2. bootupd Backend Failures

**Problem**: `bootupctl backend install` fails
**Cause**: Missing device or insufficient permissions
**Solution**: Verify device path and run with appropriate privileges

```bash
# Check device
lsblk
fdisk -l

# Run with proper device
bootupctl backend install --device=/dev/sda
```

#### 3. EFI Boot Entry Issues

**Problem**: System doesn't boot from EFI
**Cause**: Missing or incorrect EFI boot entry
**Solution**: Manually create EFI boot entry

```bash
# Create EFI boot entry
efibootmgr --create --disk /dev/sda --part 1 --loader /EFI/debian/grubx64.efi --label "Debian"

# Verify entry
efibootmgr
```

### Debugging Techniques

#### 1. GRUB Debug Mode
```bash
# Enable GRUB debug output
echo 'GRUB_TERMINAL_OUTPUT="console serial"' >> /etc/default/grub
echo 'GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"' >> /etc/default/grub

# Regenerate configuration
update-grub
```

#### 2. bootupd Verbose Mode
```bash
# Enable verbose bootupd output
bootupctl --verbose backend install --device=/dev/sda

# Check detailed status
bootupctl --verbose status
```

#### 3. EFI Debug Information
```bash
# Check EFI variables
efibootmgr -v

# Check EFI firmware
dmesg | grep -i efi

# Check mounted EFI partitions
mount | grep efi
```

## Future Considerations

### 1. Secure Boot Integration

#### Implementation Strategy
```bash
# Install secure boot components
apt-get install -y shim-signed grub-efi-amd64-signed

# Configure secure boot
mokutil --import /etc/ssl/certs/ca-certificates.crt
```

#### Key Considerations
- **Certificate Management**: Proper CA certificate handling
- **Key Signing**: Secure key generation and storage
- **Compatibility**: Ensuring bootloader compatibility with secure boot

### 2. OSTree Integration

#### bootupd OSTree Support
```bash
# OSTree deployment configuration
bootupctl backend install --device=/dev/sda --ostree-deployment=debian:debian/13/x86_64/standard
```

#### Benefits
- **Atomic Updates**: Rollback capability
- **Consistency**: Reproducible deployments
- **Efficiency**: Delta updates and compression

### 3. Multi-Architecture Support

#### ARM64 Support
```bash
# ARM64 GRUB installation
grub-install --target=arm64-efi --efi-directory=/boot/efi --boot-directory=/boot /dev/sda

# ARM64 bootupd
bootupctl backend install --device=/dev/sda --arch=arm64
```

#### RISC-V Support
```bash
# RISC-V bootloader (when available)
# Currently limited support, requires custom implementation
```

### 4. Cloud Integration

#### Cloud-Init Integration
```bash
# Install cloud-init
apt-get install -y cloud-init

# Configure cloud-init for bootloader
cat > /etc/cloud/cloud.cfg.d/99_bootloader.cfg << 'EOF'
bootcmd:
  - bootupctl backend install --device=/dev/sda
EOF
```

#### Benefits
- **Automation**: Automated bootloader configuration
- **Scalability**: Consistent deployment across instances
- **Maintenance**: Centralized configuration management

## Conclusion

The hybrid approach combining debos flexibility with osbuild patterns provides the best foundation for bootloader implementation in deb-bootc-image-builder. This approach offers:

1. **Distribution Independence**: Works across different Debian-based systems
2. **Modern Bootloader Support**: Integration with bootupd for contemporary needs
3. **Flexibility**: Customizable bootloader configurations
4. **Reliability**: Proven patterns from production systems
5. **Maintainability**: Clear structure and reusable components

### Key Success Factors

- **Thorough Testing**: Comprehensive validation of boot processes
- **Error Handling**: Robust error checking and recovery
- **Documentation**: Clear implementation guides and troubleshooting
- **Community Engagement**: Collaboration with debos and bootupd communities

### Next Steps

1. **Implement Core Components**: Create the basic bootloader installation scripts
2. **Integration Testing**: Validate the complete pipeline
3. **Documentation**: Expand this guide with specific examples
4. **Community Feedback**: Gather input from users and contributors
5. **Continuous Improvement**: Iterate based on real-world usage

This bootloader implementation strategy positions deb-bootc-image-builder as a modern, flexible tool for creating bootable container images while maintaining compatibility with existing Linux infrastructure.