# Create bootc base images without bootc

⚠️ **EXPERIMENTAL WARNING** ⚠️

**bootc on Debian is HIGHLY EXPERIMENTAL and NOT production-ready!**

- **No reproducible bootc images exist yet** - Everything is experimental
- **Use only for testing** on virtual machines or test hardware
- **Expect failures** - bootc is not stable on Debian
- **DO NOT use in production** - Data loss and system instability possible

The bootc binary is unreliable under Debian as it's mainly developed for Fedora despite it aiming to be a distro agnostic tool.

In this doc we'll go over all the little things a base bootc needs to be considered a true bootc image.

## Why Avoid bootc Binary on Debian

### Common Issues
- **Missing Dependencies**: bootc may have Fedora-specific dependencies not available in Debian
- **Compilation Issues**: Rust dependencies may not compile cleanly on Debian
- **Runtime Errors**: Even if compiled, bootc may fail at runtime on Debian systems
- **Feature Gaps**: Some bootc features may not work properly on Debian

### Alternative Validation
Instead of relying on `bootc container lint`, we'll manually validate our images against the bootc specification.

## Essential Requirements for bootc Images

### 1. Required Labels

Every bootc image must have these labels:

```dockerfile
LABEL containers.bootc 1
LABEL ostree.bootable 1
```

### 2. Systemd as Init System

bootc requires systemd as the init system:

```dockerfile
# Install systemd
RUN apt update && apt install -y systemd

# Set systemd as init
RUN ln -sf /lib/systemd/systemd /sbin/init

# Set default target
RUN systemctl set-default multi-user.target
```

### 3. Essential Systemd Services

Enable these core systemd services:

```dockerfile
# Enable essential services
RUN systemctl enable systemd-resolved.service
RUN systemctl enable systemd-networkd.service
RUN systemctl enable systemd-timesyncd.service
RUN systemctl enable systemd-journald.service
```

### 4. Proper Directory Structure

Create essential systemd directories:

```dockerfile
# Create systemd directories
RUN mkdir -p /usr/lib/systemd/system \
    /usr/lib/systemd/user \
    /etc/systemd/system \
    /etc/systemd/user \
    /var/lib/systemd \
    /run/systemd \
    /etc/systemd/network
```

### 5. Kernel and Initramfs

Install kernel and initramfs tools:

```dockerfile
# Install kernel and initramfs
RUN apt install -y \
    linux-image-amd64 \
    initramfs-tools \
    linux-base
```

### 6. Bootloader Support

Install bootloader tools:

```dockerfile
# Install GRUB2
RUN apt install -y \
    grub2 \
    grub2-common \
    grub-pc-bin \
    grub-efi-amd64-bin \
    efibootmgr
```

### 7. OSTree Support

Install OSTree for transactional updates:

```dockerfile
# Install OSTree
RUN apt install -y \
    ostree \
    libostree-1-1
```

## Manual Validation Checklist

### Filesystem Structure Validation

```bash
#!/bin/bash
# validate-filesystem.sh - Manual filesystem validation

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"

echo "🔍 Validating filesystem structure for ${IMAGE_NAME}"

# Use podman image mount for robust image inspection
echo "📦 Mounting image for inspection..."
MOUNTPOINT=$(podman image mount "${IMAGE_NAME}")
if [ -z "${MOUNTPOINT}" ]; then
    echo "❌ Failed to mount image. Trying alternative method..."
    # Fallback: extract using podman save (less robust)
    TEMP_DIR=$(mktemp -d)
    podman save "${IMAGE_NAME}" | tar -xf - -C "${TEMP_DIR}"
    LAYER_DIR=$(find "${TEMP_DIR}" -name "layer.tar" | head -1 | xargs dirname)
    tar -xf "${LAYER_DIR}/layer.tar" -C "${TEMP_DIR}/extracted"
    ROOTFS="${TEMP_DIR}/extracted"
    CLEANUP_TEMP=1
else
    ROOTFS="${MOUNTPOINT}"
    CLEANUP_TEMP=0
fi

echo "✅ Checking systemd as init..."
if [ -L "${ROOTFS}/sbin/init" ] && [ "$(readlink "${ROOTFS}/sbin/init")" = "/lib/systemd/systemd" ]; then
    echo "✅ systemd is properly set as init"
else
    echo "❌ systemd is not set as init"
    exit 1
fi

echo "✅ Checking systemd directories..."
for dir in /usr/lib/systemd/system /usr/lib/systemd/user /etc/systemd/system /etc/systemd/user; do
    if [ -d "${ROOTFS}${dir}" ]; then
        echo "✅ ${dir} exists"
    else
        echo "❌ ${dir} missing"
        exit 1
    fi
done

echo "✅ Checking essential binaries..."
for binary in /lib/systemd/systemd /usr/bin/systemctl /sbin/init; do
    if [ -f "${ROOTFS}${binary}" ]; then
        echo "✅ ${binary} exists"
    else
        echo "❌ ${binary} missing"
        exit 1
    fi
done

echo "✅ Checking kernel..."
if [ -d "${ROOTFS}/boot" ] && [ -f "${ROOTFS}/boot/vmlinuz-"* ]; then
    echo "✅ Kernel found"
else
    echo "❌ Kernel missing"
    exit 1
fi

echo "✅ Checking initramfs..."
if [ -f "${ROOTFS}/boot/initrd.img-"* ]; then
    echo "✅ Initramfs found"
else
    echo "❌ Initramfs missing"
    exit 1
fi

echo "✅ Checking GRUB..."
if [ -f "${ROOTFS}/usr/bin/grub-install" ] && [ -f "${ROOTFS}/usr/bin/grub-mkconfig" ]; then
    echo "✅ GRUB2 found"
else
    echo "❌ GRUB2 missing"
    exit 1
fi

echo "✅ Checking OSTree..."
if [ -f "${ROOTFS}/usr/bin/ostree" ]; then
    echo "✅ OSTree found"
else
    echo "❌ OSTree missing"
    exit 1
fi

# Clean up
if [ "${CLEANUP_TEMP}" = "1" ]; then
    rm -rf "${TEMP_DIR}"
else
    podman image umount "${IMAGE_NAME}"
fi

echo "🎉 All filesystem validations passed!"
```

### Label Validation

```bash
#!/bin/bash
# validate-labels.sh - Validate OCI labels

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"

echo "🔍 Validating OCI labels for ${IMAGE_NAME}"

# Check for required labels
if podman inspect "${IMAGE_NAME}" | grep -q '"containers.bootc": "1"'; then
    echo "✅ containers.bootc label found"
else
    echo "❌ containers.bootc label missing"
    exit 1
fi

if podman inspect "${IMAGE_NAME}" | grep -q '"ostree.bootable": "1"'; then
    echo "✅ ostree.bootable label found"
else
    echo "❌ ostree.bootable label missing"
    exit 1
fi

echo "🎉 All label validations passed!"
```

### Systemd Service Validation

```bash
#!/bin/bash
# validate-systemd.sh - Validate systemd configuration

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"
TEMP_DIR=$(mktemp -d)

echo "🔍 Validating systemd configuration for ${IMAGE_NAME}"

# Extract and run systemd validation
podman run --rm -v "${TEMP_DIR}:/tmp" "${IMAGE_NAME}" /bin/bash -c "
    echo 'Checking systemd version...'
    systemctl --version
    
    echo 'Checking enabled services...'
    systemctl list-unit-files --state=enabled | grep -E '(systemd-resolved|systemd-networkd|systemd-timesyncd)'
    
    echo 'Checking systemd directories...'
    ls -la /usr/lib/systemd/system/ | head -5
    ls -la /etc/systemd/system/ | head -5
    
    echo 'Checking network configuration...'
    ls -la /etc/systemd/network/
    
    echo 'Checking journal configuration...'
    ls -la /var/log/journal/ || echo 'No journal directory (normal for base image)'
"

rm -rf "${TEMP_DIR}"

echo "🎉 Systemd validation completed!"
```

## Complete Build Script Without bootc

```bash
#!/bin/bash
# build-bootc-base-wo-bootc.sh - Build bootc base without bootc binary

set -euo pipefail

IMAGE_NAME="debian-bootc-base"
TAG="${1:-latest}"

echo "🏗️  Building Debian bootc Base Image (without bootc binary)"
echo "Image: ${IMAGE_NAME}:${TAG}"

# Build the image
echo "📦 Building container image..."
podman build -f Containerfile.wo-bootc -t "${IMAGE_NAME}:${TAG}" .

echo "✅ Base image built successfully!"

# Run validations
echo "🔍 Running validations..."
./validate-labels.sh "${IMAGE_NAME}:${TAG}"
./validate-filesystem.sh "${IMAGE_NAME}:${TAG}"
./validate-systemd.sh "${IMAGE_NAME}:${TAG}"

echo "📊 Image Information:"
podman images "${IMAGE_NAME}:${TAG}"

echo ""
echo "🚀 Ready to build application layers!"
echo "Example: podman build -f examples/nginx/Containerfile -t ${IMAGE_NAME}:nginx"
```

## Containerfile Without bootc Dependency

```dockerfile
# Containerfile.wo-bootc - Build bootc base without bootc binary

FROM debian:sid-slim

# Install essential packages for bootc
RUN apt update && apt install -y \
    systemd \
    systemd-sysusers \
    systemd-tmpfiles \
    systemd-resolved \
    systemd-networkd \
    systemd-timesyncd \
    kernel \
    initramfs-tools \
    linux-base \
    grub2 \
    grub2-common \
    grub-pc-bin \
    grub-efi-amd64-bin \
    efibootmgr \
    ostree \
    libostree-1-1 \
    && apt clean

# Create essential directories
RUN mkdir -p /usr/lib/systemd/system \
    /usr/lib/systemd/user \
    /etc/systemd/system \
    /etc/systemd/user \
    /var/lib/systemd \
    /run/systemd \
    /etc/systemd/network

# Configure systemd as init
RUN ln -sf /lib/systemd/systemd /sbin/init

# Set up basic systemd configuration (using symlinks instead of systemctl)
RUN ln -sf /lib/systemd/system/multi-user.target /etc/systemd/system/default.target

# Create essential systemd services (using symlinks instead of systemctl)
RUN mkdir -p /etc/systemd/system/multi-user.target.wants \
    && ln -sf /usr/lib/systemd/system/systemd-resolved.service /etc/systemd/system/multi-user.target.wants/systemd-resolved.service \
    && ln -sf /usr/lib/systemd/system/systemd-networkd.service /etc/systemd/system/multi-user.target.wants/systemd-networkd.service \
    && ln -sf /usr/lib/systemd/system/systemd-timesyncd.service /etc/systemd/system/multi-user.target.wants/systemd-timesyncd.service

# Configure basic networking
RUN echo -e "[Match]\nName=*\n\n[Network]\nDHCP=yes" > /etc/systemd/network/80-dhcp.network

# Set up basic users and groups
RUN systemd-sysusers --create

# Configure tmpfiles
RUN systemd-tmpfiles --create

# Create essential systemd unit files
RUN echo -e "[Unit]\nDescription=Bootc Base Image\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=/bin/true\n\n[Install]\nWantedBy=multi-user.target" > /etc/systemd/system/bootc-base.service

# Enable the bootc base service (using symlink instead of systemctl)
RUN ln -sf /etc/systemd/system/bootc-base.service /etc/systemd/system/multi-user.target.wants/bootc-base.service

# Required bootc labels
LABEL containers.bootc 1
LABEL ostree.bootable 1

# Set default command
CMD ["/lib/systemd/systemd"]
```

## Advanced Validation Scripts

### Network Configuration Validation

```bash
#!/bin/bash
# validate-network.sh - Validate network configuration

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"

echo "🔍 Validating network configuration for ${IMAGE_NAME}"

podman run --rm "${IMAGE_NAME}" /bin/bash -c "
    echo 'Checking systemd-networkd configuration...'
    if [ -f /etc/systemd/network/80-dhcp.network ]; then
        echo '✅ DHCP network configuration found'
        cat /etc/systemd/network/80-dhcp.network
    else
        echo '❌ DHCP network configuration missing'
        exit 1
    fi
    
    echo 'Checking systemd-resolved configuration...'
    if [ -f /etc/systemd/resolved.conf ]; then
        echo '✅ systemd-resolved configuration found'
    else
        echo '❌ systemd-resolved configuration missing'
        exit 1
    fi
"

echo "🎉 Network validation completed!"
```

### Bootloader Validation

```bash
#!/bin/bash
# validate-bootloader.sh - Validate bootloader configuration

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"

echo "🔍 Validating bootloader configuration for ${IMAGE_NAME}"

podman run --rm "${IMAGE_NAME}" /bin/bash -c "
    echo 'Checking GRUB2 installation...'
    if command -v grub-install >/dev/null 2>&1; then
        echo '✅ grub-install found'
        grub-install --version
    else
        echo '❌ grub-install missing'
        exit 1
    fi
    
    if command -v grub-mkconfig >/dev/null 2>&1; then
        echo '✅ grub-mkconfig found'
        grub-mkconfig --version
    else
        echo '❌ grub-mkconfig missing'
        exit 1
    fi
    
    echo 'Checking EFI support...'
    if command -v efibootmgr >/dev/null 2>&1; then
        echo '✅ efibootmgr found'
        efibootmgr --version
    else
        echo '❌ efibootmgr missing'
        exit 1
    fi
"

echo "🎉 Bootloader validation completed!"
```

## Complete Validation Suite

```bash
#!/bin/bash
# validate-all.sh - Complete validation suite

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"

echo "🔍 Running complete validation suite for ${IMAGE_NAME}"
echo "=================================================="

# Run all validations
./validate-labels.sh "${IMAGE_NAME}"
echo ""

./validate-filesystem.sh "${IMAGE_NAME}"
echo ""

./validate-systemd.sh "${IMAGE_NAME}"
echo ""

./validate-network.sh "${IMAGE_NAME}"
echo ""

./validate-bootloader.sh "${IMAGE_NAME}"
echo ""

echo "🎉 All validations passed! Image is bootc-compatible."
echo "Image: ${IMAGE_NAME}"
echo "Ready for deployment with bootc!"
```

## Troubleshooting Common Issues

### Missing systemd Services
```bash
# Check if services are properly enabled
podman run --rm debian-bootc-base:latest systemctl list-unit-files --state=enabled
```

### Kernel Issues
```bash
# Check kernel installation
podman run --rm debian-bootc-base:latest ls -la /boot/
```

### GRUB Issues
```bash
# Check GRUB installation
podman run --rm debian-bootc-base:latest which grub-install
```

### OSTree Issues
```bash
# Check OSTree installation
podman run --rm debian-bootc-base:latest ostree --version
```

## Installing bootc for Management Operations

While bootc may be unreliable for installation on Debian, you can still install it for management operations like upgrades, rollbacks, and status checking.

### Installing bootc from Source

```bash
#!/bin/bash
# install-bootc-for-management.sh - Install bootc for management only

set -euo pipefail

echo "🏗️  Installing bootc for management operations on Debian"

# Install build dependencies
echo "📦 Installing build dependencies..."
sudo apt update && sudo apt install -y \
    build-essential \
    git \
    pkg-config \
    libostree-dev \
    libglib2.0-dev \
    libgpgme-dev \
    libseccomp-dev \
    cargo \
    rustc

# Install runtime dependencies
echo "📦 Installing runtime dependencies..."
sudo apt install -y \
    ostree \
    podman \
    systemd

# Clone and build bootc
echo "🔨 Building bootc from source..."
if [ ! -d "bootc" ]; then
    git clone https://github.com/containers/bootc.git
fi

cd bootc
git fetch origin
git checkout origin/main

# Build bootc
echo "🔨 Compiling bootc..."
make

# Install bootc
echo "📦 Installing bootc..."
sudo make install

# Verify installation
echo "✅ Verifying installation..."
bootc --version

echo "🎉 bootc installed for management operations!"
echo "Note: Use manual installation for deploying images"
```

### Installing bootc with Composefs Support

```bash
#!/bin/bash
# install-bootc-composefs.sh - Install bootc with composefs support

set -euo pipefail

echo "🏗️  Installing bootc with composefs support for management"

# Install additional dependencies for composefs
echo "📦 Installing composefs dependencies..."
sudo apt update && sudo apt install -y \
    build-essential \
    git \
    pkg-config \
    libostree-dev \
    libglib2.0-dev \
    libgpgme-dev \
    libseccomp-dev \
    cargo \
    rustc \
    libfuse3-dev \
    libfuse3-3

# Clone and build bootc with composefs
echo "🔨 Building bootc with composefs support..."
if [ ! -d "bootc" ]; then
    git clone https://github.com/containers/bootc.git
fi

cd bootc
git fetch origin
git checkout origin/composefs-backend

# Build with composefs feature
echo "🔨 Compiling bootc with composefs..."
cargo build --release --features composefs-backend

# Install bootc
echo "📦 Installing bootc..."
sudo cp target/release/bootc /usr/local/bin/
sudo chmod +x /usr/local/bin/bootc

# Verify installation
echo "✅ Verifying installation..."
bootc --version

echo "🎉 bootc with composefs support installed!"
```

### Management Operations with bootc

Once bootc is installed, you can use it for management operations:

```bash
#!/bin/bash
# bootc-management-examples.sh - Examples of bootc management operations

set -euo pipefail

echo "🔧 bootc Management Operations Examples"

# Check system status
echo "📊 Checking system status..."
bootc status

# Check for upgrades
echo "🔄 Checking for upgrades..."
bootc upgrade --check

# Apply upgrades (if available)
echo "⬆️  Applying upgrades..."
bootc upgrade --apply

# Switch to different image
echo "🔄 Switching image..."
# bootc switch quay.io/myorg/myapp:latest --apply

# Rollback if needed
echo "⏪ Rolling back..."
# bootc rollback --apply

# Check usr-overlay status
echo "📁 Checking usr-overlay..."
bootc usr-overlay

echo "✅ Management operations completed!"
```

### Hybrid Approach: Manual Install + bootc Management

```bash
#!/bin/bash
# hybrid-bootc-workflow.sh - Manual install + bootc management

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-nginx:latest}"
TARGET_ROOT="${2:-/mnt/sysroot}"

echo "🔄 Hybrid bootc workflow: Manual install + bootc management"
echo "Image: ${IMAGE_NAME}"
echo "Target: ${TARGET_ROOT}"

# Step 1: Manual installation (reliable)
echo "📦 Step 1: Manual installation..."
./install-bootc-manual.sh "${IMAGE_NAME}" "${TARGET_ROOT}"

# Step 2: Install bootc for management
echo "🔧 Step 2: Installing bootc for management..."
./install-bootc-for-management.sh

# Step 3: Initialize bootc management
echo "⚙️  Step 3: Initializing bootc management..."
sudo chroot "${TARGET_ROOT}" bootc status || echo "bootc status not available yet"

# Step 4: Set up bootc configuration
echo "📝 Step 4: Setting up bootc configuration..."
sudo chroot "${TARGET_ROOT}" /bin/bash -c "
    # Create bootc configuration
    mkdir -p /etc/bootc
    echo 'image: ${IMAGE_NAME}' > /etc/bootc/config.yaml
    echo 'boot_order: [\"current\", \"rollback\"]' >> /etc/bootc/config.yaml
"

echo "✅ Hybrid workflow completed!"
echo "Manual installation: ✅"
echo "bootc management: ✅"
echo "Ready for upgrades and management!"
```

### bootc Management Service

```bash
#!/bin/bash
# setup-bootc-management-service.sh - Set up bootc management service

set -euo pipefail

echo "🔧 Setting up bootc management service"

# Create systemd service for bootc management
sudo tee /etc/systemd/system/bootc-management.service > /dev/null << 'EOF'
[Unit]
Description=bootc Management Service
Documentation=man:bootc(1)
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/bootc status
ExecReload=/usr/local/bin/bootc upgrade --check
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

# Create bootc upgrade timer
sudo tee /etc/systemd/system/bootc-upgrade.timer > /dev/null << 'EOF'
[Unit]
Description=Run bootc upgrade check daily
Requires=bootc-management.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Create bootc upgrade service
sudo tee /etc/systemd/system/bootc-upgrade.service > /dev/null << 'EOF'
[Unit]
Description=bootc Upgrade Service
Documentation=man:bootc(1)
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/bootc upgrade --check
StandardOutput=journal
StandardError=journal
EOF

# Enable services
sudo systemctl daemon-reload
sudo systemctl enable bootc-management.service
sudo systemctl enable bootc-upgrade.timer

echo "✅ bootc management services configured!"
echo "Status: systemctl status bootc-management"
echo "Upgrades: systemctl status bootc-upgrade.timer"
```

### Management Scripts

```bash
#!/bin/bash
# bootc-manage.sh - Comprehensive bootc management script

set -euo pipefail

ACTION="${1:-status}"
IMAGE_NAME="${2:-}"

case "${ACTION}" in
    "status")
        echo "📊 bootc System Status"
        bootc status
        ;;
    "check-upgrades")
        echo "🔄 Checking for upgrades..."
        bootc upgrade --check
        ;;
    "upgrade")
        echo "⬆️  Applying upgrades..."
        bootc upgrade --apply
        ;;
    "switch")
        if [ -z "${IMAGE_NAME}" ]; then
            echo "❌ Error: Image name required for switch"
            echo "Usage: $0 switch <image-name>"
            exit 1
        fi
        echo "🔄 Switching to ${IMAGE_NAME}..."
        bootc switch "${IMAGE_NAME}" --apply
        ;;
    "rollback")
        echo "⏪ Rolling back..."
        bootc rollback --apply
        ;;
    "usr-overlay")
        echo "📁 Managing usr-overlay..."
        bootc usr-overlay
        ;;
    "help")
        echo "🔧 bootc Management Script"
        echo "Usage: $0 <action> [image-name]"
        echo ""
        echo "Actions:"
        echo "  status          - Show system status"
        echo "  check-upgrades  - Check for available upgrades"
        echo "  upgrade         - Apply upgrades"
        echo "  switch <image>  - Switch to different image"
        echo "  rollback        - Rollback to previous version"
        echo "  usr-overlay     - Manage usr-overlay"
        echo "  help            - Show this help"
        ;;
    *)
        echo "❌ Unknown action: ${ACTION}"
        echo "Run '$0 help' for usage information"
        exit 1
        ;;
esac
```

## Bypassing bootc for Installation

Since bootc may be unreliable on Debian, you can manually install your bootc-compatible images to the filesystem without using the `bootc install` command.

### Manual Installation Process

```bash
#!/bin/bash
# install-bootc-manual.sh - Manually install bootc image without bootc binary

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"
TARGET_ROOT="${2:-/mnt/sysroot}"

# Safety check for destructive operations
if [ "${TARGET_ROOT}" != "/mnt/sysroot" ] && [ -z "${BOOTC_CONFIRM_DISK_WIPE:-}" ]; then
    echo "🚨 CRITICAL SAFETY WARNING 🚨"
    echo "This script will perform DESTRUCTIVE operations!"
    echo "Target: ${TARGET_ROOT}"
    echo "This may WIPE existing data and partitions!"
    echo ""
    echo "THIS IS EXPERIMENTAL SOFTWARE - Expect failures and data loss!"
    echo ""
    echo "MANDATORY: Set BOOTC_CONFIRM_DISK_WIPE=1 to proceed"
    echo "Example: BOOTC_CONFIRM_DISK_WIPE=1 $0 ${IMAGE_NAME} ${TARGET_ROOT}"
    echo ""
    echo "Sleeping 10 seconds before exit..."
    sleep 10
    exit 1
fi

echo "🏗️  Manually installing bootc image without bootc binary"
echo "Image: ${IMAGE_NAME}"
echo "Target: ${TARGET_ROOT}"

# Create target directory
sudo mkdir -p "${TARGET_ROOT}"

# Extract image to temporary directory
TEMP_DIR=$(mktemp -d)
echo "📦 Extracting image to ${TEMP_DIR}..."

# Save and extract image
podman save "${IMAGE_NAME}" | tar -xf - -C "${TEMP_DIR}"

# Find the layer directory
LAYER_DIR=$(find "${TEMP_DIR}" -name "layer.tar" | head -1 | xargs dirname)

# Extract the layer to target
echo "📂 Extracting filesystem to ${TARGET_ROOT}..."
sudo tar -xf "${LAYER_DIR}/layer.tar" -C "${TARGET_ROOT}"

# Set up essential directories
echo "⚙️  Setting up essential directories..."
sudo mkdir -p "${TARGET_ROOT}/boot"
sudo mkdir -p "${TARGET_ROOT}/sysroot"
sudo mkdir -p "${TARGET_ROOT}/var/lib/ostree"

# Set up bind mounts for chroot
echo "🔗 Setting up bind mounts..."
sudo mount --bind /proc "${TARGET_ROOT}/proc"
sudo mount --bind /sys "${TARGET_ROOT}/sys"
sudo mount --bind /dev "${TARGET_ROOT}/dev"

# Configure systemd in chroot
echo "🔧 Configuring systemd..."
sudo chroot "${TARGET_ROOT}" systemctl set-default multi-user.target

# Generate initramfs
echo "🚀 Generating initramfs..."
sudo chroot "${TARGET_ROOT}" update-initramfs -u

# Install GRUB (if installing to disk)
if [ "${TARGET_ROOT}" != "/mnt/sysroot" ]; then
    echo "💾 Installing GRUB..."
    sudo chroot "${TARGET_ROOT}" grub-install /dev/sda
    sudo chroot "${TARGET_ROOT}" grub-mkconfig -o /boot/grub/grub.cfg
fi

# Clean up
sudo umount "${TARGET_ROOT}/proc" || true
sudo umount "${TARGET_ROOT}/sys" || true
sudo umount "${TARGET_ROOT}/dev" || true
rm -rf "${TEMP_DIR}"

echo "✅ Manual installation completed!"
echo "Target: ${TARGET_ROOT}"
```

### Alternative: Using systemd-nspawn

```bash
#!/bin/bash
# install-bootc-nspawn.sh - Install using systemd-nspawn

set -euo pipefail

IMAGE_NAME="${1:-debian-bootc-base:latest}"
TARGET_ROOT="${2:-/mnt/sysroot}"

echo "🏗️  Installing bootc image using systemd-nspawn"
echo "Image: ${IMAGE_NAME}"
echo "Target: ${TARGET_ROOT}"

# Extract image first
TEMP_DIR=$(mktemp -d)
podman save "${IMAGE_NAME}" | tar -xf - -C "${TEMP_DIR}"
LAYER_DIR=$(find "${TEMP_DIR}" -name "layer.tar" | head -1 | xargs dirname)
sudo tar -xf "${LAYER_DIR}/layer.tar" -C "${TARGET_ROOT}"

# Use systemd-nspawn for configuration
echo "🔧 Configuring with systemd-nspawn..."
sudo systemd-nspawn -D "${TARGET_ROOT}" /bin/bash -c "
    systemctl set-default multi-user.target
    update-initramfs -u
    systemctl enable systemd-resolved.service
    systemctl enable systemd-networkd.service
"

rm -rf "${TEMP_DIR}"
echo "✅ Installation completed with systemd-nspawn!"
```

## Essential Initramfs Integration

For proper composefs support and boot functionality, you need to include initramfs integration files in your base image. These files are essential for Debian bootc images:

### Required Files

**1. `bootc-initramfs-setup.service`** - Systemd service for initramfs:
```ini
# Copyright (C) 2013 Colin Walters <walters@verbum.org>
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2 of the License, or (at your option) any later version.

[Unit]
DefaultDependencies=no
ConditionKernelCommandLine=composefs
ConditionPathExists=/etc/initrd-release
After=sysroot.mount
Requires=sysroot.mount
Before=initrd-root-fs.target
Before=initrd-switch-root.target
OnFailure=emergency.target
OnFailureJobMode=isolate

[Service]
Type=oneshot
ExecStart=/usr/bin/bootc-initramfs-setup
StandardInput=null
StandardOutput=journal
StandardError=journal+console
RemainAfterExit=yes
```

**2. `module-setup.sh`** - Dracut module for initramfs:
```bash
#!/usr/bin/bash
check() {
    return 0
}

depends() {
    return 0
}

install() {
    inst \
        "${moddir}/bootc-initramfs-setup" /bin/bootc-initramfs-setup
    inst \
        "${moddir}/bootc-initramfs-setup.service" \
        "${systemdsystemunitdir}/bootc-initramfs-setup.service"
    $SYSTEMCTL -q --root "${initdir}" add-wants \
        'initrd-root-fs.target' 'bootc-initramfs-setup.service'
}
```

**3. `prepare-root.conf`** - Composefs configuration:
```ini
[composefs]
enabled = yes

[sysroot]
readonly = true
```

### Integration in Containerfile

Add these files to your base image:

```dockerfile
# Copy initramfs integration files
COPY files/37composefs/bootc-initramfs-setup.service /usr/lib/systemd/system/
COPY files/37composefs/module-setup.sh /usr/lib/dracut/modules.d/37composefs/
COPY files/ostree/prepare-root.conf /usr/lib/ostree-boot/prepare-root.conf

# Enable the service
RUN ln -sf /usr/lib/systemd/system/bootc-initramfs-setup.service \
    /etc/systemd/system/multi-user.target.wants/bootc-initramfs-setup.service
```

## Adding Application Layers (Example: nginx)

### Method 1: Building on Base Image

```dockerfile
# examples/nginx/Containerfile
FROM debian-bootc-base:latest

# Install nginx
RUN apt update && apt install -y nginx && apt clean

# Configure nginx
RUN echo 'server { \
    listen 80; \
    server_name _; \
    location / { \
        return 200 "Hello from bootc nginx!"; \
        add_header Content-Type text/plain; \
    } \
}' > /etc/nginx/sites-available/default

RUN ln -sf /etc/nginx/sites-available/default /etc/nginx/sites-enabled/
RUN rm -f /etc/nginx/sites-enabled/default.bak

# Enable nginx service
RUN systemctl enable nginx.service

# Create nginx user and group
RUN systemd-sysusers --create

# Required bootc labels
LABEL containers.bootc 1
LABEL ostree.bootable 1

# Set default command
CMD ["/lib/systemd/systemd"]
```

### Method 2: Multi-stage Build

```dockerfile
# examples/nginx/Containerfile.multistage
FROM debian:sid-slim as base

# Install base packages
RUN apt update && apt install -y \
    systemd \
    systemd-sysusers \
    systemd-tmpfiles \
    systemd-resolved \
    systemd-networkd \
    systemd-timesyncd \
    kernel \
    initramfs-tools \
    grub2 \
    grub2-common \
    efibootmgr \
    ostree \
    nginx \
    && apt clean

# Configure systemd
RUN ln -sf /lib/systemd/systemd /sbin/init
RUN systemctl set-default multi-user.target
RUN systemctl enable systemd-resolved.service \
    systemd-networkd.service \
    systemd-timesyncd.service \
    nginx.service

# Configure nginx
RUN echo 'server { \
    listen 80; \
    server_name _; \
    location / { \
        return 200 "Hello from bootc nginx!"; \
        add_header Content-Type text/plain; \
    } \
}' > /etc/nginx/sites-available/default

RUN ln -sf /etc/nginx/sites-available/default /etc/nginx/sites-enabled/
RUN rm -f /etc/nginx/sites-enabled/default.bak

# Create essential directories
RUN mkdir -p /usr/lib/systemd/system \
    /usr/lib/systemd/user \
    /etc/systemd/system \
    /etc/systemd/user \
    /var/lib/systemd \
    /run/systemd \
    /etc/systemd/network

# Configure networking
RUN echo -e "[Match]\nName=*\n\n[Network]\nDHCP=yes" > /etc/systemd/network/80-dhcp.network

# Set up users and groups
RUN systemd-sysusers --create
RUN systemd-tmpfiles --create

# Required bootc labels
LABEL containers.bootc 1
LABEL ostree.bootable 1

# Set default command
CMD ["/lib/systemd/systemd"]
```

### Build and Install Script

```bash
#!/bin/bash
# build-and-install-nginx.sh - Build nginx layer and install manually

set -euo pipefail

BASE_IMAGE="debian-bootc-base:latest"
NGINX_IMAGE="debian-bootc-nginx:latest"
TARGET_ROOT="${1:-/mnt/sysroot}"

echo "🏗️  Building nginx layer on bootc base"
echo "Base: ${BASE_IMAGE}"
echo "Nginx: ${NGINX_IMAGE}"
echo "Target: ${TARGET_ROOT}"

# Build nginx image
echo "📦 Building nginx image..."
podman build -f examples/nginx/Containerfile -t "${NGINX_IMAGE}" .

# Validate the image
echo "🔍 Validating nginx image..."
./validate-all.sh "${NGINX_IMAGE}"

# Install manually
echo "💾 Installing nginx image..."
./install-bootc-manual.sh "${NGINX_IMAGE}" "${TARGET_ROOT}"

echo "✅ nginx bootc image built and installed!"
echo "Image: ${NGINX_IMAGE}"
echo "Target: ${TARGET_ROOT}"
```

### Testing the nginx Layer

```bash
#!/bin/bash
# test-nginx-layer.sh - Test nginx layer

set -euo pipefail

NGINX_IMAGE="${1:-debian-bootc-nginx:latest}"

echo "🧪 Testing nginx layer: ${NGINX_IMAGE}"

# Test nginx configuration
echo "🔍 Testing nginx configuration..."
podman run --rm "${NGINX_IMAGE}" nginx -t

# Test systemd services
echo "🔍 Testing systemd services..."
podman run --rm "${NGINX_IMAGE}" systemctl list-unit-files --state=enabled | grep nginx

# Test nginx startup
echo "🔍 Testing nginx startup..."
podman run --rm "${NGINX_IMAGE}" systemctl is-enabled nginx.service

# Test HTTP response (if possible)
echo "🔍 Testing HTTP response..."
podman run --rm -d --name nginx-test -p 8080:80 "${NGINX_IMAGE}"
sleep 5

if curl -s http://localhost:8080 | grep -q "Hello from bootc nginx"; then
    echo "✅ nginx HTTP response test passed"
else
    echo "❌ nginx HTTP response test failed"
fi

podman stop nginx-test
podman rm nginx-test

echo "🎉 nginx layer testing completed!"
```

### Complete Project Structure with nginx

```
debian-bootc-project/
├── build-bootc-base-wo-bootc.sh     # Build base image
├── build-and-install-nginx.sh       # Build and install nginx
├── install-bootc-manual.sh          # Manual installation
├── install-bootc-nspawn.sh          # nspawn installation
├── validate-all.sh                  # Complete validation
├── test-nginx-layer.sh              # Test nginx layer
├── Containerfile.wo-bootc           # Base image containerfile
├── examples/
│   └── nginx/
│       ├── Containerfile            # nginx layer
│       └── Containerfile.multistage # Multi-stage nginx
└── scripts/
    ├── validate-filesystem.sh
    ├── validate-labels.sh
    ├── validate-systemd.sh
    ├── validate-network.sh
    └── validate-bootloader.sh
```

### Usage Workflow

```bash
# 1. Build base image
./build-bootc-base-wo-bootc.sh

# 2. Build and install nginx layer
./build-and-install-nginx.sh /mnt/sysroot

# 3. Test the installation
./test-nginx-layer.sh debian-bootc-nginx:latest

# 4. Boot the system (if installed to disk)
# Reboot and select the new boot entry
```

## Summary

This approach allows you to create fully bootc-compatible base images on Debian without relying on the potentially unreliable bootc binary. The manual validation scripts ensure your images meet all bootc requirements, making them ready for deployment and management with bootc once it's properly working on your system.

The key is understanding what bootc expects and manually validating those requirements rather than depending on the bootc binary for validation. You can also manually install these images without using `bootc install`, and easily add application layers like nginx on top of your base images.