debian-ostree-systems-notes/deb-ostree-builder.md

# deb-ostree-builder Project Breakdown

## Project Overview

The deb-ostree-builder is a sophisticated build system originally derived from Endless OS that creates bootable Debian systems using OSTree technology. It's designed as a "stripped down" version specifically for Debian, focusing on simplicity while maintaining the power of atomic system management.

## Repository Structure

Based on the project analysis, here's the inferred repository structure:

```
deb-ostree-builder/
├── deb-ostree-builder          # Main build script (entry point)
├── create-deployment           # Deployment creation utility
├── README.md                   # Project documentation
├── buildscript                 # Build orchestration script
├── config/                     # Configuration directory
│   ├── defaults.ini           # Default build settings
│   ├── product/               # Product-specific configurations
│   │   ├── debian.ini
│   │   ├── debianplatform.ini
│   │   └── debiansdk.ini
│   ├── branch/                # Branch-specific settings
│   │   └── *.ini
│   ├── arch/                  # Architecture-specific settings
│   │   ├── i386.ini
│   │   └── armhf.ini
│   ├── platform/              # Platform-specific settings
│   │   └── *.ini
│   └── local.ini              # Local build overrides
├── hooks/                     # Customization hooks directory
│   ├── cache/                 # Update checking hooks
│   ├── seed/                  # Pre-bootstrap hooks
│   ├── os/                    # OS customization hooks
│   │   ├── 10-debconf-set-selections.chroot
│   │   ├── 50-install-packages*
│   │   └── *.sh
│   ├── ostree/                # OSTree-specific hooks
│   ├── publish/               # Publishing hooks
│   └── error/                 # Error cleanup hooks
├── lib/                       # Library functions
│   └── eob.sh                 # Core EOB (Endless OS Builder) functions
├── helpers/                   # Helper utilities
│   └── core-packages          # Core package management
└── data/                      # Static data files
    └── debootstrap.script     # Custom debootstrap script
```

## How It Builds a Debian OSTree System

### 1. **Multi-Stage Build Process**

The build system operates in distinct stages, each with specific responsibilities:

#### Stage 1: Check Update (cache hooks)
- Determines if a build is needed by comparing cached facts
- Checks modification times of cache files
- Exits early if no changes detected

#### Stage 2: Seed (seed hooks)
- Initializes empty `${EOB_ROOTDIR}` directory
- Places any pre-bootstrap content
- Prepares foundation for system construction

#### Stage 3: OS Construction (os hooks)
- Creates the operating system using debootstrap
- Installs packages via apt
- Makes OSTree-specific modifications
- Runs customization scripts in lexical order

#### Stage 4: OSTree Commit (ostree hooks)
- Commits the prepared filesystem to OSTree repository
- Handles checksumming and integrity verification
- Creates immutable filesystem snapshots

#### Stage 5: Publish (publish hooks)
- Publishes local OSTree repository to remote servers
- Handles distribution and deployment

### 2. **Configuration System**

The build system uses a sophisticated INI-based configuration hierarchy:

```ini
# Configuration Priority (later overrides earlier):
config/defaults.ini                    # Base settings
config/product/$product.ini           # Product variants
config/branch/$branch.ini             # Branch-specific
config/arch/$arch.ini                 # Architecture-specific  
config/platform/$platform.ini        # Platform-specific
/etc/deb-ostree-builder/config.ini   # System-wide overrides
config/local.ini                      # Local build overrides
```

**Variable Interpolation Example:**
```ini
[build]
suite = sid
source = http://deb.debian.org/debian

[debian]
# Uses value from build section
packages = linux-image-${arch} ostree-boot
mirror = ${source}
```

### 3. **Package Management Integration**

The system handles Debian package management through multiple mechanisms:

#### Merged Configuration Options
For managing lists of packages and hooks:
```ini
# Adding packages
os:packages_add_base = systemd openssh-server
os:packages_add_desktop = gnome-core firefox
os:packages_del_unwanted = nano vim-tiny

# Final packages list = base + desktop - unwanted
```

#### Package Installation Process
1. **Debootstrap**: Creates minimal Debian system
2. **Hook-based Installation**: Scripts at index 50 install additional packages
3. **Chroot Operations**: Some scripts run inside the target system environment

### 4. **OSTree-Specific Transformations**

The builder implements several critical transformations to make Debian compatible with OSTree:

#### Filesystem Layout Changes
```bash
# Move /etc to /usr/etc (OSTree three-way merge requirement)
mv "${BUILDDIR}"/etc "${BUILDDIR}"/usr/etc

# Create persistent directory symlinks
ln -s /sysroot/home "${BUILDDIR}"/home
ln -s /sysroot/root "${BUILDDIR}"/root
ln -s /var/opt "${BUILDDIR}"/opt

# Move dpkg database to immutable location
mv "${BUILDDIR}"/var/lib/dpkg "${BUILDDIR}"/usr/share/dpkg/database
ln -sr "${BUILDDIR}"/usr/share/dpkg/database "${BUILDDIR}"/var/lib/dpkg
```

#### Boot File Management
```bash
# Create OSTree-compatible boot file checksums
csum=$(cat vmlinuz-* initrd.img-* | sha256sum --binary | awk '{print $1}')
mv vmlinuz-* vmlinuz-*-${csum}
mv initrd.img-* initramfs-*-${csum}
```

### 5. **Hook System Architecture**

The hook system provides extensive customization capabilities:

#### Hook Types
- **Executable Scripts**: Run directly if executable bit set
- **Bash Scripts**: Sourced with access to library functions
- **Chroot Scripts**: Run inside target system (`.chroot` suffix)

#### Hook Categories
- **cache**: Update checking and caching logic
- **seed**: Pre-bootstrap preparation
- **os**: System construction and customization
- **publish**: Repository publication
- **error**: Cleanup and error handling

#### Example Hook Structure
```bash
# hooks/os/20-configure-system.sh
#!/bin/bash
# Configure system settings for OSTree

# Access to EOB environment variables
echo "Building for architecture: ${EOB_ARCH}"
echo "Target suite: ${EOB_SUITE}"

# Use library functions
eob_info "Configuring system for OSTree compatibility"
```

### 6. **Build Environment Management**

The system carefully manages the build environment:

#### Mount Management
```bash
# Mount necessary filesystems for chroot operations
for dir in proc sys dev dev/pts; do
    mount --bind "/$dir" "$BUILDDIR/$dir"
done

# Cleanup on exit
trap cleanup_mounts EXIT
```

#### Security Measures
```bash
# Prevent daemon startup during build
cat > "$BUILDDIR"/usr/sbin/policy-rc.d <<EOF
#!/bin/sh
exit 101
EOF
```

## Key Innovations

### 1. **Simplified Build Logic**
Unlike complex build systems, deb-ostree-builder uses straightforward bash scripts with clear stage separation, making it easy to understand and modify.

### 2. **Flexible Configuration**
The hierarchical INI configuration system allows fine-grained control over different build variants while maintaining consistency.

### 3. **Hook-Based Extensibility**
The modular hook system enables easy customization without modifying core build logic.

### 4. **Debian Integration**
Seamless integration with Debian's package management while adapting to OSTree's requirements.

## Build Process Flow

```mermaid
graph TD
    A[Start Build] --> B[Load Configuration]
    B --> C[Check Update Stage]
    C --> D{Update Needed?}
    D -->|No| E[Exit Early]
    D -->|Yes| F[Seed Stage]
    F --> G[Create Build Directory]
    G --> H[Run Seed Hooks]
    H --> I[OS Construction Stage]
    I --> J[Debootstrap Base System]
    J --> K[Mount Filesystems]
    K --> L[Run OS Hooks]
    L --> M[Package Installation]
    M --> N[System Configuration]
    N --> O[OSTree Transformations]
    O --> P[OSTree Commit Stage]
    P --> Q[Create OSTree Commit]
    Q --> R[Publish Stage]
    R --> S[Run Publish Hooks]
    S --> T[Upload to Remote]
    T --> U[Build Complete]
```

## Technical Architecture

### Core Components

1. **EOB Core**: Python-based orchestration engine
2. **Hook System**: Bash script customization framework  
3. **Configuration Management**: INI-based hierarchical settings
4. **OSTree Integration**: Filesystem transformation and commitment
5. **Debian Toolchain**: Debootstrap, apt, dpkg integration

### Environment Variables

The build system communicates through environment variables:
```bash
EOB_ARCH=amd64              # Target architecture
EOB_SUITE=sid               # Debian suite
EOB_ROOTDIR=/tmp/build      # Build directory
EOB_OSTREE_REPO=/ostree/repo # OSTree repository path
```

### Library Functions

The `lib/eob.sh` provides utility functions for hooks:
```bash
eob_info()      # Informational logging
eob_error()     # Error reporting
eob_cachedir()  # Cache directory path
eob_cachefile() # Cache file naming
```

## Advantages of This Approach

1. **Simplicity**: Bash-based core with minimal abstraction
2. **Transparency**: Complete understanding of build process possible
3. **Flexibility**: Easy modification through hooks and configuration
4. **Reliability**: Atomic updates with rollback capability
5. **Security**: Immutable root filesystem with verified boot

## Use Cases

- **Development Systems**: Testing Debian packages in isolated environments
- **Production Deployments**: Atomic updates for server systems
- **Embedded Systems**: Reliable OS updates for IoT devices
- **Cloud Infrastructure**: Immutable infrastructure deployments

This architecture provides a robust foundation for building and managing Debian-based OSTree systems while maintaining the simplicity and transparency that makes the system maintainable and extensible.