particle-os/debian-ostree-systems-notes
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
debian-ostree-systems-notes/deb-ostree-builder.md

9.8 KiB
Raw Blame History

deb-ostree-builder Project Breakdown

Project Overview

The deb-ostree-builder is a sophisticated build system originally derived from Endless OS that creates bootable Debian systems using OSTree technology. It's designed as a "stripped down" version specifically for Debian, focusing on simplicity while maintaining the power of atomic system management.

Repository Structure

Based on the project analysis, here's the inferred repository structure:

deb-ostree-builder/
├── deb-ostree-builder          # Main build script (entry point)
├── create-deployment           # Deployment creation utility
├── README.md                   # Project documentation
├── buildscript                 # Build orchestration script
├── config/                     # Configuration directory
│   ├── defaults.ini           # Default build settings
│   ├── product/               # Product-specific configurations
│   │   ├── debian.ini
│   │   ├── debianplatform.ini
│   │   └── debiansdk.ini
│   ├── branch/                # Branch-specific settings
│   │   └── *.ini
│   ├── arch/                  # Architecture-specific settings
│   │   ├── i386.ini
│   │   └── armhf.ini
│   ├── platform/              # Platform-specific settings
│   │   └── *.ini
│   └── local.ini              # Local build overrides
├── hooks/                     # Customization hooks directory
│   ├── cache/                 # Update checking hooks
│   ├── seed/                  # Pre-bootstrap hooks
│   ├── os/                    # OS customization hooks
│   │   ├── 10-debconf-set-selections.chroot
│   │   ├── 50-install-packages*
│   │   └── *.sh
│   ├── ostree/                # OSTree-specific hooks
│   ├── publish/               # Publishing hooks
│   └── error/                 # Error cleanup hooks
├── lib/                       # Library functions
│   └── eob.sh                 # Core EOB (Endless OS Builder) functions
├── helpers/                   # Helper utilities
│   └── core-packages          # Core package management
└── data/                      # Static data files
    └── debootstrap.script     # Custom debootstrap script

How It Builds a Debian OSTree System

1. Multi-Stage Build Process

The build system operates in distinct stages, each with specific responsibilities:

Stage 1: Check Update (cache hooks)

  • Determines if a build is needed by comparing cached facts
  • Checks modification times of cache files
  • Exits early if no changes detected

Stage 2: Seed (seed hooks)

  • Initializes empty ${EOB_ROOTDIR} directory
  • Places any pre-bootstrap content
  • Prepares foundation for system construction

Stage 3: OS Construction (os hooks)

  • Creates the operating system using debootstrap
  • Installs packages via apt
  • Makes OSTree-specific modifications
  • Runs customization scripts in lexical order

Stage 4: OSTree Commit (ostree hooks)

  • Commits the prepared filesystem to OSTree repository
  • Handles checksumming and integrity verification
  • Creates immutable filesystem snapshots

Stage 5: Publish (publish hooks)

  • Publishes local OSTree repository to remote servers
  • Handles distribution and deployment

2. Configuration System

The build system uses a sophisticated INI-based configuration hierarchy:

# Configuration Priority (later overrides earlier):
config/defaults.ini                    # Base settings
config/product/$product.ini           # Product variants
config/branch/$branch.ini             # Branch-specific
config/arch/$arch.ini                 # Architecture-specific  
config/platform/$platform.ini        # Platform-specific
/etc/deb-ostree-builder/config.ini   # System-wide overrides
config/local.ini                      # Local build overrides

Variable Interpolation Example:

[build]
suite = sid
source = http://deb.debian.org/debian

[debian]
# Uses value from build section
packages = linux-image-${arch} ostree-boot
mirror = ${source}

3. Package Management Integration

The system handles Debian package management through multiple mechanisms:

Merged Configuration Options

For managing lists of packages and hooks:

# Adding packages
os:packages_add_base = systemd openssh-server
os:packages_add_desktop = gnome-core firefox
os:packages_del_unwanted = nano vim-tiny

# Final packages list = base + desktop - unwanted

Package Installation Process

  1. Debootstrap: Creates minimal Debian system
  2. Hook-based Installation: Scripts at index 50 install additional packages
  3. Chroot Operations: Some scripts run inside the target system environment

4. OSTree-Specific Transformations

The builder implements several critical transformations to make Debian compatible with OSTree:

Filesystem Layout Changes

# Move /etc to /usr/etc (OSTree three-way merge requirement)
mv "${BUILDDIR}"/etc "${BUILDDIR}"/usr/etc

# Create persistent directory symlinks
ln -s /sysroot/home "${BUILDDIR}"/home
ln -s /sysroot/root "${BUILDDIR}"/root
ln -s /var/opt "${BUILDDIR}"/opt

# Move dpkg database to immutable location
mv "${BUILDDIR}"/var/lib/dpkg "${BUILDDIR}"/usr/share/dpkg/database
ln -sr "${BUILDDIR}"/usr/share/dpkg/database "${BUILDDIR}"/var/lib/dpkg

Boot File Management

# Create OSTree-compatible boot file checksums
csum=$(cat vmlinuz-* initrd.img-* | sha256sum --binary | awk '{print $1}')
mv vmlinuz-* vmlinuz-*-${csum}
mv initrd.img-* initramfs-*-${csum}

5. Hook System Architecture

The hook system provides extensive customization capabilities:

Hook Types

  • Executable Scripts: Run directly if executable bit set
  • Bash Scripts: Sourced with access to library functions
  • Chroot Scripts: Run inside target system (.chroot suffix)

Hook Categories

  • cache: Update checking and caching logic
  • seed: Pre-bootstrap preparation
  • os: System construction and customization
  • publish: Repository publication
  • error: Cleanup and error handling

Example Hook Structure

# hooks/os/20-configure-system.sh
#!/bin/bash
# Configure system settings for OSTree

# Access to EOB environment variables
echo "Building for architecture: ${EOB_ARCH}"
echo "Target suite: ${EOB_SUITE}"

# Use library functions
eob_info "Configuring system for OSTree compatibility"

6. Build Environment Management

The system carefully manages the build environment:

Mount Management

# Mount necessary filesystems for chroot operations
for dir in proc sys dev dev/pts; do
    mount --bind "/$dir" "$BUILDDIR/$dir"
done

# Cleanup on exit
trap cleanup_mounts EXIT

Security Measures

# Prevent daemon startup during build
cat > "$BUILDDIR"/usr/sbin/policy-rc.d <<EOF
#!/bin/sh
exit 101
EOF

Key Innovations

1. Simplified Build Logic

Unlike complex build systems, deb-ostree-builder uses straightforward bash scripts with clear stage separation, making it easy to understand and modify.

2. Flexible Configuration

The hierarchical INI configuration system allows fine-grained control over different build variants while maintaining consistency.

3. Hook-Based Extensibility

The modular hook system enables easy customization without modifying core build logic.

4. Debian Integration

Seamless integration with Debian's package management while adapting to OSTree's requirements.

Build Process Flow

graph TD
    A[Start Build] --> B[Load Configuration]
    B --> C[Check Update Stage]
    C --> D{Update Needed?}
    D -->|No| E[Exit Early]
    D -->|Yes| F[Seed Stage]
    F --> G[Create Build Directory]
    G --> H[Run Seed Hooks]
    H --> I[OS Construction Stage]
    I --> J[Debootstrap Base System]
    J --> K[Mount Filesystems]
    K --> L[Run OS Hooks]
    L --> M[Package Installation]
    M --> N[System Configuration]
    N --> O[OSTree Transformations]
    O --> P[OSTree Commit Stage]
    P --> Q[Create OSTree Commit]
    Q --> R[Publish Stage]
    R --> S[Run Publish Hooks]
    S --> T[Upload to Remote]
    T --> U[Build Complete]

Technical Architecture

Core Components

  1. EOB Core: Python-based orchestration engine
  2. Hook System: Bash script customization framework
  3. Configuration Management: INI-based hierarchical settings
  4. OSTree Integration: Filesystem transformation and commitment
  5. Debian Toolchain: Debootstrap, apt, dpkg integration

Environment Variables

The build system communicates through environment variables:

EOB_ARCH=amd64              # Target architecture
EOB_SUITE=sid               # Debian suite
EOB_ROOTDIR=/tmp/build      # Build directory
EOB_OSTREE_REPO=/ostree/repo # OSTree repository path

Library Functions

The lib/eob.sh provides utility functions for hooks:

eob_info()      # Informational logging
eob_error()     # Error reporting
eob_cachedir()  # Cache directory path
eob_cachefile() # Cache file naming

Advantages of This Approach

  1. Simplicity: Bash-based core with minimal abstraction
  2. Transparency: Complete understanding of build process possible
  3. Flexibility: Easy modification through hooks and configuration
  4. Reliability: Atomic updates with rollback capability
  5. Security: Immutable root filesystem with verified boot

Use Cases

  • Development Systems: Testing Debian packages in isolated environments
  • Production Deployments: Atomic updates for server systems
  • Embedded Systems: Reliable OS updates for IoT devices
  • Cloud Infrastructure: Immutable infrastructure deployments

This architecture provides a robust foundation for building and managing Debian-based OSTree systems while maintaining the simplicity and transparency that makes the system maintainable and extensible.