diff --git a/deb-ostree-builder.md b/deb-ostree-builder.md new file mode 100644 index 0000000..893a336 --- /dev/null +++ b/deb-ostree-builder.md @@ -0,0 +1,293 @@ +# deb-ostree-builder Project Breakdown + +## Project Overview + +The deb-ostree-builder is a sophisticated build system originally derived from Endless OS that creates bootable Debian systems using OSTree technology. It's designed as a "stripped down" version specifically for Debian, focusing on simplicity while maintaining the power of atomic system management. + +## Repository Structure + +Based on the project analysis, here's the inferred repository structure: + +``` +deb-ostree-builder/ +├── deb-ostree-builder # Main build script (entry point) +├── create-deployment # Deployment creation utility +├── README.md # Project documentation +├── buildscript # Build orchestration script +├── config/ # Configuration directory +│ ├── defaults.ini # Default build settings +│ ├── product/ # Product-specific configurations +│ │ ├── debian.ini +│ │ ├── debianplatform.ini +│ │ └── debiansdk.ini +│ ├── branch/ # Branch-specific settings +│ │ └── *.ini +│ ├── arch/ # Architecture-specific settings +│ │ ├── i386.ini +│ │ └── armhf.ini +│ ├── platform/ # Platform-specific settings +│ │ └── *.ini +│ └── local.ini # Local build overrides +├── hooks/ # Customization hooks directory +│ ├── cache/ # Update checking hooks +│ ├── seed/ # Pre-bootstrap hooks +│ ├── os/ # OS customization hooks +│ │ ├── 10-debconf-set-selections.chroot +│ │ ├── 50-install-packages* +│ │ └── *.sh +│ ├── ostree/ # OSTree-specific hooks +│ ├── publish/ # Publishing hooks +│ └── error/ # Error cleanup hooks +├── lib/ # Library functions +│ └── eob.sh # Core EOB (Endless OS Builder) functions +├── helpers/ # Helper utilities +│ └── core-packages # Core package management +└── data/ # Static data files + └── debootstrap.script # Custom debootstrap script +``` + +## How It Builds a Debian OSTree System + +### 1. **Multi-Stage Build Process** + +The build system operates in distinct stages, each with specific responsibilities: + +#### Stage 1: Check Update (cache hooks) +- Determines if a build is needed by comparing cached facts +- Checks modification times of cache files +- Exits early if no changes detected + +#### Stage 2: Seed (seed hooks) +- Initializes empty `${EOB_ROOTDIR}` directory +- Places any pre-bootstrap content +- Prepares foundation for system construction + +#### Stage 3: OS Construction (os hooks) +- Creates the operating system using debootstrap +- Installs packages via apt +- Makes OSTree-specific modifications +- Runs customization scripts in lexical order + +#### Stage 4: OSTree Commit (ostree hooks) +- Commits the prepared filesystem to OSTree repository +- Handles checksumming and integrity verification +- Creates immutable filesystem snapshots + +#### Stage 5: Publish (publish hooks) +- Publishes local OSTree repository to remote servers +- Handles distribution and deployment + +### 2. **Configuration System** + +The build system uses a sophisticated INI-based configuration hierarchy: + +```ini +# Configuration Priority (later overrides earlier): +config/defaults.ini # Base settings +config/product/$product.ini # Product variants +config/branch/$branch.ini # Branch-specific +config/arch/$arch.ini # Architecture-specific +config/platform/$platform.ini # Platform-specific +/etc/deb-ostree-builder/config.ini # System-wide overrides +config/local.ini # Local build overrides +``` + +**Variable Interpolation Example:** +```ini +[build] +suite = sid +source = http://deb.debian.org/debian + +[debian] +# Uses value from build section +packages = linux-image-${arch} ostree-boot +mirror = ${source} +``` + +### 3. **Package Management Integration** + +The system handles Debian package management through multiple mechanisms: + +#### Merged Configuration Options +For managing lists of packages and hooks: +```ini +# Adding packages +os:packages_add_base = systemd openssh-server +os:packages_add_desktop = gnome-core firefox +os:packages_del_unwanted = nano vim-tiny + +# Final packages list = base + desktop - unwanted +``` + +#### Package Installation Process +1. **Debootstrap**: Creates minimal Debian system +2. **Hook-based Installation**: Scripts at index 50 install additional packages +3. **Chroot Operations**: Some scripts run inside the target system environment + +### 4. **OSTree-Specific Transformations** + +The builder implements several critical transformations to make Debian compatible with OSTree: + +#### Filesystem Layout Changes +```bash +# Move /etc to /usr/etc (OSTree three-way merge requirement) +mv "${BUILDDIR}"/etc "${BUILDDIR}"/usr/etc + +# Create persistent directory symlinks +ln -s /sysroot/home "${BUILDDIR}"/home +ln -s /sysroot/root "${BUILDDIR}"/root +ln -s /var/opt "${BUILDDIR}"/opt + +# Move dpkg database to immutable location +mv "${BUILDDIR}"/var/lib/dpkg "${BUILDDIR}"/usr/share/dpkg/database +ln -sr "${BUILDDIR}"/usr/share/dpkg/database "${BUILDDIR}"/var/lib/dpkg +``` + +#### Boot File Management +```bash +# Create OSTree-compatible boot file checksums +csum=$(cat vmlinuz-* initrd.img-* | sha256sum --binary | awk '{print $1}') +mv vmlinuz-* vmlinuz-*-${csum} +mv initrd.img-* initramfs-*-${csum} +``` + +### 5. **Hook System Architecture** + +The hook system provides extensive customization capabilities: + +#### Hook Types +- **Executable Scripts**: Run directly if executable bit set +- **Bash Scripts**: Sourced with access to library functions +- **Chroot Scripts**: Run inside target system (`.chroot` suffix) + +#### Hook Categories +- **cache**: Update checking and caching logic +- **seed**: Pre-bootstrap preparation +- **os**: System construction and customization +- **publish**: Repository publication +- **error**: Cleanup and error handling + +#### Example Hook Structure +```bash +# hooks/os/20-configure-system.sh +#!/bin/bash +# Configure system settings for OSTree + +# Access to EOB environment variables +echo "Building for architecture: ${EOB_ARCH}" +echo "Target suite: ${EOB_SUITE}" + +# Use library functions +eob_info "Configuring system for OSTree compatibility" +``` + +### 6. **Build Environment Management** + +The system carefully manages the build environment: + +#### Mount Management +```bash +# Mount necessary filesystems for chroot operations +for dir in proc sys dev dev/pts; do + mount --bind "/$dir" "$BUILDDIR/$dir" +done + +# Cleanup on exit +trap cleanup_mounts EXIT +``` + +#### Security Measures +```bash +# Prevent daemon startup during build +cat > "$BUILDDIR"/usr/sbin/policy-rc.d < B[Load Configuration] + B --> C[Check Update Stage] + C --> D{Update Needed?} + D -->|No| E[Exit Early] + D -->|Yes| F[Seed Stage] + F --> G[Create Build Directory] + G --> H[Run Seed Hooks] + H --> I[OS Construction Stage] + I --> J[Debootstrap Base System] + J --> K[Mount Filesystems] + K --> L[Run OS Hooks] + L --> M[Package Installation] + M --> N[System Configuration] + N --> O[OSTree Transformations] + O --> P[OSTree Commit Stage] + P --> Q[Create OSTree Commit] + Q --> R[Publish Stage] + R --> S[Run Publish Hooks] + S --> T[Upload to Remote] + T --> U[Build Complete] +``` + +## Technical Architecture + +### Core Components + +1. **EOB Core**: Python-based orchestration engine +2. **Hook System**: Bash script customization framework +3. **Configuration Management**: INI-based hierarchical settings +4. **OSTree Integration**: Filesystem transformation and commitment +5. **Debian Toolchain**: Debootstrap, apt, dpkg integration + +### Environment Variables + +The build system communicates through environment variables: +```bash +EOB_ARCH=amd64 # Target architecture +EOB_SUITE=sid # Debian suite +EOB_ROOTDIR=/tmp/build # Build directory +EOB_OSTREE_REPO=/ostree/repo # OSTree repository path +``` + +### Library Functions + +The `lib/eob.sh` provides utility functions for hooks: +```bash +eob_info() # Informational logging +eob_error() # Error reporting +eob_cachedir() # Cache directory path +eob_cachefile() # Cache file naming +``` + +## Advantages of This Approach + +1. **Simplicity**: Bash-based core with minimal abstraction +2. **Transparency**: Complete understanding of build process possible +3. **Flexibility**: Easy modification through hooks and configuration +4. **Reliability**: Atomic updates with rollback capability +5. **Security**: Immutable root filesystem with verified boot + +## Use Cases + +- **Development Systems**: Testing Debian packages in isolated environments +- **Production Deployments**: Atomic updates for server systems +- **Embedded Systems**: Reliable OS updates for IoT devices +- **Cloud Infrastructure**: Immutable infrastructure deployments + +This architecture provides a robust foundation for building and managing Debian-based OSTree systems while maintaining the simplicity and transparency that makes the system maintainable and extensible. \ No newline at end of file