added deb-ostree-builder.md
This commit is contained in:
parent
789bde1a0c
commit
7fcfe927c9
1 changed files with 293 additions and 0 deletions
293
deb-ostree-builder.md
Normal file
293
deb-ostree-builder.md
Normal file
|
|
@ -0,0 +1,293 @@
|
|||
# deb-ostree-builder Project Breakdown
|
||||
|
||||
## Project Overview
|
||||
|
||||
The deb-ostree-builder is a sophisticated build system originally derived from Endless OS that creates bootable Debian systems using OSTree technology. It's designed as a "stripped down" version specifically for Debian, focusing on simplicity while maintaining the power of atomic system management.
|
||||
|
||||
## Repository Structure
|
||||
|
||||
Based on the project analysis, here's the inferred repository structure:
|
||||
|
||||
```
|
||||
deb-ostree-builder/
|
||||
├── deb-ostree-builder # Main build script (entry point)
|
||||
├── create-deployment # Deployment creation utility
|
||||
├── README.md # Project documentation
|
||||
├── buildscript # Build orchestration script
|
||||
├── config/ # Configuration directory
|
||||
│ ├── defaults.ini # Default build settings
|
||||
│ ├── product/ # Product-specific configurations
|
||||
│ │ ├── debian.ini
|
||||
│ │ ├── debianplatform.ini
|
||||
│ │ └── debiansdk.ini
|
||||
│ ├── branch/ # Branch-specific settings
|
||||
│ │ └── *.ini
|
||||
│ ├── arch/ # Architecture-specific settings
|
||||
│ │ ├── i386.ini
|
||||
│ │ └── armhf.ini
|
||||
│ ├── platform/ # Platform-specific settings
|
||||
│ │ └── *.ini
|
||||
│ └── local.ini # Local build overrides
|
||||
├── hooks/ # Customization hooks directory
|
||||
│ ├── cache/ # Update checking hooks
|
||||
│ ├── seed/ # Pre-bootstrap hooks
|
||||
│ ├── os/ # OS customization hooks
|
||||
│ │ ├── 10-debconf-set-selections.chroot
|
||||
│ │ ├── 50-install-packages*
|
||||
│ │ └── *.sh
|
||||
│ ├── ostree/ # OSTree-specific hooks
|
||||
│ ├── publish/ # Publishing hooks
|
||||
│ └── error/ # Error cleanup hooks
|
||||
├── lib/ # Library functions
|
||||
│ └── eob.sh # Core EOB (Endless OS Builder) functions
|
||||
├── helpers/ # Helper utilities
|
||||
│ └── core-packages # Core package management
|
||||
└── data/ # Static data files
|
||||
└── debootstrap.script # Custom debootstrap script
|
||||
```
|
||||
|
||||
## How It Builds a Debian OSTree System
|
||||
|
||||
### 1. **Multi-Stage Build Process**
|
||||
|
||||
The build system operates in distinct stages, each with specific responsibilities:
|
||||
|
||||
#### Stage 1: Check Update (cache hooks)
|
||||
- Determines if a build is needed by comparing cached facts
|
||||
- Checks modification times of cache files
|
||||
- Exits early if no changes detected
|
||||
|
||||
#### Stage 2: Seed (seed hooks)
|
||||
- Initializes empty `${EOB_ROOTDIR}` directory
|
||||
- Places any pre-bootstrap content
|
||||
- Prepares foundation for system construction
|
||||
|
||||
#### Stage 3: OS Construction (os hooks)
|
||||
- Creates the operating system using debootstrap
|
||||
- Installs packages via apt
|
||||
- Makes OSTree-specific modifications
|
||||
- Runs customization scripts in lexical order
|
||||
|
||||
#### Stage 4: OSTree Commit (ostree hooks)
|
||||
- Commits the prepared filesystem to OSTree repository
|
||||
- Handles checksumming and integrity verification
|
||||
- Creates immutable filesystem snapshots
|
||||
|
||||
#### Stage 5: Publish (publish hooks)
|
||||
- Publishes local OSTree repository to remote servers
|
||||
- Handles distribution and deployment
|
||||
|
||||
### 2. **Configuration System**
|
||||
|
||||
The build system uses a sophisticated INI-based configuration hierarchy:
|
||||
|
||||
```ini
|
||||
# Configuration Priority (later overrides earlier):
|
||||
config/defaults.ini # Base settings
|
||||
config/product/$product.ini # Product variants
|
||||
config/branch/$branch.ini # Branch-specific
|
||||
config/arch/$arch.ini # Architecture-specific
|
||||
config/platform/$platform.ini # Platform-specific
|
||||
/etc/deb-ostree-builder/config.ini # System-wide overrides
|
||||
config/local.ini # Local build overrides
|
||||
```
|
||||
|
||||
**Variable Interpolation Example:**
|
||||
```ini
|
||||
[build]
|
||||
suite = sid
|
||||
source = http://deb.debian.org/debian
|
||||
|
||||
[debian]
|
||||
# Uses value from build section
|
||||
packages = linux-image-${arch} ostree-boot
|
||||
mirror = ${source}
|
||||
```
|
||||
|
||||
### 3. **Package Management Integration**
|
||||
|
||||
The system handles Debian package management through multiple mechanisms:
|
||||
|
||||
#### Merged Configuration Options
|
||||
For managing lists of packages and hooks:
|
||||
```ini
|
||||
# Adding packages
|
||||
os:packages_add_base = systemd openssh-server
|
||||
os:packages_add_desktop = gnome-core firefox
|
||||
os:packages_del_unwanted = nano vim-tiny
|
||||
|
||||
# Final packages list = base + desktop - unwanted
|
||||
```
|
||||
|
||||
#### Package Installation Process
|
||||
1. **Debootstrap**: Creates minimal Debian system
|
||||
2. **Hook-based Installation**: Scripts at index 50 install additional packages
|
||||
3. **Chroot Operations**: Some scripts run inside the target system environment
|
||||
|
||||
### 4. **OSTree-Specific Transformations**
|
||||
|
||||
The builder implements several critical transformations to make Debian compatible with OSTree:
|
||||
|
||||
#### Filesystem Layout Changes
|
||||
```bash
|
||||
# Move /etc to /usr/etc (OSTree three-way merge requirement)
|
||||
mv "${BUILDDIR}"/etc "${BUILDDIR}"/usr/etc
|
||||
|
||||
# Create persistent directory symlinks
|
||||
ln -s /sysroot/home "${BUILDDIR}"/home
|
||||
ln -s /sysroot/root "${BUILDDIR}"/root
|
||||
ln -s /var/opt "${BUILDDIR}"/opt
|
||||
|
||||
# Move dpkg database to immutable location
|
||||
mv "${BUILDDIR}"/var/lib/dpkg "${BUILDDIR}"/usr/share/dpkg/database
|
||||
ln -sr "${BUILDDIR}"/usr/share/dpkg/database "${BUILDDIR}"/var/lib/dpkg
|
||||
```
|
||||
|
||||
#### Boot File Management
|
||||
```bash
|
||||
# Create OSTree-compatible boot file checksums
|
||||
csum=$(cat vmlinuz-* initrd.img-* | sha256sum --binary | awk '{print $1}')
|
||||
mv vmlinuz-* vmlinuz-*-${csum}
|
||||
mv initrd.img-* initramfs-*-${csum}
|
||||
```
|
||||
|
||||
### 5. **Hook System Architecture**
|
||||
|
||||
The hook system provides extensive customization capabilities:
|
||||
|
||||
#### Hook Types
|
||||
- **Executable Scripts**: Run directly if executable bit set
|
||||
- **Bash Scripts**: Sourced with access to library functions
|
||||
- **Chroot Scripts**: Run inside target system (`.chroot` suffix)
|
||||
|
||||
#### Hook Categories
|
||||
- **cache**: Update checking and caching logic
|
||||
- **seed**: Pre-bootstrap preparation
|
||||
- **os**: System construction and customization
|
||||
- **publish**: Repository publication
|
||||
- **error**: Cleanup and error handling
|
||||
|
||||
#### Example Hook Structure
|
||||
```bash
|
||||
# hooks/os/20-configure-system.sh
|
||||
#!/bin/bash
|
||||
# Configure system settings for OSTree
|
||||
|
||||
# Access to EOB environment variables
|
||||
echo "Building for architecture: ${EOB_ARCH}"
|
||||
echo "Target suite: ${EOB_SUITE}"
|
||||
|
||||
# Use library functions
|
||||
eob_info "Configuring system for OSTree compatibility"
|
||||
```
|
||||
|
||||
### 6. **Build Environment Management**
|
||||
|
||||
The system carefully manages the build environment:
|
||||
|
||||
#### Mount Management
|
||||
```bash
|
||||
# Mount necessary filesystems for chroot operations
|
||||
for dir in proc sys dev dev/pts; do
|
||||
mount --bind "/$dir" "$BUILDDIR/$dir"
|
||||
done
|
||||
|
||||
# Cleanup on exit
|
||||
trap cleanup_mounts EXIT
|
||||
```
|
||||
|
||||
#### Security Measures
|
||||
```bash
|
||||
# Prevent daemon startup during build
|
||||
cat > "$BUILDDIR"/usr/sbin/policy-rc.d <<EOF
|
||||
#!/bin/sh
|
||||
exit 101
|
||||
EOF
|
||||
```
|
||||
|
||||
## Key Innovations
|
||||
|
||||
### 1. **Simplified Build Logic**
|
||||
Unlike complex build systems, deb-ostree-builder uses straightforward bash scripts with clear stage separation, making it easy to understand and modify.
|
||||
|
||||
### 2. **Flexible Configuration**
|
||||
The hierarchical INI configuration system allows fine-grained control over different build variants while maintaining consistency.
|
||||
|
||||
### 3. **Hook-Based Extensibility**
|
||||
The modular hook system enables easy customization without modifying core build logic.
|
||||
|
||||
### 4. **Debian Integration**
|
||||
Seamless integration with Debian's package management while adapting to OSTree's requirements.
|
||||
|
||||
## Build Process Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start Build] --> B[Load Configuration]
|
||||
B --> C[Check Update Stage]
|
||||
C --> D{Update Needed?}
|
||||
D -->|No| E[Exit Early]
|
||||
D -->|Yes| F[Seed Stage]
|
||||
F --> G[Create Build Directory]
|
||||
G --> H[Run Seed Hooks]
|
||||
H --> I[OS Construction Stage]
|
||||
I --> J[Debootstrap Base System]
|
||||
J --> K[Mount Filesystems]
|
||||
K --> L[Run OS Hooks]
|
||||
L --> M[Package Installation]
|
||||
M --> N[System Configuration]
|
||||
N --> O[OSTree Transformations]
|
||||
O --> P[OSTree Commit Stage]
|
||||
P --> Q[Create OSTree Commit]
|
||||
Q --> R[Publish Stage]
|
||||
R --> S[Run Publish Hooks]
|
||||
S --> T[Upload to Remote]
|
||||
T --> U[Build Complete]
|
||||
```
|
||||
|
||||
## Technical Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **EOB Core**: Python-based orchestration engine
|
||||
2. **Hook System**: Bash script customization framework
|
||||
3. **Configuration Management**: INI-based hierarchical settings
|
||||
4. **OSTree Integration**: Filesystem transformation and commitment
|
||||
5. **Debian Toolchain**: Debootstrap, apt, dpkg integration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The build system communicates through environment variables:
|
||||
```bash
|
||||
EOB_ARCH=amd64 # Target architecture
|
||||
EOB_SUITE=sid # Debian suite
|
||||
EOB_ROOTDIR=/tmp/build # Build directory
|
||||
EOB_OSTREE_REPO=/ostree/repo # OSTree repository path
|
||||
```
|
||||
|
||||
### Library Functions
|
||||
|
||||
The `lib/eob.sh` provides utility functions for hooks:
|
||||
```bash
|
||||
eob_info() # Informational logging
|
||||
eob_error() # Error reporting
|
||||
eob_cachedir() # Cache directory path
|
||||
eob_cachefile() # Cache file naming
|
||||
```
|
||||
|
||||
## Advantages of This Approach
|
||||
|
||||
1. **Simplicity**: Bash-based core with minimal abstraction
|
||||
2. **Transparency**: Complete understanding of build process possible
|
||||
3. **Flexibility**: Easy modification through hooks and configuration
|
||||
4. **Reliability**: Atomic updates with rollback capability
|
||||
5. **Security**: Immutable root filesystem with verified boot
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Development Systems**: Testing Debian packages in isolated environments
|
||||
- **Production Deployments**: Atomic updates for server systems
|
||||
- **Embedded Systems**: Reliable OS updates for IoT devices
|
||||
- **Cloud Infrastructure**: Immutable infrastructure deployments
|
||||
|
||||
This architecture provides a robust foundation for building and managing Debian-based OSTree systems while maintaining the simplicity and transparency that makes the system maintainable and extensible.
|
||||
Loading…
Add table
Add a link
Reference in a new issue