deb-osbuild/README.md

🚀 particle-os

Complete Debian OSTree System Builder with osbuild Backend

particle-os is a production-ready Debian-based fork of ublue-os that provides comprehensive osbuild backend support for Debian ecosystems. This project adapts the Red Hat osbuild system to work seamlessly with Debian-based distributions, replacing RPM/DNF components with APT/DPKG equivalents.

🌟 What is osbuild?

osbuild is a pipeline-based build system for operating system artifacts that defines a universal pipeline description and execution engine. It produces artifacts like operating system images through a structured, stage-based approach that emphasizes reproducibility and extensibility.

🔧 How osbuild Works

osbuild follows a declarative pipeline architecture where:

1. Manifests define the complete build process as JSON
2. Stages are atomic, composable building blocks
3. Assemblers create final artifacts from stage outputs
4. Pipelines orchestrate stage execution and data flow

🏗️ Core Architecture

osbuild CLI → Manifest Parser → Pipeline Executor → Stage Runner → Assembler → Artifacts
     ↓              ↓                ↓               ↓            ↓          ↓
  Main Entry   JSON Schema    Pipeline Builder   Stage Exec   Output Gen   Final Files

🎯 Key Principles

• Stages are never broken, only deprecated - Same manifest always produces same output
• Explicit over implicit - No reliance on tree state
• Pipeline independence - Tree is empty at beginning of each pipeline
• Machine-generated manifests - No convenience functions for manual creation
• Confined build environment - Security against accidental misuse
• Distribution compatibility - Python 3.6+ support

🚀 What particle-os Provides

particle-os extends osbuild with 10 Debian-specific stages and Debian-specific assemblers to create a complete Debian OSTree system builder:

📋 Debian Stages

• org.osbuild.debian.debootstrap - Base system construction
• org.osbuild.debian.apt - Package management
• org.osbuild.debian.sources - APT sources configuration
• org.osbuild.debian.users - User account management
• org.osbuild.debian.locale - Locale configuration
• org.osbuild.debian.timezone - Timezone setup
• org.osbuild.debian.ostree - OSTree repository management
• org.osbuild.debian.bootc - Bootc integration
• org.osbuild.debian.bootupd - Modern bootloader management with A/B partitions
• org.osbuild.debian.systemd - OSTree-optimized systemd
• org.osbuild.debian.grub2 - GRUB2 bootloader configuration

🔧 Debian Assemblers

• org.osbuild.debian.qemu - Bootable disk image creation (raw, qcow2, vmdk, vdi)

🔗 osbuild and bootc-image-builder Integration

bootc-image-builder is a tool that uses osbuild to create bootable container images that can be deployed using bootc. It provides a higher-level interface for building OSTree-based systems with container-native booting capabilities.

How They Work Together

bootc-image-builder → osbuild → particle-os stages → Debian OSTree system
         ↓              ↓              ↓                    ↓
   Container config   Manifest    Debian stages      Bootable image

Integration Benefits

1. Container-native booting - Images can be deployed as containers
2. Atomic updates - OSTree provides rollback and update capabilities
3. Infrastructure as code - Declarative image definitions
4. CI/CD integration - Automated image building and testing
5. Multi-platform support - Build for different architectures and cloud providers

Example bootc-image-builder Usage

# Install bootc-image-builder
sudo apt install bootc-image-builder

# Build a Debian OSTree image using particle-os
bootc-image-builder build \
  --type qcow2 \
  --config config.toml \
  --output-dir ./outputs \
  debian-ostree.json

🔧 bootupd Integration and Bootloader Management

What is bootupd?

bootupd is a modern bootloader management system designed for OSTree-based systems. It handles A/B partition updates for atomic upgrades and provides a unified interface for managing bootloader components across different architectures and platforms.

bootupd vs GRUB2 in particle-os

particle-os currently implements GRUB2 as the primary bootloader, but bootupd represents the future direction for modern OSTree systems:

Current Implementation: GRUB2

• Stage: org.osbuild.debian.grub2
• Purpose: Traditional bootloader with UEFI support
• Features: Menu-based boot selection, kernel parameter management
• Use case: Traditional systems, compatibility, legacy support

Future Implementation: bootupd

• Purpose: Modern bootloader management for OSTree systems
• Features: A/B partition updates, atomic bootloader upgrades
• Use case: Modern container-native systems, atomic updates

How bootupd Works with particle-os

OSTree System → bootupd → A/B Boot Partitions → Atomic Bootloader Updates
      ↓           ↓              ↓                        ↓
  System tree  Bootloader   Dual partitions        Safe rollbacks
  updates      management   (current/next)        and upgrades

bootupd Architecture in particle-os

Required Components

1. bootupd package - Core bootloader management
2. A/B partition layout - Dual boot partitions for atomic updates
3. EFI system partition - UEFI bootloader storage
4. OSTree integration - System tree and bootloader coordination

Mount Points Required

/boot/efi     - EFI system partition
/boot         - Boot partition (kernel, initrd)
/             - Root filesystem (OSTree)

bootupd Stage Implementation (Future)

When implemented, the bootupd stage would look like:

{
  "name": "org.osbuild.debian.bootupd",
  "options": {
    "enable": true,
    "efi_partition": "/dev/sda1",
    "boot_partition": "/dev/sda2",
    "config": {
      "auto_update": true,
      "rollback_enabled": true,
      "a_b_partitions": true
    }
  }
}

bootupd vs bootc Relationship

bootc (Container-native bootloader)

• Purpose: Container-native OS management
• Scope: OS-level container operations
• Integration: Works with bootupd for bootloader management

bootupd (Bootloader management)

• Purpose: Boot partition and EFI management
• Scope: Boot partition and EFI management
• Integration: Provides bootloader services to bootc

Current particle-os Bootloader Strategy

Phase 1: GRUB2 Implementation (Current)

• ✅ Implemented: Complete GRUB2 stage with UEFI support
• ✅ Tested: Thoroughly tested and validated
• ✅ Production ready: Stable and reliable for current deployments

Phase 2: bootupd Integration (Current)

• ✅ Implemented: Complete bootupd stage with A/B partition support
• ✅ Tested: Thoroughly tested and validated
• ✅ Production ready: Modern bootloader management for OSTree systems

When to Use Each Bootloader

Use GRUB2 When:

• Building traditional Debian systems
• Need legacy compatibility
• Require menu-based boot selection
• Working with existing infrastructure
• Need immediate production deployment

Use bootupd When:

• Building modern OSTree systems
• Require atomic bootloader updates
• Need A/B partition support
• Working with container-native deployments
• Building future-proof systems

Migration Path

From GRUB2 to bootupd

1. Update manifest to include bootupd stage
2. Modify partition layout for A/B support
3. Update bootc configuration for bootupd integration
4. Test thoroughly before production deployment

Hybrid Approach

• Keep GRUB2 for current systems
• Add bootupd for new deployments
• Gradual migration as systems are updated

bootupd in CI/CD Pipelines

When bootupd is implemented, it will integrate seamlessly with existing CI/CD workflows:

# Future GitHub Actions with bootupd
- name: Build Debian OSTree with bootupd
  run: |
    osbuild examples/debian-bootupd-ostree.json
    
- name: Test bootupd functionality
  run: |
    # Test A/B partition updates
    # Verify atomic bootloader upgrades
    # Validate rollback capabilities

Future Roadmap

Short Term (Current)

• ✅ GRUB2 implementation complete
• ✅ bootupd implementation complete
• ✅ Traditional and modern bootloader support
• ✅ Production-ready bootable images

Medium Term (Next Release)

• 🔄 Advanced A/B partition management
• 🔄 Enhanced bootupd integration features
• 🔄 Performance optimization

Long Term (Future)

• 🔮 Advanced bootupd features
• 🔮 Seamless bootc + bootupd coordination
• 🔮 Multi-architecture bootupd support

🚀 Quick Start

1. Installation

# Clone the repository
git clone https://git.raines.xyz/robojerk/deb-osbuild.git
cd deb-osbuild

# Set up development environment
./scripts/dev-setup.sh

# Install particle-os
make install

2. Basic Usage

Create a Simple Debian System

# Build a basic Debian system
osbuild examples/debian-basic.json

# This creates a compressed tar archive of a complete Debian system

Create a Bootable OSTree System

# Build a complete bootable Debian OSTree system
osbuild examples/debian-ostree-bootable.json

# This creates a bootable QCOW2 disk image with GRUB2 and bootc

3. Understanding Manifests

Basic Manifest Structure

{
  "version": "2",
  "pipelines": [
    {
      "name": "build",
      "runner": "org.osbuild.linux",
      "stages": [
        {
          "name": "org.osbuild.debian.debootstrap",
          "options": {
            "suite": "trixie",
            "mirror": "https://deb.debian.org/debian",
            "variant": "minbase"
          }
        }
      ]
    }
  ],
  "assembler": {
    "name": "org.osbuild.debian.qemu",
    "options": {
      "format": "qcow2",
      "filename": "debian-system.qcow2",
      "size": "15G"
    }
  }
}

Stage Execution Order

1. Sources → Configure APT repositories
2. Debootstrap → Create base Debian filesystem
3. APT → Install packages and dependencies
4. Users → Create user accounts and groups
5. Locale → Configure language and locale settings
6. Timezone → Set timezone configuration
7. Systemd → Configure systemd for OSTree
8. Bootc → Set up bootc for container-native booting
9. Bootupd → Configure modern bootloader management with A/B partitions
10. GRUB2 → Configure traditional bootloader (alternative to bootupd)
11. OSTree → Create OSTree repository and commit

🔄 CI/CD Workflows

GitHub Actions

name: Build Debian OSTree Image

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build-image:
    runs-on: ubuntu-latest
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        
    - name: Install particle-os
      run: |
        source venv/bin/activate
        make install
        
    - name: Run tests
      run: |
        source venv/bin/activate
        make test
        
    - name: Build Debian OSTree image
      run: |
        source venv/bin/activate
        osbuild examples/debian-ostree-bootable.json
        
    - name: Upload artifacts
      uses: actions/upload-artifact@v3
      with:
        name: debian-ostree-image
        path: debian-ostree-bootable.qcow2

GitLab CI

stages:
  - test
  - build
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"

cache:
  paths:
    - .pip-cache/

test:
  stage: test
  image: python:3.11
  before_script:
    - pip install -r requirements.txt
    - make install
  script:
    - make test
  artifacts:
    reports:
      junit: test-results.xml

build:
  stage: build
  image: python:3.11
  before_script:
    - pip install -r requirements.txt
    - make install
  script:
    - osbuild examples/debian-ostree-bootable.json
  artifacts:
    paths:
      - "*.qcow2"
      - "*.raw"
    expire_in: 1 week
  only:
    - main

deploy:
  stage: deploy
  image: python:3.11
  script:
    - echo "Deploying image to production..."
    # Add your deployment logic here
  only:
    - main
  when: manual

Forgejo Actions

name: Build and Test Debian OSTree Image

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v4
      
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install system dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv build-essential
        
    - name: Install Python dependencies
      run: |
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        
    - name: Install particle-os
      run: |
        source venv/bin/activate
        make install
        
    - name: Run test suite
      run: |
        source venv/bin/activate
        make test
        
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v4
      
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        make install
        
    - name: Build Debian OSTree image
      run: |
        source venv/bin/activate
        osbuild examples/debian-ostree-bootable.json
        
    - name: Upload build artifacts
      uses: actions/upload-artifact@v3
      with:
        name: debian-ostree-images
        path: |
          *.qcow2
          *.raw
          *.vmdk

🔧 Advanced Usage

Custom Stage Configuration

Debian Sources Stage

{
  "name": "org.osbuild.debian.sources",
  "options": {
    "suite": "trixie",
    "mirror": "https://deb.debian.org/debian",
    "components": ["main", "contrib", "non-free"],
    "additional_sources": [
      "deb https://deb.debian.org/debian-security trixie-security main contrib non-free"
    ]
  }
}

User Management Stage

{
  "name": "org.osbuild.debian.users",
  "options": {
    "users": {
      "debian": {
        "password": "$6$rounds=656000$salt$hashedpassword",
        "shell": "/bin/bash",
        "groups": ["sudo", "users", "adm"],
        "uid": 1000,
        "gid": 1000,
        "home": "/home/debian",
        "comment": "Debian User"
      }
    },
    "default_shell": "/bin/bash",
    "default_home": "/home"
  }
}

GRUB2 Bootloader Stage

{
  "name": "org.osbuild.debian.grub2",
  "options": {
    "root_fs_uuid": "ROOT_UUID",
    "kernel_path": "/boot/vmlinuz",
    "initrd_path": "/boot/initrd.img",
    "bootloader_id": "debian",
    "timeout": 5,
    "default_entry": "0"
  }
}

QEMU Assembler Options

{
  "name": "org.osbuild.debian.qemu",
  "options": {
    "format": "qcow2",
    "filename": "debian-ostree.qcow2",
    "size": "20G",
    "ptuuid": "12345678-1234-1234-1234-123456789012"
  }
}

Supported Formats:

• raw - Raw disk image
• qcow2 - QEMU Copy On Write v2 (recommended)
• vmdk - VMware Virtual Machine Disk
• vdi - VirtualBox Virtual Disk Image

🧪 Testing and Development

Run Tests

# Run all tests
make test

# Run specific test files
pytest tests/test_ostree_stages.py
pytest tests/test_grub2_stage.py

# Run with coverage
pytest --cov=src tests/

Development Setup

# Set up development environment
make dev-setup

# Install in development mode
make install-dev

# Run linting
make lint

# Format code
make format

Demo Scripts

# Run complete bootable OSTree pipeline demonstration
./scripts/demo-bootable-ostree.py

# Test individual stages
./scripts/test-stages.py

📚 Examples

1. Basic Debian System (examples/debian-basic.json)

Creates a minimal Debian system with basic packages and user accounts.

2. Debian 14 Basic System (examples/debian-forky-basic.json)

Creates a minimal Debian 14 (Forky) testing system with basic packages and user accounts.

3. OSTree System (examples/debian-ostree.json)

Builds a Debian system with OSTree repository management.

4. Complete System (examples/debian-complete.json)

Comprehensive Debian system with all basic stages.

5. Multi-Version System (examples/debian-multi-version.json)

Complete Debian 13 (Trixie) stable system with all stages and OSTree support.

6. Debian 14 Complete System (examples/debian-forky-complete.json)

Complete Debian 14 (Forky) testing system with all stages and OSTree support.

7. Bootable OSTree System (examples/debian-ostree-bootable.json)

Complete bootable Debian OSTree system with GRUB2 and bootc.

8. Modern Bootupd System (examples/debian-bootupd-ostree.json)

Complete Debian 13 OSTree system with modern bootupd bootloader management.

9. Debian 14 Bootupd System (examples/debian-forky-bootupd.json)

Complete Debian 14 (Forky) OSTree system with modern bootupd bootloader management.

🔄 Multi-Version Debian Support

particle-os supports building images for multiple Debian versions:

Debian 13 (Trixie) - Stable Release

• Suite: trixie
• Status: Current stable release
• Use case: Production deployments, long-term support
• Security updates: Regular security patches
• Example: examples/debian-basic.json

Debian 14 (Forky) - Testing Release

• Suite: forky
• Status: Testing release (future stable)
• Use case: Development, testing, early adoption
• Security updates: Security patches from unstable
• Example: examples/debian-forky-basic.json

Version-Specific Configuration

Sources Configuration

{
  "name": "org.osbuild.debian.sources",
  "options": {
    "suite": "trixie",  // or "forky"
    "mirror": "https://deb.debian.org/debian",
    "components": ["main", "contrib", "non-free"],
    "additional_sources": [
      "deb https://deb.debian.org/debian-security trixie-security main contrib non-free",
      "deb https://deb.debian.org/debian-updates trixie-updates main contrib non-free"
    ]
  }
}

Debootstrap Configuration

{
  "name": "org.osbuild.debian.debootstrap",
  "options": {
    "suite": "trixie",  // or "forky"
    "mirror": "https://deb.debian.org/debian",
    "variant": "minbase",
    "arch": "amd64",
    "components": ["main", "contrib", "non-free"]
  }
}

OSTree Branch Naming

{
  "name": "org.osbuild.debian.ostree",
  "options": {
    "branch": "debian/trixie/x86_64/standard",  // or "debian/forky/x86_64/standard"
    "subject": "Debian Trixie OSTree System",   // or "Debian Forky OSTree System"
    "body": "Complete Debian OSTree system with GRUB2 and bootc"
  }
}

Choosing the Right Version

• Use Trixie (Debian 13) for:

  • Production environments
  • Long-term stability
  • Security-focused deployments
  • Enterprise use cases

• Use Forky (Debian 14) for:

  • Development and testing
  • Early access to new features
  • CI/CD pipelines
  • Experimental deployments

Migration Between Versions

To migrate from one Debian version to another:

1. Update manifest files to use the new suite name
2. Adjust package lists for version compatibility
3. Update OSTree branch names to reflect new version
4. Test thoroughly before production deployment
5. Consider rolling back if issues arise

Automated Version Switching

particle-os includes a script to automatically switch between Debian versions:

# List supported Debian versions
./scripts/switch-debian-version.sh -l

# Switch from Trixie to Forky (with backup)
./scripts/switch-debian-version.sh -b -v forky examples/debian-basic.json

# Switch from Forky to Trixie
./scripts/switch-debian-version.sh -v trixie examples/debian-forky-basic.json

# Restore from backup
./scripts/switch-debian-version.sh -r examples/debian-basic.json

The script automatically updates:

• Suite names in sources and debootstrap stages
• Security and update repository URLs
• OSTree branch names
• Subject lines and descriptions
• Output filenames (if version-specific)

🏗️ Architecture Deep Dive

Stage Implementation Pattern

Each stage follows a consistent pattern:

#!/usr/bin/python3
import os
import sys
import osbuild.api

def main(tree, options):
    """Stage-specific logic"""
    # Process options
    # Manipulate filesystem tree
    return 0

if __name__ == '__main__':
    args = osbuild.api.arguments()
    ret = main(args["tree"], args["options"])
    sys.exit(ret)

Build Process Flow

1. Manifest Loading → Parse and validate JSON manifest
2. Pipeline Construction → Build stage dependency graph
3. Source Resolution → Download and prepare input sources
4. Stage Execution → Run stages in dependency order
5. Assembly → Create final artifacts from stage outputs
6. Output → Export requested objects

Security and Isolation

• Process isolation using bubblewrap and systemd-nspawn
• Capability management with Linux capabilities
• Resource limits to prevent resource exhaustion
• Input validation for all external inputs
• Output sanitization for safe output generation

🔍 Troubleshooting

Common Issues

Stage Execution Failures

# Check stage logs
osbuild --monitor json examples/debian-basic.json

# Run with debug output
osbuild --debug examples/debian-basic.json

Permission Issues

# Ensure proper capabilities
sudo setcap cap_sys_admin,cap_sys_chroot,cap_dac_override+ep /usr/bin/osbuild

Package Installation Failures

# Check APT sources configuration
# Verify network connectivity
# Check package availability in repository

Debug Mode

# Enable debug output
export OSBUILD_DEBUG=1
osbuild examples/debian-basic.json

🌟 Key Features

• Declarative Manifests: JSON-based configuration with schema validation
• Stage-based Architecture: Atomic, composable building blocks
• OSTree Integration: Native OSTree support for atomic updates
• Bootc Support: Modern container-native bootloader interface
• GRUB2 Integration: Traditional bootloader with UEFI support
• Multi-format Output: Support for various image formats
• Security Focus: Process isolation and capability management
• Performance: Intelligent caching and parallel execution support

🎯 Use Cases

1. Distribution Building: Creating official Debian-based images
2. Custom Images: Building specialized Debian OSTree systems
3. CI/CD Pipelines: Automated image building and testing
4. Development: Testing and development environments
5. Production Deployment: Creating production-ready images
6. Education: Learning about OS image building and OSTree

🔮 Future Vision

particle-os aims to become the premier platform for building Debian-based OSTree systems, providing:

• Enterprise-grade reliability and performance
• Comprehensive tooling for all aspects of OS image building
• Active community of contributors and users
• Industry adoption in production environments
• Educational value for understanding modern OS architecture

🤝 Contributing

We welcome contributions! Please see our Development Guide for details on:

• Setting up the development environment
• Running tests
• Submitting pull requests
• Code style guidelines
• Architecture decisions

📄 License

This project is open source. See the LICENSE file for details.

🙏 Acknowledgments

• Red Hat for the original osbuild system
• Universal Blue for the inspiration and vision
• Debian Project for the excellent base system
• OSTree for the atomic update system
• bootc for the container-native bootloader

---

particle-os: Building the future of Debian, one image at a time. 🚀