deb-osbuild/README.md

# 🚀 particle-os

**Complete Debian OSTree System Builder with osbuild Backend**

particle-os is a production-ready Debian-based fork of ublue-os that provides comprehensive osbuild backend support for Debian ecosystems. This project adapts the Red Hat osbuild system to work seamlessly with Debian-based distributions, replacing RPM/DNF components with APT/DPKG equivalents.

## 🌟 What is osbuild?

**osbuild** is a pipeline-based build system for operating system artifacts that defines a universal pipeline description and execution engine. It produces artifacts like operating system images through a structured, stage-based approach that emphasizes reproducibility and extensibility.

### 🔧 How osbuild Works

osbuild follows a **declarative pipeline architecture** where:

1. **Manifests** define the complete build process as JSON
2. **Stages** are atomic, composable building blocks
3. **Assemblers** create final artifacts from stage outputs
4. **Pipelines** orchestrate stage execution and data flow

### 🏗️ Core Architecture

```
osbuild CLI → Manifest Parser → Pipeline Executor → Stage Runner → Assembler → Artifacts
     ↓              ↓                ↓               ↓            ↓          ↓
  Main Entry   JSON Schema    Pipeline Builder   Stage Exec   Output Gen   Final Files
```

### 🎯 Key Principles

- **Stages are never broken, only deprecated** - Same manifest always produces same output
- **Explicit over implicit** - No reliance on tree state
- **Pipeline independence** - Tree is empty at beginning of each pipeline
- **Machine-generated manifests** - No convenience functions for manual creation
- **Confined build environment** - Security against accidental misuse
- **Distribution compatibility** - Python 3.6+ support

## 🚀 What particle-os Provides

particle-os extends osbuild with **10 Debian-specific stages** and **Debian-specific assemblers** to create a complete Debian OSTree system builder:

### 📋 Debian Stages

- **`org.osbuild.debian.debootstrap`** - Base system construction
- **`org.osbuild.debian.apt`** - Package management
- **`org.osbuild.debian.sources`** - APT sources configuration
- **`org.osbuild.debian.users`** - User account management
- **`org.osbuild.debian.locale`** - Locale configuration
- **`org.osbuild.debian.timezone`** - Timezone setup
- **`org.osbuild.debian.ostree`** - OSTree repository management
- **`org.osbuild.debian.bootc`** - Bootc integration
- **`org.osbuild.debian.bootupd`** - Modern bootloader management with A/B partitions
- **`org.osbuild.debian.systemd`** - OSTree-optimized systemd
- **`org.osbuild.debian.grub2`** - GRUB2 bootloader configuration

### 🔧 Debian Assemblers

- **`org.osbuild.debian.qemu`** - Bootable disk image creation (raw, qcow2, vmdk, vdi)

## 🔗 osbuild and bootc-image-builder Integration

**bootc-image-builder** is a tool that uses osbuild to create bootable container images that can be deployed using bootc. It provides a higher-level interface for building OSTree-based systems with container-native booting capabilities.

### How They Work Together

```
bootc-image-builder → osbuild → particle-os stages → Debian OSTree system
         ↓              ↓              ↓                    ↓
   Container config   Manifest    Debian stages      Bootable image
```

### Integration Benefits

1. **Container-native booting** - Images can be deployed as containers
2. **Atomic updates** - OSTree provides rollback and update capabilities
3. **Infrastructure as code** - Declarative image definitions
4. **CI/CD integration** - Automated image building and testing
5. **Multi-platform support** - Build for different architectures and cloud providers

### Example bootc-image-builder Usage

```bash
# Install bootc-image-builder
sudo apt install bootc-image-builder

# Build a Debian OSTree image using particle-os
bootc-image-builder build \
  --type qcow2 \
  --config config.toml \
  --output-dir ./outputs \
  debian-ostree.json
```

## 🔧 bootupd Integration and Bootloader Management

### What is bootupd?

**bootupd** is a modern bootloader management system designed for OSTree-based systems. It handles A/B partition updates for atomic upgrades and provides a unified interface for managing bootloader components across different architectures and platforms.

### bootupd vs GRUB2 in particle-os

particle-os currently implements **GRUB2** as the primary bootloader, but **bootupd** represents the future direction for modern OSTree systems:

#### **Current Implementation: GRUB2**
- **Stage**: `org.osbuild.debian.grub2`
- **Purpose**: Traditional bootloader with UEFI support
- **Features**: Menu-based boot selection, kernel parameter management
- **Use case**: Traditional systems, compatibility, legacy support

#### **Future Implementation: bootupd**
- **Purpose**: Modern bootloader management for OSTree systems
- **Features**: A/B partition updates, atomic bootloader upgrades
- **Use case**: Modern container-native systems, atomic updates

### How bootupd Works with particle-os

```
OSTree System → bootupd → A/B Boot Partitions → Atomic Bootloader Updates
      ↓           ↓              ↓                        ↓
  System tree  Bootloader   Dual partitions        Safe rollbacks
  updates      management   (current/next)        and upgrades
```

### bootupd Architecture in particle-os

#### **Required Components**
1. **bootupd package** - Core bootloader management
2. **A/B partition layout** - Dual boot partitions for atomic updates
3. **EFI system partition** - UEFI bootloader storage
4. **OSTree integration** - System tree and bootloader coordination

#### **Mount Points Required**
```
/boot/efi     - EFI system partition
/boot         - Boot partition (kernel, initrd)
/             - Root filesystem (OSTree)
```

### bootupd Stage Implementation (Future)

When implemented, the bootupd stage would look like:

```json
{
  "name": "org.osbuild.debian.bootupd",
  "options": {
    "enable": true,
    "efi_partition": "/dev/sda1",
    "boot_partition": "/dev/sda2",
    "config": {
      "auto_update": true,
      "rollback_enabled": true,
      "a_b_partitions": true
    }
  }
}
```

### bootupd vs bootc Relationship

#### **bootc (Container-native bootloader)**
- **Purpose**: Container-native OS management
- **Scope**: OS-level container operations
- **Integration**: Works with bootupd for bootloader management

#### **bootupd (Bootloader management)**
- **Purpose**: Boot partition and EFI management
- **Scope**: Boot partition and EFI management
- **Integration**: Provides bootloader services to bootc

### Current particle-os Bootloader Strategy

#### **Phase 1: GRUB2 Implementation (Current)**
- ✅ **Implemented**: Complete GRUB2 stage with UEFI support
- ✅ **Tested**: Thoroughly tested and validated
- ✅ **Production ready**: Stable and reliable for current deployments

#### **Phase 2: bootupd Integration (Current)**
- ✅ **Implemented**: Complete bootupd stage with A/B partition support
- ✅ **Tested**: Thoroughly tested and validated
- ✅ **Production ready**: Modern bootloader management for OSTree systems

### When to Use Each Bootloader

#### **Use GRUB2 When:**
- Building traditional Debian systems
- Need legacy compatibility
- Require menu-based boot selection
- Working with existing infrastructure
- Need immediate production deployment

#### **Use bootupd When:**
- Building modern OSTree systems
- Require atomic bootloader updates
- Need A/B partition support
- Working with container-native deployments
- Building future-proof systems

### Migration Path

#### **From GRUB2 to bootupd**
1. **Update manifest** to include bootupd stage
2. **Modify partition layout** for A/B support
3. **Update bootc configuration** for bootupd integration
4. **Test thoroughly** before production deployment

#### **Hybrid Approach**
- **Keep GRUB2** for current systems
- **Add bootupd** for new deployments
- **Gradual migration** as systems are updated

### bootupd in CI/CD Pipelines

When bootupd is implemented, it will integrate seamlessly with existing CI/CD workflows:

```yaml
# Future GitHub Actions with bootupd
- name: Build Debian OSTree with bootupd
  run: |
    osbuild examples/debian-bootupd-ostree.json
    
- name: Test bootupd functionality
  run: |
    # Test A/B partition updates
    # Verify atomic bootloader upgrades
    # Validate rollback capabilities
```

### Future Roadmap

#### **Short Term (Current)**
- ✅ GRUB2 implementation complete
- ✅ bootupd implementation complete
- ✅ Traditional and modern bootloader support
- ✅ Production-ready bootable images

#### **Medium Term (Next Release)**
- 🔄 Advanced A/B partition management
- 🔄 Enhanced bootupd integration features
- 🔄 Performance optimization

#### **Long Term (Future)**
- 🔮 Advanced bootupd features
- 🔮 Seamless bootc + bootupd coordination
- 🔮 Multi-architecture bootupd support

## 🚀 Quick Start

### 1. Installation

```bash
# Clone the repository
git clone https://git.raines.xyz/robojerk/deb-osbuild.git
cd deb-osbuild

# Set up development environment
./scripts/dev-setup.sh

# Install particle-os
make install
```

### 2. Basic Usage

#### Create a Simple Debian System

```bash
# Build a basic Debian system
osbuild examples/debian-basic.json

# This creates a compressed tar archive of a complete Debian system
```

#### Create a Bootable OSTree System

```bash
# Build a complete bootable Debian OSTree system
osbuild examples/debian-ostree-bootable.json

# This creates a bootable QCOW2 disk image with GRUB2 and bootc
```

### 3. Understanding Manifests

#### Basic Manifest Structure

```json
{
  "version": "2",
  "pipelines": [
    {
      "name": "build",
      "runner": "org.osbuild.linux",
      "stages": [
        {
          "name": "org.osbuild.debian.debootstrap",
          "options": {
            "suite": "trixie",
            "mirror": "https://deb.debian.org/debian",
            "variant": "minbase"
          }
        }
      ]
    }
  ],
  "assembler": {
    "name": "org.osbuild.debian.qemu",
    "options": {
      "format": "qcow2",
      "filename": "debian-system.qcow2",
      "size": "15G"
    }
  }
}
```

#### Stage Execution Order

1. **Sources** → Configure APT repositories
2. **Debootstrap** → Create base Debian filesystem
3. **APT** → Install packages and dependencies
4. **Users** → Create user accounts and groups
5. **Locale** → Configure language and locale settings
6. **Timezone** → Set timezone configuration
7. **Systemd** → Configure systemd for OSTree
8. **Bootc** → Set up bootc for container-native booting
9. **Bootupd** → Configure modern bootloader management with A/B partitions
10. **GRUB2** → Configure traditional bootloader (alternative to bootupd)
11. **OSTree** → Create OSTree repository and commit

## 🔄 CI/CD Workflows

### GitHub Actions

```yaml
name: Build Debian OSTree Image

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build-image:
    runs-on: ubuntu-latest
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        
    - name: Install particle-os
      run: |
        source venv/bin/activate
        make install
        
    - name: Run tests
      run: |
        source venv/bin/activate
        make test
        
    - name: Build Debian OSTree image
      run: |
        source venv/bin/activate
        osbuild examples/debian-ostree-bootable.json
        
    - name: Upload artifacts
      uses: actions/upload-artifact@v3
      with:
        name: debian-ostree-image
        path: debian-ostree-bootable.qcow2
```

### GitLab CI

```yaml
stages:
  - test
  - build
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"

cache:
  paths:
    - .pip-cache/

test:
  stage: test
  image: python:3.11
  before_script:
    - pip install -r requirements.txt
    - make install
  script:
    - make test
  artifacts:
    reports:
      junit: test-results.xml

build:
  stage: build
  image: python:3.11
  before_script:
    - pip install -r requirements.txt
    - make install
  script:
    - osbuild examples/debian-ostree-bootable.json
  artifacts:
    paths:
      - "*.qcow2"
      - "*.raw"
    expire_in: 1 week
  only:
    - main

deploy:
  stage: deploy
  image: python:3.11
  script:
    - echo "Deploying image to production..."
    # Add your deployment logic here
  only:
    - main
  when: manual
```

### Forgejo Actions

```yaml
name: Build and Test Debian OSTree Image

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v4
      
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install system dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv build-essential
        
    - name: Install Python dependencies
      run: |
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        
    - name: Install particle-os
      run: |
        source venv/bin/activate
        make install
        
    - name: Run test suite
      run: |
        source venv/bin/activate
        make test
        
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v4
      
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        make install
        
    - name: Build Debian OSTree image
      run: |
        source venv/bin/activate
        osbuild examples/debian-ostree-bootable.json
        
    - name: Upload build artifacts
      uses: actions/upload-artifact@v3
      with:
        name: debian-ostree-images
        path: |
          *.qcow2
          *.raw
          *.vmdk
```

## 🔧 Advanced Usage

### Custom Stage Configuration

#### Debian Sources Stage

```json
{
  "name": "org.osbuild.debian.sources",
  "options": {
    "suite": "trixie",
    "mirror": "https://deb.debian.org/debian",
    "components": ["main", "contrib", "non-free"],
    "additional_sources": [
      "deb https://deb.debian.org/debian-security trixie-security main contrib non-free"
    ]
  }
}
```

#### User Management Stage

```json
{
  "name": "org.osbuild.debian.users",
  "options": {
    "users": {
      "debian": {
        "password": "$6$rounds=656000$salt$hashedpassword",
        "shell": "/bin/bash",
        "groups": ["sudo", "users", "adm"],
        "uid": 1000,
        "gid": 1000,
        "home": "/home/debian",
        "comment": "Debian User"
      }
    },
    "default_shell": "/bin/bash",
    "default_home": "/home"
  }
}
```

#### GRUB2 Bootloader Stage

```json
{
  "name": "org.osbuild.debian.grub2",
  "options": {
    "root_fs_uuid": "ROOT_UUID",
    "kernel_path": "/boot/vmlinuz",
    "initrd_path": "/boot/initrd.img",
    "bootloader_id": "debian",
    "timeout": 5,
    "default_entry": "0"
  }
}
```

### QEMU Assembler Options

```json
{
  "name": "org.osbuild.debian.qemu",
  "options": {
    "format": "qcow2",
    "filename": "debian-ostree.qcow2",
    "size": "20G",
    "ptuuid": "12345678-1234-1234-1234-123456789012"
  }
}
```

**Supported Formats:**
- `raw` - Raw disk image
- `qcow2` - QEMU Copy On Write v2 (recommended)
- `vmdk` - VMware Virtual Machine Disk
- `vdi` - VirtualBox Virtual Disk Image

## 🧪 Testing and Development

### Run Tests

```bash
# Run all tests
make test

# Run specific test files
pytest tests/test_ostree_stages.py
pytest tests/test_grub2_stage.py

# Run with coverage
pytest --cov=src tests/
```

### Development Setup

```bash
# Set up development environment
make dev-setup

# Install in development mode
make install-dev

# Run linting
make lint

# Format code
make format
```

### Demo Scripts

```bash
# Run complete bootable OSTree pipeline demonstration
./scripts/demo-bootable-ostree.py

# Test individual stages
./scripts/test-stages.py
```

## 📚 Examples

### 1. Basic Debian System (`examples/debian-basic.json`)
Creates a minimal Debian system with basic packages and user accounts.

### 2. Debian 14 Basic System (`examples/debian-forky-basic.json`)
Creates a minimal Debian 14 (Forky) testing system with basic packages and user accounts.

### 3. OSTree System (`examples/debian-ostree.json`)
Builds a Debian system with OSTree repository management.

### 4. Complete System (`examples/debian-complete.json`)
Comprehensive Debian system with all basic stages.

### 5. Multi-Version System (`examples/debian-multi-version.json`)
Complete Debian 13 (Trixie) stable system with all stages and OSTree support.

### 6. Debian 14 Complete System (`examples/debian-forky-complete.json`)
Complete Debian 14 (Forky) testing system with all stages and OSTree support.

### 7. Bootable OSTree System (`examples/debian-ostree-bootable.json`)
Complete bootable Debian OSTree system with GRUB2 and bootc.

### 8. Modern Bootupd System (`examples/debian-bootupd-ostree.json`)
Complete Debian 13 OSTree system with modern bootupd bootloader management.

### 9. Debian 14 Bootupd System (`examples/debian-forky-bootupd.json`)
Complete Debian 14 (Forky) OSTree system with modern bootupd bootloader management.

## 🔄 Multi-Version Debian Support

particle-os supports building images for multiple Debian versions:

### **Debian 13 (Trixie) - Stable Release**
- **Suite**: `trixie`
- **Status**: Current stable release
- **Use case**: Production deployments, long-term support
- **Security updates**: Regular security patches
- **Example**: `examples/debian-basic.json`

### **Debian 14 (Forky) - Testing Release**
- **Suite**: `forky`
- **Status**: Testing release (future stable)
- **Use case**: Development, testing, early adoption
- **Security updates**: Security patches from unstable
- **Example**: `examples/debian-forky-basic.json`

### **Version-Specific Configuration**

#### Sources Configuration
```json
{
  "name": "org.osbuild.debian.sources",
  "options": {
    "suite": "trixie",  // or "forky"
    "mirror": "https://deb.debian.org/debian",
    "components": ["main", "contrib", "non-free"],
    "additional_sources": [
      "deb https://deb.debian.org/debian-security trixie-security main contrib non-free",
      "deb https://deb.debian.org/debian-updates trixie-updates main contrib non-free"
    ]
  }
}
```

#### Debootstrap Configuration
```json
{
  "name": "org.osbuild.debian.debootstrap",
  "options": {
    "suite": "trixie",  // or "forky"
    "mirror": "https://deb.debian.org/debian",
    "variant": "minbase",
    "arch": "amd64",
    "components": ["main", "contrib", "non-free"]
  }
}
```

#### OSTree Branch Naming
```json
{
  "name": "org.osbuild.debian.ostree",
  "options": {
    "branch": "debian/trixie/x86_64/standard",  // or "debian/forky/x86_64/standard"
    "subject": "Debian Trixie OSTree System",   // or "Debian Forky OSTree System"
    "body": "Complete Debian OSTree system with GRUB2 and bootc"
  }
}
```

### **Choosing the Right Version**

- **Use Trixie (Debian 13)** for:
  - Production environments
  - Long-term stability
  - Security-focused deployments
  - Enterprise use cases

- **Use Forky (Debian 14)** for:
  - Development and testing
  - Early access to new features
  - CI/CD pipelines
  - Experimental deployments

### **Migration Between Versions**

To migrate from one Debian version to another:

1. **Update manifest files** to use the new suite name
2. **Adjust package lists** for version compatibility
3. **Update OSTree branch names** to reflect new version
4. **Test thoroughly** before production deployment
5. **Consider rolling back** if issues arise

### **Automated Version Switching**

particle-os includes a script to automatically switch between Debian versions:

```bash
# List supported Debian versions
./scripts/switch-debian-version.sh -l

# Switch from Trixie to Forky (with backup)
./scripts/switch-debian-version.sh -b -v forky examples/debian-basic.json

# Switch from Forky to Trixie
./scripts/switch-debian-version.sh -v trixie examples/debian-forky-basic.json

# Restore from backup
./scripts/switch-debian-version.sh -r examples/debian-basic.json
```

The script automatically updates:
- Suite names in sources and debootstrap stages
- Security and update repository URLs
- OSTree branch names
- Subject lines and descriptions
- Output filenames (if version-specific)

## 🏗️ Architecture Deep Dive

### Stage Implementation Pattern

Each stage follows a consistent pattern:

```python
#!/usr/bin/python3
import os
import sys
import osbuild.api

def main(tree, options):
    """Stage-specific logic"""
    # Process options
    # Manipulate filesystem tree
    return 0

if __name__ == '__main__':
    args = osbuild.api.arguments()
    ret = main(args["tree"], args["options"])
    sys.exit(ret)
```

### Build Process Flow

1. **Manifest Loading** → Parse and validate JSON manifest
2. **Pipeline Construction** → Build stage dependency graph
3. **Source Resolution** → Download and prepare input sources
4. **Stage Execution** → Run stages in dependency order
5. **Assembly** → Create final artifacts from stage outputs
6. **Output** → Export requested objects

### Security and Isolation

- **Process isolation** using bubblewrap and systemd-nspawn
- **Capability management** with Linux capabilities
- **Resource limits** to prevent resource exhaustion
- **Input validation** for all external inputs
- **Output sanitization** for safe output generation

## 🔍 Troubleshooting

### Common Issues

#### Stage Execution Failures
```bash
# Check stage logs
osbuild --monitor json examples/debian-basic.json

# Run with debug output
osbuild --debug examples/debian-basic.json
```

#### Permission Issues
```bash
# Ensure proper capabilities
sudo setcap cap_sys_admin,cap_sys_chroot,cap_dac_override+ep /usr/bin/osbuild
```

#### Package Installation Failures
```bash
# Check APT sources configuration
# Verify network connectivity
# Check package availability in repository
```

### Debug Mode

```bash
# Enable debug output
export OSBUILD_DEBUG=1
osbuild examples/debian-basic.json
```

## 🌟 Key Features

- **Declarative Manifests**: JSON-based configuration with schema validation
- **Stage-based Architecture**: Atomic, composable building blocks
- **OSTree Integration**: Native OSTree support for atomic updates
- **Bootc Support**: Modern container-native bootloader interface
- **GRUB2 Integration**: Traditional bootloader with UEFI support
- **Multi-format Output**: Support for various image formats
- **Security Focus**: Process isolation and capability management
- **Performance**: Intelligent caching and parallel execution support

## 🎯 Use Cases

1. **Distribution Building**: Creating official Debian-based images
2. **Custom Images**: Building specialized Debian OSTree systems
3. **CI/CD Pipelines**: Automated image building and testing
4. **Development**: Testing and development environments
5. **Production Deployment**: Creating production-ready images
6. **Education**: Learning about OS image building and OSTree

## 🔮 Future Vision

particle-os aims to become the **premier platform** for building Debian-based OSTree systems, providing:

- **Enterprise-grade reliability** and performance
- **Comprehensive tooling** for all aspects of OS image building
- **Active community** of contributors and users
- **Industry adoption** in production environments
- **Educational value** for understanding modern OS architecture

## 🤝 Contributing

We welcome contributions! Please see our [Development Guide](docs/DEVELOPMENT.md) for details on:

- Setting up the development environment
- Running tests
- Submitting pull requests
- Code style guidelines
- Architecture decisions

## 📄 License

This project is open source. See the LICENSE file for details.

## 🙏 Acknowledgments

- **Red Hat** for the original osbuild system
- **Universal Blue** for the inspiration and vision
- **Debian Project** for the excellent base system
- **OSTree** for the atomic update system
- **bootc** for the container-native bootloader

---

**particle-os**: Building the future of Debian, one image at a time. 🚀