deb-osbuild/README.md

# 🚀 particle-os

**Complete Debian OSTree System Builder with osbuild Backend**

particle-os is a production-ready Debian-based fork of ublue-os that provides comprehensive osbuild backend support for Debian ecosystems. This project adapts the Red Hat osbuild system to work seamlessly with Debian-based distributions, replacing RPM/DNF components with APT/DPKG equivalents.

## 🌟 What is osbuild?

**osbuild** is a pipeline-based build system for operating system artifacts that defines a universal pipeline description and execution engine. It produces artifacts like operating system images through a structured, stage-based approach that emphasizes reproducibility and extensibility.

### 🔧 How osbuild Works

osbuild follows a **declarative pipeline architecture** where:

1. **Manifests** define the complete build process as JSON
2. **Stages** are atomic, composable building blocks
3. **Assemblers** create final artifacts from stage outputs
4. **Pipelines** orchestrate stage execution and data flow

### 🏗️ Core Architecture

```
osbuild CLI → Manifest Parser → Pipeline Executor → Stage Runner → Assembler → Artifacts
     ↓              ↓                ↓               ↓            ↓          ↓
  Main Entry   JSON Schema    Pipeline Builder   Stage Exec   Output Gen   Final Files
```

### 🎯 Key Principles

- **Stages are never broken, only deprecated** - Same manifest always produces same output
- **Explicit over implicit** - No reliance on tree state
- **Pipeline independence** - Tree is empty at beginning of each pipeline
- **Machine-generated manifests** - No convenience functions for manual creation
- **Confined build environment** - Security against accidental misuse
- **Distribution compatibility** - Python 3.6+ support

## 🚀 What particle-os Provides

particle-os extends osbuild with **10 Debian-specific stages** and **Debian-specific assemblers** to create a complete Debian OSTree system builder:

### 📋 Debian Stages

- **`org.osbuild.debian.debootstrap`** - Base system construction
- **`org.osbuild.debian.apt`** - Package management
- **`org.osbuild.debian.sources`** - APT sources configuration
- **`org.osbuild.debian.users`** - User account management
- **`org.osbuild.debian.locale`** - Locale configuration
- **`org.osbuild.debian.timezone`** - Timezone setup
- **`org.osbuild.debian.ostree`** - OSTree repository management
- **`org.osbuild.debian.bootc`** - Bootc integration
- **`org.osbuild.debian.systemd`** - OSTree-optimized systemd
- **`org.osbuild.debian.grub2`** - GRUB2 bootloader configuration

### 🔧 Debian Assemblers

- **`org.osbuild.debian.qemu`** - Bootable disk image creation (raw, qcow2, vmdk, vdi)

## 🔗 osbuild and bootc-image-builder Integration

### What is bootc-image-builder?

**bootc-image-builder** is a tool that uses osbuild to create bootable container images that can be deployed using bootc. It provides a higher-level interface for building OSTree-based systems with container-native booting capabilities.

### How They Work Together

```
bootc-image-builder → osbuild → particle-os stages → Debian OSTree system
         ↓              ↓              ↓                    ↓
   Container config   Manifest    Debian stages      Bootable image
```

### Integration Benefits

1. **Container-native booting** - Images can be deployed as containers
2. **Atomic updates** - OSTree provides rollback and update capabilities
3. **Infrastructure as code** - Declarative image definitions
4. **CI/CD integration** - Automated image building and testing
5. **Multi-platform support** - Build for different architectures and cloud providers

### Example bootc-image-builder Usage

```bash
# Install bootc-image-builder
sudo apt install bootc-image-builder

# Build a Debian OSTree image using particle-os
bootc-image-builder build \
  --type qcow2 \
  --config config.toml \
  --output-dir ./outputs \
  debian-ostree.json
```

## 🚀 Quick Start

### 1. Installation

```bash
# Clone the repository
git clone https://git.raines.xyz/robojerk/deb-osbuild.git
cd deb-osbuild

# Set up development environment
./scripts/dev-setup.sh

# Install particle-os
make install
```

### 2. Basic Usage

#### Create a Simple Debian System

```bash
# Build a basic Debian system
osbuild examples/debian-basic.json

# This creates a compressed tar archive of a complete Debian system
```

#### Create a Bootable OSTree System

```bash
# Build a complete bootable Debian OSTree system
osbuild examples/debian-ostree-bootable.json

# This creates a bootable QCOW2 disk image with GRUB2 and bootc
```

### 3. Understanding Manifests

#### Basic Manifest Structure

```json
{
  "version": "2",
  "pipelines": [
    {
      "name": "build",
      "runner": "org.osbuild.linux",
      "stages": [
        {
          "name": "org.osbuild.debian.debootstrap",
          "options": {
            "suite": "trixie",
            "mirror": "https://deb.debian.org/debian",
            "variant": "minbase"
          }
        }
      ]
    }
  ],
  "assembler": {
    "name": "org.osbuild.debian.qemu",
    "options": {
      "format": "qcow2",
      "filename": "debian-system.qcow2",
      "size": "15G"
    }
  }
}
```

#### Stage Execution Order

1. **Sources** → Configure APT repositories
2. **Debootstrap** → Create base Debian filesystem
3. **APT** → Install packages and dependencies
4. **Users** → Create user accounts and groups
5. **Locale** → Configure language and locale settings
6. **Timezone** → Set timezone configuration
7. **Systemd** → Configure systemd for OSTree
8. **Bootc** → Set up bootc for container-native booting
9. **GRUB2** → Configure bootloader
10. **OSTree** → Create OSTree repository and commit

## 🔄 CI/CD Workflows

### GitHub Actions

```yaml
name: Build Debian OSTree Image

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build-image:
    runs-on: ubuntu-latest
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        
    - name: Install particle-os
      run: |
        source venv/bin/activate
        make install
        
    - name: Run tests
      run: |
        source venv/bin/activate
        make test
        
    - name: Build Debian OSTree image
      run: |
        source venv/bin/activate
        osbuild examples/debian-ostree-bootable.json
        
    - name: Upload artifacts
      uses: actions/upload-artifact@v3
      with:
        name: debian-ostree-image
        path: debian-ostree-bootable.qcow2
```

### GitLab CI

```yaml
stages:
  - test
  - build
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"

cache:
  paths:
    - .pip-cache/

test:
  stage: test
  image: python:3.11
  before_script:
    - pip install -r requirements.txt
    - make install
  script:
    - make test
  artifacts:
    reports:
      junit: test-results.xml

build:
  stage: build
  image: python:3.11
  before_script:
    - pip install -r requirements.txt
    - make install
  script:
    - osbuild examples/debian-ostree-bootable.json
  artifacts:
    paths:
      - "*.qcow2"
      - "*.raw"
    expire_in: 1 week
  only:
    - main

deploy:
  stage: deploy
  image: python:3.11
  script:
    - echo "Deploying image to production..."
    # Add your deployment logic here
  only:
    - main
  when: manual
```

### Forgejo Actions

```yaml
name: Build and Test Debian OSTree Image

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v4
      
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install system dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv build-essential
        
    - name: Install Python dependencies
      run: |
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        
    - name: Install particle-os
      run: |
        source venv/bin/activate
        make install
        
    - name: Run test suite
      run: |
        source venv/bin/activate
        make test
        
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v4
      
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
        
    - name: Install dependencies
      run: |
        sudo apt update
        sudo apt install -y python3-pip python3-venv
        python3 -m venv venv
        source venv/bin/activate
        pip install -r requirements.txt
        make install
        
    - name: Build Debian OSTree image
      run: |
        source venv/bin/activate
        osbuild examples/debian-ostree-bootable.json
        
    - name: Upload build artifacts
      uses: actions/upload-artifact@v3
      with:
        name: debian-ostree-images
        path: |
          *.qcow2
          *.raw
          *.vmdk
```

## 🔧 Advanced Usage

### Custom Stage Configuration

#### Debian Sources Stage

```json
{
  "name": "org.osbuild.debian.sources",
  "options": {
    "suite": "trixie",
    "mirror": "https://deb.debian.org/debian",
    "components": ["main", "contrib", "non-free"],
    "additional_sources": [
      "deb https://deb.debian.org/debian-security trixie-security main contrib non-free"
    ]
  }
}
```

#### User Management Stage

```json
{
  "name": "org.osbuild.debian.users",
  "options": {
    "users": {
      "debian": {
        "password": "$6$rounds=656000$salt$hashedpassword",
        "shell": "/bin/bash",
        "groups": ["sudo", "users", "adm"],
        "uid": 1000,
        "gid": 1000,
        "home": "/home/debian",
        "comment": "Debian User"
      }
    },
    "default_shell": "/bin/bash",
    "default_home": "/home"
  }
}
```

#### GRUB2 Bootloader Stage

```json
{
  "name": "org.osbuild.debian.grub2",
  "options": {
    "root_fs_uuid": "ROOT_UUID",
    "kernel_path": "/boot/vmlinuz",
    "initrd_path": "/boot/initrd.img",
    "bootloader_id": "debian",
    "timeout": 5,
    "default_entry": "0"
  }
}
```

### QEMU Assembler Options

```json
{
  "name": "org.osbuild.debian.qemu",
  "options": {
    "format": "qcow2",
    "filename": "debian-ostree.qcow2",
    "size": "20G",
    "ptuuid": "12345678-1234-1234-1234-123456789012"
  }
}
```

**Supported Formats:**
- `raw` - Raw disk image
- `qcow2` - QEMU Copy On Write v2 (recommended)
- `vmdk` - VMware Virtual Machine Disk
- `vdi` - VirtualBox Virtual Disk Image

## 🧪 Testing and Development

### Run Tests

```bash
# Run all tests
make test

# Run specific test files
pytest tests/test_ostree_stages.py
pytest tests/test_grub2_stage.py

# Run with coverage
pytest --cov=src tests/
```

### Development Setup

```bash
# Set up development environment
make dev-setup

# Install in development mode
make install-dev

# Run linting
make lint

# Format code
make format
```

### Demo Scripts

```bash
# Run complete bootable OSTree pipeline demonstration
./scripts/demo-bootable-ostree.py

# Test individual stages
./scripts/test-stages.py
```

## 📚 Examples

### 1. Basic Debian System (`examples/debian-basic.json`)
Creates a minimal Debian system with basic packages and user accounts.

### 2. OSTree System (`examples/debian-ostree.json`)
Builds a Debian system with OSTree repository management.

### 3. Complete System (`examples/debian-complete.json`)
Comprehensive Debian system with all basic stages.

### 4. Bootable OSTree System (`examples/debian-ostree-bootable.json`)
Complete bootable Debian OSTree system with GRUB2 and bootc.

## 🏗️ Architecture Deep Dive

### Stage Implementation Pattern

Each stage follows a consistent pattern:

```python
#!/usr/bin/python3
import os
import sys
import osbuild.api

def main(tree, options):
    """Stage-specific logic"""
    # Process options
    # Manipulate filesystem tree
    return 0

if __name__ == '__main__':
    args = osbuild.api.arguments()
    ret = main(args["tree"], args["options"])
    sys.exit(ret)
```

### Build Process Flow

1. **Manifest Loading** → Parse and validate JSON manifest
2. **Pipeline Construction** → Build stage dependency graph
3. **Source Resolution** → Download and prepare input sources
4. **Stage Execution** → Run stages in dependency order
5. **Assembly** → Create final artifacts from stage outputs
6. **Output** → Export requested objects

### Security and Isolation

- **Process isolation** using bubblewrap and systemd-nspawn
- **Capability management** with Linux capabilities
- **Resource limits** to prevent resource exhaustion
- **Input validation** for all external inputs
- **Output sanitization** for safe output generation

## 🔍 Troubleshooting

### Common Issues

#### Stage Execution Failures
```bash
# Check stage logs
osbuild --monitor json examples/debian-basic.json

# Run with debug output
osbuild --debug examples/debian-basic.json
```

#### Permission Issues
```bash
# Ensure proper capabilities
sudo setcap cap_sys_admin,cap_sys_chroot,cap_dac_override+ep /usr/bin/osbuild
```

#### Package Installation Failures
```bash
# Check APT sources configuration
# Verify network connectivity
# Check package availability in repository
```

### Debug Mode

```bash
# Enable debug output
export OSBUILD_DEBUG=1
osbuild examples/debian-basic.json
```

## 🌟 Key Features

- **Declarative Manifests**: JSON-based configuration with schema validation
- **Stage-based Architecture**: Atomic, composable building blocks
- **OSTree Integration**: Native OSTree support for atomic updates
- **Bootc Support**: Modern container-native bootloader interface
- **GRUB2 Integration**: Traditional bootloader with UEFI support
- **Multi-format Output**: Support for various image formats
- **Security Focus**: Process isolation and capability management
- **Performance**: Intelligent caching and parallel execution support

## 🎯 Use Cases

1. **Distribution Building**: Creating official Debian-based images
2. **Custom Images**: Building specialized Debian OSTree systems
3. **CI/CD Pipelines**: Automated image building and testing
4. **Development**: Testing and development environments
5. **Production Deployment**: Creating production-ready images
6. **Education**: Learning about OS image building and OSTree

## 🔮 Future Vision

particle-os aims to become the **premier platform** for building Debian-based OSTree systems, providing:

- **Enterprise-grade reliability** and performance
- **Comprehensive tooling** for all aspects of OS image building
- **Active community** of contributors and users
- **Industry adoption** in production environments
- **Educational value** for understanding modern OS architecture

## 🤝 Contributing

We welcome contributions! Please see our [Development Guide](docs/DEVELOPMENT.md) for details on:

- Setting up the development environment
- Running tests
- Submitting pull requests
- Code style guidelines
- Architecture decisions

## 📄 License

This project is open source. See the LICENSE file for details.

## 🙏 Acknowledgments

- **Red Hat** for the original osbuild system
- **Universal Blue** for the inspiration and vision
- **Debian Project** for the excellent base system
- **OSTree** for the atomic update system
- **bootc** for the container-native bootloader

---

**particle-os**: Building the future of Debian, one image at a time. 🚀