particle-os/deb-orchestrator
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
deb-orchestrator/README.md
robojerk 1751544440 first commit
2025-08-18 23:45:01 -07:00

10 KiB
Raw Permalink Blame History

deb-bootc-compose

Debian's equivalent to Fedora's Pungi compose system

deb-bootc-compose is a Debian-native composition tool that orchestrates the creation of Debian bootc images from packages. It coordinates the entire compose process, similar to how Pungi coordinates Fedora's release process.

What It Does

deb-bootc-compose serves as the main orchestrator for Debian's bootc ecosystem:

  • Package Coordination: Ensures all release artifacts use identical package versions across variants
  • Multi-Artifact Generation: Creates container images, disk images, and other bootc artifacts
  • Build Orchestration: Coordinates with deb-orchestrator (Koji equivalent) and deb-mock (Mock equivalent)
  • OSTree Integration: Orchestrates bootc image creation through apt-ostree
  • Variant Management: Handles different Debian variants (minimal, server, desktop, etc.)

Architecture

The tool follows a phase-based architecture inspired by Pungi:

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ deb-bootc-      │    │ deb-orchestrator│    │    deb-mock     │
│   compose       │    │   (Koji equiv)  │    │   (Mock equiv)  │
│ Orchestrator    │    │ Build System    │    │Build Environment│
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │ Coordinates           │ Manages               │ Creates
         │ entire process        │ package building      │ isolated
         │                       │ at scale              │ environments

Core Phases

  1. init: Initialize compose environment
  2. gather: Download and organize packages
  3. build: Build packages if needed
  4. ostree: Create OSTree commits
  5. output: Generate output artifacts
  6. cleanup: Clean up temporary files

Quick Start

Prerequisites

  • Go 1.21 or later
  • Debian-based system (for development)
  • Basic understanding of Debian packaging

Installation

# Clone the repository
git clone https://git.raines.xyz/particle-os/deb-bootc-compose.git
cd deb-bootc-compose

# Install dependencies
make deps

# Build the binary
make build

# Run with help
./build/deb-bootc-compose --help

Basic Usage

# Run a compose with a treefile
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-minimal.json \
    --output ./compose-output \
    --config configs/compose.yaml

Configuration

Compose Configuration

The main configuration file (compose.yaml) controls:

  • Compose settings: Release, variants, architectures
  • Build system: sbuild, debootstrap integration
  • OSTree settings: Repository mode, signing
  • Output formats: Container, disk image, chunked OCI
  • Integration: deb-orchestrator connection settings

Treefiles

Treefiles (JSON manifests) define what gets built:

  • Package lists: Required, optional, recommended packages
  • Variants: Different flavors (minimal, server, desktop)
  • Architectures: Target CPU architectures
  • Repositories: Package sources
  • Build settings: Build system configuration

Example treefile structure:

{
  "name": "debian-bootc-minimal",
  "version": "13",
  "release": "bookworm",
  "packages": {
    "required": ["linux-image-amd64", "systemd", "ostree", "bootc"]
  },
  "architecture": ["amd64", "arm64"],
  "variants": [
    {
      "name": "minimal",
      "description": "Minimal base system"
    }
  ]
}

Development

Project Structure

deb-bootc-compose/
├── cmd/                    # Command-line interfaces
│   ├── compose/           # Main compose command
│   └── cli/              # CLI utilities
├── internal/              # Internal packages
│   ├── compose/           # Core compose engine
│   ├── config/            # Configuration management
│   ├── ostree/            # OSTree integration
│   └── build/             # Build system interface
├── pkg/                   # Public packages
├── configs/               # Configuration files
├── examples/              # Example treefiles
└── docs/                  # Documentation

Building

# Build everything
make all

# Build specific components
make build      # Main binary
make cli        # CLI tool

# Development helpers
make test       # Run tests
make fmt        # Format code
make lint       # Lint code
make clean      # Clean build artifacts

Testing

# Run all tests
make test

# Run tests with coverage
make test-coverage

# Run with sample treefile
make run-sample

Integration

With deb-orchestrator

deb-bootc-compose integrates with deb-orchestrator for:

  • Package management: Downloading packages from build system
  • Build coordination: Ensuring package availability before compose
  • Build orchestration: Submitting build requests and monitoring progress
  • Metadata integration: Using build metadata for versioning

With deb-mock

deb-bootc-compose can use deb-mock for:

  • Build environment creation: Isolated chroot environments
  • Package installation: Installing build dependencies
  • Environment isolation: Reproducible builds

Advanced Features

Multi-Variant Composes

Support for creating multiple variants in a single compose:

# Compose multiple variants
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-multi-variant.json \
    --variants minimal,standard,development \
    --output ./multi-variant-output

Parallel Processing

Efficient parallel execution of compose phases:

# Enable parallel processing
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-minimal.json \
    --parallel \
    --workers 4

Output Formats

Multiple output format support:

  • Container Images: OCI-compatible container images
  • Disk Images: QCOW2, RAW, and other disk formats
  • Tarballs: Compressed root filesystem archives
  • Metadata: JSON metadata for integration

Troubleshooting

Common Issues

  1. Package Resolution Failures

    # Check package availability
apt-cache policy <package-name>

# Verify repository configuration
cat /etc/apt/sources.list.d/*

  2. OSTree Repository Issues

    # Check OSTree repository status
ostree log <ref>

# Verify repository structure
ostree refs --repo <repo-path>

  3. Build System Connection

    # Test deb-orchestrator connection
curl -s http://localhost:8080/health

# Check build system logs
journalctl -u deb-orchestrator

Debug Mode

Enable verbose logging for troubleshooting:

# Run with debug output
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-minimal.json \
    --debug \
    --log-level debug

Performance Optimization

Caching Strategies

  • Package Cache: Local caching of downloaded packages
  • OSTree Cache: Incremental repository updates
  • Build Cache: Reuse of build artifacts

Resource Management

  • Memory Optimization: Efficient memory usage for large composes
  • Disk I/O: Optimized file operations and storage
  • Network: Parallel downloads and connection pooling

Status

Current Status: Phase 1 - Foundation Development

  • ✅ Core engine: Basic compose engine implemented
  • ✅ Treefile parser: JSON-based configuration system
  • ✅ Phase management: Simple phase execution system
  • ✅ Configuration: YAML-based configuration system
  • ✅ OSTree integration: apt-ostree integration working
  • ✅ Build system: deb-orchestrator integration functional
  • ✅ Output formats: Container, disk image, tarball generation
  • 🔄 Testing: Comprehensive testing framework in progress

Roadmap

See the main project TODO.md for the complete development roadmap.

Phase 1 (Months 1-6): Foundation ✅ COMPLETED

  • Core compose engine working
  • Basic treefile parsing and validation
  • OSTree integration functional
  • Container output working

Phase 2 (Months 7-10): Integration 🎯 CURRENT

  • Deep integration with deb-orchestrator and deb-mock
  • Advanced features and optimization
  • Production readiness features

Phase 3 (Months 11-14): Production

  • Security audit and hardening
  • Performance optimization
  • Community integration

Phase 4 (Months 15-18): Ecosystem

  • Debian Atomic and Particle-OS variants
  • Advanced use cases
  • Community adoption

Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

# Set up development environment
make dev-setup

# Install development tools
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

# Run pre-commit checks
make pre-commit

Code Style

  • Follow Go standard formatting (gofmt)
  • Use meaningful variable and function names
  • Add comprehensive tests for new features
  • Update documentation for API changes

License

This project is licensed under the same terms as Debian (GPL-2+).

Related Projects

  • deb-orchestrator: Debian's equivalent to Fedora's Koji build system
  • deb-mock: Debian's equivalent to Fedora's Mock build environment manager
  • deb-bootc-compose: This project - the main orchestrator

Support

Repository Information

Part of Debian's complete bootc ecosystem - building Debian's answer to Fedora's Pungi-Koji-Mock ecosystem.