deb-orchestrator/dev-architecture-docs/deb-bootc-compose.md

deb-bootc-compose: Debian Bootc Composition Tool

Executive Summary

deb-bootc-compose is a Debian-native composition tool designed to orchestrate the creation of Debian bootc images from packages. It's our answer to Fedora's Pungi compose system, adapted for Debian's package management ecosystem and build infrastructure.

What deb-bootc-compose is Based On

Fedora's Pungi: The Inspiration

deb-bootc-compose is directly inspired by and modeled after Fedora's Pungi compose system. Pungi is the sophisticated orchestration tool that coordinates Fedora's entire release process, including bootc image creation.

Pungi's Core Functions:

• Package Set Coordination: Ensures all release artifacts use identical package versions
• Multi-Artifact Generation: Creates ISOs, live images, container images, and cloud images
• Build Orchestration: Manages the interaction between Koji (build system) and rpm-ostree
• Dependency Management: Coordinates package dependencies across different output formats
• Release Consistency: Guarantees reproducible builds across architectures and variants

Why Pungi Works So Well:

• Declarative Configuration: Uses YAML-based compose definitions
• Isolated Builds: Each compose runs in its own environment
• Audit Trail: Complete logging of every build step
• Multi-Architecture Support: Builds for x86_64, aarch64, ppc64le, s390x
• Integration: Deep integration with Fedora's Koji build system

Adapting Pungi's Success for Debian

While Pungi is designed for Fedora's RPM-based ecosystem, deb-bootc-compose adapts the same architectural principles for Debian's DEB-based world:

Aspect	Pungi (Fedora)	deb-bootc-compose (Debian)
Package Format	RPM packages	DEB packages
Build System	Koji + Mock	sbuild + schroot
Package Manager	dnf/yum	apt/dpkg
OSTree Tool	rpm-ostree	apt-ostree
Configuration	YAML compose files	YAML compose files
Repository	RPM repositories	Debian package pools

Technical Architecture

Core Components

deb-bootc-compose consists of several interconnected components that work together to orchestrate the build process:

1. Compose Engine

The central orchestrator that manages the entire compose process.

type ComposeEngine struct {
    Config     *ComposeConfig
    BuildSystem BuildSystem
    OSTreeTool  OSTreeTool
    Output      OutputManager
    Logger      Logger
}

func (e *ComposeEngine) Compose() error {
    // 1. Parse and validate treefile
    // 2. Resolve package dependencies
    // 3. Coordinate build system
    // 4. Create OSTree commit
    // 5. Generate output artifacts
}

2. Treefile Parser

Parses and validates Debian treefile manifests (our equivalent to Pungi's compose definitions).

type Treefile struct {
    Name         string            `json:"name"`
    Version      string            `json:"version"`
    Description  string            `json:"description"`
    Packages     PackageSet        `json:"packages"`
    Exclude      []string          `json:"exclude"`
    Repositories []string          `json:"repositories"`
    Architecture []string          `json:"architecture"`
    Variants     []Variant         `json:"variants"`
}

type PackageSet struct {
    Required []string `json:"required"`
    Optional []string `json:"optional"`
    Recommended []string `json:"recommended"`
}

3. Build System Interface

Abstracts the build system (sbuild) to allow for different build backends.

type BuildSystem interface {
    Build(treefile *Treefile, arch string) (*BuildResult, error)
    InstallDependencies(packages []string, arch string) error
    CreateBuildEnvironment(arch string) (*BuildEnvironment, error)
    Cleanup(env *BuildEnvironment) error
}

type SbuildSystem struct {
    Dist        string
    Arch        string
    BuildDepResolver string
    BuildEnv        string
}

4. OSTree Tool Interface

Abstracts the OSTree tool (apt-ostree) for creating commits and containers.

type OSTreeTool interface {
    CreateCommit(packages []string, outputPath string) (*OSTreeCommit, error)
    ContainerEncapsulate(commit *OSTreeCommit, outputPath string) error
    BuildChunkedOCI(commit *OSTreeCommit, outputPath string) error
    ValidateCommit(commit *OSTreeCommit) error
}

5. Output Manager

Handles the generation of different output formats (containers, disk images, etc.).

type OutputManager struct {
    Formats []OutputFormat
    Registry RegistryConfig
    Signing  SigningConfig
}

type OutputFormat interface {
    Generate(commit *OSTreeCommit, config OutputConfig) error
    Validate(output OutputConfig) error
}

How deb-bootc-compose Works

1. Compose Initialization

# Start a compose
deb-bootc-compose --release bookworm \
    --variant debian-bootc-minimal \
    --arch amd64 \
    --treefile debian-bootc-minimal.json \
    --output /tmp/compose-output

What happens internally:

1. Parse treefile: Load and validate the JSON manifest
2. Resolve dependencies: Use apt to resolve package dependencies
3. Create build plan: Generate a build order based on dependencies
4. Initialize build environment: Set up sbuild chroots for each architecture

2. Package Resolution and Download

func (c *ComposeEngine) resolvePackages(treefile *Treefile) (*PackageResolution, error) {
    resolution := &PackageResolution{}
    
    // Resolve required packages
    for _, pkg := range treefile.Packages.Required {
        deps, err := c.aptResolver.Resolve(pkg)
        if err != nil {
            return nil, fmt.Errorf("failed to resolve %s: %w", pkg, err)
        }
        resolution.Required[pkg] = deps
    }
    
    // Resolve optional packages
    for _, pkg := range treefile.Packages.Optional {
        deps, err := c.aptResolver.Resolve(pkg)
        if err != nil {
            log.Warnf("Optional package %s failed to resolve: %v", pkg, err)
            continue
        }
        resolution.Optional[pkg] = deps
    }
    
    return resolution, nil
}

Dependency resolution process:

1. Parse package lists: Extract required, optional, and recommended packages
2. Query package database: Use apt-cache to get dependency information
3. Build dependency graph: Create a directed graph of package dependencies
4. Resolve conflicts: Handle package conflicts and alternatives
5. Generate build order: Determine the order packages should be built

3. Build Environment Creation

func (s *SbuildSystem) CreateBuildEnvironment(arch string) (*BuildEnvironment, error) {
    // Create schroot configuration
    schrootConfig := &SchrootConfig{
        Distribution: s.Dist,
        Architecture: arch,
        BuildDepResolver: s.BuildDepResolver,
        BuildEnv: s.BuildEnv,
    }
    
    // Initialize chroot
    chroot, err := s.schroot.Create(schrootConfig)
    if err != nil {
        return nil, fmt.Errorf("failed to create chroot: %w", err)
    }
    
    // Install essential packages
    essentialPkgs := []string{"build-essential", "fakeroot", "devscripts"}
    if err := chroot.Install(essentialPkgs); err != nil {
        chroot.Cleanup()
        return nil, fmt.Errorf("failed to install essential packages: %w", err)
    }
    
    return &BuildEnvironment{
        Chroot: chroot,
        Arch: arch,
        Dist: s.Dist,
    }, nil
}

Build environment setup:

1. Create schroot: Set up isolated build environment using Debian's schroot
2. Install build tools: Add build-essential, fakeroot, and other build dependencies
3. Configure package sources: Set up apt sources for the target distribution
4. Mount necessary directories: Mount source code, output directories, and package caches

4. Package Building and Installation

func (c *ComposeEngine) buildPackages(treefile *Treefile, resolution *PackageResolution) error {
    // Create build environments for each architecture
    for _, arch := range treefile.Architecture {
        env, err := c.buildSystem.CreateBuildEnvironment(arch)
        if err != nil {
            return fmt.Errorf("failed to create build environment for %s: %w", arch, err)
        }
        defer env.Cleanup()
        
        // Install packages in dependency order
        for _, pkg := range resolution.BuildOrder {
            if err := c.installPackage(env, pkg); err != nil {
                return fmt.Errorf("failed to install %s on %s: %w", pkg, arch, err)
            }
        }
        
        // Create filesystem snapshot
        if err := c.createFilesystemSnapshot(env, arch); err != nil {
            return fmt.Errorf("failed to create filesystem snapshot for %s: %w", arch, err)
        }
    }
    
    return nil
}

Package installation process:

1. Dependency order: Install packages in the correct dependency order
2. Package installation: Use apt to install packages in the chroot
3. Conflict resolution: Handle package conflicts and alternatives
4. Filesystem snapshot: Capture the complete filesystem state after installation

5. OSTree Commit Creation

func (c *ComposeEngine) createOSTreeCommit(filesystemPath string, arch string) (*OSTreeCommit, error) {
    // Initialize OSTree repository
    repo, err := ostree.InitRepo(filesystemPath, ostree.RepoModeBare)
    if err != nil {
        return nil, fmt.Errorf("failed to initialize OSTree repo: %w", err)
    }
    
    // Create commit metadata
    metadata := &ostree.CommitMetadata{
        Subject: fmt.Sprintf("Debian bootc %s %s", c.config.Release, arch),
        Body: fmt.Sprintf("Debian %s bootc image for %s architecture", c.config.Release, arch),
        Version: c.config.Version,
        Architecture: arch,
        Timestamp: time.Now(),
    }
    
    // Create the commit
    commit, err := repo.Commit(filesystemPath, metadata)
    if err != nil {
        return nil, fmt.Errorf("failed to create OSTree commit: %w", err)
    }
    
    return commit, nil
}

OSTree commit process:

1. Repository initialization: Create OSTree repository in the appropriate mode
2. Filesystem preparation: Prepare the filesystem for OSTree commit
3. Metadata creation: Generate commit metadata with version and architecture info
4. Commit creation: Use apt-ostree to create the actual commit
5. Validation: Verify the commit integrity and structure

6. Container Encapsulation

func (c *ComposeEngine) createContainer(commit *OSTreeCommit, outputPath string) error {
    // Create OCI container structure
    container := &OCIContainer{
        Config: &OCIConfig{
            Architecture: commit.Architecture,
            OS: "linux",
            Created: time.Now(),
            Labels: map[string]string{
                "containers.bootc": "1",
                "org.debian.release": c.config.Release,
                "org.debian.variant": c.config.Variant,
            },
        },
        Layers: []OCILayer{},
    }
    
    // Add OSTree commit as container layer
    ostreeLayer, err := c.createOSTreeLayer(commit)
    if err != nil {
        return fmt.Errorf("failed to create OSTree layer: %w", err)
    }
    container.Layers = append(container.Layers, ostreeLayer)
    
    // Write container to output
    if err := c.writeContainer(container, outputPath); err != nil {
        return fmt.Errorf("failed to write container: %w", err)
    }
    
    return nil
}

Container creation process:

1. OCI structure: Create OCI-compliant container structure
2. Layer creation: Convert OSTree commit to container layers
3. Metadata addition: Add Debian-specific labels and metadata
4. Container writing: Write the container to the specified output format

7. Output Generation

func (c *ComposeEngine) generateOutputs(commit *OSTreeCommit) error {
    for _, format := range c.config.OutputFormats {
        switch format {
        case "container":
            if err := c.createContainer(commit, c.config.OutputPath); err != nil {
                return fmt.Errorf("failed to create container: %w", err)
            }
        case "disk-image":
            if err := c.createDiskImage(commit, c.config.OutputPath); err != nil {
                return fmt.Errorf("failed to create disk image: %w", err)
            }
        case "chunked-oci":
            if err := c.createChunkedOCI(commit, c.config.OutputPath); err != nil {
                return fmt.Errorf("failed to create chunked OCI: %w", err)
            }
        }
    }
    
    return nil
}

Output generation:

1. Container format: Create OCI container images
2. Disk images: Generate bootable disk images (QCOW2, ISO)
3. Chunked OCI: Create optimized, chunked container images
4. Registry push: Optionally push to container registries

Configuration and Treefiles

Treefile Format

deb-bootc-compose uses JSON-based treefiles (our equivalent to Pungi's YAML compose files):

{
  "name": "debian-bootc-minimal",
  "version": "13",
  "description": "Minimal Debian bootc base image",
  "release": "bookworm",
  "architecture": ["amd64", "arm64"],
  "packages": {
    "required": [
      "linux-image-amd64",
      "systemd",
      "ostree",
      "bootc",
      "grub-pc",
      "grub-efi-amd64",
      "initramfs-tools"
    ],
    "optional": [
      "openssh-server",
      "curl",
      "vim"
    ],
    "exclude": [
      "snapd",
      "flatpak"
    ]
  },
  "repositories": [
    "deb http://deb.debian.org/debian bookworm main",
    "deb http://deb.debian.org/debian bookworm-updates main",
    "deb http://deb.debian.org/debian-security bookworm-security main"
  ],
  "build": {
    "system": "sbuild",
    "environment": "debootstrap",
    "dependencies": "aptitude"
  },
  "ostree": {
    "mode": "bare",
    "refs": ["debian/13/amd64/minimal", "debian/13/arm64/minimal"]
  },
  "output": {
    "formats": ["container", "disk-image", "chunked-oci"],
    "registry": "docker.io/debian",
    "signing": true
  }
}

Configuration Options

# Basic compose
deb-bootc-compose --treefile debian-bootc-minimal.json

# With custom options
deb-bootc-compose \
    --treefile debian-bootc-minimal.json \
    --output /tmp/compose-output \
    --build-system sbuild \
    --ostree-tool apt-ostree \
    --parallel-builds 4 \
    --cleanup-build-envs \
    --sign-output \
    --push-to-registry

Integration with Debian Infrastructure

Build System Integration

deb-bootc-compose integrates with Debian's existing build infrastructure:

• sbuild: Uses Debian's standard build environment tool
• schroot: Leverages Debian's chroot management system
• debian-installer: Integrates with Debian's installation framework
• Package repositories: Connects to Debian's package pools

Package Management Integration

• apt: Uses Debian's package manager for dependency resolution
• dpkg: Leverages Debian's package installation system
• Package sources: Integrates with Debian's repository structure
• Security updates: Incorporates Debian's security patch system

Quality Assurance Integration

• Debian policy: Ensures compliance with Debian packaging standards
• Lintian: Integrates with Debian's package quality checking
• Security scanning: Incorporates Debian's security audit tools
• Testing framework: Integrates with Debian's testing infrastructure

Performance and Optimization

Parallel Builds

func (c *ComposeEngine) parallelBuild(treefile *Treefile) error {
    // Create worker pool for parallel builds
    workers := make(chan struct{}, c.config.MaxParallelBuilds)
    var wg sync.WaitGroup
    
    for _, arch := range treefile.Architecture {
        wg.Add(1)
        go func(arch string) {
            defer wg.Done()
            
            // Acquire worker slot
            workers <- struct{}{}
            defer func() { <-workers }()
            
            // Build for this architecture
            if err := c.buildForArchitecture(treefile, arch); err != nil {
                log.Errorf("Build failed for %s: %v", arch, err)
            }
        }(arch)
    }
    
    wg.Wait()
    return nil
}

Caching and Incremental Builds

• Package cache: Cache downloaded packages between builds
• Build environment cache: Reuse build environments when possible
• OSTree cache: Cache OSTree objects for incremental builds
• Layer cache: Cache container layers for faster rebuilds

Resource Management

• Memory limits: Control memory usage during builds
• Disk space: Monitor and manage disk space usage
• CPU limits: Control CPU usage for parallel builds
• Network throttling: Limit network bandwidth for package downloads

Monitoring and Logging

Build Monitoring

type BuildMonitor struct {
    StartTime    time.Time
    CurrentStage string
    Progress     float64
    Logs         []LogEntry
    Metrics      BuildMetrics
}

func (m *BuildMonitor) UpdateProgress(stage string, progress float64) {
    m.CurrentStage = stage
    m.Progress = progress
    m.Logs = append(m.Logs, LogEntry{
        Timestamp: time.Now(),
        Stage: stage,
        Progress: progress,
    })
}

Logging and Debugging

• Structured logging: JSON-formatted logs for easy parsing
• Log levels: Debug, info, warning, error levels
• Stage tracking: Track progress through build stages
• Error reporting: Detailed error information with context

Metrics and Analytics

• Build time: Track build duration for each stage
• Resource usage: Monitor CPU, memory, and disk usage
• Success rates: Track build success rates across variants
• Performance trends: Analyze performance over time

Future Enhancements

Planned Features

• Multi-distribution support: Support for Debian testing and unstable
• Cloud integration: Direct deployment to cloud platforms
• Automated testing: Integration with Debian's testing infrastructure
• Security scanning: Automated security vulnerability scanning
• Performance optimization: Further build performance improvements

Community Integration

• Debian integration: Full integration with Debian's release process
• Upstream contribution: Contribute improvements back to the community
• Documentation: Comprehensive documentation and examples
• Training: Training materials for Debian developers

Conclusion

deb-bootc-compose represents a significant step forward for Debian's bootc ecosystem. By adapting Fedora's proven Pungi architecture for Debian's tooling and processes, we can create a robust, maintainable system for building Debian bootc images.

The tool provides:

• Complete orchestration of the Debian bootc build process
• Deep integration with Debian's existing infrastructure
• Performance optimization through parallel builds and caching
• Quality assurance through integration with Debian's tools
• Flexibility through configurable treefiles and output formats

This foundation will enable Debian to establish itself as a first-class bootc platform, providing the same benefits of immutable, atomic operating systems that Fedora users enjoy, but built on Debian's solid foundation and tooling.