debian-forge/docs/mock-integration.md

Debian Forge Mock Integration Plan

Overview

This document outlines the integration plan for deb-mock with debian-forge to create a comprehensive Debian image building ecosystem. The integration will provide isolated, reproducible build environments for Debian package and image creation.

Current State Analysis

debian-forge - The Image Building Engine ( fork of Fedora's osbuild)

• Status: Production-ready with comprehensive APT support
• Capabilities: Complete Debian/Ubuntu image building with APT stages
• Architecture: OSBuild-based pipeline system with modular stages
• Strengths: Full APT integration, cross-architecture support, comprehensive testing

deb-mock - The Build Environment Manager

• Status: Foundation development phase (Phase 1)
• Capabilities: Chroot environment management, package installation, isolation
• Architecture: Single-process, multi-stage with plugin system
• Strengths: Clean build environments, dependency management, security isolation

Integration Architecture

The Complete Debian Image Building Ecosystem

┌─────────────────────────────────────────────────────────────────┐
│                    Debian Image Building Stack                 │
├─────────────────────────────────────────────────────────────────┤
│  debian-forge (OSBuild)  │  deb-mock (Environment)  │  Output  │
│  ┌─────────────────────┐  │  ┌─────────────────────┐  │  ┌─────┐ │
│  │  Pipeline Engine    │  │  │  Chroot Manager     │  │  │ .deb│ │
│  │  - APT Stages       │  │  │  - Environment      │  │  │ .iso│ │
│  │  - Debian Support   │  │  │  - Isolation        │  │  │ .img│ │
│  │  - Cross-arch       │  │  │  - Dependencies     │  │  │ etc │ │
│  └─────────────────────┘  │  └─────────────────────┘  │  └─────┘ │
│           │                │           │                │         │
│           └────────────────┼───────────┘                │         │
│                            │                            │         │
│  ┌─────────────────────────▼─────────────────────────┐  │         │
│  │              Integration Layer                    │  │         │
│  │  - Mock Environment Provisioning                  │  │         │
│  │  - Build Command Execution                        │  │         │
│  │  - Artifact Collection                            │  │         │
│  └───────────────────────────────────────────────────┘  │         │
└─────────────────────────────────────────────────────────────────┘

Integration Phases

Phase 1: Basic Integration (Weeks 1-4)

1.1 Mock Environment Provisioning

• Goal: Integrate deb-mock as the build environment provider for debian-forge
• Implementation:
  • Create org.osbuild.deb-mock stage for environment provisioning
  • Implement mock environment lifecycle management
  • Add configuration mapping between debian-forge and deb-mock

1.2 Build Command Execution

• Goal: Execute debian-forge stages within mock environments
• Implementation:
  • Modify existing APT stages to work within mock chroots
  • Implement command execution through mock's chroot system
  • Add environment variable and mount point management

1.3 Basic Testing

• Goal: Ensure basic integration works end-to-end
• Implementation:
  • Create integration test manifests
  • Test simple Debian image builds
  • Validate artifact collection and output

Phase 2: Advanced Integration (Weeks 5-8)

2.1 Plugin System Integration

• Goal: Leverage deb-mock's plugin system for enhanced functionality
• Implementation:
  • Integrate with deb-mock's plugin architecture
  • Create debian-forge specific plugins
  • Implement caching and optimization plugins

2.2 Multi-Environment Support

• Goal: Support multiple Debian distributions and architectures
• Implementation:
  • Extend mock configuration for different Debian suites
  • Add cross-architecture build support
  • Implement environment-specific optimizations

2.3 Performance Optimization

• Goal: Optimize build performance through mock integration
• Implementation:
  • Implement build environment caching
  • Add parallel build support
  • Optimize package installation and dependency resolution

Phase 3: Production Integration (Weeks 9-12)

3.1 CI/CD Integration

• Goal: Integrate with Forgejo CI/CD for automated builds
• Implementation:
  • Update CI workflows to use mock environments
  • Add build environment management to CI
  • Implement automated testing and validation

3.2 Advanced Features

• Goal: Add advanced features for production use
• Implementation:
  • Implement build environment snapshots
  • Add debugging and troubleshooting tools
  • Create comprehensive monitoring and logging

Technical Implementation

1. Mock Stage Implementation

Create a new org.osbuild.deb-mock stage:

# stages/org.osbuild.deb-mock.py
def main(tree, options):
    """Main function for deb-mock stage"""
    config = options.get("config", {})
    environment = options.get("environment", "debian-trixie")
    arch = options.get("arch", "amd64")
    
    # Create mock environment
    mock_env = create_mock_environment(environment, arch, config)
    
    # Install build dependencies
    install_build_dependencies(mock_env, options.get("packages", []))
    
    # Execute build commands
    execute_build_commands(mock_env, options.get("commands", []))
    
    # Collect artifacts
    collect_artifacts(mock_env, tree)
    
    return 0

2. Configuration Integration

Extend debian-forge manifests to support mock configuration:

{
  "version": "2",
  "pipelines": [
    {
      "runner": "org.osbuild.linux",
      "name": "build",
      "stages": [
        {
          "type": "org.osbuild.deb-mock",
          "options": {
            "environment": "debian-trixie",
            "arch": "amd64",
            "config": {
              "mirror": "http://deb.debian.org/debian",
              "components": ["main", "contrib", "non-free"]
            },
            "packages": [
              "build-essential",
              "devscripts",
              "debhelper"
            ]
          }
        },
        {
          "type": "org.osbuild.apt",
          "options": {
            "packages": ["linux-image-amd64", "systemd"],
            "mock_environment": true
          }
        }
      ]
    }
  ]
}

3. Mock Environment Management

Implement mock environment lifecycle management:

class MockEnvironmentManager:
    def __init__(self, config):
        self.config = config
        self.environments = {}
    
    def create_environment(self, name, arch, suite):
        """Create a new mock environment"""
        # Implementation using deb-mock API
        
    def install_packages(self, env_name, packages):
        """Install packages in mock environment"""
        # Implementation using deb-mock package manager
        
    def execute_command(self, env_name, command):
        """Execute command in mock environment"""
        # Implementation using deb-mock command executor
        
    def collect_artifacts(self, env_name, output_dir):
        """Collect build artifacts from mock environment"""
        # Implementation using deb-mock artifact collection

Integration Benefits

1. Enhanced Isolation

• Clean Build Environments: Each build gets a fresh, isolated environment
• Dependency Management: Automatic handling of build dependencies
• Security: Sandboxed builds prevent host system contamination

2. Improved Reproducibility

• Consistent Environments: Identical build environments across different systems
• Version Control: Mock environments can be versioned and managed
• Debugging: Easier debugging with isolated, reproducible environments

3. Better Performance

• Environment Caching: Reuse mock environments for faster builds
• Parallel Builds: Support for multiple concurrent builds
• Optimized Dependencies: Efficient package installation and management

4. Production Readiness

• CI/CD Integration: Seamless integration with automated build systems
• Monitoring: Built-in monitoring and logging capabilities
• Scalability: Support for large-scale build operations

Migration Strategy

Phase 1: Parallel Development

• Continue developing debian-forge independently
• Develop mock integration in parallel
• Maintain compatibility with existing functionality

Phase 2: Integration Testing

• Create integration test suite
• Test mock integration with existing manifests
• Validate performance and functionality

Phase 3: Gradual Migration

• Add mock support as optional feature
• Migrate existing workflows to use mock environments
• Deprecate non-mock builds over time

Success Criteria

Technical Goals

• Mock environments successfully provisioned for debian-forge builds
• All existing APT stages work within mock environments
• Build performance improved through environment caching
• Cross-architecture builds supported through mock

Integration Goals

• Seamless integration with existing debian-forge workflows
• CI/CD pipeline updated to use mock environments
• Comprehensive documentation for mock integration
• User migration guide and examples

Production Goals

• Production-ready mock integration
• Performance benchmarks showing improvement
• Comprehensive testing and validation
• Community adoption and feedback

Implementation Responsibilities

debian-forge Project Tasks ✅ COMPLETED / 🔄 IN PROGRESS / ❌ PENDING

Phase 1: Basic Integration (Weeks 1-4)

debian-forge Responsibilities:

• Integration Plan - Comprehensive integration plan documented
• Architecture Design - Clear integration architecture defined
• Mock Stage Implementation - Create org.osbuild.deb-mock stage
  • Create stages/org.osbuild.deb-mock.py with basic functionality
  • Implement mock environment provisioning interface
  • Add configuration mapping between debian-forge and deb-mock
  • Create mock environment lifecycle management class
• APT Stage Modification - Modify existing APT stages for mock compatibility
  • Update org.osbuild.apt stage to work within mock chroots
  • Modify org.osbuild.apt.config stage for mock environments
  • Update org.osbuild.debootstrap stage for mock integration
  • Add environment variable and mount point management
• Basic Testing - Create integration test framework
  • Create integration test manifests for mock environments
  • Test simple Debian image builds with mock
  • Validate artifact collection and output from mock

deb-mock Project Dependencies:

• Python API - Stable Python API for integration
• Environment Management - Chroot environment creation and management
• Package Installation - Package installation within mock environments
• Command Execution - Command execution within mock chroots
• Artifact Collection - Artifact collection from mock environments

Phase 2: Advanced Integration (Weeks 5-8)

debian-forge Responsibilities:

• Plugin System Integration - Integrate with deb-mock's plugin system
  • Create debian-forge specific plugins for mock
  • Implement caching and optimization plugins
  • Add plugin configuration management
• Multi-Environment Support - Support multiple Debian distributions
  • Extend mock configuration for different Debian suites
  • Add cross-architecture build support through mock
  • Implement environment-specific optimizations
• Performance Optimization - Optimize build performance
  • Implement build environment caching
  • Add parallel build support with mock
  • Optimize package installation and dependency resolution

deb-mock Project Dependencies:

• Plugin Architecture - Stable plugin system for extensions
• Multi-Environment Support - Support for different Debian suites
• Cross-Architecture Support - ARM64, amd64, etc. support
• Caching System - Environment caching and reuse
• Parallel Execution - Parallel environment management

Phase 3: Production Integration (Weeks 9-12)

debian-forge Responsibilities:

• CI/CD Integration - Update CI workflows for mock
  • Update Forgejo CI workflows to use mock environments
  • Add build environment management to CI
  • Implement automated testing and validation
• Advanced Features - Production-ready features
  • Implement build environment snapshots
  • Add debugging and troubleshooting tools
  • Create comprehensive monitoring and logging

deb-mock Project Dependencies:

• Production Stability - Production-ready stability and reliability
• Monitoring Support - Built-in monitoring and logging capabilities
• Debugging Tools - Debugging and troubleshooting support
• Documentation - Comprehensive API documentation

Current Status Summary

debian-forge Project Status:

• ✅ Planning Complete - Integration plan and architecture designed
• ✅ Documentation Complete - Comprehensive integration documentation
• ❌ Implementation Pending - Mock stage and integration code needed
• ❌ Testing Pending - Integration test framework needed

deb-mock Project Status:

• 🔄 Foundation Development - Currently in Phase 1 development
• ❌ API Stability Pending - Python API needs to be stable for integration
• ❌ Production Readiness Pending - Needs to reach production-ready state
• ❌ Integration Support Pending - Integration features need to be implemented

Critical Path Dependencies

debian-forge Cannot Proceed Without:

1. Stable deb-mock Python API - Required for mock stage implementation
2. Environment Management API - Required for chroot environment creation
3. Command Execution API - Required for running debian-forge stages in mock
4. Artifact Collection API - Required for collecting build outputs

deb-mock Project Priority Items:

1. Python API Development - Create stable Python API for integration
2. Environment Management - Implement chroot environment lifecycle
3. Command Execution - Add command execution within mock environments
4. Documentation - Provide comprehensive API documentation

Recommended Next Steps

For debian-forge Project:

1. Wait for deb-mock API - Monitor deb-mock development for stable API
2. Create Mock Stage Skeleton - Create basic mock stage structure
3. Design Integration Tests - Create test framework for mock integration
4. Document Integration Requirements - Document specific API requirements

For deb-mock Project:

1. Prioritize Python API - Focus on stable Python API for integration
2. Implement Environment Management - Add chroot environment lifecycle
3. Add Command Execution - Implement command execution within mock
4. Create Integration Examples - Provide examples for debian-forge integration

Success Criteria

debian-forge Integration Complete When:

• Mock stage successfully provisions deb-mock environments
• All APT stages work within mock environments
• Build performance improved through environment caching
• CI/CD pipeline uses mock environments
• Comprehensive testing validates integration

deb-mock Project Ready When:

• Stable Python API available
• Environment management fully implemented
• Command execution working reliably
• Production-ready stability achieved
• Comprehensive documentation available

This integration requires coordinated development between both projects, with deb-mock providing the foundation infrastructure and debian-forge implementing the integration layer.