deb-bootc-image-builder/docs/ci-cd-integration.md

CI/CD Integration Guide for particle-os

particle-os is designed to be a first-class CI/CD tool for OS image building. This guide shows you how to integrate particle-os into any CI/CD system.

🚀 Why particle-os is CI/CD Friendly

Built-in CI/CD Features:

• ✅ Non-interactive mode - No prompts, pure automation
• ✅ JSON output - Machine-readable build results
• ✅ Quiet mode - Suppressed non-essential output
• ✅ Proper exit codes - Standard status codes for CI systems
• ✅ Structured output - Predictable, parseable results
• ✅ Artifact management - Organized output files

CI/CD Usage Pattern:

# Basic CI/CD build
particle-os build --json --quiet recipes/debian-server.yml

# With custom output and cleanup
particle-os build --json --quiet --output /artifacts/server.img --clean recipes/debian-server.yml

🔧 GitHub Actions Integration

Basic OS Image Build Workflow:

# .github/workflows/build-os.yml
name: Build OS Images

on:
  push:
    branches: [main]
    paths: ['recipes/**', 'src/**']
  pull_request:
    branches: [main]

jobs:
  build-images:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        recipe: [debian-server, debian-desktop, debian-gaming]
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        
      - name: Setup Go
        uses: actions/setup-go@v4
        with:
          go-version: '1.21'
          
      - name: Build particle-os
        run: |
          cd bib
          go build -o particle-os cmd/builder/main.go
          
      - name: Build OS Image
        run: |
          sudo ./bib/particle-os build --json --quiet --clean recipes/${{ matrix.recipe }}.yml
          
      - name: Parse build results
        id: build-results
        run: |
          BUILD_OUTPUT=$(sudo ./bib/particle-os build --json --quiet --clean recipes/${{ matrix.recipe }}.yml)
          echo "build_output<<EOF" >> $GITHUB_OUTPUT
          echo "$BUILD_OUTPUT" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT
          
          # Extract success status
          SUCCESS=$(echo "$BUILD_OUTPUT" | jq -r '.success')
          echo "success=$SUCCESS" >> $GITHUB_OUTPUT
          
          # Extract image path
          IMAGE_PATH=$(echo "$BUILD_OUTPUT" | jq -r '.image_path')
          echo "image_path=$IMAGE_PATH" >> $GITHUB_OUTPUT
          
      - name: Upload artifacts
        if: steps.build-results.outputs.success == 'true'
        uses: actions/upload-artifact@v3
        with:
          name: ${{ matrix.recipe }}-images
          path: |
            ${{ steps.build-results.outputs.image_path }}
            /tmp/particle-os-build/output/*.raw
            /tmp/particle-os-build/output/*.qcow2
            /tmp/particle-os-build/output/*.vmdk
          retention-days: 30
          
      - name: Build summary
        run: |
          echo "## Build Results for ${{ matrix.recipe }}" >> $GITHUB_STEP_SUMMARY
          echo "- **Status**: ${{ steps.build-results.outputs.success }}" >> $GITHUB_STEP_SUMMARY
          echo "- **Image**: ${{ steps.build-results.outputs.image_path }}" >> $GITHUB_STEP_SUMMARY
          echo "- **Recipe**: ${{ matrix.recipe }}" >> $GITHUB_STEP_SUMMARY

Advanced Multi-Stage Workflow:

# .github/workflows/advanced-os-build.yml
name: Advanced OS Image Pipeline

on:
  push:
    tags: ['v*']

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate recipes
        run: |
          cd bib && go build -o particle-os cmd/builder/main.go
          cd ..
          for recipe in recipes/*.yml; do
            echo "Validating $recipe..."
            ./bib/particle-os validate "$recipe"
          done
          
  build-base:
    needs: validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build base images
        run: |
          cd bib && go build -o particle-os cmd/builder/main.go
          cd ..
          sudo ./bib/particle-os build --json --quiet --clean recipes/debian-minimal.yml
          
  build-variants:
    needs: [validate, build-base]
    runs-on: ubuntu-latest
    strategy:
      matrix:
        variant: [server, desktop, gaming]
    steps:
      - uses: actions/checkout@v4
      - name: Build variant
        run: |
          cd bib && go build -o particle-os cmd/builder/main.go
          cd ..
          sudo ./bib/particle-os build --json --quiet --clean recipes/debian-${{ matrix.variant }}.yml
          
  test-images:
    needs: build-variants
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Download artifacts
        uses: actions/download-artifact@v3
        with:
          pattern: debian-*-images
          
      - name: Test image boot
        run: |
          # Basic image validation
          for img in *.qcow2; do
            echo "Testing $img..."
            qemu-img info "$img"
            # Add more validation as needed
          done
          
  release:
    needs: test-images
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/')
    steps:
      - uses: actions/checkout@v4
      - name: Download artifacts
        uses: actions/download-artifact@v3
        with:
          pattern: debian-*-images
          
      - name: Create release
        uses: softprops/action-gh-release@v1
        with:
          files: |
            debian-*-images/*.qcow2
            debian-*-images/*.raw
            debian-*-images/*.vmdk
          body: |
            ## OS Images Built with particle-os
            
            This release contains Debian-based OS images built using particle-os.
            
            ### Available Images:
            - **debian-server**: Server-optimized Debian
            - **debian-desktop**: Desktop Debian with GNOME
            - **debian-gaming**: Gaming-optimized Debian
            
            ### Build Details:
            - Built with particle-os v${{ github.ref_name }}
            - Base: Debian Trixie
            - Formats: raw, qcow2, vmdk

🔧 GitLab CI Integration

Basic GitLab CI Pipeline:

# .gitlab-ci.yml
stages:
  - validate
  - build
  - test
  - deploy

variables:
  WORK_DIR: "/tmp/particle-os-build"

validate-recipes:
  stage: validate
  script:
    - cd bib && go build -o particle-os cmd/builder/main.go
    - cd ..
    - for recipe in recipes/*.yml; do
        echo "Validating $recipe..."
        ./bib/particle-os validate "$recipe"
      done
  artifacts:
    paths:
      - bib/particle-os
    expire_in: 1 hour

build-server:
  stage: build
  script:
    - sudo ./bib/particle-os build --json --quiet --clean recipes/debian-server.yml
  artifacts:
    paths:
      - /tmp/particle-os-build/output/
    expire_in: 1 week
  dependencies:
    - validate-recipes

build-desktop:
  stage: build
  script:
    - sudo ./bib/particle-os build --json --quiet --clean recipes/debian-desktop.yml
  artifacts:
    paths:
      - /tmp/particle-os-build/output/
    expire_in: 1 week
  dependencies:
    - validate-recipes

test-images:
  stage: test
  script:
    - echo "Testing built images..."
    - for img in /tmp/particle-os-build/output/*.qcow2; do
        echo "Validating $img..."
        qemu-img info "$img"
      done
  dependencies:
    - build-server
    - build-desktop

deploy-images:
  stage: deploy
  script:
    - echo "Deploying images to artifact storage..."
    - # Add your deployment logic here
  dependencies:
    - test-images
  only:
    - main

🔧 Jenkins Integration

Jenkins Pipeline Script:

// Jenkinsfile
pipeline {
    agent any
    
    environment {
        WORKSPACE_DIR = '/tmp/particle-os-build'
        RECIPES_DIR = 'recipes'
    }
    
    stages {
        stage('Setup') {
            steps {
                sh 'cd bib && go build -o particle-os cmd/builder/main.go'
                sh 'cd ..'
            }
        }
        
        stage('Validate Recipes') {
            steps {
                script {
                    def recipes = sh(
                        script: "find ${RECIPES_DIR} -name '*.yml' -type f",
                        returnStdout: true
                    ).trim().split('\n')
                    
                    for (recipe in recipes) {
                        echo "Validating ${recipe}..."
                        sh "./bib/particle-os validate ${recipe}"
                    }
                }
            }
        }
        
        stage('Build Images') {
            parallel {
                stage('Build Server') {
                    steps {
                        sh '''
                            sudo ./bib/particle-os build --json --quiet --clean recipes/debian-server.yml
                            # Parse JSON output for Jenkins
                            BUILD_OUTPUT=$(sudo ./bib/particle-os build --json --quiet --clean recipes/debian-server.yml)
                            echo "BUILD_OUTPUT=${BUILD_OUTPUT}" > build.env
                        '''
                        archiveArtifacts artifacts: 'build.env'
                    }
                }
                
                stage('Build Desktop') {
                    steps {
                        sh '''
                            sudo ./bib/particle-os build --json --quiet --clean recipes/debian-desktop.yml
                            BUILD_OUTPUT=$(sudo ./bib/particle-os build --json --quiet --clean recipes/debian-desktop.yml)
                            echo "BUILD_OUTPUT=${BUILD_OUTPUT}" > build.env
                        '''
                        archiveArtifacts artifacts: 'build.env'
                    }
                }
            }
        }
        
        stage('Test Images') {
            steps {
                sh '''
                    echo "Testing built images..."
                    for img in /tmp/particle-os-build/output/*.qcow2; do
                        echo "Validating $img..."
                        qemu-img info "$img"
                    done
                '''
            }
        }
        
        stage('Archive Artifacts') {
            steps {
                archiveArtifacts artifacts: '/tmp/particle-os-build/output/**/*'
            }
        }
    }
    
    post {
        always {
            cleanWs()
        }
        success {
            echo "All OS images built successfully!"
        }
        failure {
            echo "OS image build failed!"
        }
    }
}

🔧 Generic CI/CD Integration

Shell Script for Any CI/CD System:

#!/bin/bash
# build-os-images.sh - Generic CI/CD script for particle-os

set -euo pipefail

# Configuration
RECIPE_DIR="recipes"
OUTPUT_DIR="/artifacts"
BUILD_LOG="/tmp/build.log"

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

log() {
    echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')] $1${NC}"
}

warn() {
    echo -e "${YELLOW}[$(date +'%Y-%m %H:%M:%S')] WARNING: $1${NC}"
}

error() {
    echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ERROR: $1${NC}"
    exit 1
}

# Build particle-os binary
build_particle_os() {
    log "Building particle-os binary..."
    cd bib
    if ! go build -o particle-os cmd/builder/main.go; then
        error "Failed to build particle-os binary"
    fi
    cd ..
    log "particle-os binary built successfully"
}

# Validate all recipes
validate_recipes() {
    log "Validating all recipes..."
    local failed_recipes=()
    
    for recipe in "$RECIPE_DIR"/*.yml; do
        if [[ -f "$recipe" ]]; then
            log "Validating $recipe..."
            if ! ./bib/particle-os validate "$recipe"; then
                failed_recipes+=("$recipe")
            fi
        fi
    done
    
    if [[ ${#failed_recipes[@]} -gt 0 ]]; then
        error "Recipe validation failed for: ${failed_recipes[*]}"
    fi
    
    log "All recipes validated successfully"
}

# Build OS image from recipe
build_image() {
    local recipe="$1"
    local recipe_name=$(basename "$recipe" .yml)
    
    log "Building image from recipe: $recipe_name"
    
    # Build with CI/CD flags
    local build_output
    if ! build_output=$(sudo ./bib/particle-os build --json --quiet --clean "$recipe" 2>&1); then
        error "Build failed for $recipe_name"
    fi
    
    # Parse JSON output
    local success
    local image_path
    local exit_code
    
    success=$(echo "$build_output" | jq -r '.success // false')
    image_path=$(echo "$build_output" | jq -r '.image_path // ""')
    exit_code=$(echo "$build_output" | jq -r '.exit_code // 1')
    
    if [[ "$success" != "true" ]] || [[ "$exit_code" != "0" ]]; then
        error "Build failed for $recipe_name (exit code: $exit_code)"
    fi
    
    if [[ -z "$image_path" ]]; then
        error "No image path returned for $recipe_name"
    fi
    
    log "Build successful for $recipe_name: $image_path"
    
    # Copy to artifacts directory
    if [[ -n "$OUTPUT_DIR" ]]; then
        mkdir -p "$OUTPUT_DIR/$recipe_name"
        cp "$image_path" "$OUTPUT_DIR/$recipe_name/"
        
        # Copy all output formats
        local work_dir
        work_dir=$(echo "$build_output" | jq -r '.work_directory // ""')
        if [[ -n "$work_dir" ]] && [[ -d "$work_dir/output" ]]; then
            cp "$work_dir/output"/* "$OUTPUT_DIR/$recipe_name/" 2>/dev/null || true
        fi
        
        log "Artifacts copied to $OUTPUT_DIR/$recipe_name"
    fi
    
    # Output build info for CI/CD system
    echo "BUILD_SUCCESS_$recipe_name=true" >> "$GITHUB_ENV" 2>/dev/null || true
    echo "IMAGE_PATH_$recipe_name=$image_path" >> "$GITHUB_ENV" 2>/dev/null || true
}

# Main build process
main() {
    log "Starting OS image build process..."
    
    # Check prerequisites
    if ! command -v go &> /dev/null; then
        error "Go is not installed"
    fi
    
    if ! command -v jq &> /dev/null; then
        error "jq is not installed"
    fi
    
    # Build particle-os
    build_particle_os
    
    # Validate recipes
    validate_recipes
    
    # Build all images
    local build_count=0
    local success_count=0
    
    for recipe in "$RECIPE_DIR"/*.yml; do
        if [[ -f "$recipe" ]]; then
            build_count=$((build_count + 1))
            if build_image "$recipe"; then
                success_count=$((success_count + 1))
            fi
        fi
    done
    
    # Summary
    log "Build process completed: $success_count/$build_count images built successfully"
    
    if [[ $success_count -eq $build_count ]]; then
        log "All images built successfully!"
        exit 0
    else
        error "Some images failed to build"
    fi
}

# Run main function
main "$@"

🔧 Docker Integration

Dockerfile for CI/CD:

# Dockerfile.ci
FROM debian:bookworm-slim

# Install dependencies
RUN apt-get update && apt-get install -y \
    golang-go \
    jq \
    qemu-utils \
    podman \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Copy source code
COPY . .

# Build particle-os
RUN cd bib && go build -o particle-os cmd/builder/main.go

# Create recipes directory
RUN mkdir -p /recipes

# Set entrypoint
ENTRYPOINT ["./bib/particle-os"]

# Default command
CMD ["--help"]

Docker Compose for CI/CD:

# docker-compose.ci.yml
version: '3.8'

services:
  particle-os-builder:
    build:
      context: .
      dockerfile: Dockerfile.ci
    volumes:
      - ./recipes:/recipes
      - ./artifacts:/artifacts
      - /var/lib/containers:/var/lib/containers
    environment:
      - RECIPES_DIR=/recipes
      - OUTPUT_DIR=/artifacts
    command: ["build", "--json", "--quiet", "--clean", "/recipes/debian-server.yml"]
    privileged: true  # Required for container operations

🔧 Best Practices for CI/CD

1. Always Use CI/CD Flags:

# ✅ Good - CI/CD friendly
particle-os build --json --quiet --clean recipes/debian-server.yml

# ❌ Bad - Human-friendly only
particle-os build recipes/debian-server.yml

2. Parse JSON Output:

# Extract build information
BUILD_OUTPUT=$(particle-os build --json --quiet recipes/debian-server.yml)
SUCCESS=$(echo "$BUILD_OUTPUT" | jq -r '.success')
IMAGE_PATH=$(echo "$BUILD_OUTPUT" | jq -r '.image_path')
EXIT_CODE=$(echo "$BUILD_OUTPUT" | jq -r '.exit_code')

if [[ "$SUCCESS" == "true" ]] && [[ "$EXIT_CODE" == "0" ]]; then
    echo "Build successful: $IMAGE_PATH"
else
    echo "Build failed"
    exit 1
fi

3. Handle Artifacts Properly:

# Create organized artifact structure
mkdir -p artifacts/$RECIPE_NAME
cp "$IMAGE_PATH" artifacts/$RECIPE_NAME/
cp /tmp/particle-os-build/output/* artifacts/$RECIPE_NAME/

4. Clean Up After Builds:

# Always use --clean flag in CI/CD
particle-os build --json --quiet --clean recipes/debian-server.yml

# Or manually clean up
sudo rm -rf /tmp/particle-os-build

5. Set Proper Exit Codes:

# Check exit code from particle-os
if particle-os build --json --quiet recipes/debian-server.yml; then
    echo "Build successful"
    exit 0
else
    echo "Build failed"
    exit 1
fi

🚀 Next Steps

Now that you have CI/CD integration examples, you can:

1. Choose your CI/CD system and implement the appropriate example
2. Customize the workflows for your specific needs
3. Add more validation and testing steps
4. Implement artifact management for your deployment pipeline
5. Add monitoring and alerting for build failures

particle-os is designed to work seamlessly in any CI/CD environment, providing reliable, reproducible OS image builds with machine-readable output perfect for automation.