apt-ostree/docs/CI_CD_GUIDE.md

# CI/CD Guide for Bootc OSTree Image Building

**Created**: August 21, 2024  
**Last Updated**: August 21, 2024  
**Status**: 📋 Implementation Guide

## Overview

This guide explains how to set up a CI/CD pipeline that automatically builds bootc OSTree OCI images from treefiles using apt-ostree. The pipeline runs in a container, installs the required tools, builds the image, and validates it with bootc lint.

## Requirements

### System Requirements
- **CI/CD Platform**: GitHub Actions, GitLab CI, or similar
- **Container Runtime**: Docker or Podman support
- **Base Image**: Debian 13+ or Ubuntu 24.04+
- **Memory**: Minimum 4GB RAM available
- **Storage**: Minimum 10GB free space
- **Network**: Access to Debian/Ubuntu package repositories

### Software Dependencies
- **apt-ostree**: For OSTree tree composition and container generation
- **bootc**: For image validation and linting
- **OSTree**: For tree management operations
- **APT tools**: For package management

### Repository Structure
```
your-repo/
├── .github/
│   └── workflows/
│       └── build-image.yml
├── treefile.yaml
├── Dockerfile.ci
└── README.md
```

## Implementation

### 1. CI/CD Workflow File

Create `.github/workflows/build-image.yml`:

```yaml
name: Build Bootc OSTree Image

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]
  workflow_dispatch:

jobs:
  build-image:
    runs-on: ubuntu-latest
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
      
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
      
    - name: Build CI container
      run: |
        docker build -f Dockerfile.ci -t apt-ostree-ci .
        
    - name: Build OSTree image
      run: |
        docker run --rm \
          --privileged \
          -v $(pwd):/workspace:z \
          -v /var/lib/docker:/var/lib/docker \
          apt-ostree-ci \
          /workspace/scripts/build-image.sh
          
    - name: Upload artifacts
      uses: actions/upload-artifact@v4
      with:
        name: bootc-image
        path: output/
        retention-days: 7
```

### 2. CI Container Dockerfile

Create `Dockerfile.ci`:

```dockerfile
FROM debian:bookworm-slim

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    wget \
    gnupg \
    software-properties-common \
    apt-transport-https \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Install OSTree
RUN apt-get update && apt-get install -y \
    ostree \
    libostree-1-1 \
    && rm -rf /var/lib/apt/lists/*

# Install apt-ostree
RUN wget -O /tmp/apt-ostree.deb \
    https://github.com/your-org/apt-ostree/releases/latest/download/apt-ostree_amd64.deb \
    && dpkg -i /tmp/apt-ostree.deb \
    && apt-get install -f -y \
    && rm /tmp/apt-ostree.deb

# Install bootc
RUN wget -O /tmp/bootc.deb \
    https://github.com/containers/bootc/releases/latest/download/bootc_amd64.deb \
    && dpkg -i /tmp/bootc.deb \
    && apt-get install -f -y \
    && rm /tmp/bootc.deb

# Create workspace directory
WORKDIR /workspace

# Copy build script
COPY scripts/build-image.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/build-image.sh

# Set environment variables
ENV DEBIAN_FRONTEND=noninteractive
ENV RUST_LOG=info

CMD ["/usr/local/bin/build-image.sh"]
```

### 3. Build Script

Create `scripts/build-image.sh`:

```bash
#!/bin/bash
set -e

echo "Starting OSTree image build..."

# Verify tools are available
echo "Checking required tools..."
apt-ostree --version
bootc --version
ostree --version

# Create output directory
mkdir -p /workspace/output

# Build OSTree image with container generation
echo "Building OSTree image..."
apt-ostree compose tree /workspace/treefile.yaml \
    --container \
    --verbose \
    --output-dir /workspace/output

# Verify image was created
if [ ! -f "/workspace/output/*.tar" ]; then
    echo "Error: Container image not found"
    exit 1
fi

# Run bootc lint on the generated image
echo "Running bootc lint..."
for image_file in /workspace/output/*.tar; do
    echo "Linting: $image_file"
    bootc lint "$image_file"
    
    if [ $? -eq 0 ]; then
        echo "Lint passed for $image_file"
    else
        echo "Lint failed for $image_file"
        exit 1
    fi
done

echo "Build completed successfully"
ls -la /workspace/output/
```

### 4. Example Treefile

Create `treefile.yaml`:

```yaml
ref: myapp/latest
repos:
  - name: debian
    url: http://deb.debian.org/debian
    gpg-keys: []
packages:
  include:
    - bash
    - systemd
    - curl
    - wget
  exclude: []
customizations:
  files:
    - path: /etc/hostname
      content: "myapp-container"
  scripts:
    - name: setup
      script: |
        echo "Setting up application environment"
        systemctl enable systemd-user-sessions
```

## Configuration Options

### Environment Variables
```yaml
env:
  APT_OSTREE_VERSION: "0.1.0"
  BOOTC_VERSION: "0.1.0"
  OSTREE_VERSION: "2025.2"
  DEBIAN_CODENAME: "bookworm"
```

### Build Parameters
```yaml
with:
  container-format: "docker-archive"
  output-dir: "output"
  verbose: true
  keep-artifacts: false
```

## Troubleshooting

### Common Issues

#### 1. Permission Errors
```bash
# Ensure container has proper privileges
docker run --privileged -v $(pwd):/workspace:z apt-ostree-ci
```

#### 2. Package Installation Failures
```bash
# Check repository availability
curl -I http://deb.debian.org/debian/dists/bookworm/Release

# Verify package names
apt-cache search bash
```

#### 3. OSTree Errors
```bash
# Check OSTree installation
ostree --version

# Verify repository permissions
ls -la /var/lib/ostree/
```

#### 4. Bootc Lint Failures
```bash
# Check image format
file output/*.tar

# Verify image contents
tar -tf output/*.tar | head -20
```

### Debug Mode
```bash
# Enable verbose logging
RUST_LOG=debug apt-ostree compose tree treefile.yaml --container --verbose

# Check container logs
docker logs apt-ostree-ci
```

## Performance Considerations

### Resource Limits
- **Memory**: 4-8GB recommended for builds
- **CPU**: 2-4 cores minimum
- **Storage**: 10-20GB for temporary files
- **Network**: Stable connection to package repositories

### Build Optimization
- **Caching**: Cache APT packages between builds
- **Parallel builds**: Use multiple workers if possible
- **Cleanup**: Remove temporary files after build
- **Artifact retention**: Keep only necessary outputs

## Security Notes

### Container Security
- **Privileged mode**: Required for OSTree operations
- **Volume mounts**: Limit access to necessary directories
- **Network access**: Restrict to required repositories only
- **User isolation**: Run as non-root when possible

### Package Security
- **GPG verification**: Verify package signatures
- **Repository validation**: Use trusted package sources
- **Update frequency**: Regular security updates
- **Vulnerability scanning**: Scan generated images

## Monitoring and Logging

### Build Metrics
```bash
# Build time tracking
time apt-ostree compose tree treefile.yaml --container

# Resource usage monitoring
docker stats apt-ostree-ci

# Output size tracking
du -sh output/
```

### Log Analysis
```bash
# Parse build logs
grep "ERROR\|WARN" build.log

# Extract timing information
grep "completed successfully" build.log

# Check package installation status
grep "installed successfully" build.log
```

## Alternative Implementations

### GitLab CI
```yaml
build-image:
  image: debian:bookworm-slim
  stage: build
  script:
    - apt-get update && apt-get install -y ostree apt-ostree bootc
    - apt-ostree compose tree treefile.yaml --container
    - bootc lint output/*.tar
  artifacts:
    paths:
      - output/
    expire_in: 1 week
```

### Jenkins Pipeline
```groovy
pipeline {
    agent { dockerfile true }
    stages {
        stage('Build') {
            steps {
                sh 'apt-ostree compose tree treefile.yaml --container'
            }
        }
        stage('Validate') {
            steps {
                sh 'bootc lint output/*.tar'
            }
        }
    }
}
```

## Conclusion

This CI/CD setup provides a way to automatically build and validate bootc OSTree images. The pipeline handles tool installation, image generation, and quality validation, ensuring consistent output across different environments.

Key success factors:
- Proper resource allocation
- Stable network connectivity
- Regular dependency updates
- Error handling
- Artifact management

For production use, consider adding:
- Image signing and verification
- Registry push capabilities
- Testing
- Performance monitoring
- Security scanning