particle-os/debian-forge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
debian-forge/debian-forge-docs/user-documentation.md

12 KiB
Raw Blame History

Debian Forge User Documentation

Overview

Debian Forge is a Debian-focused fork of OSBuild that enables building Debian atomic images with OSTree support. It combines OSBuild's pipeline architecture with Debian package management and atomic system capabilities.

Installation

Prerequisites

  • System: Debian 12+ or Ubuntu 22.04+
  • Python: Python 3.9+
  • Go: Go 1.21+ (for debos integration)
  • OSTree: OSTree 2023.1+
  • Build Tools: sbuild, pbuilder, debootstrap, mmdebstrap

System Dependencies

# Install required packages
sudo apt update
sudo apt install -y \
    python3-pip \
    python3-venv \
    ostree \
    debootstrap \
    mmdebstrap \
    sbuild \
    pbuilder \
    bubblewrap \
    qemu-utils \
    curl \
    skopeo \
    git

# Install Go
wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

Python Dependencies

# Create virtual environment
python3 -m venv debian-forge-env
source debian-forge-env/bin/activate

# Install Python packages
pip install -r requirements.txt

Build Environment Setup

# Run setup script
./setup-build-env.sh

# Configure apt-cacher-ng (optional)
./setup-apt-cacher.sh

Basic Usage

1. Simple Debian Base Image

# Build basic Debian image
python3 -m osbuild --libdir . test-debian-manifest.json

Manifest Example (test-debian-manifest.json):

{
  "pipeline": {
    "build": {
      "pipeline": {
        "stages": [
          {
            "name": "org.osbuild.debootstrap",
            "options": {
              "suite": "bookworm",
              "mirror": "http://deb.debian.org/debian",
              "arch": "amd64",
              "apt_proxy": "http://192.168.1.101:3142"
            }
          },
          {
            "name": "org.osbuild.apt",
            "options": {
              "packages": ["systemd", "linux-image-amd64"],
              "apt_proxy": "http://192.168.1.101:3142"
            }
          }
        ]
      }
    }
  }
}

2. Debian Atomic Image

# Build Debian atomic image with OSTree
python3 -m osbuild --libdir . test-debian-atomic-manifest.json

Atomic Manifest Example (test-debian-atomic-manifest.json):

{
  "pipelines": [
    {
      "name": "debian-base",
      "stages": [
        {
          "name": "org.osbuild.debootstrap",
          "options": {
            "suite": "bookworm",
            "mirror": "http://deb.debian.org/debian",
            "arch": "amd64",
            "apt_proxy": "http://192.168.1.101:3142"
          }
        },
        {
          "name": "org.osbuild.apt.config",
          "options": {
            "sources": {
              "main": "deb http://deb.debian.org/debian bookworm main",
              "security": "deb http://security.debian.org/debian-security bookworm-security main"
            }
          }
        },
        {
          "name": "org.osbuild.apt",
          "options": {
            "packages": ["systemd", "linux-image-amd64", "ostree"],
            "apt_proxy": "http://192.168.1.101:3142"
          }
        }
      ]
    },
    {
      "name": "ostree-commit",
      "stages": [
        {
          "name": "org.osbuild.ostree.commit",
          "options": {
            "branch": "debian/bookworm/amd64",
            "subject": "Debian Bookworm Atomic Base"
          }
        }
      ]
    }
  ]
}

Advanced Usage

1. Custom Package Building

# Build custom Debian package
python3 -m osbuild --libdir . custom-package-manifest.json

Package Build Manifest:

{
  "pipeline": {
    "build": {
      "pipeline": {
        "stages": [
          {
            "name": "org.osbuild.debootstrap",
            "options": {
              "suite": "bookworm",
              "mirror": "http://deb.debian.org/debian",
              "arch": "amd64"
            }
          },
          {
            "name": "org.osbuild.debian.source",
            "options": {
              "package": "my-package",
              "version": "1.0.0"
            }
          },
          {
            "name": "org.osbuild.sbuild",
            "options": {
              "package": "my-package",
              "arch": "amd64"
            }
          }
        ]
      }
    }
  }
}

2. Multi-Architecture Builds

# Build for multiple architectures
for arch in amd64 arm64; do
  python3 -m osbuild --libdir . \
    --options '{"arch": "'$arch'"}' \
    multi-arch-manifest.json
done

3. OSTree Repository Management

# Initialize OSTree repository
ostree init --mode=archive-z2 /path/to/repo

# List branches
ostree refs --repo=/path/to/repo

# Show commit history
ostree log --repo=/path/to/repo debian/bookworm/amd64

Build Orchestration

1. Using Build Orchestrator

from build_orchestrator import BuildOrchestrator

# Initialize orchestrator
orchestrator = BuildOrchestrator()

# Submit build
build_id = orchestrator.submit_build(
    manifest_path="test-debian-atomic-manifest.json",
    priority="high"
)

# Monitor build status
status = orchestrator.get_build_status(build_id)
print(f"Build {build_id}: {status}")

# List all builds
builds = orchestrator.list_builds()
for build in builds:
    print(f"Build {build.id}: {build.status}")

2. Resource Management

# Check system resources
resources = orchestrator.resource_manager.get_system_resources()
print(f"CPU: {resources.cpu_percent}%")
print(f"Memory: {resources.memory_percent}%")
print(f"Disk: {resources.disk_percent}%")

# Set resource limits
orchestrator.resource_manager.set_build_limits(
    max_cpu=80,
    max_memory=70,
    max_disk=85
)

3. Build Queue Management

# Submit multiple builds
builds = []
for i in range(5):
    build_id = orchestrator.submit_build(
        manifest_path=f"build-{i}.json",
        priority="normal"
    )
    builds.append(build_id)

# Monitor queue
queue_status = orchestrator.get_queue_status()
print(f"Pending: {queue_status.pending}")
print(f"Running: {queue_status.running}")
print(f"Completed: {queue_status.completed}")

# Cancel build
orchestrator.cancel_build(builds[0])

Configuration

1. APT Configuration

# Configure APT sources
cat > /etc/apt/sources.list.d/debian-forge.list << EOF
deb http://deb.debian.org/debian bookworm main
deb http://security.debian.org/debian-security bookworm-security main
deb http://deb.debian.org/debian bookworm-updates main
EOF

# Configure apt-cacher-ng proxy
cat > /etc/apt/apt.conf.d/99proxy << EOF
Acquire::http::Proxy "http://192.168.1.101:3142";
Acquire::https::Proxy "http://192.168.1.101:3142";
EOF

2. OSTree Configuration

# Configure OSTree repository
ostree config set core.min-free-space-size 1GB
ostree config set core.min-free-space-percent 5

# Set up remote for updates
ostree remote add debian-atomic \
  https://your-repo.example.com/debian-atomic \
  --set=gpg-verify=false

3. Build Environment Configuration

# Configure sbuild
sudo sbuild-adduser $USER
sudo sbuild-createchroot --include=eatmydata,ccache,distcc bookworm /var/lib/schroot/chroots/bookworm-amd64-sbuild http://deb.debian.org/debian

# Configure pbuilder
pbuilder-dist bookworm create

Troubleshooting

Common Issues

1. Build Failures

# Check build logs
tail -f /var/log/osbuild/build-*.log

# Verify manifest syntax
python3 -m osbuild --libdir . --check-only manifest.json

# Test individual stages
python3 test-apt-stage.py

2. Resource Issues

# Check system resources
python3 test-resource-allocation.py

# Monitor build processes
ps aux | grep osbuild
htop

# Check disk space
df -h
du -sh .osbuild/

3. Package Issues

# Verify package availability
apt-cache policy package-name

# Check repository configuration
apt-cache policy

# Test package installation
chroot /path/to/chroot apt update
chroot /path/to/chroot apt install package-name

Debug Mode

# Enable debug logging
export OSBUILD_DEBUG=1
export OSBUILD_VERBOSE=1

# Run with debug output
python3 -m osbuild --libdir . --verbose manifest.json

Performance Optimization

1. Caching

# Configure apt-cacher-ng
sudo systemctl enable apt-cacher-ng
sudo systemctl start apt-cacher-ng

# Use local mirror
echo 'deb http://192.168.1.101:3142/deb.debian.org/debian bookworm main' > sources.list

2. Parallel Builds

# Configure concurrent builds
orchestrator.resource_manager.set_concurrent_builds(4)

# Monitor performance
python3 test-performance-optimization.py

3. Resource Allocation

# Optimize resource allocation
orchestrator.resource_manager.set_build_limits(
    max_cpu=75,      # Leave headroom for system
    max_memory=80,   # Prevent OOM
    max_disk=90      # Maintain free space
)

Integration Examples

1. CI/CD Pipeline

# .github/workflows/debian-forge.yml
name: Debian Forge Build
on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - name: Install dependencies
        run: |
          sudo apt update
          sudo apt install -y ostree debootstrap sbuild
          pip install -r requirements.txt
      - name: Build Debian atomic image
        run: |
          python3 -m osbuild --libdir . test-debian-atomic-manifest.json

2. Docker Integration

# Dockerfile
FROM debian:bookworm-slim

# Install Debian Forge
RUN apt update && apt install -y \
    python3-pip \
    ostree \
    debootstrap \
    sbuild

COPY . /debian-forge
WORKDIR /debian-forge

RUN pip install -r requirements.txt

# Build image
CMD ["python3", "-m", "osbuild", "--libdir", ".", "manifest.json"]

3. API Integration

# REST API client
import requests

class DebianForgeAPI:
    def __init__(self, base_url):
        self.base_url = base_url
    
    def submit_build(self, manifest):
        response = requests.post(
            f"{self.base_url}/api/v1/builds",
            json={"manifest": manifest}
        )
        return response.json()
    
    def get_build_status(self, build_id):
        response = requests.get(
            f"{self.base_url}/api/v1/builds/{build_id}"
        )
        return response.json()

# Usage
api = DebianForgeAPI("http://localhost:8080")
build = api.submit_build(manifest_data)
status = api.get_build_status(build["id"])

Best Practices

1. Manifest Design

  • Use descriptive pipeline names
  • Group related stages logically
  • Include proper error handling
  • Document complex configurations

2. Resource Management

  • Monitor system resources during builds
  • Set appropriate concurrent build limits
  • Use caching for frequently accessed data
  • Implement cleanup policies

3. Security

  • Validate all input manifests
  • Use isolated build environments
  • Implement proper access controls
  • Monitor build processes

4. Performance

  • Profile build performance regularly
  • Optimize stage execution order
  • Use parallel processing where possible
  • Implement intelligent caching strategies

Support and Community

Resources

  • Documentation: This guide and project README
  • Issues: GitHub issue tracker
  • Discussions: GitHub discussions
  • Matrix: #debian-forge on matrix.org

Contributing

  • Follow OSBuild contribution guidelines
  • Maintain 1:1 compatibility where possible
  • Test changes thoroughly
  • Update documentation for new features

Getting Help

  • Check troubleshooting section first
  • Search existing issues
  • Create detailed bug reports
  • Join community discussions