524 lines
12 KiB
Markdown
524 lines
12 KiB
Markdown
# Debian Forge User Documentation
|
|
|
|
## Overview
|
|
Debian Forge is a Debian-focused fork of OSBuild that enables building Debian atomic images with OSTree support. It combines OSBuild's pipeline architecture with Debian package management and atomic system capabilities.
|
|
|
|
## Installation
|
|
|
|
### Prerequisites
|
|
- **System**: Debian 12+ or Ubuntu 22.04+
|
|
- **Python**: Python 3.9+
|
|
- **Go**: Go 1.21+ (for debos integration)
|
|
- **OSTree**: OSTree 2023.1+
|
|
- **Build Tools**: sbuild, pbuilder, debootstrap, mmdebstrap
|
|
|
|
### System Dependencies
|
|
```bash
|
|
# Install required packages
|
|
sudo apt update
|
|
sudo apt install -y \
|
|
python3-pip \
|
|
python3-venv \
|
|
ostree \
|
|
debootstrap \
|
|
mmdebstrap \
|
|
sbuild \
|
|
pbuilder \
|
|
bubblewrap \
|
|
qemu-utils \
|
|
curl \
|
|
skopeo \
|
|
git
|
|
|
|
# Install Go
|
|
wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
|
|
sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
|
|
export PATH=$PATH:/usr/local/go/bin
|
|
```
|
|
|
|
### Python Dependencies
|
|
```bash
|
|
# Create virtual environment
|
|
python3 -m venv debian-forge-env
|
|
source debian-forge-env/bin/activate
|
|
|
|
# Install Python packages
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
### Build Environment Setup
|
|
```bash
|
|
# Run setup script
|
|
./setup-build-env.sh
|
|
|
|
# Configure apt-cacher-ng (optional)
|
|
./setup-apt-cacher.sh
|
|
```
|
|
|
|
## Basic Usage
|
|
|
|
### 1. Simple Debian Base Image
|
|
```bash
|
|
# Build basic Debian image
|
|
python3 -m osbuild --libdir . test-debian-manifest.json
|
|
```
|
|
|
|
**Manifest Example** (`test-debian-manifest.json`):
|
|
```json
|
|
{
|
|
"pipeline": {
|
|
"build": {
|
|
"pipeline": {
|
|
"stages": [
|
|
{
|
|
"name": "org.osbuild.debootstrap",
|
|
"options": {
|
|
"suite": "bookworm",
|
|
"mirror": "http://deb.debian.org/debian",
|
|
"arch": "amd64",
|
|
"apt_proxy": "http://192.168.1.101:3142"
|
|
}
|
|
},
|
|
{
|
|
"name": "org.osbuild.apt",
|
|
"options": {
|
|
"packages": ["systemd", "linux-image-amd64"],
|
|
"apt_proxy": "http://192.168.1.101:3142"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 2. Debian Atomic Image
|
|
```bash
|
|
# Build Debian atomic image with OSTree
|
|
python3 -m osbuild --libdir . test-debian-atomic-manifest.json
|
|
```
|
|
|
|
**Atomic Manifest Example** (`test-debian-atomic-manifest.json`):
|
|
```json
|
|
{
|
|
"pipelines": [
|
|
{
|
|
"name": "debian-base",
|
|
"stages": [
|
|
{
|
|
"name": "org.osbuild.debootstrap",
|
|
"options": {
|
|
"suite": "bookworm",
|
|
"mirror": "http://deb.debian.org/debian",
|
|
"arch": "amd64",
|
|
"apt_proxy": "http://192.168.1.101:3142"
|
|
}
|
|
},
|
|
{
|
|
"name": "org.osbuild.apt.config",
|
|
"options": {
|
|
"sources": {
|
|
"main": "deb http://deb.debian.org/debian bookworm main",
|
|
"security": "deb http://security.debian.org/debian-security bookworm-security main"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"name": "org.osbuild.apt",
|
|
"options": {
|
|
"packages": ["systemd", "linux-image-amd64", "ostree"],
|
|
"apt_proxy": "http://192.168.1.101:3142"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"name": "ostree-commit",
|
|
"stages": [
|
|
{
|
|
"name": "org.osbuild.ostree.commit",
|
|
"options": {
|
|
"branch": "debian/bookworm/amd64",
|
|
"subject": "Debian Bookworm Atomic Base"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Advanced Usage
|
|
|
|
### 1. Custom Package Building
|
|
```bash
|
|
# Build custom Debian package
|
|
python3 -m osbuild --libdir . custom-package-manifest.json
|
|
```
|
|
|
|
**Package Build Manifest**:
|
|
```json
|
|
{
|
|
"pipeline": {
|
|
"build": {
|
|
"pipeline": {
|
|
"stages": [
|
|
{
|
|
"name": "org.osbuild.debootstrap",
|
|
"options": {
|
|
"suite": "bookworm",
|
|
"mirror": "http://deb.debian.org/debian",
|
|
"arch": "amd64"
|
|
}
|
|
},
|
|
{
|
|
"name": "org.osbuild.debian.source",
|
|
"options": {
|
|
"package": "my-package",
|
|
"version": "1.0.0"
|
|
}
|
|
},
|
|
{
|
|
"name": "org.osbuild.sbuild",
|
|
"options": {
|
|
"package": "my-package",
|
|
"arch": "amd64"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 2. Multi-Architecture Builds
|
|
```bash
|
|
# Build for multiple architectures
|
|
for arch in amd64 arm64; do
|
|
python3 -m osbuild --libdir . \
|
|
--options '{"arch": "'$arch'"}' \
|
|
multi-arch-manifest.json
|
|
done
|
|
```
|
|
|
|
### 3. OSTree Repository Management
|
|
```bash
|
|
# Initialize OSTree repository
|
|
ostree init --mode=archive-z2 /path/to/repo
|
|
|
|
# List branches
|
|
ostree refs --repo=/path/to/repo
|
|
|
|
# Show commit history
|
|
ostree log --repo=/path/to/repo debian/bookworm/amd64
|
|
```
|
|
|
|
## Build Orchestration
|
|
|
|
### 1. Using Build Orchestrator
|
|
```python
|
|
from build_orchestrator import BuildOrchestrator
|
|
|
|
# Initialize orchestrator
|
|
orchestrator = BuildOrchestrator()
|
|
|
|
# Submit build
|
|
build_id = orchestrator.submit_build(
|
|
manifest_path="test-debian-atomic-manifest.json",
|
|
priority="high"
|
|
)
|
|
|
|
# Monitor build status
|
|
status = orchestrator.get_build_status(build_id)
|
|
print(f"Build {build_id}: {status}")
|
|
|
|
# List all builds
|
|
builds = orchestrator.list_builds()
|
|
for build in builds:
|
|
print(f"Build {build.id}: {build.status}")
|
|
```
|
|
|
|
### 2. Resource Management
|
|
```python
|
|
# Check system resources
|
|
resources = orchestrator.resource_manager.get_system_resources()
|
|
print(f"CPU: {resources.cpu_percent}%")
|
|
print(f"Memory: {resources.memory_percent}%")
|
|
print(f"Disk: {resources.disk_percent}%")
|
|
|
|
# Set resource limits
|
|
orchestrator.resource_manager.set_build_limits(
|
|
max_cpu=80,
|
|
max_memory=70,
|
|
max_disk=85
|
|
)
|
|
```
|
|
|
|
### 3. Build Queue Management
|
|
```python
|
|
# Submit multiple builds
|
|
builds = []
|
|
for i in range(5):
|
|
build_id = orchestrator.submit_build(
|
|
manifest_path=f"build-{i}.json",
|
|
priority="normal"
|
|
)
|
|
builds.append(build_id)
|
|
|
|
# Monitor queue
|
|
queue_status = orchestrator.get_queue_status()
|
|
print(f"Pending: {queue_status.pending}")
|
|
print(f"Running: {queue_status.running}")
|
|
print(f"Completed: {queue_status.completed}")
|
|
|
|
# Cancel build
|
|
orchestrator.cancel_build(builds[0])
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### 1. APT Configuration
|
|
```bash
|
|
# Configure APT sources
|
|
cat > /etc/apt/sources.list.d/debian-forge.list << EOF
|
|
deb http://deb.debian.org/debian bookworm main
|
|
deb http://security.debian.org/debian-security bookworm-security main
|
|
deb http://deb.debian.org/debian bookworm-updates main
|
|
EOF
|
|
|
|
# Configure apt-cacher-ng proxy
|
|
cat > /etc/apt/apt.conf.d/99proxy << EOF
|
|
Acquire::http::Proxy "http://192.168.1.101:3142";
|
|
Acquire::https::Proxy "http://192.168.1.101:3142";
|
|
EOF
|
|
```
|
|
|
|
### 2. OSTree Configuration
|
|
```bash
|
|
# Configure OSTree repository
|
|
ostree config set core.min-free-space-size 1GB
|
|
ostree config set core.min-free-space-percent 5
|
|
|
|
# Set up remote for updates
|
|
ostree remote add debian-atomic \
|
|
https://your-repo.example.com/debian-atomic \
|
|
--set=gpg-verify=false
|
|
```
|
|
|
|
### 3. Build Environment Configuration
|
|
```bash
|
|
# Configure sbuild
|
|
sudo sbuild-adduser $USER
|
|
sudo sbuild-createchroot --include=eatmydata,ccache,distcc bookworm /var/lib/schroot/chroots/bookworm-amd64-sbuild http://deb.debian.org/debian
|
|
|
|
# Configure pbuilder
|
|
pbuilder-dist bookworm create
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### 1. Build Failures
|
|
```bash
|
|
# Check build logs
|
|
tail -f /var/log/osbuild/build-*.log
|
|
|
|
# Verify manifest syntax
|
|
python3 -m osbuild --libdir . --check-only manifest.json
|
|
|
|
# Test individual stages
|
|
python3 test-apt-stage.py
|
|
```
|
|
|
|
#### 2. Resource Issues
|
|
```bash
|
|
# Check system resources
|
|
python3 test-resource-allocation.py
|
|
|
|
# Monitor build processes
|
|
ps aux | grep osbuild
|
|
htop
|
|
|
|
# Check disk space
|
|
df -h
|
|
du -sh .osbuild/
|
|
```
|
|
|
|
#### 3. Package Issues
|
|
```bash
|
|
# Verify package availability
|
|
apt-cache policy package-name
|
|
|
|
# Check repository configuration
|
|
apt-cache policy
|
|
|
|
# Test package installation
|
|
chroot /path/to/chroot apt update
|
|
chroot /path/to/chroot apt install package-name
|
|
```
|
|
|
|
### Debug Mode
|
|
```bash
|
|
# Enable debug logging
|
|
export OSBUILD_DEBUG=1
|
|
export OSBUILD_VERBOSE=1
|
|
|
|
# Run with debug output
|
|
python3 -m osbuild --libdir . --verbose manifest.json
|
|
```
|
|
|
|
### Performance Optimization
|
|
|
|
#### 1. Caching
|
|
```bash
|
|
# Configure apt-cacher-ng
|
|
sudo systemctl enable apt-cacher-ng
|
|
sudo systemctl start apt-cacher-ng
|
|
|
|
# Use local mirror
|
|
echo 'deb http://192.168.1.101:3142/deb.debian.org/debian bookworm main' > sources.list
|
|
```
|
|
|
|
#### 2. Parallel Builds
|
|
```python
|
|
# Configure concurrent builds
|
|
orchestrator.resource_manager.set_concurrent_builds(4)
|
|
|
|
# Monitor performance
|
|
python3 test-performance-optimization.py
|
|
```
|
|
|
|
#### 3. Resource Allocation
|
|
```python
|
|
# Optimize resource allocation
|
|
orchestrator.resource_manager.set_build_limits(
|
|
max_cpu=75, # Leave headroom for system
|
|
max_memory=80, # Prevent OOM
|
|
max_disk=90 # Maintain free space
|
|
)
|
|
```
|
|
|
|
## Integration Examples
|
|
|
|
### 1. CI/CD Pipeline
|
|
```yaml
|
|
# .github/workflows/debian-forge.yml
|
|
name: Debian Forge Build
|
|
on: [push, pull_request]
|
|
|
|
jobs:
|
|
build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v3
|
|
- name: Setup Python
|
|
uses: actions/setup-python@v4
|
|
with:
|
|
python-version: '3.9'
|
|
- name: Install dependencies
|
|
run: |
|
|
sudo apt update
|
|
sudo apt install -y ostree debootstrap sbuild
|
|
pip install -r requirements.txt
|
|
- name: Build Debian atomic image
|
|
run: |
|
|
python3 -m osbuild --libdir . test-debian-atomic-manifest.json
|
|
```
|
|
|
|
### 2. Docker Integration
|
|
```dockerfile
|
|
# Dockerfile
|
|
FROM debian:bookworm-slim
|
|
|
|
# Install Debian Forge
|
|
RUN apt update && apt install -y \
|
|
python3-pip \
|
|
ostree \
|
|
debootstrap \
|
|
sbuild
|
|
|
|
COPY . /debian-forge
|
|
WORKDIR /debian-forge
|
|
|
|
RUN pip install -r requirements.txt
|
|
|
|
# Build image
|
|
CMD ["python3", "-m", "osbuild", "--libdir", ".", "manifest.json"]
|
|
```
|
|
|
|
### 3. API Integration
|
|
```python
|
|
# REST API client
|
|
import requests
|
|
|
|
class DebianForgeAPI:
|
|
def __init__(self, base_url):
|
|
self.base_url = base_url
|
|
|
|
def submit_build(self, manifest):
|
|
response = requests.post(
|
|
f"{self.base_url}/api/v1/builds",
|
|
json={"manifest": manifest}
|
|
)
|
|
return response.json()
|
|
|
|
def get_build_status(self, build_id):
|
|
response = requests.get(
|
|
f"{self.base_url}/api/v1/builds/{build_id}"
|
|
)
|
|
return response.json()
|
|
|
|
# Usage
|
|
api = DebianForgeAPI("http://localhost:8080")
|
|
build = api.submit_build(manifest_data)
|
|
status = api.get_build_status(build["id"])
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
### 1. Manifest Design
|
|
- Use descriptive pipeline names
|
|
- Group related stages logically
|
|
- Include proper error handling
|
|
- Document complex configurations
|
|
|
|
### 2. Resource Management
|
|
- Monitor system resources during builds
|
|
- Set appropriate concurrent build limits
|
|
- Use caching for frequently accessed data
|
|
- Implement cleanup policies
|
|
|
|
### 3. Security
|
|
- Validate all input manifests
|
|
- Use isolated build environments
|
|
- Implement proper access controls
|
|
- Monitor build processes
|
|
|
|
### 4. Performance
|
|
- Profile build performance regularly
|
|
- Optimize stage execution order
|
|
- Use parallel processing where possible
|
|
- Implement intelligent caching strategies
|
|
|
|
## Support and Community
|
|
|
|
### Resources
|
|
- **Documentation**: This guide and project README
|
|
- **Issues**: GitHub issue tracker
|
|
- **Discussions**: GitHub discussions
|
|
- **Matrix**: #debian-forge on matrix.org
|
|
|
|
### Contributing
|
|
- Follow OSBuild contribution guidelines
|
|
- Maintain 1:1 compatibility where possible
|
|
- Test changes thoroughly
|
|
- Update documentation for new features
|
|
|
|
### Getting Help
|
|
- Check troubleshooting section first
|
|
- Search existing issues
|
|
- Create detailed bug reports
|
|
- Join community discussions
|