Complete file structure reorganization for 1:1 osbuild compatibility
This commit is contained in:
robojerk
parent
61e7caaddb
commit
56f029cbc0
77 changed files with 5 additions and 956 deletions
524
docs/debian/user-documentation.md
Normal file
524
docs/debian/user-documentation.md
Normal file
|
|
@ -0,0 +1,524 @@
|
|||
# Debian Forge User Documentation
|
||||
|
||||
## Overview
|
||||
Debian Forge is a Debian-focused fork of OSBuild that enables building Debian atomic images with OSTree support. It combines OSBuild's pipeline architecture with Debian package management and atomic system capabilities.
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
- **System**: Debian 12+ or Ubuntu 22.04+
|
||||
- **Python**: Python 3.9+
|
||||
- **Go**: Go 1.21+ (for debos integration)
|
||||
- **OSTree**: OSTree 2023.1+
|
||||
- **Build Tools**: sbuild, pbuilder, debootstrap, mmdebstrap
|
||||
|
||||
### System Dependencies
|
||||
```bash
|
||||
# Install required packages
|
||||
sudo apt update
|
||||
sudo apt install -y \
|
||||
python3-pip \
|
||||
python3-venv \
|
||||
ostree \
|
||||
debootstrap \
|
||||
mmdebstrap \
|
||||
sbuild \
|
||||
pbuilder \
|
||||
bubblewrap \
|
||||
qemu-utils \
|
||||
curl \
|
||||
skopeo \
|
||||
git
|
||||
|
||||
# Install Go
|
||||
wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
|
||||
sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
|
||||
export PATH=$PATH:/usr/local/go/bin
|
||||
```
|
||||
|
||||
### Python Dependencies
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python3 -m venv debian-forge-env
|
||||
source debian-forge-env/bin/activate
|
||||
|
||||
# Install Python packages
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### Build Environment Setup
|
||||
```bash
|
||||
# Run setup script
|
||||
./setup-build-env.sh
|
||||
|
||||
# Configure apt-cacher-ng (optional)
|
||||
./setup-apt-cacher.sh
|
||||
```
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### 1. Simple Debian Base Image
|
||||
```bash
|
||||
# Build basic Debian image
|
||||
python3 -m osbuild --libdir . test-debian-manifest.json
|
||||
```
|
||||
|
||||
**Manifest Example** (`test-debian-manifest.json`):
|
||||
```json
|
||||
{
|
||||
"pipeline": {
|
||||
"build": {
|
||||
"pipeline": {
|
||||
"stages": [
|
||||
{
|
||||
"name": "org.osbuild.debootstrap",
|
||||
"options": {
|
||||
"suite": "bookworm",
|
||||
"mirror": "http://deb.debian.org/debian",
|
||||
"arch": "amd64",
|
||||
"apt_proxy": "http://192.168.1.101:3142"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "org.osbuild.apt",
|
||||
"options": {
|
||||
"packages": ["systemd", "linux-image-amd64"],
|
||||
"apt_proxy": "http://192.168.1.101:3142"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Debian Atomic Image
|
||||
```bash
|
||||
# Build Debian atomic image with OSTree
|
||||
python3 -m osbuild --libdir . test-debian-atomic-manifest.json
|
||||
```
|
||||
|
||||
**Atomic Manifest Example** (`test-debian-atomic-manifest.json`):
|
||||
```json
|
||||
{
|
||||
"pipelines": [
|
||||
{
|
||||
"name": "debian-base",
|
||||
"stages": [
|
||||
{
|
||||
"name": "org.osbuild.debootstrap",
|
||||
"options": {
|
||||
"suite": "bookworm",
|
||||
"mirror": "http://deb.debian.org/debian",
|
||||
"arch": "amd64",
|
||||
"apt_proxy": "http://192.168.1.101:3142"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "org.osbuild.apt.config",
|
||||
"options": {
|
||||
"sources": {
|
||||
"main": "deb http://deb.debian.org/debian bookworm main",
|
||||
"security": "deb http://security.debian.org/debian-security bookworm-security main"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "org.osbuild.apt",
|
||||
"options": {
|
||||
"packages": ["systemd", "linux-image-amd64", "ostree"],
|
||||
"apt_proxy": "http://192.168.1.101:3142"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "ostree-commit",
|
||||
"stages": [
|
||||
{
|
||||
"name": "org.osbuild.ostree.commit",
|
||||
"options": {
|
||||
"branch": "debian/bookworm/amd64",
|
||||
"subject": "Debian Bookworm Atomic Base"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### 1. Custom Package Building
|
||||
```bash
|
||||
# Build custom Debian package
|
||||
python3 -m osbuild --libdir . custom-package-manifest.json
|
||||
```
|
||||
|
||||
**Package Build Manifest**:
|
||||
```json
|
||||
{
|
||||
"pipeline": {
|
||||
"build": {
|
||||
"pipeline": {
|
||||
"stages": [
|
||||
{
|
||||
"name": "org.osbuild.debootstrap",
|
||||
"options": {
|
||||
"suite": "bookworm",
|
||||
"mirror": "http://deb.debian.org/debian",
|
||||
"arch": "amd64"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "org.osbuild.debian.source",
|
||||
"options": {
|
||||
"package": "my-package",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "org.osbuild.sbuild",
|
||||
"options": {
|
||||
"package": "my-package",
|
||||
"arch": "amd64"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Multi-Architecture Builds
|
||||
```bash
|
||||
# Build for multiple architectures
|
||||
for arch in amd64 arm64; do
|
||||
python3 -m osbuild --libdir . \
|
||||
--options '{"arch": "'$arch'"}' \
|
||||
multi-arch-manifest.json
|
||||
done
|
||||
```
|
||||
|
||||
### 3. OSTree Repository Management
|
||||
```bash
|
||||
# Initialize OSTree repository
|
||||
ostree init --mode=archive-z2 /path/to/repo
|
||||
|
||||
# List branches
|
||||
ostree refs --repo=/path/to/repo
|
||||
|
||||
# Show commit history
|
||||
ostree log --repo=/path/to/repo debian/bookworm/amd64
|
||||
```
|
||||
|
||||
## Build Orchestration
|
||||
|
||||
### 1. Using Build Orchestrator
|
||||
```python
|
||||
from build_orchestrator import BuildOrchestrator
|
||||
|
||||
# Initialize orchestrator
|
||||
orchestrator = BuildOrchestrator()
|
||||
|
||||
# Submit build
|
||||
build_id = orchestrator.submit_build(
|
||||
manifest_path="test-debian-atomic-manifest.json",
|
||||
priority="high"
|
||||
)
|
||||
|
||||
# Monitor build status
|
||||
status = orchestrator.get_build_status(build_id)
|
||||
print(f"Build {build_id}: {status}")
|
||||
|
||||
# List all builds
|
||||
builds = orchestrator.list_builds()
|
||||
for build in builds:
|
||||
print(f"Build {build.id}: {build.status}")
|
||||
```
|
||||
|
||||
### 2. Resource Management
|
||||
```python
|
||||
# Check system resources
|
||||
resources = orchestrator.resource_manager.get_system_resources()
|
||||
print(f"CPU: {resources.cpu_percent}%")
|
||||
print(f"Memory: {resources.memory_percent}%")
|
||||
print(f"Disk: {resources.disk_percent}%")
|
||||
|
||||
# Set resource limits
|
||||
orchestrator.resource_manager.set_build_limits(
|
||||
max_cpu=80,
|
||||
max_memory=70,
|
||||
max_disk=85
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Build Queue Management
|
||||
```python
|
||||
# Submit multiple builds
|
||||
builds = []
|
||||
for i in range(5):
|
||||
build_id = orchestrator.submit_build(
|
||||
manifest_path=f"build-{i}.json",
|
||||
priority="normal"
|
||||
)
|
||||
builds.append(build_id)
|
||||
|
||||
# Monitor queue
|
||||
queue_status = orchestrator.get_queue_status()
|
||||
print(f"Pending: {queue_status.pending}")
|
||||
print(f"Running: {queue_status.running}")
|
||||
print(f"Completed: {queue_status.completed}")
|
||||
|
||||
# Cancel build
|
||||
orchestrator.cancel_build(builds[0])
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. APT Configuration
|
||||
```bash
|
||||
# Configure APT sources
|
||||
cat > /etc/apt/sources.list.d/debian-forge.list << EOF
|
||||
deb http://deb.debian.org/debian bookworm main
|
||||
deb http://security.debian.org/debian-security bookworm-security main
|
||||
deb http://deb.debian.org/debian bookworm-updates main
|
||||
EOF
|
||||
|
||||
# Configure apt-cacher-ng proxy
|
||||
cat > /etc/apt/apt.conf.d/99proxy << EOF
|
||||
Acquire::http::Proxy "http://192.168.1.101:3142";
|
||||
Acquire::https::Proxy "http://192.168.1.101:3142";
|
||||
EOF
|
||||
```
|
||||
|
||||
### 2. OSTree Configuration
|
||||
```bash
|
||||
# Configure OSTree repository
|
||||
ostree config set core.min-free-space-size 1GB
|
||||
ostree config set core.min-free-space-percent 5
|
||||
|
||||
# Set up remote for updates
|
||||
ostree remote add debian-atomic \
|
||||
https://your-repo.example.com/debian-atomic \
|
||||
--set=gpg-verify=false
|
||||
```
|
||||
|
||||
### 3. Build Environment Configuration
|
||||
```bash
|
||||
# Configure sbuild
|
||||
sudo sbuild-adduser $USER
|
||||
sudo sbuild-createchroot --include=eatmydata,ccache,distcc bookworm /var/lib/schroot/chroots/bookworm-amd64-sbuild http://deb.debian.org/debian
|
||||
|
||||
# Configure pbuilder
|
||||
pbuilder-dist bookworm create
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. Build Failures
|
||||
```bash
|
||||
# Check build logs
|
||||
tail -f /var/log/osbuild/build-*.log
|
||||
|
||||
# Verify manifest syntax
|
||||
python3 -m osbuild --libdir . --check-only manifest.json
|
||||
|
||||
# Test individual stages
|
||||
python3 test-apt-stage.py
|
||||
```
|
||||
|
||||
#### 2. Resource Issues
|
||||
```bash
|
||||
# Check system resources
|
||||
python3 test-resource-allocation.py
|
||||
|
||||
# Monitor build processes
|
||||
ps aux | grep osbuild
|
||||
htop
|
||||
|
||||
# Check disk space
|
||||
df -h
|
||||
du -sh .osbuild/
|
||||
```
|
||||
|
||||
#### 3. Package Issues
|
||||
```bash
|
||||
# Verify package availability
|
||||
apt-cache policy package-name
|
||||
|
||||
# Check repository configuration
|
||||
apt-cache policy
|
||||
|
||||
# Test package installation
|
||||
chroot /path/to/chroot apt update
|
||||
chroot /path/to/chroot apt install package-name
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
```bash
|
||||
# Enable debug logging
|
||||
export OSBUILD_DEBUG=1
|
||||
export OSBUILD_VERBOSE=1
|
||||
|
||||
# Run with debug output
|
||||
python3 -m osbuild --libdir . --verbose manifest.json
|
||||
```
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
#### 1. Caching
|
||||
```bash
|
||||
# Configure apt-cacher-ng
|
||||
sudo systemctl enable apt-cacher-ng
|
||||
sudo systemctl start apt-cacher-ng
|
||||
|
||||
# Use local mirror
|
||||
echo 'deb http://192.168.1.101:3142/deb.debian.org/debian bookworm main' > sources.list
|
||||
```
|
||||
|
||||
#### 2. Parallel Builds
|
||||
```python
|
||||
# Configure concurrent builds
|
||||
orchestrator.resource_manager.set_concurrent_builds(4)
|
||||
|
||||
# Monitor performance
|
||||
python3 test-performance-optimization.py
|
||||
```
|
||||
|
||||
#### 3. Resource Allocation
|
||||
```python
|
||||
# Optimize resource allocation
|
||||
orchestrator.resource_manager.set_build_limits(
|
||||
max_cpu=75, # Leave headroom for system
|
||||
max_memory=80, # Prevent OOM
|
||||
max_disk=90 # Maintain free space
|
||||
)
|
||||
```
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### 1. CI/CD Pipeline
|
||||
```yaml
|
||||
# .github/workflows/debian-forge.yml
|
||||
name: Debian Forge Build
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
sudo apt update
|
||||
sudo apt install -y ostree debootstrap sbuild
|
||||
pip install -r requirements.txt
|
||||
- name: Build Debian atomic image
|
||||
run: |
|
||||
python3 -m osbuild --libdir . test-debian-atomic-manifest.json
|
||||
```
|
||||
|
||||
### 2. Docker Integration
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM debian:bookworm-slim
|
||||
|
||||
# Install Debian Forge
|
||||
RUN apt update && apt install -y \
|
||||
python3-pip \
|
||||
ostree \
|
||||
debootstrap \
|
||||
sbuild
|
||||
|
||||
COPY . /debian-forge
|
||||
WORKDIR /debian-forge
|
||||
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
# Build image
|
||||
CMD ["python3", "-m", "osbuild", "--libdir", ".", "manifest.json"]
|
||||
```
|
||||
|
||||
### 3. API Integration
|
||||
```python
|
||||
# REST API client
|
||||
import requests
|
||||
|
||||
class DebianForgeAPI:
|
||||
def __init__(self, base_url):
|
||||
self.base_url = base_url
|
||||
|
||||
def submit_build(self, manifest):
|
||||
response = requests.post(
|
||||
f"{self.base_url}/api/v1/builds",
|
||||
json={"manifest": manifest}
|
||||
)
|
||||
return response.json()
|
||||
|
||||
def get_build_status(self, build_id):
|
||||
response = requests.get(
|
||||
f"{self.base_url}/api/v1/builds/{build_id}"
|
||||
)
|
||||
return response.json()
|
||||
|
||||
# Usage
|
||||
api = DebianForgeAPI("http://localhost:8080")
|
||||
build = api.submit_build(manifest_data)
|
||||
status = api.get_build_status(build["id"])
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Manifest Design
|
||||
- Use descriptive pipeline names
|
||||
- Group related stages logically
|
||||
- Include proper error handling
|
||||
- Document complex configurations
|
||||
|
||||
### 2. Resource Management
|
||||
- Monitor system resources during builds
|
||||
- Set appropriate concurrent build limits
|
||||
- Use caching for frequently accessed data
|
||||
- Implement cleanup policies
|
||||
|
||||
### 3. Security
|
||||
- Validate all input manifests
|
||||
- Use isolated build environments
|
||||
- Implement proper access controls
|
||||
- Monitor build processes
|
||||
|
||||
### 4. Performance
|
||||
- Profile build performance regularly
|
||||
- Optimize stage execution order
|
||||
- Use parallel processing where possible
|
||||
- Implement intelligent caching strategies
|
||||
|
||||
## Support and Community
|
||||
|
||||
### Resources
|
||||
- **Documentation**: This guide and project README
|
||||
- **Issues**: GitHub issue tracker
|
||||
- **Discussions**: GitHub discussions
|
||||
- **Matrix**: #debian-forge on matrix.org
|
||||
|
||||
### Contributing
|
||||
- Follow OSBuild contribution guidelines
|
||||
- Maintain 1:1 compatibility where possible
|
||||
- Test changes thoroughly
|
||||
- Update documentation for new features
|
||||
|
||||
### Getting Help
|
||||
- Check troubleshooting section first
|
||||
- Search existing issues
|
||||
- Create detailed bug reports
|
||||
- Join community discussions
|
||||
Loading…
Add table
Add a link
Reference in a new issue