robojerk 56f029cbc0 Complete file structure reorganization for 1:1 osbuild compatibility

2025-08-26 15:38:59 -07:00

20 KiB

Raw Blame History

Debian Forge Deployment Documentation

Overview

This document covers deploying Debian Forge in production environments, including system requirements, configuration, monitoring, and maintenance procedures.

System Requirements

Minimum Requirements

CPU: 4 cores (8 recommended)
Memory: 8GB RAM (16GB recommended)
Storage: 100GB available space (500GB recommended)
Network: Stable internet connection for package downloads
OS: Debian 12+ or Ubuntu 22.04+

Recommended Production Specs

CPU: 8+ cores with good single-thread performance
Memory: 32GB+ RAM for concurrent builds
Storage: 1TB+ SSD with good I/O performance
Network: Gigabit connection with low latency
OS: Debian 12+ with LTS support

Storage Requirements

/var/lib/debian-forge/          # Build artifacts and cache
├── builds/                     # Build outputs (50-200GB)
├── cache/                      # Package cache (20-100GB)
├── ostree/                     # OSTree repositories (100-500GB)
└── logs/                       # Build logs (10-50GB)

/tmp/                          # Temporary build space (50-100GB)
.osbuild/                      # OSBuild cache (20-100GB)

Production Deployment

1. System Preparation

# Update system
sudo apt update && sudo apt upgrade -y

# Install essential packages
sudo apt install -y \
    python3-pip \
    python3-venv \
    python3-dev \
    build-essential \
    ostree \
    debootstrap \
    mmdebstrap \
    sbuild \
    pbuilder \
    bubblewrap \
    qemu-utils \
    curl \
    skopeo \
    git \
    nginx \
    postgresql \
    redis-server \
    supervisor \
    logrotate \
    fail2ban \
    ufw

# Configure firewall
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 8080/tcp  # Debian Forge API
sudo ufw enable

2. User and Security Setup

# Create dedicated user
sudo useradd -m -s /bin/bash debian-forge
sudo usermod -aG sudo debian-forge
sudo usermod -aG sbuild debian-forge

# Configure sudo access
echo "debian-forge ALL=(ALL) NOPASSWD: /usr/bin/apt, /usr/bin/apt-get, /usr/bin/dpkg" | sudo tee /etc/sudoers.d/debian-forge

# Set up SSH keys
sudo mkdir -p /home/debian-forge/.ssh
sudo chown debian-forge:debian-forge /home/debian-forge/.ssh
sudo chmod 700 /home/debian-forge/.ssh

# Copy your SSH key
sudo -u debian-forge ssh-keygen -t ed25519 -C "debian-forge@$(hostname)"

3. Database Setup

# Configure PostgreSQL
sudo -u postgres createuser debian-forge
sudo -u postgres createdb debian_forge
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE debian_forge TO debian_forge;"

# Configure Redis
sudo systemctl enable redis-server
sudo systemctl start redis-server

4. Application Deployment

# Switch to debian-forge user
sudo su - debian-forge

# Clone repository
git clone https://github.com/your-org/debian-forge.git
cd debian-forge

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Install additional production packages
pip install gunicorn uwsgi psycopg2-binary redis supervisor

5. Configuration Files

Environment Configuration

# /home/debian-forge/debian-forge/.env
DEBIAN_FORGE_ENV=production
DEBIAN_FORGE_DEBUG=false
DEBIAN_FORGE_SECRET_KEY=your-secret-key-here
DEBIAN_FORGE_DATABASE_URL=postgresql://debian-forge@localhost/debian_forge
DEBIAN_FORGE_REDIS_URL=redis://localhost:6379/0
DEBIAN_FORGE_LOG_LEVEL=INFO
DEBIAN_FORGE_MAX_CONCURRENT_BUILDS=4
DEBIAN_FORGE_BUILD_TIMEOUT=3600
DEBIAN_FORGE_CACHE_SIZE=50GB
DEBIAN_FORGE_OSTREE_REPO_PATH=/var/lib/debian-forge/ostree

Build Environment Configuration

# /home/debian-forge/debian-forge/config/build-env.conf
[build_environment]
max_concurrent_builds = 4
build_timeout = 3600
resource_limits_cpu = 80
resource_limits_memory = 85
resource_limits_disk = 90
cleanup_after_build = true
cache_retention_days = 30

[ostree]
repo_path = /var/lib/debian-forge/ostree
max_repo_size = 100GB
cleanup_old_commits = true
commit_retention_days = 90

[apt]
proxy_url = http://192.168.1.101:3142
mirror_url = http://deb.debian.org/debian
security_url = http://security.debian.org/debian-security
updates_url = http://deb.debian.org/debian

6. Service Configuration

Supervisor Configuration

# /etc/supervisor/conf.d/debian-forge.conf
[program:debian-forge-api]
command=/home/debian-forge/debian-forge/venv/bin/gunicorn -w 4 -b 127.0.0.1:8080 --timeout 300 --max-requests 1000 --max-requests-jitter 100 app:app
directory=/home/debian-forge/debian-forge
user=debian-forge
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/debian-forge/api.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

[program:debian-forge-worker]
command=/home/debian-forge/debian-forge/venv/bin/python -m build_orchestrator
directory=/home/debian-forge/debian-forge
user=debian-forge
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/debian-forge/worker.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

[program:debian-forge-cleanup]
command=/home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager
directory=/home/debian-forge/debian-forge
user=debian-forge
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/debian-forge/cleanup.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

Nginx Configuration

# /etc/nginx/sites-available/debian-forge
server {
    listen 80;
    server_name your-domain.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name your-domain.com;

    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    client_max_body_size 100M;
    proxy_read_timeout 300s;
    proxy_connect_timeout 75s;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /static/ {
        alias /home/debian-forge/debian-forge/static/;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }

    location /logs/ {
        alias /var/log/debian-forge/;
        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

7. SSL Certificate Setup

# Install Certbot
sudo apt install -y certbot python3-certbot-nginx

# Obtain SSL certificate
sudo certbot --nginx -d your-domain.com

# Test auto-renewal
sudo certbot renew --dry-run

8. Monitoring and Logging

Logrotate Configuration

# /etc/logrotate.d/debian-forge
/var/log/debian-forge/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 debian-forge debian-forge
    postrotate
        systemctl reload supervisor
    endscript
}

Monitoring Scripts

#!/bin/bash
# /home/debian-forge/debian-forge/scripts/monitor.sh

# Check service status
check_service() {
    local service=$1
    if ! systemctl is-active --quiet $service; then
        echo "ERROR: $service is not running"
        systemctl restart $service
        echo "$(date): Restarted $service" >> /var/log/debian-forge/monitor.log
    fi
}

# Check disk space
check_disk() {
    local usage=$(df /var/lib/debian-forge | tail -1 | awk '{print $5}' | sed 's/%//')
    if [ $usage -gt 90 ]; then
        echo "WARNING: Disk usage is ${usage}%"
        # Trigger cleanup
        /home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager --force
    fi
}

# Check memory usage
check_memory() {
    local usage=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100.0}')
    if [ $usage -gt 90 ]; then
        echo "WARNING: Memory usage is ${usage}%"
    fi
}

# Main monitoring loop
while true; do
    check_service debian-forge-api
    check_service debian-forge-worker
    check_service debian-forge-cleanup
    check_disk
    check_memory
    sleep 300  # Check every 5 minutes
done

Production Configuration

1. Performance Tuning

System Tuning

# /etc/sysctl.conf
# Increase file descriptor limits
fs.file-max = 65536
fs.inotify.max_user_watches = 524288

# Network tuning
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 65535

# Memory tuning
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5

Application Tuning

# /home/debian-forge/debian-forge/config/performance.conf
[performance]
max_workers = 4
worker_timeout = 300
max_requests = 1000
max_requests_jitter = 100
keepalive = 2
worker_connections = 1000

[caching]
cache_size = 50GB
cache_ttl = 86400
cache_cleanup_interval = 3600

[build_optimization]
parallel_stages = true
stage_cache_enabled = true
artifact_compression = true

2. Security Configuration

Fail2ban Configuration

# /etc/fail2ban/jail.local
[debian-forge-api]
enabled = true
port = 8080
filter = debian-forge-api
logpath = /var/log/debian-forge/api.log
maxretry = 5
bantime = 3600
findtime = 600

[debian-forge-ssh]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
findtime = 600

Access Control

# /etc/nginx/.htpasswd (for log access)
sudo htpasswd -c /etc/nginx/.htpasswd admin

# SSH key-based authentication only
sudo sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl reload ssh

3. Backup Configuration

Backup Script

#!/bin/bash
# /home/debian-forge/debian-forge/scripts/backup.sh

BACKUP_DIR="/var/backups/debian-forge"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=30

# Create backup directory
mkdir -p $BACKUP_DIR

# Database backup
pg_dump -U debian-forge debian_forge > $BACKUP_DIR/db_$DATE.sql

# Configuration backup
tar -czf $BACKUP_DIR/config_$DATE.tar.gz \
    /home/debian-forge/debian-forge/config \
    /etc/supervisor/conf.d/debian-forge.conf \
    /etc/nginx/sites-available/debian-forge

# OSTree repository backup
rsync -av --delete /var/lib/debian-forge/ostree/ $BACKUP_DIR/ostree_$DATE/

# Cleanup old backups
find $BACKUP_DIR -type f -mtime +$RETENTION_DAYS -delete
find $BACKUP_DIR -type d -mtime +$RETENTION_DAYS -exec rm -rf {} +

echo "Backup completed: $DATE" >> /var/log/debian-forge/backup.log

Automated Backup

# /etc/cron.daily/debian-forge-backup
#!/bin/bash
/home/debian-forge/debian-forge/scripts/backup.sh

Maintenance Procedures

1. Regular Maintenance

Daily Tasks

# Check service status
sudo supervisorctl status

# Monitor logs
tail -f /var/log/debian-forge/*.log

# Check disk space
df -h /var/lib/debian-forge

# Check build queue
curl -s http://localhost:8080/api/v1/queue/status

Weekly Tasks

# Update system packages
sudo apt update && sudo apt upgrade -y

# Clean old build artifacts
/home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager --force

# Rotate logs
sudo logrotate -f /etc/logrotate.d/debian-forge

# Check SSL certificate expiration
sudo certbot certificates

Monthly Tasks

# Review and clean old OSTree commits
/home/debian-forge/debian-forge/venv/bin/python -m ostree_cleanup --older-than 90

# Update application dependencies
cd /home/debian-forge/debian-forge
source venv/bin/activate
pip install --upgrade -r requirements.txt

# Review and update security configurations
sudo fail2ban-client status
sudo ufw status

2. Troubleshooting

Common Issues

Service Not Starting

# Check supervisor status
sudo supervisorctl status

# Check logs
sudo tail -f /var/log/supervisor/supervisord.log
sudo tail -f /var/log/debian-forge/*.log

# Restart services
sudo supervisorctl restart debian-forge-api
sudo supervisorctl restart debian-forge-worker

Build Failures

# Check build logs
tail -f /var/log/debian-forge/worker.log

# Check system resources
htop
df -h
free -h

# Restart worker
sudo supervisorctl restart debian-forge-worker

Database Issues

# Check PostgreSQL status
sudo systemctl status postgresql

# Check connection
sudo -u debian-forge psql -d debian_forge -c "SELECT version();"

# Restart database
sudo systemctl restart postgresql

3. Recovery Procedures

Service Recovery

#!/bin/bash
# /home/debian-forge/debian-forge/scripts/recovery.sh

echo "Starting Debian Forge recovery..."

# Stop all services
sudo supervisorctl stop all

# Clean up temporary files
sudo rm -rf /tmp/debian-forge-*
sudo rm -rf /var/tmp/debian-forge-*

# Restart database
sudo systemctl restart postgresql
sudo systemctl restart redis-server

# Wait for services to be ready
sleep 10

# Start services
sudo supervisorctl start all

# Check status
sudo supervisorctl status

echo "Recovery completed"

Data Recovery

#!/bin/bash
# /home/debian-forge/debian-forge/scripts/data-recovery.sh

BACKUP_DIR="/var/backups/debian-forge"
LATEST_BACKUP=$(ls -t $BACKUP_DIR/db_*.sql | head -1)

if [ -n "$LATEST_BACKUP" ]; then
    echo "Restoring from backup: $LATEST_BACKUP"
    
    # Stop services
    sudo supervisorctl stop all
    
    # Restore database
    sudo -u postgres dropdb debian_forge
    sudo -u postgres createdb debian_forge
    sudo -u postgres psql debian_forge < $LATEST_BACKUP
    
    # Restart services
    sudo supervisorctl start all
    
    echo "Data recovery completed"
else
    echo "No backup found for recovery"
    exit 1
fi

Scaling Considerations

1. Horizontal Scaling

Load Balancer Configuration

# /etc/nginx/sites-available/debian-forge-cluster
upstream debian_forge_backend {
    server 192.168.1.10:8080;
    server 192.168.1.11:8080;
    server 192.168.1.12:8080;
}

server {
    listen 443 ssl http2;
    server_name your-domain.com;
    
    location / {
        proxy_pass http://debian_forge_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Shared Storage

# NFS configuration for shared storage
# /etc/exports
/var/lib/debian-forge 192.168.1.0/24(rw,sync,no_subtree_check)

# Mount on worker nodes
# /etc/fstab
192.168.1.10:/var/lib/debian-forge /var/lib/debian-forge nfs defaults 0 0

2. Vertical Scaling

Resource Optimization

# /home/debian-forge/debian-forge/config/scaling.conf
[scaling]
max_concurrent_builds = 8
worker_processes = 8
memory_limit = 32GB
cpu_limit = 8

[cache]
cache_size = 100GB
cache_ttl = 172800  # 48 hours

Monitoring and Alerting

1. Health Checks

Application Health

# /home/debian-forge/debian-forge/health_check.py
import requests
import psutil
import os

def check_health():
    health_status = {
        "status": "healthy",
        "checks": {}
    }
    
    # Check API endpoint
    try:
        response = requests.get("http://localhost:8080/health", timeout=5)
        health_status["checks"]["api"] = "healthy" if response.status_code == 200 else "unhealthy"
    except:
        health_status["checks"]["api"] = "unhealthy"
    
    # Check system resources
    cpu_percent = psutil.cpu_percent()
    memory_percent = psutil.virtual_memory().percent
    disk_percent = psutil.disk_usage('/var/lib/debian-forge').percent
    
    health_status["checks"]["cpu"] = "healthy" if cpu_percent < 90 else "warning"
    health_status["checks"]["memory"] = "healthy" if memory_percent < 90 else "warning"
    health_status["checks"]["disk"] = "healthy" if disk_percent < 90 else "warning"
    
    # Overall status
    if any(check == "unhealthy" for check in health_status["checks"].values()):
        health_status["status"] = "unhealthy"
    elif any(check == "warning" for check in health_status["checks"].values()):
        health_status["status"] = "degraded"
    
    return health_status

Monitoring Dashboard

<!-- /home/debian-forge/debian-forge/templates/monitor.html -->
<!DOCTYPE html>
<html>
<head>
    <title>Debian Forge Monitor</title>
    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
</head>
<body>
    <h1>Debian Forge System Monitor</h1>
    
    <div class="status-grid">
        <div class="status-card">
            <h3>API Status</h3>
            <div id="api-status">Checking...</div>
        </div>
        
        <div class="status-card">
            <h3>Build Queue</h3>
            <div id="queue-status">Checking...</div>
        </div>
        
        <div class="status-card">
            <h3>System Resources</h3>
            <canvas id="resource-chart"></canvas>
        </div>
    </div>
    
    <script>
        // Update status every 30 seconds
        setInterval(updateStatus, 30000);
        
        function updateStatus() {
            fetch('/api/v1/health')
                .then(response => response.json())
                .then(data => {
                    document.getElementById('api-status').textContent = data.status;
                    // Update other status elements
                });
        }
    </script>
</body>
</html>

2. Alerting Configuration

Email Alerts

# /home/debian-forge/debian-forge/scripts/alert.py
import smtplib
from email.mime.text import MIMEText
import os

def send_alert(subject, message, severity="INFO"):
    smtp_server = os.getenv("SMTP_SERVER", "localhost")
    smtp_port = int(os.getenv("SMTP_PORT", "587"))
    smtp_user = os.getenv("SMTP_USER")
    smtp_password = os.getenv("SMTP_PASSWORD")
    alert_email = os.getenv("ALERT_EMAIL")
    
    msg = MIMEText(message)
    msg['Subject'] = f"[{severity}] Debian Forge: {subject}"
    msg['From'] = smtp_user
    msg['To'] = alert_email
    
    try:
        with smtplib.SMTP(smtp_server, smtp_port) as server:
            server.starttls()
            server.login(smtp_user, smtp_password)
            server.send_message(msg)
        print(f"Alert sent: {subject}")
    except Exception as e:
        print(f"Failed to send alert: {e}")

Alert Rules

# /home/debian-forge/debian-forge/config/alerts.yaml
alerts:
  - name: "High CPU Usage"
    condition: "cpu_percent > 90"
    severity: "WARNING"
    cooldown: 300
    
  - name: "High Memory Usage"
    condition: "memory_percent > 90"
    severity: "WARNING"
    cooldown: 300
    
  - name: "High Disk Usage"
    condition: "disk_percent > 90"
    severity: "CRITICAL"
    cooldown: 60
    
  - name: "Service Down"
    condition: "service_status != 'running'"
    severity: "CRITICAL"
    cooldown: 0
    
  - name: "Build Queue Full"
    condition: "queue_size > 100"
    severity: "WARNING"
    cooldown: 600

Conclusion

This deployment guide provides a comprehensive approach to deploying Debian Forge in production. Key points to remember:

Security First: Always configure firewalls, fail2ban, and SSL certificates
Monitoring: Implement comprehensive monitoring and alerting
Backups: Regular automated backups with tested recovery procedures
Maintenance: Scheduled maintenance windows for updates and cleanup
Scaling: Plan for both horizontal and vertical scaling from the start
Documentation: Keep deployment and maintenance procedures up to date

For additional support, refer to the troubleshooting section or create an issue in the project repository.

20 KiB Raw Blame History