Add comprehensive testing framework, performance monitoring, and plugin system

- Add complete pytest testing framework with conftest.py and test files - Add performance monitoring and benchmarking capabilities - Add plugin system with ccache plugin example - Add comprehensive documentation (API, deployment, testing, etc.) - Add Docker API wrapper for service deployment - Add advanced configuration examples - Remove old wget package file - Update core modules with enhanced functionality
2025-08-19 20:49:32 -07:00 · 2025-08-19 20:49:32 -07:00 · c51819c836
commit c51819c836
parent 4c0dcb2522
30 changed files with 11141 additions and 105 deletions
--- a/docs/DEPLOYMENT.md
+++ b/docs/DEPLOYMENT.md
@ -0,0 +1,764 @@
+# Deb-Mock Deployment Guide
+
+## Overview
+
+This guide covers the deployment of `deb-mock` in various environments, from development to production. `deb-mock` is a sophisticated build environment management tool that provides isolated, reproducible package builds with advanced features like performance monitoring, plugin systems, and comprehensive testing.
+
+## Table of Contents
+
+1. [System Requirements](#system-requirements)
+2. [Installation Methods](#installation-methods)
+3. [Configuration](#configuration)
+4. [Environment Setup](#environment-setup)
+5. [Service Deployment](#service-deployment)
+6. [Production Deployment](#production-deployment)
+7. [Monitoring and Maintenance](#monitoring-and-maintenance)
+8. [Troubleshooting](#troubleshooting)
+9. [Security Considerations](#security-considerations)
+10. [Backup and Recovery](#backup-and-recovery)
+
+## System Requirements
+
+### Minimum Requirements
+
+- **Operating System**: Debian 13+ (Trixie) or Ubuntu 22.04+
+- **CPU**: 2 cores (4 recommended)
+- **Memory**: 4GB RAM (8GB recommended)
+- **Storage**: 20GB available space (50GB recommended)
+- **Python**: 3.8+ (3.10+ recommended)
+
+### Recommended Requirements
+
+- **Operating System**: Debian 13+ (Trixie) or Ubuntu 22.04+
+- **CPU**: 8+ cores
+- **Memory**: 16GB+ RAM
+- **Storage**: 100GB+ available space (SSD recommended)
+- **Python**: 3.10+
+
+### Required System Packages
+
+```bash
+# Debian/Ubuntu
+sudo apt update
+sudo apt install -y \
+    python3 \
+    python3-pip \
+    python3-venv \
+    python3-dev \
+    build-essential \
+    debootstrap \
+    schroot \
+    sbuild \
+    ccache \
+    rsync \
+    curl \
+    wget \
+    git \
+    sudo \
+    procps \
+    sysstat \
+    iotop \
+    htop
+
+# For advanced features
+sudo apt install -y \
+    python3-psutil \
+    python3-yaml \
+    python3-click \
+    python3-rich \
+    python3-pytest \
+    python3-pytest-cov \
+    python3-pytest-mock \
+    python3-pytest-xdist \
+    python3-pytest-timeout \
+    python3-pytest-html \
+    python3-pytest-json-report \
+    python3-coverage
+```
+
+## Installation Methods
+
+### Method 1: Python Package Installation (Recommended)
+
+```bash
+# Create virtual environment
+python3 -m venv deb-mock-env
+source deb-mock-env/bin/activate
+
+# Install from source
+git clone https://github.com/your-org/deb-mock.git
+cd deb-mock
+pip install -e .
+
+# Or install from PyPI (when available)
+pip install deb-mock
+```
+
+### Method 2: System-wide Installation
+
+```bash
+# Install system-wide (requires root)
+sudo pip3 install deb-mock
+
+# Or install from source
+sudo pip3 install -e .
+```
+
+### Method 3: Docker Installation
+
+```dockerfile
+FROM debian:13-slim
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    python3 \
+    python3-pip \
+    debootstrap \
+    schroot \
+    sbuild \
+    ccache \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install deb-mock
+COPY . /app/deb-mock
+WORKDIR /app/deb-mock
+RUN pip3 install -e .
+
+# Set up entry point
+ENTRYPOINT ["deb-mock"]
+```
+
+## Configuration
+
+### Configuration File Structure
+
+`deb-mock` uses YAML configuration files. The main configuration file is typically located at:
+
+- **User config**: `~/.config/deb-mock/config.yaml`
+- **System config**: `/etc/deb-mock/config.yaml`
+- **Project config**: `./deb-mock.yaml`
+
+### Basic Configuration Example
+
+```yaml
+# deb-mock.yaml
+chroot:
+  base_dir: /var/lib/deb-mock/chroots
+  suite: trixie
+  architecture: amd64
+  mirror: http://deb.debian.org/debian/
+  components: [main, contrib, non-free]
+
+cache:
+  enabled: true
+  base_dir: /var/cache/deb-mock
+  ccache_size_mb: 2048
+  root_cache_size_mb: 5120
+  package_cache_size_mb: 1024
+
+sbuild:
+  enabled: true
+  user: sbuild
+  group: sbuild
+  chroot_suffix: -sbuild
+  build_user: buildd
+
+performance:
+  enable_performance_monitoring: true
+  performance_metrics_dir: /var/log/deb-mock/performance
+  performance_retention_days: 30
+  performance_auto_optimization: true
+  performance_benchmark_iterations: 10
+  performance_reporting: true
+
+plugins:
+  enabled: true
+  plugin_dir: /usr/local/lib/deb-mock/plugins
+  auto_load: true
+
+parallel:
+  enabled: true
+  max_parallel_builds: 4
+  max_parallel_chroots: 8
+
+mounts:
+  proc: true
+  sys: true
+  dev: true
+  tmpfs: true
+  bind_mounts:
+    - source: /var/cache/apt/archives
+      target: /var/cache/apt/archives
+      options: [ro]
+  overlay_mounts:
+    - source: /var/cache/deb-mock/overlay
+      target: /var/cache/deb-mock/overlay
+
+uid_management:
+  enabled: true
+  create_users: true
+  copy_host_users: true
+  privilege_escalation: true
+```
+
+### Environment Variables
+
+```bash
+# Core configuration
+export DEB_MOCK_CONFIG=/path/to/config.yaml
+export DEB_MOCK_CHROOT_DIR=/var/lib/deb-mock/chroots
+export DEB_MOCK_CACHE_DIR=/var/cache/deb-mock
+
+# Performance monitoring
+export DEB_MOCK_PERFORMANCE_DIR=/var/log/deb-mock/performance
+export DEB_MOCK_ENABLE_PERFORMANCE_MONITORING=true
+
+# Plugin system
+export DEB_MOCK_PLUGIN_DIR=/usr/local/lib/deb-mock/plugins
+export DEB_MOCK_AUTO_LOAD_PLUGINS=true
+
+# Logging
+export DEB_MOCK_LOG_LEVEL=INFO
+export DEB_MOCK_LOG_FILE=/var/log/deb-mock/deb-mock.log
+```
+
+## Environment Setup
+
+### User Setup
+
+```bash
+# Create deb-mock user
+sudo useradd -m -s /bin/bash deb-mock
+sudo usermod -aG sbuild deb-mock
+
+# Set up user environment
+sudo -u deb-mock mkdir -p ~/.config/deb-mock
+sudo -u deb-mock mkdir -p ~/.cache/deb-mock
+sudo -u deb-mock mkdir -p ~/deb-mock-workspace
+
+# Configure sbuild for the user
+sudo -u deb-mock sbuild-update --keygen
+sudo -u deb-mock sbuild-adduser $USER
+```
+
+### Directory Structure Setup
+
+```bash
+# Create necessary directories
+sudo mkdir -p /var/lib/deb-mock/chroots
+sudo mkdir -p /var/cache/deb-mock/{ccache,root,packages,overlay}
+sudo mkdir -p /var/log/deb-mock/{performance,logs}
+sudo mkdir -p /usr/local/lib/deb-mock/plugins
+
+# Set proper permissions
+sudo chown -R deb-mock:deb-mock /var/lib/deb-mock
+sudo chown -R deb-mock:deb-mock /var/cache/deb-mock
+sudo chown -R deb-mock:deb-mock /var/log/deb-mock
+sudo chown -R deb-mock:deb-mock /usr/local/lib/deb-mock
+
+# Set proper permissions for sbuild
+sudo chown -R deb-mock:sbuild /var/lib/deb-mock/chroots
+sudo chmod 775 /var/lib/deb-mock/chroots
+```
+
+### Sbuild Configuration
+
+```bash
+# Configure sbuild for deb-mock user
+sudo -u deb-mock mkdir -p ~/.config/sbuild
+sudo -u deb-mock cat > ~/.config/sbuild/config.pl << 'EOF'
+$build_arch = 'amd64';
+$build_arch_all = 1;
+$build_source = 1;
+$build_binary = 1;
+$build_arch_indep = 1;
+$build_arch_all = 1;
+$build_profiles = ['default'];
+$build_environment = ['debian'];
+$build_suite = 'trixie';
+$build_components = ['main', 'contrib', 'non-free'];
+$build_mirror = 'http://deb.debian.org/debian/';
+$build_indep = 1;
+$build_arch_all = 1;
+$build_source = 1;
+$build_binary = 1;
+$build_arch_indep = 1;
+$build_arch_all = 1;
+$build_profiles = ['default'];
+$build_environment = ['debian'];
+$build_suite = 'trixie';
+$build_components = ['main', 'contrib', 'non-free'];
+$build_mirror = 'http://deb.debian.org/debian/';
+EOF
+```
+
+## Service Deployment
+
+### Systemd Service (Recommended)
+
+Create a systemd service file for production deployments:
+
+```ini
+# /etc/systemd/system/deb-mock.service
+[Unit]
+Description=Deb-Mock Build Service
+After=network.target
+Wants=network.target
+
+[Service]
+Type=simple
+User=deb-mock
+Group=deb-mock
+Environment=DEB_MOCK_CONFIG=/etc/deb-mock/config.yaml
+Environment=DEB_MOCK_LOG_LEVEL=INFO
+Environment=DEB_MOCK_LOG_FILE=/var/log/deb-mock/deb-mock.log
+WorkingDirectory=/var/lib/deb-mock
+ExecStart=/usr/local/bin/deb-mock service start
+ExecReload=/bin/kill -HUP $MAINPID
+Restart=always
+RestartSec=10
+StandardOutput=journal
+StandardError=journal
+
+# Security settings
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+ReadWritePaths=/var/lib/deb-mock /var/cache/deb-mock /var/log/deb-mock
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### Service Management
+
+```bash
+# Enable and start the service
+sudo systemctl daemon-reload
+sudo systemctl enable deb-mock.service
+sudo systemctl start deb-mock.service
+
+# Check service status
+sudo systemctl status deb-mock.service
+
+# View logs
+sudo journalctl -u deb-mock.service -f
+
+# Restart service
+sudo systemctl restart deb-mock.service
+```
+
+### Docker Compose Deployment
+
+```yaml
+# docker-compose.yml
+version: '3.8'
+
+services:
+  deb-mock:
+    build: .
+    container_name: deb-mock
+    restart: unless-stopped
+    environment:
+      - DEB_MOCK_CONFIG=/etc/deb-mock/config.yaml
+      - DEB_MOCK_LOG_LEVEL=INFO
+    volumes:
+      - ./config:/etc/deb-mock:ro
+      - deb-mock-chroots:/var/lib/deb-mock/chroots
+      - deb-mock-cache:/var/cache/deb-mock
+      - deb-mock-logs:/var/log/deb-mock
+    ports:
+      - "8080:8080"
+    networks:
+      - deb-mock-network
+
+volumes:
+  deb-mock-chroots:
+  deb-mock-cache:
+  deb-mock-logs:
+
+networks:
+  deb-mock-network:
+    driver: bridge
+```
+
+## Production Deployment
+
+### High Availability Setup
+
+```bash
+# Load balancer configuration (nginx)
+sudo apt install nginx
+
+# Create nginx configuration
+sudo tee /etc/nginx/sites-available/deb-mock << 'EOF'
+upstream deb-mock_backend {
+    server 127.0.0.1:8080;
+    server 127.0.0.1:8081;
+    server 127.0.0.1:8082;
+}
+
+server {
+    listen 80;
+    server_name deb-mock.yourdomain.com;
+
+    location / {
+        proxy_pass http://deb-mock_backend;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}
+EOF
+
+# Enable site
+sudo ln -s /etc/nginx/sites-available/deb-mock /etc/nginx/sites-enabled/
+sudo nginx -t
+sudo systemctl reload nginx
+```
+
+### Monitoring Setup
+
+```bash
+# Install monitoring tools
+sudo apt install -y prometheus node-exporter grafana
+
+# Configure Prometheus
+sudo tee /etc/prometheus/prometheus.yml << 'EOF'
+global:
+  scrape_interval: 15s
+
+scrape_configs:
+  - job_name: 'deb-mock'
+    static_configs:
+      - targets: ['localhost:8080']
+    metrics_path: /metrics
+    scrape_interval: 5s
+
+  - job_name: 'node-exporter'
+    static_configs:
+      - targets: ['localhost:9100']
+EOF
+
+# Start monitoring services
+sudo systemctl enable prometheus node-exporter grafana-server
+sudo systemctl start prometheus node-exporter grafana-server
+```
+
+### Backup Strategy
+
+```bash
+# Create backup script
+sudo tee /usr/local/bin/deb-mock-backup << 'EOF'
+#!/bin/bash
+
+BACKUP_DIR="/var/backups/deb-mock"
+DATE=$(date +%Y%m%d_%H%M%S)
+
+# Create backup directory
+mkdir -p "$BACKUP_DIR"
+
+# Backup configuration
+tar -czf "$BACKUP_DIR/config_$DATE.tar.gz" -C /etc deb-mock
+
+# Backup chroots (excluding temporary files)
+tar -czf "$BACKUP_DIR/chroots_$DATE.tar.gz" \
+    --exclude='*/tmp/*' \
+    --exclude='*/var/tmp/*' \
+    -C /var/lib deb-mock/chroots
+
+# Backup cache
+tar -czf "$BACKUP_DIR/cache_$DATE.tar.gz" -C /var/cache deb-mock
+
+# Backup logs
+tar -czf "$BACKUP_DIR/logs_$DATE.tar.gz" -C /var/log deb-mock
+
+# Clean up old backups (keep last 7 days)
+find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete
+
+echo "Backup completed: $BACKUP_DIR"
+EOF
+
+# Make executable and set up cron
+sudo chmod +x /usr/local/bin/deb-mock-backup
+sudo crontab -e
+# Add: 0 2 * * * /usr/local/bin/deb-mock-backup
+```
+
+## Monitoring and Maintenance
+
+### Health Checks
+
+```bash
+# Create health check script
+sudo tee /usr/local/bin/deb-mock-health << 'EOF'
+#!/bin/bash
+
+# Check service status
+if ! systemctl is-active --quiet deb-mock.service; then
+    echo "ERROR: deb-mock service is not running"
+    exit 1
+fi
+
+# Check disk space
+DISK_USAGE=$(df /var/lib/deb-mock | tail -1 | awk '{print $5}' | sed 's/%//')
+if [ "$DISK_USAGE" -gt 90 ]; then
+    echo "WARNING: Disk usage is ${DISK_USAGE}%"
+fi
+
+# Check memory usage
+MEM_USAGE=$(free | grep Mem | awk '{printf("%.0f", $3/$2 * 100.0)}')
+if [ "$MEM_USAGE" -gt 90 ]; then
+    echo "WARNING: Memory usage is ${MEM_USAGE}%"
+fi
+
+# Check chroot health
+if ! deb-mock status >/dev/null 2>&1; then
+    echo "ERROR: deb-mock status check failed"
+    exit 1
+fi
+
+echo "OK: All health checks passed"
+EOF
+
+sudo chmod +x /usr/local/bin/deb-mock-health
+```
+
+### Log Rotation
+
+```bash
+# Configure log rotation
+sudo tee /etc/logrotate.d/deb-mock << 'EOF'
+/var/log/deb-mock/*.log {
+    daily
+    missingok
+    rotate 52
+    compress
+    delaycompress
+    notifempty
+    create 644 deb-mock deb-mock
+    postrotate
+        systemctl reload deb-mock.service >/dev/null 2>&1 || true
+    endscript
+}
+EOF
+```
+
+### Performance Monitoring
+
+```bash
+# Set up performance monitoring
+sudo -u deb-mock mkdir -p /var/log/deb-mock/performance
+
+# Create performance monitoring script
+sudo tee /usr/local/bin/deb-mock-performance << 'EOF'
+#!/bin/bash
+
+# Generate performance report
+deb-mock performance-report --output /var/log/deb-mock/performance/report_$(date +%Y%m%d_%H%M%S).html
+
+# Clean up old reports (keep last 30 days)
+find /var/log/deb-mock/performance -name "report_*.html" -mtime +30 -delete
+
+# Generate benchmark report if needed
+if [ "$1" = "benchmark" ]; then
+    deb-mock benchmark --template standard --iterations 20
+fi
+EOF
+
+sudo chmod +x /usr/local/bin/deb-mock-performance
+```
+
+## Troubleshooting
+
+### Common Issues
+
+#### Service Won't Start
+
+```bash
+# Check service status
+sudo systemctl status deb-mock.service
+
+# Check logs
+sudo journalctl -u deb-mock.service -n 50
+
+# Check configuration
+deb-mock --config /etc/deb-mock/config.yaml validate
+
+# Check permissions
+sudo ls -la /var/lib/deb-mock/
+sudo ls -la /var/cache/deb-mock/
+```
+
+#### Chroot Issues
+
+```bash
+# List chroots
+deb-mock list-chroots
+
+# Check chroot status
+deb-mock status
+
+# Clean up broken chroots
+deb-mock cleanup --force
+
+# Rebuild chroot
+deb-mock create-chroot --suite trixie --architecture amd64
+```
+
+#### Performance Issues
+
+```bash
+# Check performance metrics
+deb-mock performance-summary
+
+# Run performance analysis
+deb-mock performance-analysis
+
+# Generate performance report
+deb-mock performance-report
+
+# Run benchmarks
+deb-mock benchmark --template comprehensive
+```
+
+### Debug Mode
+
+```bash
+# Enable debug logging
+export DEB_MOCK_LOG_LEVEL=DEBUG
+export DEB_MOCK_DEBUG=true
+
+# Run with verbose output
+deb-mock --verbose --debug build package-name
+
+# Check system resources
+deb-mock --debug status
+```
+
+## Security Considerations
+
+### User Isolation
+
+```bash
+# Create dedicated user for deb-mock
+sudo useradd -r -s /bin/false -d /var/lib/deb-mock deb-mock
+
+# Set up proper file permissions
+sudo chown -R deb-mock:deb-mock /var/lib/deb-mock
+sudo chmod 750 /var/lib/deb-mock/chroots
+sudo chmod 640 /var/log/deb-mock/*.log
+```
+
+### Network Security
+
+```bash
+# Configure firewall
+sudo ufw allow from 192.168.1.0/24 to any port 8080
+sudo ufw enable
+
+# Use reverse proxy with SSL
+sudo apt install certbot python3-certbot-nginx
+sudo certbot --nginx -d deb-mock.yourdomain.com
+```
+
+### Access Control
+
+```bash
+# Set up API key authentication
+export DEB_MOCK_API_KEY=your-secure-api-key
+
+# Configure RBAC
+sudo tee /etc/deb-mock/rbac.yaml << 'EOF'
+roles:
+  admin:
+    permissions: ["*"]
+  builder:
+    permissions: ["build", "status", "logs"]
+  viewer:
+    permissions: ["status", "logs"]
+
+users:
+  admin@example.com:
+    role: admin
+  builder@example.com:
+    role: builder
+  viewer@example.com:
+    role: viewer
+EOF
+```
+
+## Backup and Recovery
+
+### Automated Backups
+
+```bash
+# Create backup script
+sudo tee /usr/local/bin/deb-mock-backup-full << 'EOF'
+#!/bin/bash
+
+BACKUP_DIR="/var/backups/deb-mock/full"
+DATE=$(date +%Y%m%d_%H%M%S)
+
+# Create backup directory
+mkdir -p "$BACKUP_DIR"
+
+# Stop service
+systemctl stop deb-mock.service
+
+# Create full backup
+tar -czf "$BACKUP_DIR/full_backup_$DATE.tar.gz" \
+    -C /var lib/deb-mock \
+    -C /var cache/deb-mock \
+    -C /var log/deb-mock \
+    -C /etc deb-mock
+
+# Start service
+systemctl start deb-mock.service
+
+# Clean up old backups (keep last 30 days)
+find "$BACKUP_DIR" -name "full_backup_*.tar.gz" -mtime +30 -delete
+
+echo "Full backup completed: $BACKUP_DIR/full_backup_$DATE.tar.gz"
+EOF
+
+sudo chmod +x /usr/local/bin/deb-mock-backup-full
+```
+
+### Recovery Procedures
+
+```bash
+# Restore from backup
+sudo systemctl stop deb-mock.service
+
+# Extract backup
+sudo tar -xzf /var/backups/deb-mock/full/full_backup_YYYYMMDD_HHMMSS.tar.gz -C /
+
+# Fix permissions
+sudo chown -R deb-mock:deb-mock /var/lib/deb-mock
+sudo chown -R deb-mock:deb-mock /var/cache/deb-mock
+sudo chown -R deb-mock:deb-mock /var/log/deb-mock
+
+# Start service
+sudo systemctl start deb-mock.service
+
+# Verify recovery
+deb-mock status
+```
+
+## Conclusion
+
+This deployment guide provides comprehensive instructions for deploying `deb-mock` in various environments. For production deployments, ensure you have proper monitoring, backup, and security measures in place.
+
+For additional support and troubleshooting, refer to the main documentation or contact the development team.
+
+## Additional Resources
+
+- [Main Documentation](../README.md)
+- [Configuration Guide](CONFIGURATION.md)
+- [Performance Monitoring](PERFORMANCE_MONITORING.md)
+- [Plugin System](PLUGIN_SYSTEM.md)
+- [Testing Guide](TESTING.md)
+- [API Reference](API.md)