deb-mock/docs/PERFORMANCE_MONITORING.md

# Deb-Mock Performance Monitoring and Optimization

## Overview

The `deb-mock` performance monitoring and optimization system provides comprehensive insights into build performance, automatic optimization suggestions, and detailed performance analytics. This system enables users to identify bottlenecks, optimize build configurations, and maintain optimal performance across different build environments.

## Features

- **Real-time Performance Monitoring** - Track CPU, memory, disk I/O, and network usage during builds
- **Build Profiling** - Detailed analysis of each build phase with performance metrics
- **Automatic Optimization** - AI-driven suggestions for improving build performance
- **Benchmarking** - Multi-iteration performance testing for accurate measurements
- **Performance Reporting** - Comprehensive reports and visualizations
- **Resource Utilization Analysis** - Identify resource bottlenecks and optimization opportunities
- **Cache Performance Tracking** - Monitor cache hit rates and effectiveness
- **Automatic Tuning** - Apply optimization recommendations automatically

## Architecture

### Core Components

1. **PerformanceMonitor** - Real-time monitoring and metrics collection
2. **PerformanceOptimizer** - Analysis and optimization recommendations
3. **PerformanceReporter** - Report generation and data export
4. **Build Profiles** - Detailed performance tracking for individual builds
5. **System Monitoring** - Background system resource tracking

### Data Flow

```
Build Operation → Performance Monitor → Metrics Collection → Build Profile → Analysis → Optimization Suggestions
```

## Configuration

### Performance Monitoring Settings

```yaml
# Performance monitoring configuration
enable_performance_monitoring: true
performance_metrics_dir: "./performance-metrics"
performance_retention_days: 30
performance_auto_optimization: false
performance_benchmark_iterations: 3
performance_reporting: true
```

### Configuration Options

- **`enable_performance_monitoring`** - Enable/disable performance monitoring (default: true)
- **`performance_metrics_dir`** - Directory for storing performance data (default: "./performance-metrics")
- **`performance_retention_days`** - How long to keep performance data (default: 30)
- **`performance_auto_optimization`** - Automatically apply optimization suggestions (default: false)
- **`performance_benchmark_iterations`** - Number of iterations for benchmarking (default: 3)
- **`performance_reporting`** - Enable performance reporting (default: true)

## Usage

### CLI Commands

#### Performance Summary

```bash
# Show overall performance statistics
deb-mock performance-summary
```

**Output Example:**
```
=== Performance Summary ===
Total Operations: 15
Total Duration: 1250.45s
Average Duration: 83.36s
Active Operations: 0

=== Operation Statistics ===
package_build:
  Count: 5
  Avg Duration: 45.23s
  Min Duration: 32.10s
  Max Duration: 67.89s

chroot_creation:
  Count: 3
  Avg Duration: 12.45s
  Min Duration: 8.90s
  Max Duration: 18.20s
```

#### Benchmarking Operations

```bash
# Benchmark a specific operation
deb-mock benchmark "chroot_creation" --function "init_chroot" --iterations 5

# Benchmark build operations
deb-mock benchmark "package_build" --function "build" --iterations 3
```

**Output Example:**
```
=== Benchmark Results for chroot_creation ===
Iterations: 5
Average Duration: 12.45s
Min Duration: 8.90s
Max Duration: 18.20s
Variance: 12.3456
```

#### Performance Reports

```bash
# Generate comprehensive performance report
deb-mock performance-report

# Generate report with custom output file
deb-mock performance-report --output-file "my_performance_report.txt"
```

#### Build Profile Reports

```bash
# Generate detailed report for a specific build
deb-mock build-profile-report "build_1234567890"

# Generate report with custom output file
deb-mock build-profile-report "build_1234567890" --output-file "build_analysis.txt"
```

#### Performance Analysis

```bash
# Analyze performance and generate optimization suggestions
deb-mock performance-analysis
```

**Output Example:**
```
=== Analysis 1: test-package ===
Performance Score: 85/100

Optimization Suggestions:
  • Consider enabling parallel builds for faster execution
  • Review chroot caching strategy for better performance

Automatic Tuning Recommendations:
  • Low CPU utilization suggests room for more parallel builds
    Current: 2
    Suggested: 3

Manual Optimization Recommendations:
  • Consider using tmpfs for /tmp to improve I/O performance
  • Review and optimize chroot package selection
```

#### Optimization

```bash
# Show optimization recommendations
deb-mock optimize

# Automatically apply optimizations
deb-mock optimize --auto-apply
```

#### Metrics Management

```bash
# Export performance metrics
deb-mock export-metrics

# Export to specific file
deb-mock export-metrics --output-file "performance_data.json"

# Clean up old metrics
deb-mock cleanup-metrics
```

### Programmatic Usage

#### Basic Performance Monitoring

```python
from deb_mock.config import Config
from deb_mock.core import DebMock

# Create configuration with performance monitoring enabled
config = Config(
    enable_performance_monitoring=True,
    performance_auto_optimization=True
)

# Initialize deb-mock
deb_mock = DebMock(config)

# Build a package (performance monitoring happens automatically)
result = deb_mock.build("source-package")

# Get performance summary
summary = deb_mock.performance_monitor.get_performance_summary()
print(f"Total operations: {summary['total_operations']}")
```

#### Custom Performance Monitoring

```python
# Monitor custom operations
with deb_mock.performance_monitor.monitor_operation("custom_operation") as op_id:
    # Your custom operation here
    result = perform_custom_operation()
    
    # Add metrics to build profile
    if hasattr(deb_mock, 'current_build_profile'):
        deb_mock.performance_monitor.add_phase_metrics(
            deb_mock.current_build_profile, "custom_operation", metrics
        )
```

#### Benchmarking Custom Functions

```python
def my_custom_function():
    # Your function implementation
    pass

# Benchmark the function
result = deb_mock.performance_monitor.benchmark_operation(
    "my_custom_function", my_custom_function, iterations=5
)

print(f"Average duration: {result['average_duration']:.2f}s")
```

## Performance Metrics

### Collected Metrics

#### Operation Metrics

- **Duration** - Time taken for the operation
- **CPU Usage** - CPU utilization during operation
- **Memory Usage** - Memory consumption and changes
- **Disk I/O** - Read/write operations and data transfer
- **Network I/O** - Network data transfer
- **Chroot Size** - Size of chroot environment
- **Cache Hit Rate** - Effectiveness of caching
- **Parallel Efficiency** - Efficiency of parallel operations
- **Resource Utilization** - Overall resource usage

#### System Metrics

- **CPU Percentage** - Overall system CPU usage
- **Memory Percentage** - System memory utilization
- **Disk Usage** - Available disk space
- **Active Operations** - Currently running operations

### Build Profile Structure

```python
@dataclass
class BuildProfile:
    build_id: str                    # Unique build identifier
    package_name: str                # Name of the package being built
    architecture: str                # Target architecture
    suite: str                       # Debian suite
    total_duration: float            # Total build time
    phases: Dict[str, PerformanceMetrics]  # Performance data for each phase
    resource_peak: Dict[str, float]  # Peak resource usage
    cache_performance: Dict[str, float]  # Cache performance metrics
    optimization_suggestions: List[str]  # Generated optimization suggestions
    timestamp: datetime              # When the build was performed
```

## Optimization System

### Automatic Optimization Rules

#### Parallel Build Optimization

- **Low CPU Usage (< 60%)** - Increase parallel builds
- **High CPU Usage (> 90%)** - Decrease parallel builds
- **Optimal Range** - 70-85% CPU utilization

#### Cache Optimization

- **Low Hit Rate (< 30%)** - Increase cache size
- **Medium Hit Rate (30-70%)** - Review cache strategy
- **High Hit Rate (> 70%)** - Optimal performance

#### Resource Optimization

- **Memory Usage > 2GB** - Enable ccache, review dependencies
- **Disk I/O High** - Use tmpfs for temporary files
- **Network Usage High** - Review mirror configuration

### Optimization Suggestions

#### Performance-Based

1. **Duration > 5 minutes** - Enable parallel builds, optimize chroot caching
2. **Duration > 10 minutes** - Review entire build process, consider system upgrades

#### Resource-Based

1. **CPU > 80%** - Limit parallel jobs, optimize build processes
2. **Memory > 2GB** - Enable ccache, review package dependencies
3. **Disk I/O High** - Use tmpfs, optimize chroot structure

#### Cache-Based

1. **Hit Rate < 50%** - Increase cache size, review retention policy
2. **Cache Misses High** - Optimize cache invalidation strategy

## Reporting and Analysis

### Performance Reports

#### Comprehensive Report

- **Performance Summary** - Overall statistics and trends
- **Operation Breakdown** - Detailed analysis of each operation type
- **System Statistics** - Resource utilization patterns
- **Optimization History** - Applied optimizations and their effects

#### Build Profile Report

- **Build Information** - Package details and build parameters
- **Phase Breakdown** - Performance analysis of each build phase
- **Resource Peaks** - Maximum resource usage during build
- **Cache Performance** - Cache effectiveness metrics
- **Optimization Suggestions** - Specific recommendations for improvement

### Data Export

#### JSON Export

```json
{
  "export_timestamp": "2024-08-19T12:00:00",
  "summary": {
    "total_operations": 15,
    "total_duration": 1250.45,
    "average_duration": 83.36
  },
  "build_profiles": {
    "profile_1": {
      "build_id": "build_1234567890",
      "package_name": "test-package",
      "total_duration": 45.23
    }
  },
  "operation_history": [
    {
      "operation": "package_build",
      "duration": 45.23,
      "cpu_percent": 75.5
    }
  ]
}
```

#### Text Reports

- **Human-readable format** - Easy to understand performance summaries
- **Detailed breakdowns** - Phase-by-phase performance analysis
- **Optimization suggestions** - Actionable recommendations

## Best Practices

### Performance Monitoring

1. **Enable by Default** - Always enable performance monitoring in production
2. **Regular Analysis** - Analyze performance data weekly
3. **Trend Tracking** - Monitor performance trends over time
4. **Resource Planning** - Use performance data for capacity planning

### Optimization

1. **Start Conservative** - Begin with conservative optimization settings
2. **Monitor Effects** - Track the impact of optimizations
3. **Gradual Changes** - Apply optimizations incrementally
4. **Test Thoroughly** - Validate optimizations in test environments

### Data Management

1. **Regular Cleanup** - Clean up old metrics monthly
2. **Data Retention** - Keep performance data for at least 30 days
3. **Export Important Data** - Export critical performance data before cleanup
4. **Backup Metrics** - Include performance metrics in system backups

## Troubleshooting

### Common Issues

#### Performance Monitoring Not Working

**Symptoms**: No performance data available, commands return empty results

**Solutions**:
1. Check if `enable_performance_monitoring` is set to `true`
2. Verify `psutil` package is installed
3. Check permissions for metrics directory
4. Restart deb-mock to reinitialize monitoring

#### High Memory Usage

**Symptoms**: Memory usage > 2GB, build failures due to memory

**Solutions**:
1. Enable ccache to reduce compilation memory usage
2. Reduce parallel build count
3. Review and optimize chroot package selection
4. Increase system swap space

#### Slow Build Performance

**Symptoms**: Build duration > 5 minutes, high resource utilization

**Solutions**:
1. Enable parallel builds (if CPU usage allows)
2. Optimize chroot caching strategy
3. Use tmpfs for temporary files
4. Review build dependencies for unnecessary packages

#### Cache Performance Issues

**Symptoms**: Low cache hit rate, frequent cache misses

**Solutions**:
1. Increase cache size
2. Review cache retention policy
3. Optimize cache invalidation strategy
4. Check disk space availability

### Debug Mode

Enable debug mode for detailed performance information:

```bash
export DEB_MOCK_DEBUG=1
deb-mock performance-summary
```

### Getting Help

- **Performance Reports** - Generate detailed reports for analysis
- **Log Files** - Check deb-mock logs for performance-related errors
- **System Monitoring** - Use system tools to verify resource usage
- **Community Support** - Check project issues and discussions

## Future Enhancements

### Planned Features

- **Real-time Dashboard** - Web-based performance monitoring interface
- **Machine Learning Optimization** - AI-driven optimization suggestions
- **Performance Alerts** - Automated alerts for performance issues
- **Integration with Monitoring Systems** - Prometheus, Grafana, etc.
- **Performance Regression Detection** - Automatic detection of performance degradation

### Extension Points

- **Custom Metrics** - User-defined performance metrics
- **Performance Plugins** - Extensible performance monitoring
- **External Integrations** - Third-party monitoring system support
- **Performance Testing Framework** - Automated performance testing

## Integration Examples

### CI/CD Integration

```yaml
# GitHub Actions example
- name: Performance Analysis
  run: |
    deb-mock performance-analysis
    deb-mock performance-report --output-file "performance_report.txt"
  
- name: Upload Performance Report
  uses: actions/upload-artifact@v2
  with:
    name: performance-report
    path: performance_report.txt
```

### Monitoring System Integration

```python
# Prometheus metrics export
from prometheus_client import Gauge, Histogram

# Create metrics
build_duration = Histogram('deb_mock_build_duration_seconds', 'Build duration in seconds')
cpu_usage = Gauge('deb_mock_cpu_usage_percent', 'CPU usage percentage')

# Export metrics
def export_prometheus_metrics(performance_monitor):
    summary = performance_monitor.get_performance_summary()
    
    for metrics in performance_monitor._operation_history:
        build_duration.observe(metrics.duration)
        cpu_usage.set(metrics.cpu_percent)
```

### Performance Testing

```python
# Automated performance testing
def test_build_performance():
    config = Config(enable_performance_monitoring=True)
    deb_mock = DebMock(config)
    
    # Run benchmark
    result = deb_mock.performance_monitor.benchmark_operation(
        "test_build", deb_mock.build, iterations=5
    )
    
    # Assert performance requirements
    assert result['average_duration'] < 60.0, "Build too slow"
    assert result['variance'] < 10.0, "Build performance inconsistent"
    
    print("✅ Performance requirements met")
```

This comprehensive performance monitoring and optimization system provides the tools needed to maintain optimal build performance, identify bottlenecks, and continuously improve the deb-mock build system.