deb-bootc-image-builder/docs/validation-guide.md

# Validation Guide

## Overview

This document consolidates all validation information for the Debian bootc-image-builder project. It covers testing strategies, validation procedures, quality assurance, and best practices for ensuring the reliability and correctness of the system.

## Testing Strategy

### Testing Pyramid

1. **Unit Tests** (Foundation)
   - Individual component testing
   - Function and method validation
   - Mock and stub usage
   - Fast execution (< 1 second per test)

2. **Integration Tests** (Middle)
   - Component interaction testing
   - API endpoint validation
   - Database and external service integration
   - Moderate execution time (1-10 seconds per test)

3. **End-to-End Tests** (Top)
   - Complete workflow validation
   - Real environment testing
   - Image building and deployment
   - Longer execution time (10+ seconds per test)

### Test Coverage Goals

- **Unit Tests**: 90%+ code coverage
- **Integration Tests**: 80%+ API coverage
- **End-to-End Tests**: Critical path coverage
- **Overall Coverage**: 85%+ combined coverage

## Unit Testing

### Go Testing Framework

#### Test Structure
```go
package debos

import (
    "testing"
    "github.com/stretchr/testify/assert"
)

func TestFunctionName(t *testing.T) {
    // Arrange
    input := "test input"
    expected := "expected output"
    
    // Act
    result := FunctionName(input)
    
    // Assert
    assert.Equal(t, expected, result)
}
```

#### Test Naming Convention
- `TestFunctionName`: Test for specific function
- `TestFunctionName_Scenario`: Test for specific scenario
- `TestFunctionName_Error`: Test for error conditions

#### Mock and Stub Usage
```go
// Mock interface
type MockDebosRunner struct {
    ExecuteFunc func(template *DebosTemplate, outputDir string) (*DebosResult, error)
}

func (m *MockDebosRunner) Execute(template *DebosTemplate, outputDir string) (*DebosResult, error) {
    if m.ExecuteFunc != nil {
        return m.ExecuteFunc(template, outputDir)
    }
    return nil, nil
}

// Test with mock
func TestBuilderWithMock(t *testing.T) {
    mockRunner := &MockDebosRunner{
        ExecuteFunc: func(template *DebosTemplate, outputDir string) (*DebosResult, error) {
            return &DebosResult{Success: true}, nil
        },
    }
    
    // Test implementation
}
```

### Test Categories

#### 1. Functionality Tests
- **Input Validation**: Test with valid and invalid inputs
- **Edge Cases**: Test boundary conditions
- **Error Handling**: Test error scenarios and recovery
- **Success Paths**: Test normal operation flows

#### 2. Performance Tests
- **Memory Usage**: Test memory allocation and cleanup
- **Execution Time**: Test function performance
- **Resource Usage**: Test CPU and I/O usage
- **Scalability**: Test with different input sizes

#### 3. Security Tests
- **Input Sanitization**: Test for injection attacks
- **Access Control**: Test permission validation
- **Data Validation**: Test for malicious input
- **Secure Defaults**: Test default security settings

## Integration Testing

### API Testing

#### HTTP Endpoint Testing
```go
func TestAPIIntegration(t *testing.T) {
    // Setup test server
    server := httptest.NewServer(http.HandlerFunc(handleRequest))
    defer server.Close()
    
    // Test API calls
    resp, err := http.Get(server.URL + "/api/test")
    assert.NoError(t, err)
    assert.Equal(t, http.StatusOK, resp.StatusCode)
}
```

#### Database Integration Testing
```go
func TestDatabaseIntegration(t *testing.T) {
    // Setup test database
    db := setupTestDB(t)
    defer cleanupTestDB(t, db)
    
    // Test database operations
    result, err := db.Query("SELECT * FROM test_table")
    assert.NoError(t, err)
    // Validate results
}
```

### External Service Testing

#### Container Runtime Testing
```go
func TestContainerIntegration(t *testing.T) {
    // Test with real container runtime
    if testing.Short() {
        t.Skip("Skipping container integration test in short mode")
    }
    
    // Test container operations
    container, err := createTestContainer()
    assert.NoError(t, err)
    defer cleanupContainer(container)
    
    // Validate container functionality
}
```

#### Package Repository Testing
```go
func TestPackageRepository(t *testing.T) {
    // Test with real Debian repositories
    if testing.Short() {
        t.Skip("Skipping repository test in short mode")
    }
    
    // Test package resolution
    packages, err := resolvePackages([]string{"systemd", "bash"})
    assert.NoError(t, err)
    assert.NotEmpty(t, packages)
}
```

## End-to-End Testing

### Image Building Validation

#### Complete Build Pipeline
```bash
#!/bin/bash
# End-to-end build test script

set -e

echo "Starting end-to-end build test..."

# 1. Setup test environment
echo "Setting up test environment..."
mkdir -p test-output
cd test-output

# 2. Generate manifest
echo "Generating manifest..."
bootc-image-builder manifest --config /dev/null debian:trixie > manifest.json

# 3. Build image
echo "Building image..."
bootc-image-builder build --type qcow2 --output . debian:trixie

# 4. Validate output
echo "Validating output..."
if [ -f "*.qcow2" ]; then
    echo "✅ Image file created successfully"
    
    # Check file size
    file_size=$(stat -c%s *.qcow2)
    if [ $file_size -gt 1000000 ]; then
        echo "✅ Image file size is reasonable: $file_size bytes"
    else
        echo "❌ Image file size is too small: $file_size bytes"
        exit 1
    fi
else
    echo "❌ Image file not found"
    exit 1
fi

echo "🎉 End-to-end test completed successfully!"
```

#### Image Validation
```bash
#!/bin/bash
# Image validation script

set -e

IMAGE_FILE="$1"
if [ -z "$IMAGE_FILE" ]; then
    echo "Usage: $0 <image-file>"
    exit 1
fi

echo "Validating image: $IMAGE_FILE"

# Check file exists and is readable
if [ ! -r "$IMAGE_FILE" ]; then
    echo "❌ Image file not readable: $IMAGE_FILE"
    exit 1
fi

# Check file size
file_size=$(stat -c%s "$IMAGE_FILE")
echo "📏 Image file size: $file_size bytes"

# Validate image format
if command -v qemu-img >/dev/null 2>&1; then
    echo "🔍 Validating image format with qemu-img..."
    qemu-img info "$IMAGE_FILE"
    
    # Check if image is valid
    if qemu-img check "$IMAGE_FILE" 2>/dev/null; then
        echo "✅ Image format validation passed"
    else
        echo "❌ Image format validation failed"
        exit 1
    fi
else
    echo "⚠️  qemu-img not available, skipping format validation"
fi

# Test image mounting (if possible)
echo "🔧 Testing image mounting..."
if command -v guestmount >/dev/null 2>&1; then
    mount_point=$(mktemp -d)
    if guestmount -a "$IMAGE_FILE" -i "$mount_point" 2>/dev/null; then
        echo "✅ Image mounting successful"
        
        # Check basic filesystem structure
        if [ -d "$mount_point/etc" ] && [ -d "$mount_point/boot" ]; then
            echo "✅ Basic filesystem structure validated"
        else
            echo "❌ Basic filesystem structure missing"
            exit 1
        fi
        
        guestunmount "$mount_point"
        rmdir "$mount_point"
    else
        echo "❌ Image mounting failed"
        exit 1
    fi
else
    echo "⚠️  guestmount not available, skipping mount test"
fi

echo "🎉 Image validation completed successfully!"
```

### OSTree Validation

#### Repository Validation
```bash
#!/bin/bash
# OSTree repository validation

set -e

REPO_PATH="$1"
if [ -z "$REPO_PATH" ]; then
    echo "Usage: $0 <ostree-repo-path>"
    exit 1
fi

echo "Validating OSTree repository: $REPO_PATH"

# Check if repository exists
if [ ! -d "$REPO_PATH" ]; then
    echo "❌ Repository directory not found: $REPO_PATH"
    exit 1
fi

# Check repository structure
if [ -d "$REPO_PATH/refs" ] && [ -d "$REPO_PATH/objects" ]; then
    echo "✅ Repository structure validated"
else
    echo "❌ Invalid repository structure"
    exit 1
fi

# List available branches
echo "📋 Available branches:"
ostree refs --repo="$REPO_PATH" || {
    echo "❌ Failed to list repository references"
    exit 1
}

# Check repository integrity
echo "🔍 Checking repository integrity..."
ostree fsck --repo="$REPO_PATH" || {
    echo "❌ Repository integrity check failed"
    exit 1
}

echo "🎉 OSTree repository validation completed successfully!"
```

## Quality Assurance

### Code Quality

#### Linting and Formatting
```bash
# Go formatting
go fmt ./...

# Go linting
golangci-lint run

# Code complexity analysis
gocyclo -over 15 .

# Security scanning
gosec ./...
```

#### Static Analysis
```bash
# Go vet
go vet ./...

# Staticcheck
staticcheck ./...

# Ineffassign
ineffassign ./...
```

### Performance Testing

#### Benchmark Tests
```go
func BenchmarkFunctionName(b *testing.B) {
    for i := 0; i < b.N; i++ {
        FunctionName("test input")
    }
}
```

#### Memory Profiling
```bash
# Run with memory profiling
go test -memprofile=mem.prof ./...

# Analyze memory profile
go tool pprof mem.prof
```

#### CPU Profiling
```bash
# Run with CPU profiling
go test -cpuprofile=cpu.prof ./...

# Analyze CPU profile
go tool pprof cpu.prof
```

## Continuous Integration

### CI/CD Pipeline

#### GitHub Actions Example
```yaml
name: Test and Validate

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Go
      uses: actions/setup-go@v3
      with:
        go-version: '1.21'
    
    - name: Install dependencies
      run: go mod download
    
    - name: Run unit tests
      run: go test -v -race -coverprofile=coverage.txt ./...
    
    - name: Run integration tests
      run: go test -v -tags=integration ./...
    
    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        file: ./coverage.txt
```

#### Pre-commit Hooks
```bash
#!/bin/bash
# Pre-commit validation script

set -e

echo "Running pre-commit validation..."

# Format code
go fmt ./...

# Run tests
go test -v ./...

# Run linting
golangci-lint run

# Check for security issues
gosec ./...

echo "✅ Pre-commit validation passed"
```

## Validation Checklist

### Before Release

- [ ] All unit tests pass
- [ ] All integration tests pass
- [ ] End-to-end tests pass
- [ ] Code coverage meets targets
- [ ] Security scan passes
- [ ] Performance benchmarks meet targets
- [ ] Documentation is complete and accurate
- [ ] Changelog is updated
- [ ] Version numbers are correct

### Before Deployment

- [ ] Staging environment tests pass
- [ ] Load testing completed
- [ ] Security testing completed
- [ ] Backup and recovery tested
- [ ] Monitoring and alerting configured
- [ ] Rollback plan tested

## Troubleshooting

### Common Test Issues

#### Test Environment Problems
```bash
# Check Go environment
go env

# Verify dependencies
go mod verify

# Clean test cache
go clean -testcache

# Check for race conditions
go test -race ./...
```

#### Integration Test Failures
```bash
# Check external services
systemctl status docker
systemctl status containerd

# Verify network connectivity
ping -c 1 deb.debian.org

# Check disk space
df -h
```

#### Performance Test Issues
```bash
# Check system resources
htop
iostat
iotop

# Monitor system during tests
watch -n 1 'ps aux | grep test'
```

## Resources

### Testing Tools

- **Go Testing**: Built-in testing framework
- **Testify**: Assertion and mocking library
- **GolangCI-Lint**: Comprehensive linting
- **Gosec**: Security scanning
- **pprof**: Performance profiling

### Documentation

- [Go Testing Documentation](https://golang.org/pkg/testing/)
- [Testify Documentation](https://github.com/stretchr/testify)
- [GolangCI-Lint Documentation](https://golangci-lint.run/)

---

**Status**: Active Development  
**Last Updated**: August 2025  
**Maintainer**: Debian Bootc Image Builder Team