deb-bootc-image-builder/docs/TESTING_STRATEGY.md

# Testing Strategy - deb-bootc-image-builder

**Date**: August 17, 2025  
**Status**: 🚧 **Ready for Testing - Binary Needs Recompilation**

---

## 🎯 **Testing Overview**

This document outlines the comprehensive testing strategy for `deb-bootc-image-builder` once the binary is recompiled with the sudo fixes. The goal is to validate that all critical functionality works correctly and identify any remaining issues.

---

## 🧪 **Testing Phases**

### **Phase 1: Basic Functionality Testing** 🔥 **HIGHEST PRIORITY**
**Goal**: Verify that the core infrastructure works correctly

#### **1.1 Container Operations**
```bash
# Test container listing
./bib/particle-os container list

# Test container inspection
./bib/particle-os container inspect debian:trixie-slim

# Test container extraction (without full build)
./bib/particle-os build --work-dir /tmp/test-container recipes/minimal-debug.yml --verbose
```

**Expected Results**:
- ✅ Container listing works
- ✅ Container inspection provides correct metadata
- ✅ Container extraction completes successfully
- ✅ Rootfs is created with correct ownership

#### **1.2 Recipe System**
```bash
# Test recipe listing
./bib/particle-os list

# Test recipe validation
./bib/particle-os validate recipes/minimal-debug.yml
./bib/particle-os validate recipes/simple-cli-bootable.yml

# Test recipe parsing
./bib/particle-os build --work-dir /tmp/test-recipe recipes/minimal-debug.yml --verbose
```

**Expected Results**:
- ✅ Recipe listing shows all available recipes
- ✅ Recipe validation passes for valid recipes
- ✅ Recipe parsing works correctly
- ✅ Build starts successfully

### **Phase 2: Stage Execution Testing** 🔥 **HIGHEST PRIORITY**
**Goal**: Verify that all stages execute correctly with the sudo fixes

#### **2.1 Package Management Stage**
```bash
# Test apt stage with minimal recipe
./bib/particle-os build --work-dir /tmp/test-apt recipes/minimal-debug.yml --verbose
```

**Expected Results**:
- ✅ Container extraction completes
- ✅ Package installation works (locales, tzdata, bash, coreutils)
- ✅ Package cache cleanup works
- ✅ Stage completes successfully

#### **2.2 Locale Stage**
```bash
# Test locale stage with minimal-debug-locale.yml
./bib/particle-os build --work-dir /tmp/test-locale recipes/minimal-debug-locale.yml --verbose
```

**Expected Results**:
- ✅ Locale stage executes without permission errors
- ✅ `/etc/locale.gen` is written correctly
- ✅ `locale-gen` command runs successfully
- ✅ Default locale is set correctly
- ✅ Stage completes successfully

#### **2.3 Timezone Stage**
```bash
# Test timezone stage with a recipe that includes it
./bib/particle-os build --work-dir /tmp/test-timezone recipes/simple-cli-bootable.yml --verbose
```

**Expected Results**:
- ✅ Timezone stage executes without permission errors
- ✅ `/etc/timezone` is written correctly
- ✅ `/etc/localtime` symlink is created correctly
- ✅ Stage completes successfully

#### **2.4 Users Stage**
```bash
# Test users stage with a recipe that includes it
./bib/particle-os build --work-dir /tmp/test-users recipes/simple-cli-bootable.yml --verbose
```

**Expected Results**:
- ✅ Users stage executes without permission errors
- ✅ User accounts are created correctly
- ✅ Passwords are set correctly
- ✅ Groups are assigned correctly
- ✅ Stage completes successfully

### **Phase 3: End-to-End Workflow Testing** 🔥 **HIGH PRIORITY**
**Goal**: Verify that complete recipes execute successfully

#### **3.1 Minimal Debug Recipe**
```bash
# Test complete minimal recipe
./bib/particle-os build --work-dir /tmp/test-minimal recipes/minimal-debug.yml --verbose
```

**Expected Results**:
- ✅ All stages execute successfully
- ✅ Final image creation is reached
- ✅ Bootable image is created
- ✅ Build completes with success

#### **3.2 Simple CLI Bootable Recipe**
```bash
# Test complete CLI recipe
./bib/particle-os build --work-dir /tmp/test-cli recipes/simple-cli-bootable.yml --verbose
```

**Expected Results**:
- ✅ All stages execute successfully
- ✅ Package installation completes
- ✅ System configuration applies correctly
- ✅ Final image creation is reached
- ✅ Bootable image is created
- ✅ Build completes with success

### **Phase 4: Image Creation Testing** 📋 **MEDIUM PRIORITY**
**Goal**: Verify that all image formats and creation methods work

#### **4.1 QEMU Stage Testing**
```bash
# Test QEMU stage with different formats
./bib/particle-os build --work-dir /tmp/test-qemu recipes/qemu-test.yml --verbose
```

**Expected Results**:
- ✅ Raw image creation works
- ✅ QCOW2 image creation works (if qemu-img available)
- ✅ VMDK image creation works (if qemu-img available)
- ✅ VDI image creation works (if qemu-img available)
- ✅ Size parsing works correctly
- ✅ Filename generation works correctly

#### **4.2 Bootable Image Creation**
```bash
# Test bootable image creation
./bib/particle-os build --work-dir /tmp/test-bootable recipes/simple-cli-bootable.yml --verbose
```

**Expected Results**:
- ✅ GPT partition table is created
- ✅ ext4 filesystem is formatted
- ✅ Rootfs content is copied correctly
- ✅ Bootloader (extlinux) is installed
- ✅ Boot configuration files are created
- ✅ Image is recognized as bootable by QEMU

### **Phase 5: Error Handling Testing** 📋 **MEDIUM PRIORITY**
**Goal**: Verify that error handling works correctly

#### **5.1 Invalid Recipe Testing**
```bash
# Test with invalid recipe
./bib/particle-os build --work-dir /tmp/test-error invalid-recipe.yml --verbose
```

**Expected Results**:
- ✅ Clear error message is displayed
- ✅ Helpful debugging information is provided
- ✅ Work directory cleanup occurs
- ✅ Proper exit code is returned

#### **5.2 Resource Limitation Testing**
```bash
# Test with insufficient disk space
./bib/particle-os build --work-dir /tmp/test-space recipes/minimal-debug.yml --verbose
```

**Expected Results**:
- ✅ Disk space check prevents build
- ✅ Clear error message about space requirements
- ✅ Helpful suggestions for resolution
- ✅ Build fails gracefully

---

## 🔧 **Test Environment Setup**

### **Prerequisites**
```bash
# Install required tools
sudo apt update
sudo apt install -y parted e2fsprogs syslinux-common extlinux qemu-utils

# Verify tools are available
which parted mkfs.ext4 extlinux qemu-img

# Check sudo access
sudo -n true || echo "Sudo access required"
```

### **Disk Space Requirements**
```bash
# Check available space (need minimum 5GB)
df -h /tmp
df -h /home

# Use custom work directory if needed
mkdir -p /home/joe/particle-os-test
```

### **Test Recipes**
- `recipes/minimal-debug.yml` - Basic apt stage testing
- `recipes/minimal-debug-locale.yml` - Locale stage testing
- `recipes/simple-cli-bootable.yml` - Complete workflow testing
- `recipes/qemu-test.yml` - QEMU stage testing (create this)

---

## 📊 **Test Results Tracking**

### **Test Matrix**
| Test Category | Test Case | Expected Result | Actual Result | Status | Notes |
|---------------|-----------|-----------------|---------------|--------|-------|
| **Container** | List containers | Shows available containers | | | |
| **Container** | Inspect container | Provides metadata | | | |
| **Container** | Extract container | Creates rootfs | | | |
| **Recipe** | List recipes | Shows all recipes | | | |
| **Recipe** | Validate recipe | Passes validation | | | |
| **Stage** | apt stage | Installs packages | | | |
| **Stage** | locale stage | Configures locale | | | |
| **Stage** | timezone stage | Configures timezone | | | |
| **Stage** | users stage | Creates users | | | |
| **QEMU** | raw format | Creates raw image | | | |
| **QEMU** | qcow2 format | Creates qcow2 image | | | |
| **Image** | Bootable creation | Creates bootable image | | | |
| **Error** | Invalid recipe | Handles gracefully | | | |
| **Error** | Insufficient space | Prevents build | | | |

### **Success Criteria**
- **Phase 1**: All basic functionality tests pass
- **Phase 2**: All stage execution tests pass
- **Phase 3**: All end-to-end workflow tests pass
- **Phase 4**: All image creation tests pass
- **Phase 5**: All error handling tests pass

---

## 🚨 **Known Issues to Monitor**

### **Permission Issues (Should Be Fixed)**
- ❌ Locale stage permission denied errors
- ❌ Timezone stage permission denied errors
- ❌ Users stage permission denied errors

### **Disk Space Issues (Should Be Handled)**
- ❌ "No space left on device" errors
- ❌ Build failures due to insufficient space

### **Stage Execution Issues (Should Be Fixed)**
- ❌ Stage failures stopping entire build
- ❌ Poor error messages and debugging info

---

## 🎯 **Testing Priorities**

### **Immediate (After Binary Recompilation)**
1. **Test stage execution** - Verify sudo fixes work
2. **Test end-to-end workflow** - Validate complete builds
3. **Test error handling** - Verify graceful failure handling

### **Short Term (1-2 weeks)**
1. **Test all image formats** - Validate QEMU stage
2. **Test bootable image creation** - Verify bootloader installation
3. **Test edge cases** - Invalid recipes, resource limitations

### **Medium Term (2-4 weeks)**
1. **Performance testing** - Build times and resource usage
2. **Integration testing** - CI/CD workflows
3. **User acceptance testing** - Real-world usage scenarios

---

## 📝 **Test Execution Commands**

### **Quick Test Suite**
```bash
#!/bin/bash
# Quick test suite for basic functionality

echo "🧪 Running quick test suite..."

# Test 1: Basic functionality
echo "📋 Test 1: Basic functionality"
./bib/particle-os --version
./bib/particle-os --help
./bib/particle-os list

# Test 2: Container operations
echo "📋 Test 2: Container operations"
./bib/particle-os container list
./bib/particle-os container inspect debian:trixie-slim

# Test 3: Recipe validation
echo "📋 Test 3: Recipe validation"
./bib/particle-os validate recipes/minimal-debug.yml

echo "✅ Quick test suite completed"
```

### **Full Test Suite**
```bash
#!/bin/bash
# Full test suite for comprehensive validation

echo "🧪 Running full test suite..."

# Create test work directory
TEST_DIR="/tmp/particle-os-test-$(date +%s)"
mkdir -p "$TEST_DIR"

# Test all stages individually
echo "📋 Testing individual stages..."
./bib/particle-os build --work-dir "$TEST_DIR/apt" recipes/minimal-debug.yml --verbose
./bib/particle-os build --work-dir "$TEST_DIR/locale" recipes/minimal-debug-locale.yml --verbose
./bib/particle-os build --work-dir "$TEST_DIR/cli" recipes/simple-cli-bootable.yml --verbose

# Cleanup
rm -rf "$TEST_DIR"

echo "✅ Full test suite completed"
```

---

## 🔍 **Debugging and Troubleshooting**

### **Common Issues and Solutions**

#### **Stage Execution Failures**
```bash
# Enable verbose logging
./bib/particle-os build --verbose recipes/minimal-debug.yml

# Check work directory
ls -la /tmp/particle-os-build/

# Check stage logs
find /tmp/particle-os-build/stages/ -name "*.log" -exec cat {} \;
```

#### **Permission Issues**
```bash
# Check sudo access
sudo -l

# Check file ownership in rootfs
ls -la /tmp/particle-os-build/rootfs/etc/

# Check chroot permissions
sudo chroot /tmp/particle-os-build/rootfs ls -la /etc/
```

#### **Disk Space Issues**
```bash
# Check available space
df -h /tmp
df -h /home

# Use disk space management script
./scripts/manage-disk-space.sh status
./scripts/manage-disk-space.sh cleanup
```

---

## 📚 **Additional Resources**

### **Documentation**
- `HOW-TO-USE.md`: Command-line usage guide
- `HOW-TO-USE-AS-CICD.md`: CI/CD integration guide
- `todo`: Detailed project status and roadmap

### **Test Scripts**
- `test-improvements.sh`: Basic functionality testing
- `test-sudo-fix.sh`: Sudo approach validation
- `scripts/manage-disk-space.sh`: Disk space management

---

**Last Updated**: August 17, 2025  
**Status**: 🚧 **Ready for Testing - Binary Needs Recompilation**  
**Next Milestone**: Execute Testing Strategy After Binary Recompilation  
**Testing Readiness**: 100% (strategy complete, waiting for binary)