particle-os-tools/ROLLBACK_DEPLOYMENT_TESTING_GUIDE.md

# Particle-OS Rollback and Deployment Activation Testing Guide

This guide provides comprehensive testing procedures for Particle-OS rollback and deployment activation functionality. These tests validate that the atomic rollback system works correctly and that deployments activate properly on system reboot.

## 🎯 Testing Objectives

### Primary Goals
1. **Validate OSTree Atomic Commits** - Ensure atomic package operations create proper commits
2. **Test Rollback Preparation** - Verify rollback commands prepare deployments correctly
3. **Test Deployment Activation** - Confirm rollbacks activate properly on reboot
4. **Verify System Integrity** - Ensure system remains functional after rollbacks
5. **Test Live Overlay Rollback** - Validate live overlay rollback functionality

### Success Criteria
- ✅ All rollback preparations complete successfully
- ✅ Deployments activate correctly on reboot
- ✅ System functionality preserved after rollback
- ✅ Package states match expected rollback targets
- ✅ No data loss or system corruption

## 🧪 Test Environment Requirements

### System Requirements
- **Ubuntu 24.04+** or compatible Debian system
- **sudo access** for test execution
- **Internet connectivity** for package downloads
- **2GB+ free disk space** for test operations
- **jq** package installed (`sudo apt install jq`)

### Prerequisites
```bash
# Install required packages
sudo apt update
sudo apt install jq squashfs-tools

# Ensure Particle-OS tools are installed
which apt-layer.sh
which bootc-alternative.sh
which bootupd-alternative.sh
```

### Safety Considerations
- **Backup important data** before testing
- **Test in VM or disposable environment** when possible
- **Have recovery plan** ready in case of issues
- **Monitor system resources** during testing

## 📋 Test Suite Overview

### Test Scripts Available

1. **`test-rollback-deployment.sh`** - Comprehensive rollback testing (no reboot)
2. **`test-deployment-activation-reboot.sh`** - Full deployment activation testing (with reboot)

### Test Categories

#### Category 1: Basic Functionality Tests
- OSTree status and deployment database
- Commit creation and management
- Log and diff functionality
- Cleanup operations

#### Category 2: Rollback Preparation Tests
- Rollback preparation without activation
- Live overlay rollback
- BootC alternative rollback
- System recovery verification

#### Category 3: Deployment Activation Tests
- Actual deployment activation via reboot
- Rollback verification after reboot
- Package state validation
- System functionality verification

## 🚀 Quick Start Testing

### Option 1: Safe Testing (No Reboot)
```bash
# Run comprehensive rollback tests without rebooting
chmod +x test-rollback-deployment.sh
./test-rollback-deployment.sh
```

### Option 2: Full Testing (With Reboot)
```bash
# Run complete deployment activation test with reboot
chmod +x test-deployment-activation-reboot.sh
./test-deployment-activation-reboot.sh
```

## 📊 Detailed Test Procedures

### Phase 1: Basic Functionality Testing

#### Test 1.1: OSTree Status and Deployment Database
```bash
# Check if deployment database exists
ls -la /var/lib/particle-os/deployment-db.json

# Test OSTree status command
sudo apt-layer.sh ostree status

# Verify status output format
sudo apt-layer.sh ostree status | grep -E "(Current|Pending) Deployment"
```

**Expected Results:**
- Deployment database file exists
- Status command executes without errors
- Status shows deployment information (may be empty for new systems)

#### Test 1.2: Commit Creation
```bash
# Create initial commit
sudo apt-layer.sh ostree compose install htop curl

# Create second commit
sudo apt-layer.sh ostree compose install wget tree

# Verify commits were created
sudo apt-layer.sh ostree log
```

**Expected Results:**
- Commits created successfully
- Log shows multiple commits
- No error messages

#### Test 1.3: Log and Diff Functionality
```bash
# Test log command
sudo apt-layer.sh ostree log

# Get commit list for diff testing
commits=($(sudo apt-layer.sh ostree log | grep -o "ostree-[0-9]*-[0-9]*"))

# Test diff between commits
if [[ ${#commits[@]} -ge 2 ]]; then
    sudo apt-layer.sh ostree diff "${commits[0]}" "${commits[1]}"
fi
```

**Expected Results:**
- Log command shows commit history
- Diff command works between commits
- Package differences are visible

### Phase 2: Rollback Preparation Testing

#### Test 2.1: Rollback Preparation (Without Activation)
```bash
# Get current deployment
current=$(sudo apt-layer.sh ostree status | grep "Current Deployment:" | cut -d: -f2 | tr -d ' ')

# Get previous commit for rollback
commits=($(sudo apt-layer.sh ostree log | grep -o "ostree-[0-9]*-[0-9]*"))
target_commit="${commits[1]}"  # Second commit (previous)

# Prepare rollback
sudo apt-layer.sh ostree rollback "$target_commit"

# Verify pending deployment
sudo apt-layer.sh ostree status
```

**Expected Results:**
- Rollback preparation completes successfully
- Pending deployment is set
- Status shows pending deployment information

#### Test 2.2: Live Overlay Rollback
```bash
# Start live overlay
sudo apt-layer.sh --live-overlay start

# Install package in overlay
sudo apt-layer.sh --live-install sl

# Verify package is available
which sl

# Test rollback
sudo apt-layer.sh --live-overlay rollback

# Verify package is no longer available
which sl

# Stop overlay
sudo apt-layer.sh --live-overlay stop
```

**Expected Results:**
- Live overlay starts successfully
- Package installs in overlay
- Rollback removes package
- Overlay stops cleanly

#### Test 2.3: BootC Alternative Rollback
```bash
# Test BootC status
sudo bootc-alternative.sh status

# Test BootC rollback (if deployments exist)
sudo bootc-alternative.sh rollback
```

**Expected Results:**
- BootC status command works
- Rollback command executes (may not have deployments to rollback)

### Phase 3: Deployment Activation Testing

#### Test 3.1: Full Deployment Activation Test
```bash
# Run the complete deployment activation test
./test-deployment-activation-reboot.sh
```

**This test will:**
1. Create multiple OSTree commits
2. Prepare a rollback to a previous commit
3. Reboot the system to activate the rollback
4. Verify the rollback was successful
5. Confirm system functionality

**Expected Results:**
- System reboots successfully
- Rollback activates correctly
- Package states match expected rollback target
- System remains functional

#### Test 3.2: Manual Deployment Activation Verification
```bash
# After a rollback reboot, verify the deployment

# Check current deployment
sudo apt-layer.sh ostree status

# Verify package availability
command -v htop && echo "htop available" || echo "htop not available"
command -v curl && echo "curl available" || echo "curl not available"
command -v wget && echo "wget available" || echo "wget not available"

# Test system functionality
sudo apt-layer.sh status
sudo apt-layer.sh ostree log
```

**Expected Results:**
- Current deployment matches rollback target
- Expected packages are available
- Unexpected packages are not available
- System commands work correctly

## 🔍 Troubleshooting Guide

### Common Issues and Solutions

#### Issue 1: "Command not found" errors
```bash
# Check if tools are installed
which apt-layer.sh
which bootc-alternative.sh

# Reinstall if missing
sudo ./install-particle-os.sh
```

#### Issue 2: Permission denied errors
```bash
# Check sudo access
sudo -n true

# Fix directory permissions
sudo chown -R root:root /var/lib/particle-os
sudo chmod -R 755 /var/lib/particle-os
```

#### Issue 3: Deployment database not found
```bash
# Initialize apt-layer
sudo apt-layer.sh --init

# Check if database was created
ls -la /var/lib/particle-os/deployment-db.json
```

#### Issue 4: Rollback preparation fails
```bash
# Check if commits exist
sudo apt-layer.sh ostree log

# Verify target commit exists
sudo apt-layer.sh ostree status

# Check system resources
df -h
free -h
```

#### Issue 5: Live overlay issues
```bash
# Check overlay status
sudo apt-layer.sh --live-overlay status

# Force stop overlay if stuck
sudo apt-layer.sh --live-overlay stop

# Check for mount points
mount | grep overlay
```

### Emergency Recovery

#### If system becomes unresponsive after rollback
```bash
# Boot into recovery mode or single user mode
# Check deployment status
sudo apt-layer.sh ostree status

# Force rollback to known good deployment
sudo apt-layer.sh ostree rollback <known-good-commit>

# Reboot to activate
sudo reboot
```

#### If rollback verification fails
```bash
# Check what actually happened
sudo apt-layer.sh ostree log
sudo apt-layer.sh ostree status

# Manual package verification
dpkg -l | grep -E "(htop|curl|wget)"

# Check system logs
sudo journalctl -b | grep -i particle
```

## 📈 Performance Testing

### Test Rollback Performance
```bash
# Time rollback preparation
time sudo apt-layer.sh ostree rollback <target-commit>

# Monitor system resources during rollback
htop  # or top
iotop # if available
```

### Test Deployment Activation Performance
```bash
# Measure reboot time
time sudo reboot

# Check boot time after rollback
systemd-analyze
systemd-analyze blame
```

## 📝 Test Reporting

### Test Results Template
```
Test Date: _______________
Test Environment: _______________
Test Script: _______________

Results:
□ Basic functionality tests passed
□ Rollback preparation tests passed
□ Deployment activation tests passed
□ System integrity verified
□ Performance acceptable

Issues Found:
- Issue 1: _______________
- Issue 2: _______________

Recommendations:
- Recommendation 1: _______________
- Recommendation 2: _______________

Overall Assessment: □ PASS □ FAIL □ PARTIAL
```

### Log Analysis
```bash
# Analyze test logs
grep -E "(ERROR|FAILED)" /tmp/particle-os-rollback-test.log
grep -E "(SUCCESS|PASSED)" /tmp/particle-os-rollback-test.log

# Check for warnings
grep -E "(WARNING|WARN)" /tmp/particle-os-rollback-test.log
```

## 🎯 Next Steps After Testing

### If All Tests Pass
1. **Document successful configuration**
2. **Create production deployment plan**
3. **Train users on rollback procedures**
4. **Set up monitoring and alerting**

### If Tests Fail
1. **Analyze failure patterns**
2. **Check system requirements**
3. **Review error logs**
4. **Test in different environment**
5. **Report issues to development team**

### Production Readiness Checklist
- [ ] All basic functionality tests pass
- [ ] Rollback preparation works reliably
- [ ] Deployment activation verified with reboot
- [ ] System integrity maintained after rollback
- [ ] Performance meets requirements
- [ ] Documentation completed
- [ ] User training materials ready
- [ ] Monitoring and alerting configured

## 🔗 Related Documentation

- [Particle-OS Installation Guide](INSTALLATION.md)
- [Particle-OS Testing Guide](TESTING_GUIDE.md)
- [Particle-OS Configuration Guide](docs/README.md)
- [apt-layer User Guide](src/apt-layer/README.md)

## 📞 Support

If you encounter issues during testing:

1. **Check this troubleshooting guide**
2. **Review the test logs**
3. **Verify system requirements**
4. **Test in a different environment**
5. **Report issues with detailed logs**

For additional support, refer to the main Particle-OS documentation or create an issue in the project repository.