particle-os-tools/ROLLBACK_DEPLOYMENT_TESTING_GUIDE.md

Particle-OS Rollback and Deployment Activation Testing Guide

This guide provides comprehensive testing procedures for Particle-OS rollback and deployment activation functionality. These tests validate that the atomic rollback system works correctly and that deployments activate properly on system reboot.

🎯 Testing Objectives

Primary Goals

1. Validate OSTree Atomic Commits - Ensure atomic package operations create proper commits
2. Test Rollback Preparation - Verify rollback commands prepare deployments correctly
3. Test Deployment Activation - Confirm rollbacks activate properly on reboot
4. Verify System Integrity - Ensure system remains functional after rollbacks
5. Test Live Overlay Rollback - Validate live overlay rollback functionality

Success Criteria

• ✅ All rollback preparations complete successfully
• ✅ Deployments activate correctly on reboot
• ✅ System functionality preserved after rollback
• ✅ Package states match expected rollback targets
• ✅ No data loss or system corruption

🧪 Test Environment Requirements

System Requirements

• Ubuntu 24.04+ or compatible Debian system
• sudo access for test execution
• Internet connectivity for package downloads
• 2GB+ free disk space for test operations
• jq package installed (sudo apt install jq)

Prerequisites

# Install required packages
sudo apt update
sudo apt install jq squashfs-tools

# Ensure Particle-OS tools are installed
which apt-layer.sh
which bootc-alternative.sh
which bootupd-alternative.sh

Safety Considerations

• Backup important data before testing
• Test in VM or disposable environment when possible
• Have recovery plan ready in case of issues
• Monitor system resources during testing

📋 Test Suite Overview

Test Scripts Available

1. test-rollback-deployment.sh - Comprehensive rollback testing (no reboot)
2. test-deployment-activation-reboot.sh - Full deployment activation testing (with reboot)

Test Categories

Category 1: Basic Functionality Tests

• OSTree status and deployment database
• Commit creation and management
• Log and diff functionality
• Cleanup operations

Category 2: Rollback Preparation Tests

• Rollback preparation without activation
• Live overlay rollback
• BootC alternative rollback
• System recovery verification

Category 3: Deployment Activation Tests

• Actual deployment activation via reboot
• Rollback verification after reboot
• Package state validation
• System functionality verification

🚀 Quick Start Testing

Option 1: Safe Testing (No Reboot)

# Run comprehensive rollback tests without rebooting
chmod +x test-rollback-deployment.sh
./test-rollback-deployment.sh

Option 2: Full Testing (With Reboot)

# Run complete deployment activation test with reboot
chmod +x test-deployment-activation-reboot.sh
./test-deployment-activation-reboot.sh

📊 Detailed Test Procedures

Phase 1: Basic Functionality Testing

Test 1.1: OSTree Status and Deployment Database

# Check if deployment database exists
ls -la /var/lib/particle-os/deployment-db.json

# Test OSTree status command
sudo apt-layer.sh ostree status

# Verify status output format
sudo apt-layer.sh ostree status | grep -E "(Current|Pending) Deployment"

Expected Results:

• Deployment database file exists
• Status command executes without errors
• Status shows deployment information (may be empty for new systems)

Test 1.2: Commit Creation

# Create initial commit
sudo apt-layer.sh ostree compose install htop curl

# Create second commit
sudo apt-layer.sh ostree compose install wget tree

# Verify commits were created
sudo apt-layer.sh ostree log

Expected Results:

• Commits created successfully
• Log shows multiple commits
• No error messages

Test 1.3: Log and Diff Functionality

# Test log command
sudo apt-layer.sh ostree log

# Get commit list for diff testing
commits=($(sudo apt-layer.sh ostree log | grep -o "ostree-[0-9]*-[0-9]*"))

# Test diff between commits
if [[ ${#commits[@]} -ge 2 ]]; then
    sudo apt-layer.sh ostree diff "${commits[0]}" "${commits[1]}"
fi

Expected Results:

• Log command shows commit history
• Diff command works between commits
• Package differences are visible

Phase 2: Rollback Preparation Testing

Test 2.1: Rollback Preparation (Without Activation)

# Get current deployment
current=$(sudo apt-layer.sh ostree status | grep "Current Deployment:" | cut -d: -f2 | tr -d ' ')

# Get previous commit for rollback
commits=($(sudo apt-layer.sh ostree log | grep -o "ostree-[0-9]*-[0-9]*"))
target_commit="${commits[1]}"  # Second commit (previous)

# Prepare rollback
sudo apt-layer.sh ostree rollback "$target_commit"

# Verify pending deployment
sudo apt-layer.sh ostree status

Expected Results:

• Rollback preparation completes successfully
• Pending deployment is set
• Status shows pending deployment information

Test 2.2: Live Overlay Rollback

# Start live overlay
sudo apt-layer.sh --live-overlay start

# Install package in overlay
sudo apt-layer.sh --live-install sl

# Verify package is available
which sl

# Test rollback
sudo apt-layer.sh --live-overlay rollback

# Verify package is no longer available
which sl

# Stop overlay
sudo apt-layer.sh --live-overlay stop

Expected Results:

• Live overlay starts successfully
• Package installs in overlay
• Rollback removes package
• Overlay stops cleanly

Test 2.3: BootC Alternative Rollback

# Test BootC status
sudo bootc-alternative.sh status

# Test BootC rollback (if deployments exist)
sudo bootc-alternative.sh rollback

Expected Results:

• BootC status command works
• Rollback command executes (may not have deployments to rollback)

Phase 3: Deployment Activation Testing

Test 3.1: Full Deployment Activation Test

# Run the complete deployment activation test
./test-deployment-activation-reboot.sh

This test will:

1. Create multiple OSTree commits
2. Prepare a rollback to a previous commit
3. Reboot the system to activate the rollback
4. Verify the rollback was successful
5. Confirm system functionality

Expected Results:

• System reboots successfully
• Rollback activates correctly
• Package states match expected rollback target
• System remains functional

Test 3.2: Manual Deployment Activation Verification

# After a rollback reboot, verify the deployment

# Check current deployment
sudo apt-layer.sh ostree status

# Verify package availability
command -v htop && echo "htop available" || echo "htop not available"
command -v curl && echo "curl available" || echo "curl not available"
command -v wget && echo "wget available" || echo "wget not available"

# Test system functionality
sudo apt-layer.sh status
sudo apt-layer.sh ostree log

Expected Results:

• Current deployment matches rollback target
• Expected packages are available
• Unexpected packages are not available
• System commands work correctly

🔍 Troubleshooting Guide

Common Issues and Solutions

Issue 1: "Command not found" errors

# Check if tools are installed
which apt-layer.sh
which bootc-alternative.sh

# Reinstall if missing
sudo ./install-particle-os.sh

Issue 2: Permission denied errors

# Check sudo access
sudo -n true

# Fix directory permissions
sudo chown -R root:root /var/lib/particle-os
sudo chmod -R 755 /var/lib/particle-os

Issue 3: Deployment database not found

# Initialize apt-layer
sudo apt-layer.sh --init

# Check if database was created
ls -la /var/lib/particle-os/deployment-db.json

Issue 4: Rollback preparation fails

# Check if commits exist
sudo apt-layer.sh ostree log

# Verify target commit exists
sudo apt-layer.sh ostree status

# Check system resources
df -h
free -h

Issue 5: Live overlay issues

# Check overlay status
sudo apt-layer.sh --live-overlay status

# Force stop overlay if stuck
sudo apt-layer.sh --live-overlay stop

# Check for mount points
mount | grep overlay

Emergency Recovery

If system becomes unresponsive after rollback

# Boot into recovery mode or single user mode
# Check deployment status
sudo apt-layer.sh ostree status

# Force rollback to known good deployment
sudo apt-layer.sh ostree rollback <known-good-commit>

# Reboot to activate
sudo reboot

If rollback verification fails

# Check what actually happened
sudo apt-layer.sh ostree log
sudo apt-layer.sh ostree status

# Manual package verification
dpkg -l | grep -E "(htop|curl|wget)"

# Check system logs
sudo journalctl -b | grep -i particle

📈 Performance Testing

Test Rollback Performance

# Time rollback preparation
time sudo apt-layer.sh ostree rollback <target-commit>

# Monitor system resources during rollback
htop  # or top
iotop # if available

Test Deployment Activation Performance

# Measure reboot time
time sudo reboot

# Check boot time after rollback
systemd-analyze
systemd-analyze blame

📝 Test Reporting

Test Results Template

Test Date: _______________
Test Environment: _______________
Test Script: _______________

Results:
□ Basic functionality tests passed
□ Rollback preparation tests passed
□ Deployment activation tests passed
□ System integrity verified
□ Performance acceptable

Issues Found:
- Issue 1: _______________
- Issue 2: _______________

Recommendations:
- Recommendation 1: _______________
- Recommendation 2: _______________

Overall Assessment: □ PASS □ FAIL □ PARTIAL

Log Analysis

# Analyze test logs
grep -E "(ERROR|FAILED)" /tmp/particle-os-rollback-test.log
grep -E "(SUCCESS|PASSED)" /tmp/particle-os-rollback-test.log

# Check for warnings
grep -E "(WARNING|WARN)" /tmp/particle-os-rollback-test.log

🎯 Next Steps After Testing

If All Tests Pass

1. Document successful configuration
2. Create production deployment plan
3. Train users on rollback procedures
4. Set up monitoring and alerting

If Tests Fail

1. Analyze failure patterns
2. Check system requirements
3. Review error logs
4. Test in different environment
5. Report issues to development team

Production Readiness Checklist

• All basic functionality tests pass
• Rollback preparation works reliably
• Deployment activation verified with reboot
• System integrity maintained after rollback
• Performance meets requirements
• Documentation completed
• User training materials ready
• Monitoring and alerting configured

🔗 Related Documentation

• Particle-OS Installation Guide
• Particle-OS Testing Guide
• Particle-OS Configuration Guide
• apt-layer User Guide

📞 Support

If you encounter issues during testing:

1. Check this troubleshooting guide
2. Review the test logs
3. Verify system requirements
4. Test in a different environment
5. Report issues with detailed logs

For additional support, refer to the main Particle-OS documentation or create an issue in the project repository.