bootc-docs/state/bootc-state-examples-troubleshooting.md

# bootc state - Examples and Troubleshooting

## Overview

This document provides practical examples and troubleshooting guidance for the `bootc state` system, covering common use cases, error scenarios, and debugging techniques.

## Common Use Cases

### 1. System Recovery

#### Complete System Reset

```bash
#!/bin/bash
# Complete system reset script

echo "=== Bootc System Reset ==="
echo "Date: $(date)"
echo

# Check current state
echo "Current system state:"
bootc status

# Confirm reset
read -p "Are you sure you want to reset the system? (yes/no): " confirm
if [ "$confirm" != "yes" ]; then
    echo "Reset cancelled"
    exit 1
fi

# Wipe all OSTree deployments
echo "Wiping all OSTree deployments..."
bootc state wipe-ostree

# Verify wipe
echo "Verifying wipe..."
bootc status

# Check OSTree status
echo "OSTree status:"
ostree admin status

echo "System reset complete"
```

#### Recovery from Corrupted State

```bash
#!/bin/bash
# Recovery from corrupted state

echo "=== Bootc System Recovery ==="
echo "Date: $(date)"
echo

# Check system state
echo "Checking system state..."
if ! bootc status > /dev/null 2>&1; then
    echo "System state is corrupted, attempting recovery..."
    
    # Try to fix OSTree repository
    echo "Attempting to fix OSTree repository..."
    ostree fsck --repair
    
    # If repair fails, wipe and reinstall
    if [ $? -ne 0 ]; then
        echo "Repair failed, wiping system..."
        bootc state wipe-ostree
        
        echo "System wiped, ready for fresh installation"
        echo "Run: bootc install to-disk /dev/sda"
    else
        echo "Repository repaired successfully"
    fi
else
    echo "System state is healthy"
fi
```

### 2. Development and Testing

#### Test Environment Reset

```bash
#!/bin/bash
# Test environment reset script

echo "=== Test Environment Reset ==="
echo "Date: $(date)"
echo

# Check if running in test environment
if [ ! -f /etc/test-environment ]; then
    echo "ERROR: Not running in test environment"
    exit 1
fi

# Wipe test deployments
echo "Wiping test deployments..."
bootc state wipe-ostree

# Clean up test data
echo "Cleaning up test data..."
rm -rf /var/lib/bootc/test-*
rm -rf /tmp/bootc-test-*

# Reset test configuration
echo "Resetting test configuration..."
rm -f /etc/bootc/test-config.toml

echo "Test environment reset complete"
```

#### Automated Testing

```bash
#!/bin/bash
# Automated testing script

echo "=== Automated Testing ==="
echo "Date: $(date)"
echo

# Test 1: Normal wipe
echo "Test 1: Normal wipe"
bootc state wipe-ostree
if [ $? -eq 0 ]; then
    echo "✓ Normal wipe test passed"
else
    echo "✗ Normal wipe test failed"
    exit 1
fi

# Test 2: Wipe with no deployments
echo "Test 2: Wipe with no deployments"
bootc state wipe-ostree
if [ $? -eq 0 ]; then
    echo "✓ Wipe with no deployments test passed"
else
    echo "✗ Wipe with no deployments test failed"
    exit 1
fi

# Test 3: Verify system state
echo "Test 3: Verify system state"
bootc status
if [ $? -eq 0 ]; then
    echo "✓ System state verification passed"
else
    echo "✗ System state verification failed"
    exit 1
fi

echo "All tests passed"
```

### 3. System Maintenance

#### Scheduled Maintenance

```bash
#!/bin/bash
# Scheduled maintenance script

echo "=== Scheduled Maintenance ==="
echo "Date: $(date)"
echo

# Check system health
echo "Checking system health..."
if ! bootc status > /dev/null 2>&1; then
    echo "System health check failed, attempting recovery..."
    
    # Try to fix system
    ostree fsck --repair
    
    # If still failing, wipe and reinstall
    if [ $? -ne 0 ]; then
        echo "System repair failed, wiping and reinstalling..."
        bootc state wipe-ostree
        
        # Reinstall system
        bootc install to-disk /dev/sda
        
        echo "System reinstalled successfully"
    else
        echo "System repaired successfully"
    fi
else
    echo "System health check passed"
fi

# Log maintenance
echo "Maintenance completed at $(date)" >> /var/log/bootc-maintenance.log
```

#### Emergency Recovery

```bash
#!/bin/bash
# Emergency recovery script

echo "=== Emergency Recovery ==="
echo "Date: $(date)"
echo

# Check if system is bootable
if ! systemctl is-system-running > /dev/null 2>&1; then
    echo "System is not running, attempting emergency recovery..."
    
    # Mount root filesystem
    mount /dev/sda1 /mnt
    
    # Change root and wipe deployments
    chroot /mnt bootc state wipe-ostree
    
    # Unmount
    umount /mnt
    
    echo "Emergency recovery completed"
    echo "System should be bootable now"
else
    echo "System is running normally"
fi
```

### 4. System Migration

#### Migrate to New Configuration

```bash
#!/bin/bash
# System migration script

echo "=== System Migration ==="
echo "Date: $(date)"
echo

# Backup current configuration
echo "Backing up current configuration..."
cp -r /etc/bootc /etc/bootc.backup
cp -r /var/lib/bootc /var/lib/bootc.backup

# Wipe current deployments
echo "Wiping current deployments..."
bootc state wipe-ostree

# Install new configuration
echo "Installing new configuration..."
bootc install to-disk /dev/sda

# Verify new installation
echo "Verifying new installation..."
bootc status

echo "Migration completed successfully"
```

#### Clean Installation

```bash
#!/bin/bash
# Clean installation script

echo "=== Clean Installation ==="
echo "Date: $(date)"
echo

# Wipe existing deployments
echo "Wiping existing deployments..."
bootc state wipe-ostree

# Clean up system
echo "Cleaning up system..."
rm -rf /var/lib/bootc/*
rm -rf /etc/bootc/*

# Install fresh system
echo "Installing fresh system..."
bootc install to-disk /dev/sda

# Configure system
echo "Configuring system..."
bootc edit

echo "Clean installation completed"
```

## Troubleshooting Guide

### 1. Common Error Scenarios

#### Permission Denied Errors

**Error**: `Permission denied: This command must be executed as the root user`

**Cause**: Command executed without root privileges

**Solution**:
```bash
# Check current user
whoami

# Switch to root
sudo su -

# Or use sudo
sudo bootc state wipe-ostree
```

**Prevention**:
```bash
# Check if running as root
if [ "$EUID" -ne 0 ]; then
    echo "This script must be run as root"
    exit 1
fi
```

#### OSTree Not Available

**Error**: `OSTree not found`

**Cause**: OSTree package not installed

**Solution**:
```bash
# Install OSTree
apt update
apt install ostree

# Verify installation
ostree --version
```

**Prevention**:
```bash
# Check OSTree availability
if ! command -v ostree &> /dev/null; then
    echo "OSTree is not installed"
    exit 1
fi
```

#### System Not OSTree-Based

**Error**: `No OSTree deployments found`

**Cause**: System is not OSTree-based

**Solution**:
```bash
# Check if system is OSTree-based
ostree admin status

# If not OSTree-based, install OSTree
bootc install to-disk /dev/sda
```

**Prevention**:
```bash
# Check system type
if ! ostree admin status > /dev/null 2>&1; then
    echo "System is not OSTree-based"
    exit 1
fi
```

#### OSTree Repository Corruption

**Error**: `OSTree repository corruption detected`

**Cause**: OSTree repository is corrupted

**Solution**:
```bash
# Try to repair repository
ostree fsck --repair

# If repair fails, wipe and reinstall
if [ $? -ne 0 ]; then
    bootc state wipe-ostree
    bootc install to-disk /dev/sda
fi
```

**Prevention**:
```bash
# Regular repository checks
ostree fsck

# Backup important deployments
ostree admin pin <deployment-id>
```

### 2. Debugging Techniques

#### Enable Debug Logging

```bash
# Set debug log level
export RUST_LOG=debug

# Run command with debug output
bootc state wipe-ostree

# Check debug logs
journalctl -u bootc-* --since "1 hour ago" | grep DEBUG
```

#### Verbose Output

```bash
# Enable verbose output
bootc state wipe-ostree --verbose

# Check verbose logs
journalctl -u bootc-* --since "1 hour ago" | grep -v INFO
```

#### System Information

```bash
# Gather system information
uname -a
lsb_release -a
systemctl --version
ostree --version
bootc --version

# Check system configuration
cat /etc/os-release
cat /proc/version
cat /proc/cpuinfo | head -20
```

#### OSTree Diagnostics

```bash
# Check OSTree status
ostree admin status

# Check repository integrity
ostree fsck

# List all references
ostree refs

# Show commit details
ostree show <commit-hash>

# Check repository contents
ostree ls <commit-hash>
```

### 3. Recovery Procedures

#### System Recovery

```bash
#!/bin/bash
# System recovery script

echo "=== System Recovery ==="
echo "Date: $(date)"
echo

# Check system status
echo "Checking system status..."
bootc status

# Check OSTree status
echo "Checking OSTree status..."
ostree admin status

# Check repository integrity
echo "Checking repository integrity..."
ostree fsck

# If errors found, attempt repair
if [ $? -ne 0 ]; then
    echo "Repository errors detected, attempting repair..."
    ostree fsck --repair
    
    # If repair fails, wipe and reinstall
    if [ $? -ne 0 ]; then
        echo "Repair failed, wiping and reinstalling..."
        bootc state wipe-ostree
        bootc install to-disk /dev/sda
    fi
fi

# Verify system is working
echo "Verifying system is working..."
bootc status
bootc status

echo "Recovery complete"
```

#### Data Recovery

```bash
#!/bin/bash
# Data recovery script

echo "=== Data Recovery ==="
echo "Date: $(date)"
echo

# Check for available backups
echo "Checking for available backups..."
ls -la /var/lib/bootc/backups/

# List available deployments
echo "Available deployments:"
ostree admin status

# Check deployment history
echo "Deployment history:"
ostree log <deployment-id>

# Attempt to recover specific deployment
echo "Attempting to recover deployment..."
ostree admin deploy <deployment-id>

# Verify recovery
echo "Verifying recovery..."
bootc status
```

### 4. Performance Analysis

#### System Performance

```bash
#!/bin/bash
# System performance analysis

echo "=== System Performance Analysis ==="
echo "Date: $(date)"
echo

# Check system load
echo "System load:"
uptime

# Check memory usage
echo "Memory usage:"
free -h

# Check disk usage
echo "Disk usage:"
df -h

# Check I/O usage
echo "I/O usage:"
iotop -bn1

# Check process usage
echo "Process usage:"
top -bn1 | head -20
```

#### Command Performance

```bash
#!/bin/bash
# Command performance analysis

echo "=== Command Performance Analysis ==="
echo "Date: $(date)"
echo

# Time wipe operation
echo "Timing wipe operation..."
time bootc state wipe-ostree

# Check resource usage
echo "Resource usage:"
ps aux | grep bootc | awk '{sum+=$6} END {print sum/1024 " MB"}'

# Check system load
echo "System load:"
uptime
```

### 5. Monitoring and Alerting

#### Health Check Script

```bash
#!/bin/bash
# Health check script

HEALTH_STATUS=0

echo "=== Bootc State Health Check ==="
echo "Date: $(date)"
echo

# Check system status
echo "Checking system status..."
if ! bootc status > /dev/null 2>&1; then
    echo "ERROR: System status check failed"
    HEALTH_STATUS=1
fi

# Check OSTree status
echo "Checking OSTree status..."
if ! ostree admin status > /dev/null 2>&1; then
    echo "ERROR: OSTree status check failed"
    HEALTH_STATUS=1
fi

# Check repository integrity
echo "Checking repository integrity..."
if ! ostree fsck > /dev/null 2>&1; then
    echo "ERROR: Repository integrity check failed"
    HEALTH_STATUS=1
fi

# Check storage usage
echo "Checking storage usage..."
STORAGE_USAGE=$(df /var/lib/bootc | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $STORAGE_USAGE -gt 90 ]; then
    echo "ERROR: Storage usage is critical: ${STORAGE_USAGE}%"
    HEALTH_STATUS=1
elif [ $STORAGE_USAGE -gt 80 ]; then
    echo "WARNING: Storage usage is high: ${STORAGE_USAGE}%"
fi

# Report health status
if [ $HEALTH_STATUS -eq 0 ]; then
    echo "Health check passed"
else
    echo "Health check failed"
fi

exit $HEALTH_STATUS
```

#### Alerting Script

```bash
#!/bin/bash
# Alerting script

# Send alert to monitoring system
send_alert() {
    local severity=$1
    local message=$2
    
    curl -X POST "https://monitoring.example.com/alerts" \
        -H "Content-Type: application/json" \
        -d "{
            \"service\": \"bootc-state\",
            \"severity\": \"$severity\",
            \"message\": \"$message\",
            \"timestamp\": \"$(date -Iseconds)\"
        }"
}

# Check system health
if ! /usr/local/bin/bootc-state-health-check.sh; then
    send_alert "critical" "Bootc state system health check failed"
fi

# Check storage usage
STORAGE_USAGE=$(df /var/lib/bootc | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $STORAGE_USAGE -gt 90 ]; then
    send_alert "critical" "Storage usage is critical: ${STORAGE_USAGE}%"
elif [ $STORAGE_USAGE -gt 80 ]; then
    send_alert "warning" "Storage usage is high: ${STORAGE_USAGE}%"
fi
```

## Best Practices

### 1. Usage Guidelines

- **Backup First**: Always backup important data before wiping
- **Verify System**: Ensure system is OSTree-based before wiping
- **Plan Recovery**: Have recovery plan ready
- **Test First**: Test in non-production environment

### 2. Safety Measures

- **Confirmation**: Consider adding confirmation prompt
- **Logging**: Log all wipe operations
- **Monitoring**: Monitor system after wipe
- **Documentation**: Document wipe procedures

### 3. Recovery Procedures

- **Fresh Install**: Plan for fresh installation
- **Data Recovery**: Ensure data is backed up
- **System Restore**: Have system restore plan
- **Testing**: Test recovery procedures

This comprehensive examples and troubleshooting guide provides practical solutions for common issues and advanced debugging techniques for the bootc state system.