bootc-docs/state/bootc-state-examples-troubleshooting.md

bootc state - Examples and Troubleshooting

Overview

This document provides practical examples and troubleshooting guidance for the bootc state system, covering common use cases, error scenarios, and debugging techniques.

Common Use Cases

1. System Recovery

Complete System Reset

#!/bin/bash
# Complete system reset script

echo "=== Bootc System Reset ==="
echo "Date: $(date)"
echo

# Check current state
echo "Current system state:"
bootc status

# Confirm reset
read -p "Are you sure you want to reset the system? (yes/no): " confirm
if [ "$confirm" != "yes" ]; then
    echo "Reset cancelled"
    exit 1
fi

# Wipe all OSTree deployments
echo "Wiping all OSTree deployments..."
bootc state wipe-ostree

# Verify wipe
echo "Verifying wipe..."
bootc status

# Check OSTree status
echo "OSTree status:"
ostree admin status

echo "System reset complete"

Recovery from Corrupted State

#!/bin/bash
# Recovery from corrupted state

echo "=== Bootc System Recovery ==="
echo "Date: $(date)"
echo

# Check system state
echo "Checking system state..."
if ! bootc status > /dev/null 2>&1; then
    echo "System state is corrupted, attempting recovery..."
    
    # Try to fix OSTree repository
    echo "Attempting to fix OSTree repository..."
    ostree fsck --repair
    
    # If repair fails, wipe and reinstall
    if [ $? -ne 0 ]; then
        echo "Repair failed, wiping system..."
        bootc state wipe-ostree
        
        echo "System wiped, ready for fresh installation"
        echo "Run: bootc install to-disk /dev/sda"
    else
        echo "Repository repaired successfully"
    fi
else
    echo "System state is healthy"
fi

2. Development and Testing

Test Environment Reset

#!/bin/bash
# Test environment reset script

echo "=== Test Environment Reset ==="
echo "Date: $(date)"
echo

# Check if running in test environment
if [ ! -f /etc/test-environment ]; then
    echo "ERROR: Not running in test environment"
    exit 1
fi

# Wipe test deployments
echo "Wiping test deployments..."
bootc state wipe-ostree

# Clean up test data
echo "Cleaning up test data..."
rm -rf /var/lib/bootc/test-*
rm -rf /tmp/bootc-test-*

# Reset test configuration
echo "Resetting test configuration..."
rm -f /etc/bootc/test-config.toml

echo "Test environment reset complete"

Automated Testing

#!/bin/bash
# Automated testing script

echo "=== Automated Testing ==="
echo "Date: $(date)"
echo

# Test 1: Normal wipe
echo "Test 1: Normal wipe"
bootc state wipe-ostree
if [ $? -eq 0 ]; then
    echo "✓ Normal wipe test passed"
else
    echo "✗ Normal wipe test failed"
    exit 1
fi

# Test 2: Wipe with no deployments
echo "Test 2: Wipe with no deployments"
bootc state wipe-ostree
if [ $? -eq 0 ]; then
    echo "✓ Wipe with no deployments test passed"
else
    echo "✗ Wipe with no deployments test failed"
    exit 1
fi

# Test 3: Verify system state
echo "Test 3: Verify system state"
bootc status
if [ $? -eq 0 ]; then
    echo "✓ System state verification passed"
else
    echo "✗ System state verification failed"
    exit 1
fi

echo "All tests passed"

3. System Maintenance

Scheduled Maintenance

#!/bin/bash
# Scheduled maintenance script

echo "=== Scheduled Maintenance ==="
echo "Date: $(date)"
echo

# Check system health
echo "Checking system health..."
if ! bootc status > /dev/null 2>&1; then
    echo "System health check failed, attempting recovery..."
    
    # Try to fix system
    ostree fsck --repair
    
    # If still failing, wipe and reinstall
    if [ $? -ne 0 ]; then
        echo "System repair failed, wiping and reinstalling..."
        bootc state wipe-ostree
        
        # Reinstall system
        bootc install to-disk /dev/sda
        
        echo "System reinstalled successfully"
    else
        echo "System repaired successfully"
    fi
else
    echo "System health check passed"
fi

# Log maintenance
echo "Maintenance completed at $(date)" >> /var/log/bootc-maintenance.log

Emergency Recovery

#!/bin/bash
# Emergency recovery script

echo "=== Emergency Recovery ==="
echo "Date: $(date)"
echo

# Check if system is bootable
if ! systemctl is-system-running > /dev/null 2>&1; then
    echo "System is not running, attempting emergency recovery..."
    
    # Mount root filesystem
    mount /dev/sda1 /mnt
    
    # Change root and wipe deployments
    chroot /mnt bootc state wipe-ostree
    
    # Unmount
    umount /mnt
    
    echo "Emergency recovery completed"
    echo "System should be bootable now"
else
    echo "System is running normally"
fi

4. System Migration

Migrate to New Configuration

#!/bin/bash
# System migration script

echo "=== System Migration ==="
echo "Date: $(date)"
echo

# Backup current configuration
echo "Backing up current configuration..."
cp -r /etc/bootc /etc/bootc.backup
cp -r /var/lib/bootc /var/lib/bootc.backup

# Wipe current deployments
echo "Wiping current deployments..."
bootc state wipe-ostree

# Install new configuration
echo "Installing new configuration..."
bootc install to-disk /dev/sda

# Verify new installation
echo "Verifying new installation..."
bootc status

echo "Migration completed successfully"

Clean Installation

#!/bin/bash
# Clean installation script

echo "=== Clean Installation ==="
echo "Date: $(date)"
echo

# Wipe existing deployments
echo "Wiping existing deployments..."
bootc state wipe-ostree

# Clean up system
echo "Cleaning up system..."
rm -rf /var/lib/bootc/*
rm -rf /etc/bootc/*

# Install fresh system
echo "Installing fresh system..."
bootc install to-disk /dev/sda

# Configure system
echo "Configuring system..."
bootc edit

echo "Clean installation completed"

Troubleshooting Guide

1. Common Error Scenarios

Permission Denied Errors

Error: Permission denied: This command must be executed as the root user

Cause: Command executed without root privileges

Solution:

# Check current user
whoami

# Switch to root
sudo su -

# Or use sudo
sudo bootc state wipe-ostree

Prevention:

# Check if running as root
if [ "$EUID" -ne 0 ]; then
    echo "This script must be run as root"
    exit 1
fi

OSTree Not Available

Error: OSTree not found

Cause: OSTree package not installed

Solution:

# Install OSTree
apt update
apt install ostree

# Verify installation
ostree --version

Prevention:

# Check OSTree availability
if ! command -v ostree &> /dev/null; then
    echo "OSTree is not installed"
    exit 1
fi

System Not OSTree-Based

Error: No OSTree deployments found

Cause: System is not OSTree-based

Solution:

# Check if system is OSTree-based
ostree admin status

# If not OSTree-based, install OSTree
bootc install to-disk /dev/sda

Prevention:

# Check system type
if ! ostree admin status > /dev/null 2>&1; then
    echo "System is not OSTree-based"
    exit 1
fi

OSTree Repository Corruption

Error: OSTree repository corruption detected

Cause: OSTree repository is corrupted

Solution:

# Try to repair repository
ostree fsck --repair

# If repair fails, wipe and reinstall
if [ $? -ne 0 ]; then
    bootc state wipe-ostree
    bootc install to-disk /dev/sda
fi

Prevention:

# Regular repository checks
ostree fsck

# Backup important deployments
ostree admin pin <deployment-id>

2. Debugging Techniques

Enable Debug Logging

# Set debug log level
export RUST_LOG=debug

# Run command with debug output
bootc state wipe-ostree

# Check debug logs
journalctl -u bootc-* --since "1 hour ago" | grep DEBUG

Verbose Output

# Enable verbose output
bootc state wipe-ostree --verbose

# Check verbose logs
journalctl -u bootc-* --since "1 hour ago" | grep -v INFO

System Information

# Gather system information
uname -a
lsb_release -a
systemctl --version
ostree --version
bootc --version

# Check system configuration
cat /etc/os-release
cat /proc/version
cat /proc/cpuinfo | head -20

OSTree Diagnostics

# Check OSTree status
ostree admin status

# Check repository integrity
ostree fsck

# List all references
ostree refs

# Show commit details
ostree show <commit-hash>

# Check repository contents
ostree ls <commit-hash>

3. Recovery Procedures

System Recovery

#!/bin/bash
# System recovery script

echo "=== System Recovery ==="
echo "Date: $(date)"
echo

# Check system status
echo "Checking system status..."
bootc status

# Check OSTree status
echo "Checking OSTree status..."
ostree admin status

# Check repository integrity
echo "Checking repository integrity..."
ostree fsck

# If errors found, attempt repair
if [ $? -ne 0 ]; then
    echo "Repository errors detected, attempting repair..."
    ostree fsck --repair
    
    # If repair fails, wipe and reinstall
    if [ $? -ne 0 ]; then
        echo "Repair failed, wiping and reinstalling..."
        bootc state wipe-ostree
        bootc install to-disk /dev/sda
    fi
fi

# Verify system is working
echo "Verifying system is working..."
bootc status
bootc status

echo "Recovery complete"

Data Recovery

#!/bin/bash
# Data recovery script

echo "=== Data Recovery ==="
echo "Date: $(date)"
echo

# Check for available backups
echo "Checking for available backups..."
ls -la /var/lib/bootc/backups/

# List available deployments
echo "Available deployments:"
ostree admin status

# Check deployment history
echo "Deployment history:"
ostree log <deployment-id>

# Attempt to recover specific deployment
echo "Attempting to recover deployment..."
ostree admin deploy <deployment-id>

# Verify recovery
echo "Verifying recovery..."
bootc status

4. Performance Analysis

System Performance

#!/bin/bash
# System performance analysis

echo "=== System Performance Analysis ==="
echo "Date: $(date)"
echo

# Check system load
echo "System load:"
uptime

# Check memory usage
echo "Memory usage:"
free -h

# Check disk usage
echo "Disk usage:"
df -h

# Check I/O usage
echo "I/O usage:"
iotop -bn1

# Check process usage
echo "Process usage:"
top -bn1 | head -20

Command Performance

#!/bin/bash
# Command performance analysis

echo "=== Command Performance Analysis ==="
echo "Date: $(date)"
echo

# Time wipe operation
echo "Timing wipe operation..."
time bootc state wipe-ostree

# Check resource usage
echo "Resource usage:"
ps aux | grep bootc | awk '{sum+=$6} END {print sum/1024 " MB"}'

# Check system load
echo "System load:"
uptime

5. Monitoring and Alerting

Health Check Script

#!/bin/bash
# Health check script

HEALTH_STATUS=0

echo "=== Bootc State Health Check ==="
echo "Date: $(date)"
echo

# Check system status
echo "Checking system status..."
if ! bootc status > /dev/null 2>&1; then
    echo "ERROR: System status check failed"
    HEALTH_STATUS=1
fi

# Check OSTree status
echo "Checking OSTree status..."
if ! ostree admin status > /dev/null 2>&1; then
    echo "ERROR: OSTree status check failed"
    HEALTH_STATUS=1
fi

# Check repository integrity
echo "Checking repository integrity..."
if ! ostree fsck > /dev/null 2>&1; then
    echo "ERROR: Repository integrity check failed"
    HEALTH_STATUS=1
fi

# Check storage usage
echo "Checking storage usage..."
STORAGE_USAGE=$(df /var/lib/bootc | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $STORAGE_USAGE -gt 90 ]; then
    echo "ERROR: Storage usage is critical: ${STORAGE_USAGE}%"
    HEALTH_STATUS=1
elif [ $STORAGE_USAGE -gt 80 ]; then
    echo "WARNING: Storage usage is high: ${STORAGE_USAGE}%"
fi

# Report health status
if [ $HEALTH_STATUS -eq 0 ]; then
    echo "Health check passed"
else
    echo "Health check failed"
fi

exit $HEALTH_STATUS

Alerting Script

#!/bin/bash
# Alerting script

# Send alert to monitoring system
send_alert() {
    local severity=$1
    local message=$2
    
    curl -X POST "https://monitoring.example.com/alerts" \
        -H "Content-Type: application/json" \
        -d "{
            \"service\": \"bootc-state\",
            \"severity\": \"$severity\",
            \"message\": \"$message\",
            \"timestamp\": \"$(date -Iseconds)\"
        }"
}

# Check system health
if ! /usr/local/bin/bootc-state-health-check.sh; then
    send_alert "critical" "Bootc state system health check failed"
fi

# Check storage usage
STORAGE_USAGE=$(df /var/lib/bootc | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $STORAGE_USAGE -gt 90 ]; then
    send_alert "critical" "Storage usage is critical: ${STORAGE_USAGE}%"
elif [ $STORAGE_USAGE -gt 80 ]; then
    send_alert "warning" "Storage usage is high: ${STORAGE_USAGE}%"
fi

Best Practices

1. Usage Guidelines

• Backup First: Always backup important data before wiping
• Verify System: Ensure system is OSTree-based before wiping
• Plan Recovery: Have recovery plan ready
• Test First: Test in non-production environment

2. Safety Measures

• Confirmation: Consider adding confirmation prompt
• Logging: Log all wipe operations
• Monitoring: Monitor system after wipe
• Documentation: Document wipe procedures

3. Recovery Procedures

• Fresh Install: Plan for fresh installation
• Data Recovery: Ensure data is backed up
• System Restore: Have system restore plan
• Testing: Test recovery procedures

This comprehensive examples and troubleshooting guide provides practical solutions for common issues and advanced debugging techniques for the bootc state system.