Initial commit: Comprehensive Debian bootc documentation

- Complete documentation for all bootc commands and subcommands - Debian-specific adaptations and workarounds - Manual installation methods to bypass bootc reliability issues - Technical guides with Rust source code analysis - Flowcharts and external command references - Hidden command documentation (bootc internals, state, etc.) - Composefs integration analysis - Base image creation guides (with and without bootc binary) - Management scripts and automation - Comprehensive troubleshooting and examples
2025-09-15 14:02:28 -07:00 · 2025-09-15 14:02:28 -07:00 · 526f1c1afd
commit 526f1c1afd
67 changed files with 34174 additions and 0 deletions
--- a/rollback/bootc-rollback-technical-guide.md
+++ b/rollback/bootc-rollback-technical-guide.md
@ -0,0 +1,632 @@
+# bootc rollback - Technical Guide
+
+## Overview
+
+`bootc rollback` is a command for reverting to a previous deployment in a bootc-managed system. It changes the bootloader entry ordering to make the rollback deployment the next boot target, while preserving the current deployment as the new rollback target. This provides a safety mechanism for quickly reverting problematic updates.
+
+## Purpose
+
+The rollback command serves several critical functions:
+
+1. **Deployment Reversion**: Revert to a previous working deployment
+2. **Safety Mechanism**: Quick recovery from problematic updates
+3. **Boot Order Management**: Change bootloader entry ordering
+4. **State Preservation**: Maintain deployment history
+5. **Emergency Recovery**: Fast system recovery from failures
+
+## Command Syntax
+
+```bash
+bootc rollback [OPTIONS...]
+```
+
+### Basic Usage
+
+```bash
+# Rollback to previous deployment
+bootc rollback
+
+# Rollback and apply immediately
+bootc rollback --apply
+
+# Rollback with soft reboot
+bootc rollback --apply --soft-reboot=auto
+```
+
+## Command Options
+
+| Option | Description | Default | Required |
+|--------|-------------|---------|----------|
+| `--apply` | Restart/reboot into rollback image | `false` | No |
+| `--soft-reboot` | Configure soft reboot behavior | `None` | No |
+
+### Soft Reboot Modes
+
+| Mode | Description | Behavior |
+|------|-------------|----------|
+| `required` | Fail if soft reboot unavailable | Error if not supported |
+| `auto` | Use soft reboot if available | Fallback to regular reboot |
+
+## Architecture Overview
+
+### 1. Rollback Command Structure
+
+```rust
+pub(crate) struct RollbackOpts {
+    pub(crate) apply: bool,                           // Apply and reboot
+    pub(crate) soft_reboot: Option<SoftRebootMode>,  // Soft reboot behavior
+}
+```
+
+### 2. Rollback Process Flow
+
+```rust
+async fn rollback(opts: RollbackOpts) -> Result<()> {
+    // 1. Get system storage and OSTree
+    let sysroot = &get_storage().await?;
+    let ostree = sysroot.get_ostree()?;
+    
+    // 2. Execute rollback operation
+    crate::deploy::rollback(sysroot).await?;
+    
+    // 3. Handle soft reboot if requested
+    if opts.soft_reboot.is_some() {
+        let host = crate::status::get_status_require_booted(ostree)?.2;
+        handle_soft_reboot(
+            opts.soft_reboot,
+            host.status.rollback.as_ref(),
+            "rollback",
+            || soft_reboot_rollback(ostree),
+        )?;
+    }
+    
+    // 4. Apply changes if requested
+    if opts.apply {
+        crate::reboot::reboot()?;
+    }
+    
+    Ok(())
+}
+```
+
+### 3. Core Rollback Implementation
+
+```rust
+pub(crate) async fn rollback(sysroot: &Storage) -> Result<()> {
+    const ROLLBACK_JOURNAL_ID: &str = "26f3b1eb24464d12aa5e7b544a6b5468";
+    let ostree = sysroot.get_ostree()?;
+    let (booted_deployment, deployments, host) = 
+        crate::status::get_status_require_booted(ostree)?;
+    
+    // 1. Create new specification with swapped boot order
+    let new_spec = {
+        let mut new_spec = host.spec.clone();
+        new_spec.boot_order = new_spec.boot_order.swap();
+        new_spec
+    };
+    
+    // 2. Verify transition is valid
+    host.spec.verify_transition(&new_spec)?;
+    
+    // 3. Determine if reverting or rolling back
+    let reverting = new_spec.boot_order == BootOrder::Default;
+    if reverting {
+        println!("notice: Reverting queued rollback state");
+    }
+    
+    // 4. Get rollback deployment information
+    let rollback_status = host.status.rollback
+        .ok_or_else(|| anyhow!("No rollback available"))?;
+    let rollback_image = rollback_status
+        .query_image(repo)?
+        .ok_or_else(|| anyhow!("Rollback is not container image based"))?;
+    
+    // 5. Get current booted image for comparison
+    let current_image = host.status.booted
+        .as_ref()
+        .and_then(|b| b.query_image(repo).ok()?);
+    
+    // 6. Log rollback operation
+    tracing::info!(
+        message_id = ROLLBACK_JOURNAL_ID,
+        bootc.manifest_digest = rollback_image.manifest_digest.as_ref(),
+        bootc.ostree_commit = &rollback_image.merge_commit,
+        bootc.rollback_type = if reverting { "revert" } else { "rollback" },
+        bootc.current_manifest_digest = current_image
+            .as_ref()
+            .map(|i| i.manifest_digest.as_ref())
+            .unwrap_or("none"),
+        "Rolling back to image: {}",
+        rollback_image.manifest_digest
+    );
+    
+    // 7. Reorder deployments
+    let rollback_deployment = deployments.rollback.expect("rollback deployment");
+    let new_deployments = if reverting {
+        [booted_deployment, rollback_deployment]
+    } else {
+        [rollback_deployment, booted_deployment]
+    };
+    let new_deployments = new_deployments
+        .into_iter()
+        .chain(deployments.other)
+        .collect::<Vec<_>>();
+    
+    // 8. Write new deployment order
+    ostree.write_deployments(&new_deployments, gio::Cancellable::NONE)?;
+    
+    // 9. Provide user feedback
+    if reverting {
+        println!("Next boot: current deployment");
+    } else {
+        println!("Next boot: rollback deployment");
+    }
+    
+    // 10. Write reboot required marker
+    write_reboot_required(rollback_image.manifest_digest.as_ref())?;
+    
+    // 11. Update system status
+    sysroot.update_mtime()?;
+    
+    Ok(())
+}
+```
+
+## Boot Order Management
+
+### 1. Boot Order States
+
+The rollback system manages two boot order states:
+
+#### Default Boot Order
+- **Current Deployment**: First in boot order
+- **Rollback Deployment**: Second in boot order
+- **State**: Normal operation
+
+#### Rollback Boot Order
+- **Rollback Deployment**: First in boot order
+- **Current Deployment**: Second in boot order
+- **State**: Rollback queued for next boot
+
+### 2. Boot Order Swapping
+
+```rust
+let new_spec = {
+    let mut new_spec = host.spec.clone();
+    new_spec.boot_order = new_spec.boot_order.swap();
+    new_spec
+};
+```
+
+**Process**:
+1. **Clone Current Spec**: Copy existing host specification
+2. **Swap Boot Order**: Change boot order state
+3. **Preserve Other Settings**: Keep image and other settings
+
+### 3. Reversion Detection
+
+```rust
+let reverting = new_spec.boot_order == BootOrder::Default;
+if reverting {
+    println!("notice: Reverting queued rollback state");
+}
+```
+
+**Purpose**: Detect if rolling back to normal state or rolling back to previous deployment
+**Behavior**:
+- **Rollback**: Switch to rollback deployment
+- **Revert**: Return to normal boot order
+
+## Deployment Reordering
+
+### 1. Deployment Array Management
+
+```rust
+let new_deployments = if reverting {
+    [booted_deployment, rollback_deployment]
+} else {
+    [rollback_deployment, booted_deployment]
+};
+let new_deployments = new_deployments
+    .into_iter()
+    .chain(deployments.other)
+    .collect::<Vec<_>>();
+```
+
+**Process**:
+1. **Determine Order**: Based on reverting flag
+2. **Reorder Deployments**: Put target deployment first
+3. **Preserve Others**: Keep other deployments in order
+4. **Create Array**: Build new deployment array
+
+### 2. OSTree Deployment Writing
+
+```rust
+ostree.write_deployments(&new_deployments, gio::Cancellable::NONE)?;
+```
+
+**Purpose**: Write new deployment order to OSTree
+**Process**: Update bootloader configuration with new deployment order
+
+## Logging and Monitoring
+
+### 1. Rollback Operation Logging
+
+```rust
+const ROLLBACK_JOURNAL_ID: &str = "26f3b1eb24464d12aa5e7b544a6b5468";
+
+tracing::info!(
+    message_id = ROLLBACK_JOURNAL_ID,
+    bootc.manifest_digest = rollback_image.manifest_digest.as_ref(),
+    bootc.ostree_commit = &rollback_image.merge_commit,
+    bootc.rollback_type = if reverting { "revert" } else { "rollback" },
+    bootc.current_manifest_digest = current_image
+        .as_ref()
+        .map(|i| i.manifest_digest.as_ref())
+        .unwrap_or("none"),
+    "Rolling back to image: {}",
+    rollback_image.manifest_digest
+);
+```
+
+**Purpose**: Track rollback operations
+**Fields**:
+- `message_id`: Unique identifier for rollback operations
+- `bootc.manifest_digest`: Target rollback image digest
+- `bootc.ostree_commit`: OSTree commit hash
+- `bootc.rollback_type`: Operation type (rollback or revert)
+- `bootc.current_manifest_digest`: Current image digest
+
+### 2. User Feedback
+
+```rust
+if reverting {
+    println!("Next boot: current deployment");
+} else {
+    println!("Next boot: rollback deployment");
+}
+```
+
+**Purpose**: Inform user about next boot target
+**Messages**:
+- **Reverting**: "Next boot: current deployment"
+- **Rolling Back**: "Next boot: rollback deployment"
+
+## Soft Reboot Support
+
+### 1. Soft Reboot Rollback
+
+```rust
+fn soft_reboot_rollback(sysroot: &SysrootLock) -> Result<()> {
+    println!("Rollback deployment is soft-reboot capable, preparing for soft-reboot...");
+    
+    let deployments_list = sysroot.deployments();
+    let target_deployment = deployments_list
+        .first()
+        .ok_or_else(|| anyhow::anyhow!("No rollback deployment found!"))?;
+    
+    prepare_soft_reboot(sysroot, target_deployment)
+}
+```
+
+**Purpose**: Prepare rollback deployment for soft reboot
+**Process**:
+1. **Check Capability**: Verify soft reboot support
+2. **Get Target**: Find rollback deployment
+3. **Prepare**: Set up soft reboot environment
+
+### 2. Soft Reboot Handling
+
+```rust
+if opts.soft_reboot.is_some() {
+    let host = crate::status::get_status_require_booted(ostree)?.2;
+    handle_soft_reboot(
+        opts.soft_reboot,
+        host.status.rollback.as_ref(),
+        "rollback",
+        || soft_reboot_rollback(ostree),
+    )?;
+}
+```
+
+**Purpose**: Handle soft reboot configuration
+**Process**:
+1. **Check Status**: Get current system status
+2. **Handle Soft Reboot**: Configure if supported
+3. **Fallback**: Use regular reboot if not supported
+
+## State Management
+
+### 1. Rollback Availability Check
+
+```rust
+let rollback_status = host.status.rollback
+    .ok_or_else(|| anyhow!("No rollback available"))?;
+```
+
+**Purpose**: Ensure rollback deployment exists
+**Error**: Fails if no rollback deployment available
+
+### 2. Image Validation
+
+```rust
+let rollback_image = rollback_status
+    .query_image(repo)?
+    .ok_or_else(|| anyhow!("Rollback is not container image based"))?;
+```
+
+**Purpose**: Validate rollback deployment is container-based
+**Error**: Fails if rollback is not container image based
+
+### 3. Transition Verification
+
+```rust
+host.spec.verify_transition(&new_spec)?;
+```
+
+**Purpose**: Ensure rollback transition is valid
+**Process**: Validate state transition is allowed
+
+## Error Handling
+
+### 1. Rollback Availability Errors
+
+```rust
+// No rollback available
+Error: No rollback available
+
+// Rollback not container-based
+Error: Rollback is not container image based
+```
+
+### 2. System State Errors
+
+```rust
+// System not bootc compatible
+Error: System is not bootc compatible
+
+// Invalid transition
+Error: Invalid state transition
+```
+
+### 3. Soft Reboot Errors
+
+```rust
+// Soft reboot not supported
+Error: Soft reboot not available
+
+// No rollback deployment found
+Error: No rollback deployment found!
+```
+
+## Usage Patterns
+
+### 1. Emergency Rollback
+
+```bash
+# Quick rollback to previous deployment
+bootc rollback --apply
+
+# Rollback with soft reboot
+bootc rollback --apply --soft-reboot=auto
+```
+
+### 2. Rollback Preparation
+
+```bash
+# Prepare rollback for next boot
+bootc rollback
+
+# Check status
+bootc status
+```
+
+### 3. Reversion
+
+```bash
+# Revert rollback state
+bootc rollback
+
+# Apply reversion
+bootc rollback --apply
+```
+
+## Integration with Other Commands
+
+### 1. Relationship to bootc upgrade
+
+**Complementary**:
+- `upgrade`: Deploy new updates
+- `rollback`: Revert to previous deployment
+
+**Usage**:
+```bash
+# Deploy update
+bootc upgrade --apply
+
+# If issues, rollback
+bootc rollback --apply
+```
+
+### 2. Relationship to bootc switch
+
+**Complementary**:
+- `switch`: Change image source
+- `rollback`: Revert to previous deployment
+
+**Usage**:
+```bash
+# Switch to new image
+bootc switch quay.io/myorg/debian-bootc:v2.0
+
+# If issues, rollback
+bootc rollback --apply
+```
+
+### 3. Relationship to bootc status
+
+**Status Display**:
+```bash
+# Check current status
+bootc status
+
+# Perform rollback
+bootc rollback
+
+# Check updated status
+bootc status
+```
+
+## /etc Directory Behavior
+
+### 1. /etc State Preservation
+
+**Important Note**: When performing a rollback, changes made to files in the `/etc` directory won't carry over to the rolled-back deployment. The `/etc` files will revert to their state from that previous deployment.
+
+### 2. Why /etc Changes Don't Persist
+
+```rust
+// This is because `bootc rollback` just reorders the existing
+// deployments. It doesn't create new deployments. The `/etc`
+// merges happen when new deployments are created.
+```
+
+**Reason**:
+- **Deployment Reordering**: Rollback only changes boot order
+- **No New Deployment**: Doesn't create new deployment
+- **No /etc Merge**: /etc merges happen during deployment creation
+
+### 3. /etc State Management
+
+**Current Deployment**:
+- `/etc` changes are preserved
+- Local modifications remain
+
+**Rollback Deployment**:
+- `/etc` state from previous deployment
+- No local modifications from current deployment
+
+## Security Considerations
+
+### 1. Rollback Validation
+
+- **Deployment Existence**: Verify rollback deployment exists
+- **Image Compatibility**: Ensure rollback is container-based
+- **Transition Safety**: Validate state transitions
+
+### 2. State Consistency
+
+- **Atomic Operations**: All-or-nothing rollback operations
+- **Status Updates**: Consistent system status updates
+- **Logging**: Comprehensive operation logging
+
+### 3. Access Control
+
+- **Privilege Requirements**: Appropriate system privileges
+- **Deployment Access**: Access to deployment information
+- **Bootloader Control**: Bootloader configuration access
+
+## Performance Considerations
+
+### 1. Rollback Speed
+
+- **No Image Download**: Rollback doesn't download images
+- **Deployment Reordering**: Only changes boot order
+- **Fast Execution**: Quick rollback operations
+
+### 2. State Management
+
+- **Minimal Changes**: Only reorders existing deployments
+- **Preserved State**: Maintains deployment history
+- **Efficient Updates**: Minimal system updates
+
+### 3. Boot Time
+
+- **No Impact**: Rollback doesn't affect boot time
+- **Same Images**: Uses existing deployments
+- **Normal Boot**: Standard boot process
+
+## Troubleshooting
+
+### 1. Common Issues
+
+#### No Rollback Available
+```bash
+# Check system status
+bootc status
+
+# Check deployment history
+ostree admin status
+```
+
+#### Rollback Not Container-Based
+```bash
+# Check rollback deployment
+bootc status --json | jq '.status.rollback'
+
+# Check deployment type
+ostree admin status
+```
+
+#### Soft Reboot Fails
+```bash
+# Use regular reboot
+bootc rollback --apply
+
+# Check soft reboot support
+bootc rollback --apply --soft-reboot=required
+```
+
+### 2. Debug Commands
+
+```bash
+# Enable debug logging
+RUST_LOG=debug bootc rollback
+
+# Check system logs
+journalctl -u bootc-fetch-apply-updates.service
+
+# Check deployment status
+ostree admin status
+```
+
+## Best Practices
+
+### 1. Rollback Strategy
+
+- **Test Rollbacks**: Test rollback procedures regularly
+- **Monitor Deployments**: Keep track of deployment history
+- **Quick Response**: Be prepared for quick rollbacks
+- **Documentation**: Document rollback procedures
+
+### 2. System Management
+
+- **Regular Backups**: Backup important /etc configurations
+- **Deployment Monitoring**: Monitor deployment health
+- **Rollback Testing**: Test rollback procedures
+- **Recovery Planning**: Plan for rollback scenarios
+
+### 3. Integration
+
+- **Automated Monitoring**: Monitor for rollback needs
+- **Alert Systems**: Alert on rollback operations
+- **Documentation**: Document rollback procedures
+- **Training**: Train operators on rollback procedures
+
+## Future Enhancements
+
+### 1. Planned Features
+
+- **Userspace Restart**: For kernel-unchanged rollbacks
+- **Rollback Automation**: Automatic rollback on failure
+- **Rollback Scheduling**: Time-based rollback operations
+- **Health Checks**: Pre/post rollback validation
+
+### 2. Integration Improvements
+
+- **API Support**: REST API for rollback operations
+- **Web Interface**: Web-based rollback management
+- **Configuration Management**: Declarative rollback control
+- **Audit Logging**: Comprehensive audit trails
+
+This technical guide provides comprehensive understanding of the bootc rollback system's architecture, implementation, and usage patterns.