bootc-docs/rollback/bootc-rollback-technical-guide.md

bootc rollback - Technical Guide

Overview

bootc rollback is a command for reverting to a previous deployment in a bootc-managed system. It changes the bootloader entry ordering to make the rollback deployment the next boot target, while preserving the current deployment as the new rollback target. This provides a safety mechanism for quickly reverting problematic updates.

Purpose

The rollback command serves several critical functions:

1. Deployment Reversion: Revert to a previous working deployment
2. Safety Mechanism: Quick recovery from problematic updates
3. Boot Order Management: Change bootloader entry ordering
4. State Preservation: Maintain deployment history
5. Emergency Recovery: Fast system recovery from failures

Command Syntax

bootc rollback [OPTIONS...]

Basic Usage

# Rollback to previous deployment
bootc rollback

# Rollback and apply immediately
bootc rollback --apply

# Rollback with soft reboot
bootc rollback --apply --soft-reboot=auto

Command Options

Option	Description	Default	Required
--apply	Restart/reboot into rollback image	false	No
--soft-reboot	Configure soft reboot behavior	None	No

Soft Reboot Modes

Mode	Description	Behavior
required	Fail if soft reboot unavailable	Error if not supported
auto	Use soft reboot if available	Fallback to regular reboot

Architecture Overview

1. Rollback Command Structure

pub(crate) struct RollbackOpts {
    pub(crate) apply: bool,                           // Apply and reboot
    pub(crate) soft_reboot: Option<SoftRebootMode>,  // Soft reboot behavior
}

2. Rollback Process Flow

async fn rollback(opts: RollbackOpts) -> Result<()> {
    // 1. Get system storage and OSTree
    let sysroot = &get_storage().await?;
    let ostree = sysroot.get_ostree()?;
    
    // 2. Execute rollback operation
    crate::deploy::rollback(sysroot).await?;
    
    // 3. Handle soft reboot if requested
    if opts.soft_reboot.is_some() {
        let host = crate::status::get_status_require_booted(ostree)?.2;
        handle_soft_reboot(
            opts.soft_reboot,
            host.status.rollback.as_ref(),
            "rollback",
            || soft_reboot_rollback(ostree),
        )?;
    }
    
    // 4. Apply changes if requested
    if opts.apply {
        crate::reboot::reboot()?;
    }
    
    Ok(())
}

3. Core Rollback Implementation

pub(crate) async fn rollback(sysroot: &Storage) -> Result<()> {
    const ROLLBACK_JOURNAL_ID: &str = "26f3b1eb24464d12aa5e7b544a6b5468";
    let ostree = sysroot.get_ostree()?;
    let (booted_deployment, deployments, host) = 
        crate::status::get_status_require_booted(ostree)?;
    
    // 1. Create new specification with swapped boot order
    let new_spec = {
        let mut new_spec = host.spec.clone();
        new_spec.boot_order = new_spec.boot_order.swap();
        new_spec
    };
    
    // 2. Verify transition is valid
    host.spec.verify_transition(&new_spec)?;
    
    // 3. Determine if reverting or rolling back
    let reverting = new_spec.boot_order == BootOrder::Default;
    if reverting {
        println!("notice: Reverting queued rollback state");
    }
    
    // 4. Get rollback deployment information
    let rollback_status = host.status.rollback
        .ok_or_else(|| anyhow!("No rollback available"))?;
    let rollback_image = rollback_status
        .query_image(repo)?
        .ok_or_else(|| anyhow!("Rollback is not container image based"))?;
    
    // 5. Get current booted image for comparison
    let current_image = host.status.booted
        .as_ref()
        .and_then(|b| b.query_image(repo).ok()?);
    
    // 6. Log rollback operation
    tracing::info!(
        message_id = ROLLBACK_JOURNAL_ID,
        bootc.manifest_digest = rollback_image.manifest_digest.as_ref(),
        bootc.ostree_commit = &rollback_image.merge_commit,
        bootc.rollback_type = if reverting { "revert" } else { "rollback" },
        bootc.current_manifest_digest = current_image
            .as_ref()
            .map(|i| i.manifest_digest.as_ref())
            .unwrap_or("none"),
        "Rolling back to image: {}",
        rollback_image.manifest_digest
    );
    
    // 7. Reorder deployments
    let rollback_deployment = deployments.rollback.expect("rollback deployment");
    let new_deployments = if reverting {
        [booted_deployment, rollback_deployment]
    } else {
        [rollback_deployment, booted_deployment]
    };
    let new_deployments = new_deployments
        .into_iter()
        .chain(deployments.other)
        .collect::<Vec<_>>();
    
    // 8. Write new deployment order
    ostree.write_deployments(&new_deployments, gio::Cancellable::NONE)?;
    
    // 9. Provide user feedback
    if reverting {
        println!("Next boot: current deployment");
    } else {
        println!("Next boot: rollback deployment");
    }
    
    // 10. Write reboot required marker
    write_reboot_required(rollback_image.manifest_digest.as_ref())?;
    
    // 11. Update system status
    sysroot.update_mtime()?;
    
    Ok(())
}

Boot Order Management

1. Boot Order States

The rollback system manages two boot order states:

Default Boot Order

• Current Deployment: First in boot order
• Rollback Deployment: Second in boot order
• State: Normal operation

Rollback Boot Order

• Rollback Deployment: First in boot order
• Current Deployment: Second in boot order
• State: Rollback queued for next boot

2. Boot Order Swapping

let new_spec = {
    let mut new_spec = host.spec.clone();
    new_spec.boot_order = new_spec.boot_order.swap();
    new_spec
};

Process:

1. Clone Current Spec: Copy existing host specification
2. Swap Boot Order: Change boot order state
3. Preserve Other Settings: Keep image and other settings

3. Reversion Detection

let reverting = new_spec.boot_order == BootOrder::Default;
if reverting {
    println!("notice: Reverting queued rollback state");
}

Purpose: Detect if rolling back to normal state or rolling back to previous deployment Behavior:

• Rollback: Switch to rollback deployment
• Revert: Return to normal boot order

Deployment Reordering

1. Deployment Array Management

let new_deployments = if reverting {
    [booted_deployment, rollback_deployment]
} else {
    [rollback_deployment, booted_deployment]
};
let new_deployments = new_deployments
    .into_iter()
    .chain(deployments.other)
    .collect::<Vec<_>>();

Process:

1. Determine Order: Based on reverting flag
2. Reorder Deployments: Put target deployment first
3. Preserve Others: Keep other deployments in order
4. Create Array: Build new deployment array

2. OSTree Deployment Writing

ostree.write_deployments(&new_deployments, gio::Cancellable::NONE)?;

Purpose: Write new deployment order to OSTree Process: Update bootloader configuration with new deployment order

Logging and Monitoring

1. Rollback Operation Logging

const ROLLBACK_JOURNAL_ID: &str = "26f3b1eb24464d12aa5e7b544a6b5468";

tracing::info!(
    message_id = ROLLBACK_JOURNAL_ID,
    bootc.manifest_digest = rollback_image.manifest_digest.as_ref(),
    bootc.ostree_commit = &rollback_image.merge_commit,
    bootc.rollback_type = if reverting { "revert" } else { "rollback" },
    bootc.current_manifest_digest = current_image
        .as_ref()
        .map(|i| i.manifest_digest.as_ref())
        .unwrap_or("none"),
    "Rolling back to image: {}",
    rollback_image.manifest_digest
);

Purpose: Track rollback operations Fields:

• message_id: Unique identifier for rollback operations
• bootc.manifest_digest: Target rollback image digest
• bootc.ostree_commit: OSTree commit hash
• bootc.rollback_type: Operation type (rollback or revert)
• bootc.current_manifest_digest: Current image digest

2. User Feedback

if reverting {
    println!("Next boot: current deployment");
} else {
    println!("Next boot: rollback deployment");
}

Purpose: Inform user about next boot target Messages:

• Reverting: "Next boot: current deployment"
• Rolling Back: "Next boot: rollback deployment"

Soft Reboot Support

1. Soft Reboot Rollback

fn soft_reboot_rollback(sysroot: &SysrootLock) -> Result<()> {
    println!("Rollback deployment is soft-reboot capable, preparing for soft-reboot...");
    
    let deployments_list = sysroot.deployments();
    let target_deployment = deployments_list
        .first()
        .ok_or_else(|| anyhow::anyhow!("No rollback deployment found!"))?;
    
    prepare_soft_reboot(sysroot, target_deployment)
}

Purpose: Prepare rollback deployment for soft reboot Process:

1. Check Capability: Verify soft reboot support
2. Get Target: Find rollback deployment
3. Prepare: Set up soft reboot environment

2. Soft Reboot Handling

if opts.soft_reboot.is_some() {
    let host = crate::status::get_status_require_booted(ostree)?.2;
    handle_soft_reboot(
        opts.soft_reboot,
        host.status.rollback.as_ref(),
        "rollback",
        || soft_reboot_rollback(ostree),
    )?;
}

Purpose: Handle soft reboot configuration Process:

1. Check Status: Get current system status
2. Handle Soft Reboot: Configure if supported
3. Fallback: Use regular reboot if not supported

State Management

1. Rollback Availability Check

let rollback_status = host.status.rollback
    .ok_or_else(|| anyhow!("No rollback available"))?;

Purpose: Ensure rollback deployment exists Error: Fails if no rollback deployment available

2. Image Validation

let rollback_image = rollback_status
    .query_image(repo)?
    .ok_or_else(|| anyhow!("Rollback is not container image based"))?;

Purpose: Validate rollback deployment is container-based Error: Fails if rollback is not container image based

3. Transition Verification

host.spec.verify_transition(&new_spec)?;

Purpose: Ensure rollback transition is valid Process: Validate state transition is allowed

Error Handling

1. Rollback Availability Errors

// No rollback available
Error: No rollback available

// Rollback not container-based
Error: Rollback is not container image based

2. System State Errors

// System not bootc compatible
Error: System is not bootc compatible

// Invalid transition
Error: Invalid state transition

3. Soft Reboot Errors

// Soft reboot not supported
Error: Soft reboot not available

// No rollback deployment found
Error: No rollback deployment found!

Usage Patterns

1. Emergency Rollback

# Quick rollback to previous deployment
bootc rollback --apply

# Rollback with soft reboot
bootc rollback --apply --soft-reboot=auto

2. Rollback Preparation

# Prepare rollback for next boot
bootc rollback

# Check status
bootc status

3. Reversion

# Revert rollback state
bootc rollback

# Apply reversion
bootc rollback --apply

Integration with Other Commands

1. Relationship to bootc upgrade

Complementary:

• upgrade: Deploy new updates
• rollback: Revert to previous deployment

Usage:

# Deploy update
bootc upgrade --apply

# If issues, rollback
bootc rollback --apply

2. Relationship to bootc switch

Complementary:

• switch: Change image source
• rollback: Revert to previous deployment

Usage:

# Switch to new image
bootc switch quay.io/myorg/debian-bootc:v2.0

# If issues, rollback
bootc rollback --apply

3. Relationship to bootc status

Status Display:

# Check current status
bootc status

# Perform rollback
bootc rollback

# Check updated status
bootc status

/etc Directory Behavior

1. /etc State Preservation

Important Note: When performing a rollback, changes made to files in the /etc directory won't carry over to the rolled-back deployment. The /etc files will revert to their state from that previous deployment.

2. Why /etc Changes Don't Persist

// This is because `bootc rollback` just reorders the existing
// deployments. It doesn't create new deployments. The `/etc`
// merges happen when new deployments are created.

Reason:

• Deployment Reordering: Rollback only changes boot order
• No New Deployment: Doesn't create new deployment
• No /etc Merge: /etc merges happen during deployment creation

3. /etc State Management

Current Deployment:

• /etc changes are preserved
• Local modifications remain

Rollback Deployment:

• /etc state from previous deployment
• No local modifications from current deployment

Security Considerations

1. Rollback Validation

• Deployment Existence: Verify rollback deployment exists
• Image Compatibility: Ensure rollback is container-based
• Transition Safety: Validate state transitions

2. State Consistency

• Atomic Operations: All-or-nothing rollback operations
• Status Updates: Consistent system status updates
• Logging: Comprehensive operation logging

3. Access Control

• Privilege Requirements: Appropriate system privileges
• Deployment Access: Access to deployment information
• Bootloader Control: Bootloader configuration access

Performance Considerations

1. Rollback Speed

• No Image Download: Rollback doesn't download images
• Deployment Reordering: Only changes boot order
• Fast Execution: Quick rollback operations

2. State Management

• Minimal Changes: Only reorders existing deployments
• Preserved State: Maintains deployment history
• Efficient Updates: Minimal system updates

3. Boot Time

• No Impact: Rollback doesn't affect boot time
• Same Images: Uses existing deployments
• Normal Boot: Standard boot process

Troubleshooting

1. Common Issues

No Rollback Available

# Check system status
bootc status

# Check deployment history
ostree admin status

Rollback Not Container-Based

# Check rollback deployment
bootc status --json | jq '.status.rollback'

# Check deployment type
ostree admin status

Soft Reboot Fails

# Use regular reboot
bootc rollback --apply

# Check soft reboot support
bootc rollback --apply --soft-reboot=required

2. Debug Commands

# Enable debug logging
RUST_LOG=debug bootc rollback

# Check system logs
journalctl -u bootc-fetch-apply-updates.service

# Check deployment status
ostree admin status

Best Practices

1. Rollback Strategy

• Test Rollbacks: Test rollback procedures regularly
• Monitor Deployments: Keep track of deployment history
• Quick Response: Be prepared for quick rollbacks
• Documentation: Document rollback procedures

2. System Management

• Regular Backups: Backup important /etc configurations
• Deployment Monitoring: Monitor deployment health
• Rollback Testing: Test rollback procedures
• Recovery Planning: Plan for rollback scenarios

3. Integration

• Automated Monitoring: Monitor for rollback needs
• Alert Systems: Alert on rollback operations
• Documentation: Document rollback procedures
• Training: Train operators on rollback procedures

Future Enhancements

1. Planned Features

• Userspace Restart: For kernel-unchanged rollbacks
• Rollback Automation: Automatic rollback on failure
• Rollback Scheduling: Time-based rollback operations
• Health Checks: Pre/post rollback validation

2. Integration Improvements

• API Support: REST API for rollback operations
• Web Interface: Web-based rollback management
• Configuration Management: Declarative rollback control
• Audit Logging: Comprehensive audit trails

This technical guide provides comprehensive understanding of the bootc rollback system's architecture, implementation, and usage patterns.