apt-tx/docs/dev-rollback-discussion.md

# Rollback Implementation Discussion

** This file was used to keep track of ideas for doing rollbacks**
It is not an official document. It's kept here in case we want to relitigate our process

## Current Issues

### Problem 1: Tries to install specific versions that may no longer be available
```rust
// Current problematic code in rollback():
packages_to_restore.push(format!("{}={}", package, before_version));
```

**Issue**: If the old version is no longer in repositories, this will fail.

### Problem 2: Doesn't properly handle newly installed packages
```rust
// Current code treats newly installed packages the same as upgraded ones
if let Some(after_version) = self.after_versions.get(package) {
    // Only rollback if version changed
    if before_version != after_version {
        packages_to_restore.push(format!("{}={}", package, before_version));
    }
} else {
    // Package was newly installed, remove it
    packages_to_restore.push(format!("{}", package));
}
```

**Issue**: Newly installed packages should be removed with `apt remove`, not downgraded.

## Proposed Solutions

### Solution 1: Separate handling for new vs upgraded packages
```rust
// Track what was newly installed vs upgraded
let mut packages_to_remove = Vec::new();
let mut packages_to_downgrade = Vec::new();

for (package, before_version) in &self.before_versions {
    if let Some(after_version) = self.after_versions.get(package) {
        if before_version != after_version {
            packages_to_downgrade.push(format!("{}={}", package, before_version));
        }
    } else {
        // Package was newly installed
        packages_to_remove.push(package.clone());
    }
}
```

### Solution 2: Use apt remove for newly installed packages
```rust
// Remove newly installed packages
if !packages_to_remove.is_empty() {
    let output = Command::new("apt")
        .args(&["remove", "-y"])
        .args(&packages_to_remove)
        .output()
        .context("Failed to execute APT remove command")?;
    
    if !output.status.success() {
        let error = String::from_utf8_lossy(&output.stderr);
        return Err(anyhow!("APT removal failed: {}", error)
            .context(format!("Failed to remove packages: {:?}", packages_to_remove)));
    }
}
```

### Solution 3: Fallback for unavailable versions
```rust
// Try to downgrade, with fallback to removal
for package_downgrade in packages_to_downgrade {
    let output = Command::new("apt")
        .args(&["install", "-y", &package_downgrade])
        .output()
        .context("Failed to execute APT downgrade command")?;
    
    if !output.status.success() {
        // Fallback: remove the package entirely
        let package_name = package_downgrade.split('=').next().unwrap();
        let remove_output = Command::new("apt")
            .args(&["remove", "-y", package_name])
            .output()
            .context("Failed to execute APT remove fallback")?;
        
        if !remove_output.status.success() {
            // Log warning but continue
            eprintln!("Warning: Could not rollback package {}", package_name);
        }
    }
}
```

## Questions for Discussion

1. **Should we track package installation state more explicitly?**
   - Currently we infer it from before/after version comparison
   - Could track "newly_installed" packages separately

2. **How to handle partial rollback failures?**
   - Continue with other packages?
   - Stop and report what failed?

3. **Should we use apt-mark for version pinning?**
   - Pin versions before changes to prevent upgrades
   - More complex but potentially more reliable

4. **What about dependency conflicts during rollback?**
   - Some packages might have been installed as dependencies
   - Removing them might break other packages

## Discussion & Decision

### **Scope: "Good Enough" vs. "Complex"**
**Decision: Aim for "good enough" (80/20 rule)**

- Vast majority of package updates are simple and don't involve complex dependency trees
- Complex dependency graph analysis would push code well beyond line count goals
- Acknowledging apt's inherent limitations is better than fragile workarounds

### **Failure Mode: "Fail Fast"**
**Decision: Stop completely and let user fix manually**

- Most honest approach - communicates exactly what failed
- Prevents further damage from partial rollback
- Gives user control to make informed decisions
- Aligns with "fail with loud, clear error" principle

### **Future Direction: Stepping Stone**
**Decision: This is a stepping stone toward more robust solutions**

- Document limitations clearly in README
- Demonstrate value of rollback feature
- Highlight problems that snapshot-based solutions would solve
- Create foundation for more advanced approaches

## **Final Approach: Hybrid with Explicit Tracking**

```rust
pub struct AptTransaction {
    packages: Vec<String>,
    newly_installed: HashSet<String>,  // Explicit tracking
    upgraded: HashMap<String, String>, // package -> old_version
}
```

**Benefits:**
- Handles 80% of common cases correctly
- Clear failure modes
- Maintains simplicity goals
- Sets stage for future improvements

## **Refined Scope: "Good Enough + Safety"**

**Decision: Aim for common cases with robust safety measures**

- Handle 80% of rollbacks correctly
- Add essential safety features (locking, critical package detection)
- Avoid complex dependency graph analysis
- Stop short of filesystem snapshots or full atomicity

**Key Design Principle**: The tracked state represents the explicitly managed packages in the transaction. Due to cascading dependencies, the actual system changes may be larger. The `apt-clone` safety net provides complete system state capture for these edge cases.

## Additional Considerations

### **Package Locking & Concurrency**
- **Lock apt during operations** to prevent concurrent package management
- Use `dpkg --configure -a` before operations, unlock after
- Prevents race conditions with other package managers

### **Critical Package Handling**
- **Use apt's built-in essential package detection**: `dpkg -l | grep 'ii' | grep -E '^E'`
- **Prevent rollback of essential packages** to avoid system breakage
- **No hardcoded whitelist** - rely on apt's own package classification

### **State Persistence & Retry**
- **Simple JSON file**: `/var/lib/apt-wrapper/rollback-state.json`
- **Store before each operation**: Package lists, versions, operation type
- **Enable retry**: Read state file to resume from last successful operation
- **Crash recovery**: Detect incomplete rollback and offer retry option
- **Corruption handling**: Simple checksum validation, fallback to clean state if corrupted

### **Enhanced Error Handling**
- **Fail fast on critical errors**: Package not found, version unavailable, permission denied
- **Continue with warnings on soft errors**: Package already removed, non-critical dependency warnings
- **Clear distinction**: Critical = system-breaking, Soft = recoverable or ignorable
- **Accumulate warnings**: Collect all soft errors and display summary at end
- **Informative fallback messages**: Explain why package removal occurred (version unavailable)

### **Soft Error Definition**
**Soft errors are non-critical issues that don't break the rollback:**
- Package already removed (no action needed)
- Non-essential dependency warnings (system still functional)
- Version downgrade warnings (package still works)

**Critical errors stop the entire rollback:**
- Package not found in repository
- Permission denied
- Essential package conflicts

### **Exit Code Strategy**
- **0**: Rollback successful, maybe with warnings
- **1**: Rollback failed on critical error
- **2**: Rollback succeeded but with unresolved soft issues

## **Rollback Command Strategy**

**Decision: Separate commands for remove vs downgrade (non-atomic)**

- **Phase 1**: `apt remove -y package1 package2` (newly installed packages)
- **Phase 2**: `apt install -y package1=oldver package2=oldver` (upgraded packages)
- **Acknowledge**: This is not atomic, but simpler and more reliable than complex single commands

## **Implementation Details**

### **Warning Accumulation**
```rust
let mut warnings = Vec::new();

// During rollback operations
if some_error_condition {
    warnings.push(format!("Warning: Could not remove package {}", package));
}

// After the rollback
if !warnings.is_empty() {
    eprintln!("Warnings during rollback:");
    for warning in warnings {
        eprintln!("{}", warning);
    }
}
```

### **Informative Fallback Messages**
```rust
// This is the "continue with warnings" path for a non-critical error
if !output.status.success() {
    let package_name = package_downgrade.split('=').next().unwrap();
    eprintln!("Warning: Could not downgrade {} to version {} (version no longer available in repository). Removing instead.", package_name, before_version);
    
    let remove_output = Command::new("apt")
        .args(&["remove", "-y", package_name])
        .output()
        .context("Failed to execute APT remove fallback")?;
}
```

### **Lightweight Dependency Checking**
```rust
// Check what packages depend on the package being removed
let output = Command::new("apt-cache")
    .arg("rdepends")
    .arg(removed_package)
    .output()
    .expect("Failed to check reverse dependencies");

if !output.stdout.is_empty() {
    eprintln!("Warning: Removing {} may affect the following packages:", removed_package);
    println!("{}", String::from_utf8_lossy(&output.stdout));
}
```

### **User-Installed Package Awareness**
```rust
// Quick check for user-installed rdepends before removal
let output = Command::new("apt-mark")
    .args(&["showmanual"])
    .output()?;

let manual_packages: HashSet<_> = String::from_utf8_lossy(&output.stdout)
    .lines()
    .collect();

for pkg in &packages_to_remove {
    let rdepends = get_reverse_dependencies(pkg)?;
    // Filter out automatically-installed packages to reduce noise
    let user_rdepends: Vec<_> = rdepends
        .into_iter()
        .filter(|dep| manual_packages.contains(dep))
        .collect();

    if !user_rdepends.is_empty() {
        eprintln!("Warning: Removing {} may break manually installed packages: {:?}", pkg, user_rdepends);
    }
}
```

### **Meta-Package Detection**
```rust
// Detect meta-packages and warn about cascade removals
for pkg in &packages_to_remove {
    let output = Command::new("apt-cache")
        .args(&["show", pkg])
        .output()?;
    
    if output.status.success() {
        let info = String::from_utf8_lossy(&output.stdout);
        if info.contains("Meta package") || info.contains("Virtual package") {
            eprintln!("Warning: {} is a meta-package, removal may trigger cascade removals", pkg);
        }
    }
}
```

### **Dry-Run JSON Output**
```json
{
  "would_remove": ["pkg1", "pkg2"],
  "would_downgrade": ["pkg3=1.2.3"],
  "warnings": ["pkg1 has user-installed reverse dependencies: foo, bar"]
}
```

## **Strategic Improvements**

### **Pre-Transaction Safety Net**
- **Use `apt-clone` (preferred) or `dpkg --get-selections`** to create full system snapshot before rollback
- **apt-clone**: Creates complete package state backup including sources and keyrings
- **dpkg --get-selections**: Simpler but only captures package selections
- **Complete restore point** if rollback fails catastrophically
- **Better than partial rollback** - can restore entire system state

### **Enhanced Dependency Awareness**
- **Warn on non-auto-installed reverse dependencies** (user-installed packages that depend on removed package)
- **Prevent breaking user workflows** by highlighting critical dependencies
- **Interactive confirmation** for potentially destructive operations

### **Version Handling Sophistication**
- **Find closest available older version** instead of exact match or remove
- **Interactive fallback**: "Version X not found. Install nearest available version Y? [Y/n]"
- **User-configurable fallback strategy**:
  - `--strict`: fail if exact version not found
  - `--nearest`: choose closest available lower version
  - `--remove`: remove if version missing (default)
- **Less destructive** than complete package removal

### **Concurrency Safety Improvements**
- **Wait for lock with timeout** instead of immediate failure
- **Check `/var/lib/dpkg/lock-frontend`** and wait up to 30 seconds
- **Clear error message** if timeout expires: "APT is locked by another process, aborting rollback"
- **Optional `--wait-indefinitely` flag** for CI/automation use cases
- **Better user experience** when apt is busy with other operations

### **Enhanced State Management**
- **Schema versioning** in rollback-state.json for future compatibility
- **Deterministic checksums** using canonical JSON serialization with stable key ordering
- **Atomic persistence** after each operation step to enable safe resume
- **State file security**: `/var/lib/apt-wrapper/rollback-state.json` owned by `root:root` with mode `0600`
- **Manual state clearing**: `apt-wrapper rollback --abort`
- **Dry-run mode**: `apt-wrapper rollback --dry-run` to preview changes
- **Machine-readable dry-run output** (JSON format) for automation/CI integration

## **Future Considerations**

### **Auditing & Troubleshooting**
- Track rollback metadata (timestamps, user IDs) for auditing
- Integration with larger system management tools
- Logging for troubleshooting and system analysis

### **Scalability Planning**
- Handle multiple system states simultaneously
- Related package update rollbacks
- Service-level rollback coordination

## **Implementation Phases**

The following phased plan will incrementally implement the features outlined in the 'Final Approach' and 'Strategic Improvements' sections.

### **Phase 1: Core Rollback Logic**
1. Implement separate tracking for new vs upgraded packages
2. Use `apt remove` for newly installed packages
3. **Fail fast on critical errors**, continue with warnings on soft errors

### **Phase 2: Robustness Enhancements**
4. **Add warning accumulation** for better user experience
5. **Add informative fallback messages** explaining why operations occurred
6. **Add lightweight dependency checking** for removed packages
7. Add proper error handling and logging
8. **Add package locking with timeout** to prevent concurrent operations
9. **Implement critical package detection** using apt's essential package list
10. **Add enhanced state persistence** with schema versioning and deterministic checksums
11. **Add dry-run mode** for previewing rollback operations
12. **Add CLI ergonomics**: `--resume`, `--abort`, `--dry-run --json` vs `--dry-run --pretty`
13. **Add exit code strategy** for automation/CI integration

### **Phase 2.5: Advanced Safety Features**
14. **Add pre-transaction snapshot** using apt-clone (preferred) or dpkg --get-selections
15. **Enhanced dependency awareness** for non-auto-installed packages
16. **Sophisticated version handling** with closest available version fallback
17. **Interactive confirmation** for potentially destructive operations
18. **Add meta-package detection** and cascade removal warnings

### **Phase 3: Testing and Documentation**
19. Test with various scenarios
20. Document limitations clearly
21. **Add JSON schema documentation** for rollback-state.json

## **Rollback State JSON Schema**

```json
{
  "schema_version": "1.0",
  "checksum": "sha256:abc123...",
  "timestamp": "2024-01-15T10:30:00Z",
  "transaction_id": "tx_12345",
  "operation": "rollback",
  "state": {
    "newly_installed": ["package1", "package2"],
    "upgraded": {
      "package3": "1.0.0",
      "package4": "2.1.0"
    },
    "before_versions": {
      "package3": "0.9.0",
      "package4": "2.0.0"
    },
    "after_versions": {
      "package3": "1.0.0",
      "package4": "2.1.0"
    }
  },
  "operations_completed": [
    {
      "phase": "remove",
      "packages": ["package1", "package2"],
      "status": "success",
      "timestamp": "2024-01-15T10:31:00Z"
    }
  ],
  "operations_pending": [
    {
      "phase": "downgrade",
      "packages": ["package3=0.9.0", "package4=2.0.0"],
      "status": "pending"
    }
  ]
}
```

**Key Features:**
- **Schema versioning**: Future compatibility
- **Checksum**: Corruption detection
- **Transaction tracking**: Unique ID for each rollback
- **Phase tracking**: What's completed vs pending
- **State preservation**: Complete before/after package states