bootc-docs/usr-overlay/bootc-usr-overlay-technical-guide.md

# bootc usr-overlay - Technical Guide

## Overview

`bootc usr-overlay` is a command for adding a transient writable overlay filesystem on the `/usr` directory. This overlay allows temporary modifications to the `/usr` directory that will be discarded on reboot, enabling package installation and system modifications while maintaining the integrity of the base system.

## Purpose

The usr-overlay command serves several critical functions:

1. **Temporary Package Installation**: Install packages that aren't in the base image
2. **Debugging Tools**: Install tracing and debugging tools like `strace`
3. **System Modifications**: Make temporary changes to `/usr` without persistence
4. **Development Workflows**: Enable development and testing workflows
5. **System Maintenance**: Allow maintenance operations without permanent changes

## Command Syntax

```bash
bootc usr-overlay [OPTIONS...]
```

### Basic Usage

```bash
# Add transient writable overlay on /usr
bootc usr-overlay

# Alternative command name
bootc usroverlay
```

## Command Options

Currently, `bootc usr-overlay` has no command-line options. It's a simple command that either succeeds or fails.

## Architecture Overview

### 1. Implementation Structure

```rust
/// Implementation of `bootc usroverlay`
async fn usroverlay() -> Result<()> {
    // This is just a pass-through today.  At some point we may make this a libostree API
    // or even oxidize it.
    Err(Command::new("ostree")
        .args(["admin", "unlock"])
        .exec()
        .into())
}
```

**Current Implementation**:
- **Pass-through Command**: Currently delegates to `ostree admin unlock`
- **Future Plans**: May become a native libostree API or Rust implementation
- **Simple Interface**: No complex options or configuration

### 2. OSTree Integration

The command currently uses `ostree admin unlock` which:
- **Unlocks the OSTree deployment**: Makes `/usr` writable
- **Creates Overlay**: Sets up a transient overlay filesystem
- **Preserves Base**: Keeps the original `/usr` intact
- **Enables Modifications**: Allows temporary changes

### 3. Overlay Filesystem Mechanism

```bash
# What happens internally (simplified)
ostree admin unlock
# This creates an overlay filesystem on /usr
# - Lower layer: Original read-only /usr
# - Upper layer: Writable overlay
# - Work layer: Temporary files during operations
```

## Use Cases

### 1. Debugging and Tracing Tools

**Scenario**: Need to install debugging tools not in base image

```bash
# Add overlay
bootc usr-overlay

# Install debugging tools
apt update
apt install -y strace gdb valgrind

# Use tools
strace -p 1234
gdb /path/to/process

# Changes discarded on reboot
```

### 2. Package Manager Operations

**Scenario**: Install packages temporarily for testing

```bash
# Add overlay
bootc usr-overlay

# Install packages
apt install -y python3-pip
pip3 install requests

# Test application
python3 -c "import requests; print('Success')"

# Changes discarded on reboot
```

### 3. Development Workflows

**Scenario**: Development and testing without permanent changes

```bash
# Add overlay
bootc usr-overlay

# Install development tools
apt install -y build-essential git

# Clone and build project
git clone https://github.com/user/project.git
cd project
make

# Test and develop
./project

# Changes discarded on reboot
```

### 4. System Maintenance

**Scenario**: Temporary system modifications for maintenance

```bash
# Add overlay
bootc usr-overlay

# Install maintenance tools
apt install -y htop iotop nethogs

# Perform maintenance
htop
iotop
nethogs

# Changes discarded on reboot
```

## Filesystem Behavior

### 1. /usr Directory

**Before Overlay**:
- **Read-only**: `/usr` is mounted read-only
- **Immutable**: Cannot be modified
- **Base System**: Contains original system files

**After Overlay**:
- **Writable**: `/usr` becomes writable
- **Overlay**: Changes go to overlay layer
- **Transient**: Changes discarded on reboot

### 2. /etc and /var Directories

**Important**: The overlay only affects `/usr`, not `/etc` or `/var`

**/etc Directory**:
- **Persistent**: Changes persist across reboots
- **Configuration**: System configuration files
- **Not Affected**: Overlay doesn't apply here

**/var Directory**:
- **Persistent**: Changes persist across reboots
- **Runtime Data**: Logs, caches, temporary files
- **Not Affected**: Overlay doesn't apply here

### 3. Overlay Structure

```
/usr (overlay mount point)
├── lower (original read-only /usr)
├── upper (writable overlay layer)
└── work (temporary work directory)
```

## OSTree Integration

### 1. OSTree Admin Unlock

The command delegates to `ostree admin unlock` which:

```bash
ostree admin unlock
```

**What it does**:
- **Unlocks Deployment**: Makes the current deployment writable
- **Creates Overlay**: Sets up overlay filesystem
- **Preserves Base**: Keeps original content intact
- **Enables Changes**: Allows modifications

### 2. OSTree Deployment State

**Before Unlock**:
- **Read-only**: Deployment is read-only
- **Immutable**: Cannot be modified
- **Base State**: Original system state

**After Unlock**:
- **Writable**: Deployment becomes writable
- **Overlay Active**: Overlay filesystem active
- **Changes Allowed**: Modifications possible

### 3. OSTree Lock Management

**Lock States**:
- **Locked**: Read-only, immutable
- **Unlocked**: Writable, overlay active
- **Auto-lock**: Locks on reboot

## Package Manager Integration

### 1. APT Integration

**Debian/Ubuntu Systems**:

```bash
# Add overlay
bootc usr-overlay

# Update package lists
apt update

# Install packages
apt install -y package-name

# Packages installed to overlay
# Changes discarded on reboot
```

**Package Installation**:
- **Overlay Target**: Packages installed to overlay
- **Transient**: Changes don't persist
- **Base Preserved**: Original system unchanged

### 2. DNF Integration

**Fedora/RHEL Systems**:

```bash
# Add overlay
bootc usr-overlay

# Install packages
dnf install -y package-name

# Packages installed to overlay
# Changes discarded on reboot
```

**Package Installation**:
- **Overlay Target**: Packages installed to overlay
- **Transient**: Changes don't persist
- **Base Preserved**: Original system unchanged

### 3. Package Manager Detection

**Recommended Integration**:
Package managers should detect overlay state and inform users:

```bash
# Example error message
error: read-only /usr detected, refusing to operate. 
See `man apt-image-based` for more information.
```

**Detection Methods**:
- **Read-only Check**: Check if `/usr` is read-only
- **Overlay Detection**: Detect overlay filesystem
- **Bootc Detection**: Check if system is bootc-managed

## Unmounting the Overlay

### 1. Lazy Unmount

**Standard Unmount**:
```bash
umount /usr
# May fail if processes hold references
```

**Lazy Unmount**:
```bash
umount -l /usr
# Unmounts when references are released
```

### 2. Process References

**Common Holders**:
- **Package Managers**: `apt`, `dnf` processes
- **System Services**: Services using `/usr` files
- **User Processes**: Applications with open files

**Solution**:
- **Lazy Unmount**: Use `umount -l /usr`
- **Process Termination**: Stop processes holding references
- **Reboot**: Automatic unmount on reboot

### 3. Unmount Process

```bash
# Check what's using /usr
lsof /usr

# Lazy unmount
umount -l /usr

# Verify unmount
mount | grep /usr
```

## Error Handling

### 1. Common Errors

#### OSTree Not Available
```bash
Error: ostree command not found
```

**Solution**: Install OSTree
```bash
apt install -y ostree
# or
dnf install -y ostree
```

#### Permission Denied
```bash
Error: Permission denied
```

**Solution**: Run with appropriate privileges
```bash
sudo bootc usr-overlay
```

#### Already Unlocked
```bash
Error: Deployment already unlocked
```

**Solution**: Check current state
```bash
ostree admin status
```

### 2. Troubleshooting

#### Check Overlay Status
```bash
# Check if overlay is active
mount | grep overlay

# Check OSTree status
ostree admin status

# Check /usr mount
mount | grep /usr
```

#### Verify Changes
```bash
# Test if /usr is writable
touch /usr/test-file

# Check if file exists
ls -la /usr/test-file

# File should exist until reboot
```

## Security Considerations

### 1. Transient Nature

**Security Benefits**:
- **No Persistence**: Changes don't persist
- **Base Integrity**: Original system unchanged
- **Clean State**: System returns to known state

**Security Implications**:
- **Temporary Access**: Allows temporary modifications
- **Package Installation**: Can install arbitrary packages
- **System Changes**: Can modify system behavior

### 2. Access Control

**Privilege Requirements**:
- **Root Access**: Requires root privileges
- **OSTree Access**: Needs OSTree admin access
- **Mount Privileges**: Requires mount capabilities

**Best Practices**:
- **Minimal Use**: Use only when necessary
- **Audit Changes**: Monitor what's installed
- **Clean Up**: Unmount when done

### 3. Overlay Security

**Overlay Isolation**:
- **Separate Layer**: Changes in separate layer
- **Base Protection**: Original system protected
- **Clean Separation**: Clear boundary between layers

## Performance Considerations

### 1. Overlay Overhead

**Performance Impact**:
- **Overlay Layer**: Additional filesystem layer
- **Copy-on-Write**: COW operations for modifications
- **Metadata Overhead**: Additional metadata for overlay

**Mitigation**:
- **Minimal Changes**: Make only necessary changes
- **Unmount When Done**: Remove overlay when not needed
- **Efficient Operations**: Use efficient package operations

### 2. Storage Usage

**Storage Impact**:
- **Overlay Storage**: Additional storage for overlay
- **Temporary Files**: Work directory usage
- **Package Caches**: Package manager caches

**Management**:
- **Clean Caches**: Clean package caches
- **Unmount Overlay**: Remove overlay when done
- **Monitor Usage**: Monitor storage usage

## Integration Patterns

### 1. Development Workflows

**Development Setup**:
```bash
# Add overlay
bootc usr-overlay

# Install development tools
apt install -y build-essential git python3-pip

# Install project dependencies
pip3 install -r requirements.txt

# Develop and test
# Changes discarded on reboot
```

### 2. Debugging Workflows

**Debugging Setup**:
```bash
# Add overlay
bootc usr-overlay

# Install debugging tools
apt install -y strace gdb valgrind

# Debug application
strace -f ./application
gdb ./application

# Changes discarded on reboot
```

### 3. Testing Workflows

**Testing Setup**:
```bash
# Add overlay
bootc usr-overlay

# Install test tools
apt install -y test-package

# Run tests
./run-tests.sh

# Changes discarded on reboot
```

## Best Practices

### 1. Usage Guidelines

**When to Use**:
- **Debugging**: Install debugging tools
- **Development**: Temporary development setup
- **Testing**: Test new packages
- **Maintenance**: System maintenance tasks

**When Not to Use**:
- **Production**: Avoid in production systems
- **Persistent Changes**: Don't use for permanent changes
- **System Modifications**: Avoid core system changes

### 2. Workflow Management

**Before Overlay**:
- **Plan Changes**: Know what you need to install
- **Check Dependencies**: Verify package dependencies
- **Prepare Commands**: Have commands ready

**During Overlay**:
- **Minimal Changes**: Make only necessary changes
- **Document Changes**: Keep track of what's installed
- **Test Thoroughly**: Test all functionality

**After Overlay**:
- **Clean Up**: Remove unnecessary packages
- **Unmount**: Unmount overlay when done
- **Document**: Document what was done

### 3. System Management

**Monitoring**:
- **Check Status**: Monitor overlay status
- **Track Changes**: Keep track of modifications
- **Monitor Performance**: Watch for performance impact

**Maintenance**:
- **Regular Cleanup**: Clean up regularly
- **Unmount When Done**: Remove overlay when not needed
- **Reboot Periodically**: Reboot to clean state

## Future Enhancements

### 1. Planned Features

**Native Implementation**:
- **Rust Implementation**: Native Rust implementation
- **libostree API**: Direct libostree API usage
- **Better Integration**: Improved OSTree integration

**Enhanced Functionality**:
- **Selective Overlay**: Overlay specific directories
- **Persistent Overlay**: Option for persistent overlays
- **Overlay Management**: Better overlay management

### 2. Integration Improvements

**Package Manager Integration**:
- **Automatic Detection**: Auto-detect overlay state
- **User Warnings**: Warn about transient changes
- **Better Error Messages**: Improved error messages

**System Integration**:
- **Service Integration**: Systemd service integration
- **Configuration**: Configuration file support
- **Logging**: Better logging and monitoring

## Troubleshooting

### 1. Common Issues

#### Overlay Not Working
```bash
# Check if overlay is active
mount | grep overlay

# Check OSTree status
ostree admin status

# Check /usr mount
mount | grep /usr
```

#### Package Installation Fails
```bash
# Check if /usr is writable
touch /usr/test-file

# Check package manager status
apt update
# or
dnf check-update
```

#### Cannot Unmount
```bash
# Check what's using /usr
lsof /usr

# Use lazy unmount
umount -l /usr
```

### 2. Debug Commands

```bash
# Check overlay status
mount | grep overlay

# Check OSTree status
ostree admin status

# Check /usr mount
mount | grep /usr

# Check processes using /usr
lsof /usr

# Check overlay filesystem
cat /proc/mounts | grep overlay
```

This technical guide provides comprehensive understanding of the bootc usr-overlay system's architecture, implementation, and usage patterns.