robojerk 526f1c1afd Initial commit: Comprehensive Debian bootc documentation

- Complete documentation for all bootc commands and subcommands
- Debian-specific adaptations and workarounds
- Manual installation methods to bypass bootc reliability issues
- Technical guides with Rust source code analysis
- Flowcharts and external command references
- Hidden command documentation (bootc internals, state, etc.)
- Composefs integration analysis
- Base image creation guides (with and without bootc binary)
- Management scripts and automation
- Comprehensive troubleshooting and examples

2025-09-15 14:02:28 -07:00

13 KiB

Raw Blame History

bootc usr-overlay - Technical Guide

Overview

bootc usr-overlay is a command for adding a transient writable overlay filesystem on the /usr directory. This overlay allows temporary modifications to the /usr directory that will be discarded on reboot, enabling package installation and system modifications while maintaining the integrity of the base system.

Purpose

The usr-overlay command serves several critical functions:

Temporary Package Installation: Install packages that aren't in the base image
Debugging Tools: Install tracing and debugging tools like strace
System Modifications: Make temporary changes to /usr without persistence
Development Workflows: Enable development and testing workflows
System Maintenance: Allow maintenance operations without permanent changes

Command Syntax

bootc usr-overlay [OPTIONS...]

Basic Usage

# Add transient writable overlay on /usr
bootc usr-overlay

# Alternative command name
bootc usroverlay

Command Options

Currently, bootc usr-overlay has no command-line options. It's a simple command that either succeeds or fails.

Architecture Overview

1. Implementation Structure

/// Implementation of `bootc usroverlay`
async fn usroverlay() -> Result<()> {
    // This is just a pass-through today.  At some point we may make this a libostree API
    // or even oxidize it.
    Err(Command::new("ostree")
        .args(["admin", "unlock"])
        .exec()
        .into())
}

Current Implementation:

Pass-through Command: Currently delegates to ostree admin unlock
Future Plans: May become a native libostree API or Rust implementation
Simple Interface: No complex options or configuration

2. OSTree Integration

The command currently uses ostree admin unlock which:

Unlocks the OSTree deployment: Makes /usr writable
Creates Overlay: Sets up a transient overlay filesystem
Preserves Base: Keeps the original /usr intact
Enables Modifications: Allows temporary changes

3. Overlay Filesystem Mechanism

# What happens internally (simplified)
ostree admin unlock
# This creates an overlay filesystem on /usr
# - Lower layer: Original read-only /usr
# - Upper layer: Writable overlay
# - Work layer: Temporary files during operations

Use Cases

1. Debugging and Tracing Tools

Scenario: Need to install debugging tools not in base image

# Add overlay
bootc usr-overlay

# Install debugging tools
apt update
apt install -y strace gdb valgrind

# Use tools
strace -p 1234
gdb /path/to/process

# Changes discarded on reboot

2. Package Manager Operations

Scenario: Install packages temporarily for testing

# Add overlay
bootc usr-overlay

# Install packages
apt install -y python3-pip
pip3 install requests

# Test application
python3 -c "import requests; print('Success')"

# Changes discarded on reboot

3. Development Workflows

Scenario: Development and testing without permanent changes

# Add overlay
bootc usr-overlay

# Install development tools
apt install -y build-essential git

# Clone and build project
git clone https://github.com/user/project.git
cd project
make

# Test and develop
./project

# Changes discarded on reboot

4. System Maintenance

Scenario: Temporary system modifications for maintenance

# Add overlay
bootc usr-overlay

# Install maintenance tools
apt install -y htop iotop nethogs

# Perform maintenance
htop
iotop
nethogs

# Changes discarded on reboot

Filesystem Behavior

1. /usr Directory

Before Overlay:

Read-only: /usr is mounted read-only
Immutable: Cannot be modified
Base System: Contains original system files

After Overlay:

Writable: /usr becomes writable
Overlay: Changes go to overlay layer
Transient: Changes discarded on reboot

2. /etc and /var Directories

Important: The overlay only affects /usr, not /etc or /var

/etc Directory:

Persistent: Changes persist across reboots
Configuration: System configuration files
Not Affected: Overlay doesn't apply here

/var Directory:

Persistent: Changes persist across reboots
Runtime Data: Logs, caches, temporary files
Not Affected: Overlay doesn't apply here

3. Overlay Structure

/usr (overlay mount point)
├── lower (original read-only /usr)
├── upper (writable overlay layer)
└── work (temporary work directory)

OSTree Integration

1. OSTree Admin Unlock

The command delegates to ostree admin unlock which:

ostree admin unlock

What it does:

Unlocks Deployment: Makes the current deployment writable
Creates Overlay: Sets up overlay filesystem
Preserves Base: Keeps original content intact
Enables Changes: Allows modifications

2. OSTree Deployment State

Before Unlock:

Read-only: Deployment is read-only
Immutable: Cannot be modified
Base State: Original system state

After Unlock:

Writable: Deployment becomes writable
Overlay Active: Overlay filesystem active
Changes Allowed: Modifications possible

3. OSTree Lock Management

Lock States:

Locked: Read-only, immutable
Unlocked: Writable, overlay active
Auto-lock: Locks on reboot

Package Manager Integration

1. APT Integration

Debian/Ubuntu Systems:

# Add overlay
bootc usr-overlay

# Update package lists
apt update

# Install packages
apt install -y package-name

# Packages installed to overlay
# Changes discarded on reboot

Package Installation:

Overlay Target: Packages installed to overlay
Transient: Changes don't persist
Base Preserved: Original system unchanged

2. DNF Integration

Fedora/RHEL Systems:

# Add overlay
bootc usr-overlay

# Install packages
dnf install -y package-name

# Packages installed to overlay
# Changes discarded on reboot

Package Installation:

Overlay Target: Packages installed to overlay
Transient: Changes don't persist
Base Preserved: Original system unchanged

3. Package Manager Detection

Recommended Integration: Package managers should detect overlay state and inform users:

# Example error message
error: read-only /usr detected, refusing to operate. 
See `man apt-image-based` for more information.

Detection Methods:

Read-only Check: Check if /usr is read-only
Overlay Detection: Detect overlay filesystem
Bootc Detection: Check if system is bootc-managed

Unmounting the Overlay

1. Lazy Unmount

Standard Unmount:

umount /usr
# May fail if processes hold references

Lazy Unmount:

umount -l /usr
# Unmounts when references are released

2. Process References

Common Holders:

Package Managers: apt, dnf processes
System Services: Services using /usr files
User Processes: Applications with open files

Solution:

Lazy Unmount: Use umount -l /usr
Process Termination: Stop processes holding references
Reboot: Automatic unmount on reboot

3. Unmount Process

# Check what's using /usr
lsof /usr

# Lazy unmount
umount -l /usr

# Verify unmount
mount | grep /usr

Error Handling

1. Common Errors

OSTree Not Available

Error: ostree command not found

Solution: Install OSTree

apt install -y ostree
# or
dnf install -y ostree

Permission Denied

Error: Permission denied

Solution: Run with appropriate privileges

sudo bootc usr-overlay

Already Unlocked

Error: Deployment already unlocked

Solution: Check current state

ostree admin status

2. Troubleshooting

Check Overlay Status

# Check if overlay is active
mount | grep overlay

# Check OSTree status
ostree admin status

# Check /usr mount
mount | grep /usr

Verify Changes

# Test if /usr is writable
touch /usr/test-file

# Check if file exists
ls -la /usr/test-file

# File should exist until reboot

Security Considerations

1. Transient Nature

Security Benefits:

No Persistence: Changes don't persist
Base Integrity: Original system unchanged
Clean State: System returns to known state

Security Implications:

Temporary Access: Allows temporary modifications
Package Installation: Can install arbitrary packages
System Changes: Can modify system behavior

2. Access Control

Privilege Requirements:

Root Access: Requires root privileges
OSTree Access: Needs OSTree admin access
Mount Privileges: Requires mount capabilities

Best Practices:

Minimal Use: Use only when necessary
Audit Changes: Monitor what's installed
Clean Up: Unmount when done

3. Overlay Security

Overlay Isolation:

Separate Layer: Changes in separate layer
Base Protection: Original system protected
Clean Separation: Clear boundary between layers

Performance Considerations

1. Overlay Overhead

Performance Impact:

Overlay Layer: Additional filesystem layer
Copy-on-Write: COW operations for modifications
Metadata Overhead: Additional metadata for overlay

Mitigation:

Minimal Changes: Make only necessary changes
Unmount When Done: Remove overlay when not needed
Efficient Operations: Use efficient package operations

2. Storage Usage

Storage Impact:

Overlay Storage: Additional storage for overlay
Temporary Files: Work directory usage
Package Caches: Package manager caches

Management:

Clean Caches: Clean package caches
Unmount Overlay: Remove overlay when done
Monitor Usage: Monitor storage usage

Integration Patterns

1. Development Workflows

Development Setup:

# Add overlay
bootc usr-overlay

# Install development tools
apt install -y build-essential git python3-pip

# Install project dependencies
pip3 install -r requirements.txt

# Develop and test
# Changes discarded on reboot

2. Debugging Workflows

Debugging Setup:

# Add overlay
bootc usr-overlay

# Install debugging tools
apt install -y strace gdb valgrind

# Debug application
strace -f ./application
gdb ./application

# Changes discarded on reboot

3. Testing Workflows

Testing Setup:

# Add overlay
bootc usr-overlay

# Install test tools
apt install -y test-package

# Run tests
./run-tests.sh

# Changes discarded on reboot

Best Practices

1. Usage Guidelines

When to Use:

Debugging: Install debugging tools
Development: Temporary development setup
Testing: Test new packages
Maintenance: System maintenance tasks

When Not to Use:

Production: Avoid in production systems
Persistent Changes: Don't use for permanent changes
System Modifications: Avoid core system changes

2. Workflow Management