apt-ostree/.notes/research/atomic-filesystems.md

Understanding Atomic Filesystems: rpm-ostree, apt-ostree, and Ubuntu Integration

Overview

Fedora Atomic Desktops, including spins like Silverblue, Kinoite (KDE Plasma), Bazzite, and Bluefin, leverage rpm-ostree to provide a unique approach to operating system management built around an immutable core filesystem. This differs significantly from traditional Linux distributions and introduces some nuances in how the filesystem is structured and interact with applications.

NEW: apt-ostree Integration The apt-ostree project brings similar atomic filesystem concepts to Ubuntu/Debian systems, adapting OSTree's immutable filesystem model for APT package management.

1. The Immutable Root Filesystem

Read-Only Core

• The core operating system (located at / and everything under /usr) is mounted as read-only
• This enhances stability and security by preventing accidental or malicious modifications to the base system
• apt-ostree adaptation: Same principle applies, but with DEB packages instead of RPM packages

Version Control

• rpm-ostree functions like "Git for operating system binaries", allowing for atomic updates and rollbacks
• apt-ostree equivalent: Uses OSTree for atomic commits of DEB package layers
• Updates are applied as a whole, transactional unit, rather than piecemeal package installations

Transactional Updates

• When you perform an OS update on a Fedora Atomic Desktop, rpm-ostree downloads and prepares the new version in the background
• apt-ostree process: Similar workflow but with APT package resolution and DEB extraction
• New combined image is created, with previous version available for rollback

2. Writable Directories and User Data

Separate Writable Areas

• While the core OS is immutable, directories like /etc and /var remain writable
• Ubuntu considerations: Same structure, but Ubuntu may have additional writable locations

User Data Preservation

• User data is stored separately (typically in /var/home, symlinked to /home)
• apt-ostree mapping:

  /home → /var/home (symlink)
  /opt → /var/opt (symlink)
  /usr/local → /var/usrlocal (symlink)
  /srv → /var/srv (symlink)
  /root → /var/roothome (symlink)
  /mnt → /var/mnt (symlink)
  /tmp → /var/tmp (symlink)
  

Symlinks for Compatibility

• Fedora Atomic Desktops utilize symlinks to redirect expected writable locations into /var
• apt-ostree implementation: Same symlink structure for Ubuntu compatibility

3. Application Management and Layering

Containerized Applications

• Flatpaks: Core philosophy for most software installations
• Ubuntu alternatives: Snap packages and Flatpaks both work with apt-ostree
• Applications run in isolated environments and are not part of the base filesystem

Package Layering

• rpm-ostree: Allows "layering" additional RPM packages on top of the base OS image
• apt-ostree: Layers DEB packages on top of the base Ubuntu system
• mmdebstrap integration: Used for creating clean base images and package layers

Development Environments

• Toolbox/Devcontainers: Containerized development environments
• Ubuntu equivalents: Docker containers, LXD containers, or chroot environments
• Keeps development tools isolated from the host system

4. Filesystem Structure for apt-ostree

Base Filesystem Layout

/
├── ostree/                    # OSTree repository and deployments
│   ├── repo/                 # OSTree repository
│   ├── deploy/               # Deployed systems
│   └── boot/                 # Boot configurations
├── var/                      # Writable data (shared across deployments)
│   ├── home/                 # User home directories
│   ├── opt/                  # Optional application software
│   ├── usrlocal/             # Locally installed software
│   ├── etc/                  # System configuration (merged on upgrade)
│   └── tmp/                  # Temporary files
├── etc/                      # System configuration (writable)
└── usr/                      # Read-only system software

mmdebstrap Integration

# Create base system with mmdebstrap
sudo mmdebstrap --arch=amd64 --variant=minbase \
    --include=systemd,systemd-sysv,ostree \
    noble /tmp/ubuntu-base

# Create package layer
sudo mmdebstrap --arch=amd64 \
    --include=ubuntu-desktop-minimal,gnome-shell \
    noble /tmp/ubuntu-desktop /tmp/ubuntu-base

# Create OSTree commit
sudo ostree commit --repo=/ostree/repo \
    --branch=ubuntu/24.04/x86_64/desktop \
    --parent=ubuntu/24.04/x86_64/base \
    /tmp/ubuntu-desktop

5. Ubuntu-Specific Considerations

Package Management Differences

• APT vs DNF: Different package managers with different dependency resolution
• DEB vs RPM: Different package formats and metadata
• Repository structure: Ubuntu uses different repository organization

Filesystem Choices

• Btrfs: Recommended for root and /var/home (same as Fedora)
• Ext4: Alternative for simpler setups
• LVM: Supported for advanced partitioning

Bootloader Configuration

• GRUB: Primary bootloader for Ubuntu systems
• Secure Boot: Ubuntu-specific key management
• UEFI: Modern boot method support

6. OSTree Atomic Filesystem Best Practices (Debian/Ubuntu Focus)

Core Principles

• Root and /usr are always read-only; only /etc and /var are writable
• Use symlinks/bind mounts for: /home, /opt, /srv, /root, /usr/local, /mnt, /tmp
• /var is shared across deployments; initial content is copied on first boot, not overwritten on upgrade
• /etc is merged on upgrade; defaults should be in /usr/etc
• Package layering creates new deployments; all changes are atomic and require reboot

User and Group Management

• Static users/groups: use nss-altfiles or systemd-sysusers
• Dynamic user creation: handled through systemd services
• apt-ostree consideration: DEB package scripts may create users/groups

Package Script Execution

• Pre-installation scripts: Run during package extraction
• Post-installation scripts: Run after filesystem assembly
• Sandboxing: Scripts run in controlled environment with bubblewrap
• Rollback: Failed scripts trigger automatic rollback

7. Testing and Validation

Filesystem Validation Tasks

• Validate all symlinks/bind mounts at boot and after upgrade
• Test package install/remove/upgrade for packages writing to /var, /opt, /usr/local
• Test /etc merge behavior with complex configurations
• Test user/group management and persistence
• Test container support and isolation
• Document any Debian/Ubuntu-specific quirks

mmdebstrap Testing

# Test base system creation
time sudo mmdebstrap --arch=amd64 --variant=minbase noble /tmp/test-base

# Test package layering
time sudo mmdebstrap --arch=amd64 --include=systemd,ostree noble /tmp/test-layered /tmp/test-base

# Test OSTree integration
sudo ostree commit --repo=/ostree/repo --branch=test/base /tmp/test-layered

Performance Benchmarks

# mmdebstrap vs debootstrap timing
time sudo debootstrap --arch=amd64 --variant=minbase noble /tmp/debootstrap-test
# Typical: 3-5 minutes

time sudo mmdebstrap --arch=amd64 --variant=minbase noble /tmp/mmdebstrap-test
# Typical: 1-2 minutes

8. Development Workflow with apt-ostree

Local Development Environment

# Set up isolated OSTree repository
mkdir -p ~/ostree-dev/repo
ostree init --repo=~/ostree-dev/repo --mode=archive-z2

# Create test packages
mkdir -p ~/ostree-dev/test-packages
apt download hello cowsay fortune-mod

# Extract and commit packages
for pkg in *.deb; do
    dpkg-deb -R "$pkg" "extracted-${pkg%.deb}"
done

ostree commit --repo=~/ostree-dev/repo \
    --branch=test/packages \
    --subject="Test Packages" \
    extracted-*

Integration Testing

# Test apt-ostree with local repository
cd ~/ostree-dev/apt-ostree
cargo build --release
./target/release/apt-ostree --repo=~/ostree-dev/repo status
./target/release/apt-ostree --repo=~/ostree-dev/repo install hello

9. Security Considerations

Filesystem Security

• Read-only root filesystem prevents tampering
• Symlink structure prevents privilege escalation
• OSTree commits are cryptographically verified

Package Security

• APT package verification (GPG signatures)
• OSTree commit verification
• Sandboxed script execution

Runtime Security

• Container isolation for applications
• Systemd security features
• AppArmor/SELinux integration

10. Migration and Deployment

From Traditional Ubuntu

• User data migration from /home to /var/home
• Configuration migration from /etc to /usr/etc
• Package state migration to OSTree layers

Deployment Strategies

• Incremental: Layer packages on existing system
• Full: Complete system replacement
• Hybrid: Combine with snap/Flatpak applications

Rollback Procedures

• Automatic rollback on failed updates
• Manual rollback to previous deployments
• Emergency recovery procedures

Conclusion

Fedora Atomic Desktops and their Ubuntu equivalents through apt-ostree offer a robust and reliable computing experience built around an immutable core. The filesystem structure and application handling are distinct from traditional Linux distributions, with a strong emphasis on containerization and clear separation between the base operating system and user data.

apt-ostree Integration Benefits:

• Atomic Updates: Transactional system updates with rollback capability
• Immutable Core: Enhanced security and stability
• Package Layering: Flexible software installation
• Ubuntu Compatibility: Works with existing Ubuntu ecosystem
• mmdebstrap Integration: Fast and reliable system creation

While this approach may require some adjustment for users accustomed to traditional package management, the benefits in terms of stability, security, and reproducibility are substantial. The integration of mmdebstrap provides significant performance improvements for development and testing workflows.

---

Based on upstream OSTree documentation, Fedora Atomic Desktop research, and apt-ostree development experience. Adapted for Ubuntu/Debian ecosystem integration.