# The Filesystem Architecture of Modern Immutable Systems

The filesystem structure of modern immutable operating systems like Fedora Atomic Desktops and ublue-os represents a carefully engineered solution to a fundamental challenge: maintaining system stability and security through immutability while enabling normal system operations and user workflows.

This document explores the architectural foundations that enable these systems to maintain consistency, reliability, and user functionality through strategic filesystem design.

## The OSTree Foundation

Modern immutable desktop systems don't simply make `/usr` read-only—they implement a more sophisticated approach using **OSTree**, a Git-like versioning system for operating system trees.

### Core OSTree Concepts

**Commit-Based Deployments:** OSTree manages the entire root filesystem as versioned "commits." Each commit represents a complete, bootable system state, similar to how Git commits represent code snapshots.

**Content-Addressable Storage:** Files are stored as objects in a content-addressable store. Identical files across commits share the same storage, dramatically reducing space requirements and update sizes.

**Hardlinked Trees:** When deploying a new commit, OSTree creates the filesystem tree using hardlinks to existing objects where possible. Only new or changed files require additional storage. This makes updates extremely fast and efficient—there's no "wholesale filesystem replacement."

**Atomic Updates:** The entire system update is atomic. Either the new commit is successfully deployed, or the system remains on the previous commit. Partial updates that could leave the system in an inconsistent state are impossible.

## The Read-Only Base and Writable Overlays

The immutable architecture separates the system into distinct layers with different mutability characteristics.

### The Immutable Root

The **entire root filesystem (`/`)** forms the immutable base, not just `/usr`. This includes:
- `/bin`, `/sbin` (essential system binaries)
- `/lib`, `/lib64` (essential system libraries)  
- `/usr` (user programs, libraries, documentation)
- Base versions of `/etc` (system configuration templates)

During normal operation, this entire tree is effectively read-only, preventing accidental or malicious corruption of the core system.

### Strategic Writable Spaces

The system maintains functionality through carefully designed writable overlays:

**`/var`: The Primary Mutable Space**
- Mounted as a separate, writable partition
- Contains all data that changes during normal system operation
- Houses system logs (`/var/log`), caches (`/var/cache`), temporary files (`/var/tmp`)
- Stores container images and data (`/var/lib/containers`, `/var/lib/flatpak`)
- Provides isolation—runaway processes filling `/var` won't crash the core system

**`/etc`: The Configuration Challenge**
The handling of `/etc` is particularly sophisticated, involving a three-way merge strategy:
- **Base Layer:** Default configurations from the OSTree commit
- **Local Modifications:** User and system customizations preserved across updates  
- **Runtime Resolution:** During boot, tools like `rpm-ostree` merge base configurations with local changes, detecting and handling conflicts

This approach allows system updates to deploy new default configurations while preserving user customizations.

## User Data Architecture: The `/home` Solution

The treatment of user home directories illustrates the practical elegance of this architecture.

### The Traditional Challenge
User home directories are inherently mutable—users constantly create, modify, and delete files. However, `/home` traditionally exists at the filesystem root, which is now immutable.

### The Symbolic Link Approach
Most immutable systems solve this through strategic redirection:

1. **Physical Storage:** User data is physically stored in `/var/home` on the writable `/var` partition
2. **Transparent Access:** A symbolic link (`/home -> /var/home`) in the immutable root filesystem redirects all `/home` access
3. **Seamless Experience:** Users and applications interact with the familiar `/home/username` paths without awareness of the underlying redirection

This is simpler and more reliable than bind mounts for this use case, while maintaining full compatibility with existing software expectations.

## Architectural Benefits and Trade-offs

This design delivers several key advantages while making specific trade-offs:

### Advantages
- **Corruption Resistance:** The immutable base cannot be accidentally damaged
- **Reliable Updates:** Atomic commits eliminate partial update failures
- **Easy Rollbacks:** Previous commits remain available for instant rollback
- **Consistency:** All systems with the same commit are identical
- **Container Compatibility:** Perfect foundation for containerized applications

### Design Considerations
- **Complexity:** The overlay system is more complex than traditional filesystems
- **Storage Efficiency:** While OSTree is efficient, it requires understanding of its deduplication model
- **Application Compatibility:** Some software expecting traditional write access to system directories requires adaptation

## Why This Approach Over Alternatives?

Several other technologies could theoretically provide similar immutability:

**OverlayFS:** Used extensively in containers, but designed for temporary overlays rather than persistent system management.

**Btrfs Snapshots:** Excellent for filesystem-level snapshots, but lack the atomic deployment and deduplication optimizations of OSTree.

**Traditional Package Management:** Cannot provide the same level of atomic updates and rollback guarantees.

OSTree was specifically designed for immutable operating system management, providing optimizations and guarantees that general-purpose filesystem technologies cannot match.

## Container Integration

This architecture provides an ideal foundation for containerized applications:

- **Flatpak applications** store their data in `/var/lib/flatpak`, maintaining isolation from the host system
- **Container runtimes** like Podman use `/var/lib/containers` for images and persistent data
- **Application sandboxing** works seamlessly within the immutable host constraints

## Implications for System Administration

Understanding this architecture is crucial for effective system administration:

- **Customizations** must be made through proper channels (`rpm-ostree`, layered packages, or container-based applications)
- **System logs and diagnostics** are centralized in `/var/log`
- **Persistent data** should always use `/var` or proper user directories
- **Updates** are fundamentally different—they deploy new system images rather than modifying existing installations

## From Theory to Practice: Implementation Challenges

While understanding the architectural principles is essential, building a functional immutable system requires confronting significant implementation challenges that theory alone cannot address.

### Critical Implementation Gaps

**OSTree Repository Management:**
The conceptual "commits" become concrete through repository management. In practice, this means:
- Using `bootc image-compose` or `apt-ostree` commands to initialize local OSTree repositories
- Orchestrating builds through `justfile` recipes that transform Containerfile package definitions into OSTree commits
- Managing the complex dependency chain where kernel updates trigger NVIDIA driver rebuilds and full system recomposition

**Overlay Creation Reality:**
The "automatic" writable overlays require explicit implementation:
- `/var` partitioning happens during `bootc install to-disk`, but disk layout and sizing decisions are crucial
- The `/home -> /var/home` symbolic link must be explicitly created in the Containerfile with commands like `RUN ln -sf ../var/home /home`
- Boot process complexity involving initramfs generation and bootloader configuration that `bootc` handles but may require debugging

**The `/etc` Merge Challenge:**
The three-way merge strategy faces real-world complexity:
- Unlike mature `rpm-ostree` tooling on Fedora, `apt-ostree` may have less-developed `/etc` handling
- Configuration conflicts during updates require resolution strategies that may need custom scripting
- Testing merge behavior across system updates becomes a critical validation requirement

### Tooling Maturity and Ecosystem Gaps

**Debian-Specific Challenges:**
- `bootc` integration with Debian is newer than with Fedora, meaning fewer community examples and potential undocumented edge cases
- Package management integration may require custom solutions where Fedora has established patterns
- Driver and kernel module handling (crucial for the NVIDIA kmods goal) lacks the mature toolchain available in Fedora

**Build Pipeline Complexity:**
- Every kernel update triggers a complete rebuild chain: kernel → NVIDIA drivers → system image
- Version management requires clear strategies for tagging, releasing, and managing container registry artifacts
- Testing strategies must validate not just functionality but atomic updates, rollbacks, and configuration merges

### Strategic Implementation Considerations

**Build Manifest Design:**
The Containerfile becomes the authoritative system definition, requiring:
- Comprehensive package lists defining base system, desktop environment, and opinionated additions
- Explicit handling of symbolic links, directory structures, and system integration points
- Version pinning strategies that balance stability with security updates

**Operational Validation:**
Production readiness requires robust testing frameworks:
- Automated rollback testing to ensure recovery mechanisms actually work
- Configuration merge validation across update scenarios  
- Integration testing for containerized applications within the immutable host

**Release Engineering:**
Managing immutable system releases involves:
- Clear versioning schemes (e.g., `2025.2.1-1`) that track both base system and customization layers
- Container registry management for storing and distributing system images
- Update delivery mechanisms that maintain the atomic properties users depend on

## Bridging the Gap

The filesystem architecture of modern immutable systems represents both sophisticated engineering and practical implementation challenges. While the principles of OSTree commits, writable overlays, and strategic use of `/var` provide the foundation, successful implementation requires:

- Deep understanding of toolchain limitations and workarounds
- Robust build and testing automation to handle complex dependency chains
- Clear operational procedures for managing updates, rollbacks, and system customization

Building systems like Particle OS means not just understanding *why* this architecture works, but mastering *how* to implement it reliably in practice. The gap between architectural elegance and implementation reality is where most immutable system projects succeed or fail.
