particle-os/filesystem.md

The Filesystem Architecture of Modern Immutable Systems

The filesystem structure of modern immutable operating systems like Fedora Atomic Desktops and ublue-os represents a carefully engineered solution to a fundamental challenge: maintaining system stability and security through immutability while enabling normal system operations and user workflows.

This document explores the architectural foundations that enable these systems to maintain consistency, reliability, and user functionality through strategic filesystem design.

The OSTree Foundation

Modern immutable desktop systems don't simply make /usr read-only—they implement a more sophisticated approach using OSTree, a Git-like versioning system for operating system trees.

Core OSTree Concepts

Commit-Based Deployments: OSTree manages the entire root filesystem as versioned "commits." Each commit represents a complete, bootable system state, similar to how Git commits represent code snapshots.

Content-Addressable Storage: Files are stored as objects in a content-addressable store. Identical files across commits share the same storage, dramatically reducing space requirements and update sizes.

Hardlinked Trees: When deploying a new commit, OSTree creates the filesystem tree using hardlinks to existing objects where possible. Only new or changed files require additional storage. This makes updates extremely fast and efficient—there's no "wholesale filesystem replacement."

Atomic Updates: The entire system update is atomic. Either the new commit is successfully deployed, or the system remains on the previous commit. Partial updates that could leave the system in an inconsistent state are impossible.

The Read-Only Base and Writable Overlays

The immutable architecture separates the system into distinct layers with different mutability characteristics.

The Immutable Root

The entire root filesystem (/) forms the immutable base, not just /usr. This includes:

• /bin, /sbin (essential system binaries)
• /lib, /lib64 (essential system libraries)
• /usr (user programs, libraries, documentation)
• Base versions of /etc (system configuration templates)

During normal operation, this entire tree is effectively read-only, preventing accidental or malicious corruption of the core system.

Strategic Writable Spaces

The system maintains functionality through carefully designed writable overlays:

/var: The Primary Mutable Space

• Mounted as a separate, writable partition
• Contains all data that changes during normal system operation
• Houses system logs (/var/log), caches (/var/cache), temporary files (/var/tmp)
• Stores container images and data (/var/lib/containers, /var/lib/flatpak)
• Provides isolation—runaway processes filling /var won't crash the core system

/etc: The Configuration Challenge The handling of /etc is particularly sophisticated, involving a three-way merge strategy:

• Base Layer: Default configurations from the OSTree commit
• Local Modifications: User and system customizations preserved across updates
• Runtime Resolution: During boot, tools like rpm-ostree merge base configurations with local changes, detecting and handling conflicts

This approach allows system updates to deploy new default configurations while preserving user customizations.

User Data Architecture: The /home Solution

The treatment of user home directories illustrates the practical elegance of this architecture.

The Traditional Challenge

User home directories are inherently mutable—users constantly create, modify, and delete files. However, /home traditionally exists at the filesystem root, which is now immutable.

The Symbolic Link Approach

Most immutable systems solve this through strategic redirection:

1. Physical Storage: User data is physically stored in /var/home on the writable /var partition
2. Transparent Access: A symbolic link (/home -> /var/home) in the immutable root filesystem redirects all /home access
3. Seamless Experience: Users and applications interact with the familiar /home/username paths without awareness of the underlying redirection

This is simpler and more reliable than bind mounts for this use case, while maintaining full compatibility with existing software expectations.

Architectural Benefits and Trade-offs

This design delivers several key advantages while making specific trade-offs:

Advantages

• Corruption Resistance: The immutable base cannot be accidentally damaged
• Reliable Updates: Atomic commits eliminate partial update failures
• Easy Rollbacks: Previous commits remain available for instant rollback
• Consistency: All systems with the same commit are identical
• Container Compatibility: Perfect foundation for containerized applications

Design Considerations

• Complexity: The overlay system is more complex than traditional filesystems
• Storage Efficiency: While OSTree is efficient, it requires understanding of its deduplication model
• Application Compatibility: Some software expecting traditional write access to system directories requires adaptation

Why This Approach Over Alternatives?

Several other technologies could theoretically provide similar immutability:

OverlayFS: Used extensively in containers, but designed for temporary overlays rather than persistent system management.

Btrfs Snapshots: Excellent for filesystem-level snapshots, but lack the atomic deployment and deduplication optimizations of OSTree.

Traditional Package Management: Cannot provide the same level of atomic updates and rollback guarantees.

OSTree was specifically designed for immutable operating system management, providing optimizations and guarantees that general-purpose filesystem technologies cannot match.

Container Integration

This architecture provides an ideal foundation for containerized applications:

• Flatpak applications store their data in /var/lib/flatpak, maintaining isolation from the host system
• Container runtimes like Podman use /var/lib/containers for images and persistent data
• Application sandboxing works seamlessly within the immutable host constraints

Implications for System Administration

Understanding this architecture is crucial for effective system administration:

• Customizations must be made through proper channels (rpm-ostree, layered packages, or container-based applications)
• System logs and diagnostics are centralized in /var/log
• Persistent data should always use /var or proper user directories
• Updates are fundamentally different—they deploy new system images rather than modifying existing installations

From Theory to Practice: Implementation Challenges

While understanding the architectural principles is essential, building a functional immutable system requires confronting significant implementation challenges that theory alone cannot address.

Critical Implementation Gaps

OSTree Repository Management: The conceptual "commits" become concrete through repository management. In practice, this means:

• Using bootc image-compose or apt-ostree commands to initialize local OSTree repositories
• Orchestrating builds through justfile recipes that transform Containerfile package definitions into OSTree commits
• Managing the complex dependency chain where kernel updates trigger NVIDIA driver rebuilds and full system recomposition

Overlay Creation Reality: The "automatic" writable overlays require explicit implementation:

• /var partitioning happens during bootc install to-disk, but disk layout and sizing decisions are crucial
• The /home -> /var/home symbolic link must be explicitly created in the Containerfile with commands like RUN ln -sf ../var/home /home
• Boot process complexity involving initramfs generation and bootloader configuration that bootc handles but may require debugging

The /etc Merge Challenge: The three-way merge strategy faces real-world complexity:

• Unlike mature rpm-ostree tooling on Fedora, apt-ostree may have less-developed /etc handling
• Configuration conflicts during updates require resolution strategies that may need custom scripting
• Testing merge behavior across system updates becomes a critical validation requirement

Tooling Maturity and Ecosystem Gaps

Debian-Specific Challenges:

• bootc integration with Debian is newer than with Fedora, meaning fewer community examples and potential undocumented edge cases
• Package management integration may require custom solutions where Fedora has established patterns
• Driver and kernel module handling (crucial for the NVIDIA kmods goal) lacks the mature toolchain available in Fedora

Build Pipeline Complexity:

• Every kernel update triggers a complete rebuild chain: kernel → NVIDIA drivers → system image
• Version management requires clear strategies for tagging, releasing, and managing container registry artifacts
• Testing strategies must validate not just functionality but atomic updates, rollbacks, and configuration merges

Strategic Implementation Considerations

Build Manifest Design: The Containerfile becomes the authoritative system definition, requiring:

• Comprehensive package lists defining base system, desktop environment, and opinionated additions
• Explicit handling of symbolic links, directory structures, and system integration points
• Version pinning strategies that balance stability with security updates

Operational Validation: Production readiness requires robust testing frameworks:

• Automated rollback testing to ensure recovery mechanisms actually work
• Configuration merge validation across update scenarios
• Integration testing for containerized applications within the immutable host

Release Engineering: Managing immutable system releases involves:

• Clear versioning schemes (e.g., 2025.2.1-1) that track both base system and customization layers
• Container registry management for storing and distributing system images
• Update delivery mechanisms that maintain the atomic properties users depend on

Bridging the Gap

The filesystem architecture of modern immutable systems represents both sophisticated engineering and practical implementation challenges. While the principles of OSTree commits, writable overlays, and strategic use of /var provide the foundation, successful implementation requires:

• Deep understanding of toolchain limitations and workarounds
• Robust build and testing automation to handle complex dependency chains
• Clear operational procedures for managing updates, rollbacks, and system customization

Building systems like Particle OS means not just understanding why this architecture works, but mastering how to implement it reliably in practice. The gap between architectural elegance and implementation reality is where most immutable system projects succeed or fail.