apt-ostree/.notes/rpm-ostree/how-commands-work/explainer.md

What Actually Goes On Behind the Scenes with rpm-ostree and Fedora Atomic Desktops

rpm-ostree is a sophisticated hybrid system that brings together the best of traditional package management (RPMs) with image-based, atomic updates (OSTree), forming the core of Fedora Atomic Desktops (like Silverblue, Kinoite, Bazzite, and Bluefin). It provides a unique approach to operating system management built around an immutable core filesystem, enhancing stability, security, and reproducibility.

The Core Idea: Immutability, Version Control, and Layering

1. The Immutable Root Filesystem (/ and /usr):

   • Read-Only Core: The core operating system (primarily /usr and, by extension, the entire / hierarchy) is fundamentally read-only. This is a cornerstone of the atomic desktop, preventing accidental or malicious modifications to the base system and ensuring that the OS always matches a known, tested state.
   • Version Control (Git for OS Binaries): rpm-ostree functions much like Git. It manages an OSTree repository (/ostree/repo) that stores different versions (commits) of the entire OS image. Each commit is a complete snapshot of the root filesystem.
   • Transactional Updates: Updates are applied as whole, transactional units. rpm-ostree downloads and prepares a new version in the background, creating a new combined image (an OSTree commit). You then reboot into this new image, with the previous version still available for instant rollback if needed.

2. Writable Directories and User Data:

   • Separate Writable Areas: While the core OS is immutable, directories like /etc and /var remain writable to store configurations and runtime state.
     • /etc is designed for configuration. On upgrade, rpm-ostree performs a 3-way merge, combining your local changes with upstream defaults and changes, ensuring configuration persistence. Defaults should ideally be in /usr/etc.
     • /var stores variable data and is largely shared across deployments. Its initial content is copied on first boot and is not overwritten on subsequent upgrades, ensuring persistence of logs, caches, and other variable data.
   • User Data Preservation: User data is stored separately (typically in /var/home, which is symlinked to /home by default), ensuring that rollbacks or system re-installations don't impact personal files or settings.
   • Symlinks for Compatibility: To maintain compatibility with traditional Linux software expectations, Fedora Atomic Desktops utilize symlinks to redirect some expected writable locations from the read-only /usr into /var. For instance:
     • /opt becomes /var/opt
     • /usr/local becomes /var/usrlocal
     • /srv and /root are also typically symlinked or bind-mounted into /var.
     • /mnt and /tmp are standard temporary or mount points and are handled appropriately (e.g., tmpfs for /tmp).

Behind the Scenes: The Key Components & Their Orchestration

1. libostree (The Foundational Layer for Immutability)

• Content-Addressable Storage: libostree manages a Git-like repository (/ostree/repo) on your system. It breaks down the filesystem tree into individual files and directories, hashes them, and stores them in an object store.
• Deduplication: Identical files (even across different OS versions or applications, or layered packages) are stored only once via hardlinks. This is a core feature that saves significant disk space and allows for fast deployment of new OS versions.
• Deployments: When rpm-ostree deploys a new OS version, libostree creates a new "deployment" in /ostree/deploy/. This deployment is primarily a collection of hardlinks pointing back to the objects in /ostree/repo, effectively being a thin overlay. This makes deployments very fast and space-efficient.
• Atomic Switching: libostree handles the atomic switch between deployments by updating bootloader entries (like GRUB) to point to the new root filesystem.

2. libdnf / RPM (The Package Intelligence)

• Hybrid Nature: rpm-ostree integrates deeply with RPM package management using libdnf (the library that powers DNF). It leverages libdnf to understand RPM metadata, resolve dependencies, and manage package conflicts.
• Server-Side Composes (Primary Model for Base OS): For base OS images (e.g., Fedora CoreOS, Fedora Silverblue), the entire OS is pre-built on a build server. This server uses DNF/RPM to resolve all dependencies and assemble the complete filesystem tree from RPMs, then commits it to an OSTree repository. Clients then pull these pre-composed, immutable OSTree commits.
• Client-Side Layering (The "Layered" Packages): This is where rpm-ostree provides flexibility for users. When you run rpm-ostree install <package>, it doesn't directly install the RPM onto your running system. Instead:
  1. rpm-ostree downloads the specified RPM(s) and their dependencies using libdnf.
  2. It takes your current deployed OSTree commit as the base.
  3. It then uses libdnf to simulate an RPM transaction to layer these packages on top of the base OSTree commit. This involves unpacking the RPMs and carefully integrating their contents into a new filesystem tree.
  4. The result is a new, modified filesystem tree, which rpm-ostree then commits to your local OSTree repository.
  5. This new commit becomes your "pending deployment," which will be used on the next reboot. This is generally recommended only when absolutely necessary, as it can potentially complicate updates and rollbacks compared to using Flatpaks.

3. bubblewrap (Sandboxing Script Execution)

• Safe Script Execution: RPMs contain %post, %pre, %posttrans scripts. In rpm-ostree, these scripts cannot directly modify the read-only /usr.
• Isolated Environment: rpm-ostree uses bubblewrap (a lightweight, unprivileged sandboxing tool) to run these RPM scriptlets in a confined environment.
• Simulated Root: Inside the bubblewrap sandbox, a temporary, mutable root filesystem is created that simulates the target environment. The script runs within this isolated space. Any changes the script would make to /usr are instead captured by rpm-ostree, which then integrates these changes into the new OSTree commit.
• Security: This sandboxing prevents potentially malicious or misbehaving package scripts from directly affecting the running host system or the OSTree repository itself during the layering process.

4. podman (Container Integration - The Preferred Application Management)

• rpm-ostree and Fedora Atomic Desktops strongly encourage the use of containerized applications, particularly Flatpaks, for most software installations. podman is the underlying container engine that powers many of these container-related workflows.
• Why?
  • Keeps /usr Clean: By using containers, applications run in isolated environments and are not part of the base filesystem. This keeps your base OS lean and closer to the upstream image, significantly improving reproducibility and upgrade reliability.
  • Isolation: Applications in containers are isolated from the host OS, preventing dependency conflicts and ensuring stability.
  • Portability: Containers provide consistent environments across different machines.
• Development Environments (Toolbox/Devcontainers): For developers, Fedora Atomic Desktops promote using containerized development environments like Toolbox or devcontainers (often leveraging podman). This keeps development tools and dependencies isolated from the host system, avoiding conflicts and ensuring a clean environment.
• Integration Points: While rpm-ostree doesn't directly use podman for its core OS update mechanism, operating systems built with rpm-ostree come with podman pre-installed and promote its use for user-installed software and development. Projects like Bluefin leverage bootc which in turn uses OCI container features (often via podman-like runtimes) to compose and build the OS image itself.

5. OverlayFS (Used for Live Overlays / Debugging, Not Core Persistence)

• Primary Model: It's important to clarify that rpm-ostree's core mechanism primarily uses hardlinks into the OSTree repository to construct and deploy filesystem trees, not OverlayFS for the persistent, base filesystem.
• Specific OverlayFS Uses: OverlayFS is utilized by rpm-ostree in specific scenarios, primarily for temporary, "live" changes or development:
  • rpm-ostree install --apply-live: This experimental feature attempts to apply a layered package immediately to the running system without a reboot. It achieves this by creating a transient overlayfs mount over /usr (or other parts of the root filesystem), allowing the changes to appear live. This is not the primary transactional mechanism for persistent changes and is generally discouraged for long-term use.
  • rpm-ostree usroverlay: This command explicitly creates a writable overlayfs over /usr for debugging or temporary modifications. These changes are not persistent across reboots.
  • /var for Mutable Data: While the core OSTree /usr is read-only, /var is designed to be writable and is where most mutable system state resides. This is typically managed by standard filesystems like Btrfs or XFS, not generally OverlayFS within the core rpm-ostree design for /var.

Filesystem Choices

While the immutable nature is central, the underlying filesystem used for / and /var/home can vary:

• Btrfs: Fedora Workstation and its Atomic spins often use Btrfs as the default, offering features like transparent compression and snapshots. Btrfs subvolumes are utilized to separate the root and home directories.
• Other options: Manual partitioning also supports LVM, standard partitions, or XFS.

The Update Workflow (rpm-ostree upgrade):

1. Fetch: rpm-ostree contacts configured remote OSTree repositories to fetch new base OS versions (and potentially RPM repositories if client-side layering is involved).
2. Resolve & Compose (Client-Side for Layering): If you have layered packages, rpm-ostree uses libdnf to calculate the new desired state (new base OS + your layered packages). This involves downloading necessary RPMs and resolving dependencies.
3. Synthesize OSTree Commit: rpm-ostree takes the new base OS OSTree commit and "applies" the RPM changes (using sandboxed bubblewrap for script execution) to produce a new, complete filesystem tree. This new tree is then committed to the local /ostree/repo.
4. Stage Deployment: libostree is used to create a new deployment entry, essentially a new set of hardlinks to the objects in the local repository.
5. Bootloader Update: The system's bootloader (GRUB) is updated to include an entry for the new deployment, usually making it the default boot target. The previous deployment is always kept as a fallback.
6. Reboot: The user reboots the system to apply the update atomically.
7. Atomic Swap: On reboot, the bootloader directs the kernel to the new root filesystem. If the boot is successful (often confirmed by health checks via greenboot), the new deployment becomes the active one. If not, the system can automatically fall back to the previous known-good state, ensuring a highly reliable update process.

In conclusion, Fedora Atomic Desktops and their derivatives offer a robust and reliable computing experience built around an immutable core. The filesystem structure and the way applications are handled are distinct from traditional Linux distributions, with a strong emphasis on containerization and a clear separation between the base operating system and user data. While this approach may require some adjustment for users accustomed to traditional package management, the benefits in terms of stability, security, and reproducibility are substantial.