apt-ostree/.notes/research/ostree.md

https://ostreedev.github.io/ostree/

You've hit on the core of what makes rpm-ostree (and systems like it) unique. While rpm-ostree adds the RPM layer, OSTree itself is the underlying technology that handles the immutable filesystem and atomic updates.

Let's break down how OSTree works, covering both the client and server sides.

How OSTree Works: The "Git for Operating System Binaries"

OSTree (often referred to as libostree for the library or ostree for the CLI tool) is a system for atomically deploying filesystem trees. It's designed to manage operating system content in a highly efficient, reliable, and version-controlled manner.

Core Concepts

1. Repository (/ostree/repo): This is the central storage location for all OS versions (commits) on a system. It's content-addressable, similar to a Git repository. Files are stored once and referenced by their cryptographic hash (SHA256). This enables massive deduplication across different OS versions and even different "branches" of the OS.
   • Object Store: Inside the repository, files and metadata (like directories and their permissions) are stored as "objects," each identified by its hash.
   • Repository Modes: OSTree repositories can operate in different modes (e.g., bare for read/write access, archive for static HTTP serving).
2. Commit: A commit in OSTree is an immutable snapshot of an entire filesystem tree. It's analogous to a Git commit, but for an entire operating system. Each commit contains:
   • A unique SHA256 checksum (its ID).
   • References to the actual file and directory objects in the repository.
   • Metadata: timestamp, commit message, parent commit (for history), and other custom key-value pairs.
   • Crucially, an OSTree commit is not directly bootable on its own; it's a blueprint.
3. Ref (Branch): A "ref" (short for reference) is a symbolic pointer to a specific commit. It's like a Git branch. For example, fedora/39/x86_64/silverblue might be a ref pointing to the latest Fedora Silverblue 39 commit for x86_64. Refs make it easy to follow a specific "stream" of updates.
4. Deployment: A deployment is an actual, bootable instance of an OSTree commit on the filesystem.
   • Deployments are typically located under /ostree/deploy/$STATEROOT/$CHECKSUM.
   • They are created primarily using hardlinks back to the objects in the central /ostree/repo. This means that deploying a new OS version is extremely fast and consumes minimal additional disk space (only for the files that actually changed between commits).
   • An OSTree system always has at least one active deployment (the one currently booted) and often one or more older deployments for rollback.
   • OSTree manages the bootloader (e.g., GRUB) to point to the desired deployment.
5. Read-Only /usr: OSTree strongly promotes a read-only /usr filesystem. When a deployment is active, /usr is mounted read-only (often via a bind mount), preventing accidental or malicious changes to the core OS binaries and libraries.
6. Mutable /etc and /var: OSTree specifically excludes /etc (system configuration) and /var (variable data like logs, caches, user data) from the immutable content of a commit.
   • /etc is handled with a 3-way merge on upgrade: it merges the old deployment's /etc, the new deployment's /etc (from the commit), and any local changes the user made. This preserves user customizations.
   • /var is simply shared across deployments within the same "stateroot" (/ostree/deploy/$STATEROOT/var). OSTree does not manage /var's contents; it's up to the OS and applications to manage their data there.

Server-Side Operations (Composing and Serving Images)

The server side of OSTree is primarily concerned with creating and distributing immutable OS commits. This is typically done by distribution maintainers or system administrators building custom images.

1. Image Composition:
   • This is the process of assembling a complete operating system from its source components (e.g., RPMs in the rpm-ostree context, or .deb packages in other OSTree-based systems like Endless OS or Torizon).
   • Tools like rpm-ostree compose (or higher-level tools like Red Hat's Image Builder, CoreOS Assembler, or BlueBuild) take a manifest of desired packages/files, resolve dependencies, extract content, and build a complete filesystem tree.
2. ostree commit:
   • Once a filesystem tree is assembled, the ostree commit command is used to capture this tree and all its associated metadata (permissions, ownership, xattrs) into the OSTree repository.
   • It generates a new commit object and stores all the unique file and metadata objects.
   • This commit can be optionally signed with a GPG key for cryptographic verification by clients.
3. ostree summary:
   • To make repositories efficient for clients, servers generate a summary file. This file contains a list of all available refs (branches) and their latest commit checksums, along with information about static deltas. Clients can download this small file to quickly see what's available without having to browse the entire repository.
4. ostree static-delta (Optimization):
   • Servers can pre-calculate "static deltas" between common commits. These deltas are compressed bundles of only the changed files between two specific commits.
   • When a client requests an update, if a static delta is available for their current commit and the target commit, they can download just the delta, significantly reducing network bandwidth. If not, OSTree defaults to a "pull-everything-unique" approach.
5. Serving the Repository:
   • An OSTree repository, once composed, is typically served over HTTP(S). Because all objects are content-addressable and immutable, a simple static web server (like Nginx or Apache) is sufficient.
   • The archive mode of an OSTree repository is specifically designed for static HTTP serving.
   • Specialized tools like Pulp (for Red Hat) or custom services can also serve OSTree content, often with additional features like content synchronization and access control.

Client-Side Operations (Consuming and Managing Images)

The client side is where end-users and administrators interact with OSTree-based systems to update, manage, and rollback their operating systems.

1. Local Repository (/ostree/repo): Each OSTree client maintains its own local repository, which stores the commits relevant to its deployments.
2. ostree remote:
   • Clients configure "remotes" (similar to Git remotes) that point to server-side OSTree repositories. These configurations specify the URL, GPG verification keys, and other settings.
   • ostree remote add <name> <url>: Adds a new remote.
   • ostree remote refs <name>: Lists the available branches/refs on a remote.
3. ostree pull:
   • When a client wants to update, it uses ostree pull <remote> <refspec> (e.g., ostree pull fedora fedora/39/x86_64/silverblue).
   • This command downloads new commit objects and any unique file/metadata objects from the remote repository into the client's local /ostree/repo.
   • It leverages the summary file and static deltas for efficient, incremental downloads.
   • The pull operation is cryptographic: all downloaded content is verified against its checksum, and commits are verified against GPG signatures.
4. ostree deploy:
   • After a new commit has been pulled, ostree deploy <commit-checksum> --os=<osname> creates a new deployment on the filesystem (e.g., in /ostree/deploy/fedora-silverblue/).
   • This involves creating a directory structure and filling it with hardlinks pointing back to the objects in /ostree/repo.
   • It also handles the 3-way merge for /etc and prepares the bootloader configuration.
   • This operation happens offline; it does not affect the currently running system.
5. ostree admin (for management):
   • ostree admin deploy <refspec>: A common high-level command that combines pull and deploy to get and stage the latest commit for a ref.
   • ostree admin switch <refspec>: Changes the default deployment for the next boot.
   • ostree admin undeploy <checksum>: Removes an old, unneeded deployment.
   • ostree admin cleanup: Removes unreferenced objects from the local repository and prunes old deployments to save space.
   • ostree admin boot: Manages bootloader entries.
6. ostree rebase (Conceptual in rpm-ostree): While ostree itself has commands for direct ref manipulation, in rpm-ostree, rebase is a specific operation that switches the base OSTree image while reapplying any client-side layered packages. It involves pulling the new base and then creating a new client-side derived commit.
7. Atomic Rollback: If a new deployment causes issues, the client can use ostree admin deploy --rollback (or rpm-ostree rollback) to tell the bootloader to simply boot the previous known-good deployment. Since the old deployment is still fully present on disk, this is instantaneous and extremely reliable.

In essence, OSTree provides a robust, efficient, and secure "operating system delivery mechanism" that treats the entire OS as a versioned artifact. This allows for highly reliable updates, easy rollbacks, and efficient storage, forming the immutable foundation for systems like Fedora Silverblue, Fedora CoreOS, and others.