debian-ostree-systems-notes/endlessos.md

# The Complete EndlessOS Build and Deployment Lifecycle

## 1. Introduction to the EndlessOS Immutable Paradigm

### 1.1. EndlessOS: Purpose, Target Audience, and Core Philosophy

EndlessOS is a Linux distribution engineered with a distinct purpose: to provide a simple, secure, and robust computing experience for first-time computer users, particularly in emerging markets where internet access may be intermittent or nonexistent. The design philosophy of EndlessOS diverges sharply from traditional Linux distributions that prioritize user-level flexibility. Instead, the developers have elected to shift complexity away from the end-user by creating a system that is fundamentally stable and self-maintaining.

This is achieved by delivering a comprehensive, "offline-ready" operating system that includes over 100 pre-installed applications and educational content, eliminating the need for extensive internet downloads or complex package management. The user interface, a heavily modified GNOME desktop environment, is designed to be as intuitive as a smartphone or tablet, further reducing the learning curve for its target audience.

### 1.2. The Immutable System Model and Filesystem Layout

The foundational architectural decision that underpins EndlessOS is the use of an immutable, read-only root filesystem. In contrast to traditional Linux systems, where the entire filesystem is writable by default, the core of EndlessOS, including the root directory (/) and the /usr directory, is mounted as read-only. This design choice is a direct response to the common problem in traditional package-based systems where a user or an application can inadvertently corrupt the operating system by modifying or deleting critical files.

A strict separation is maintained between the immutable core and the writable portions of the system. Directories that require write access for user data, configuration, or runtime state are located on a separate, writable partition, typically /var. For example, the user's home directory is accessed via a symlink from /home to /var/home, and the system's runtime state is stored in /var/usrlocal. The /etc directory, which holds system-wide configuration files, is also writable.

This dual-partition approach ensures that the core operating system files are always protected from accidental or malicious modification, while still allowing for necessary user and system state persistence.

This immutable model is also the enabler for "atomic updates," which are a cornerstone of the EndlessOS update system. Instead of live-patching the active system, which can leave it in a broken or inconsistent state if the update is interrupted, a new version of the entire OS is downloaded in the background as a complete, new filesystem tree. Upon the next reboot, the system simply "switches" to the new tree. This all-or-nothing approach guarantees that a system is always in a pristine, functional state. The immutability of the base OS also eliminates the need for a dedicated recovery partition, as there is nothing for the user to "reset" to a pristine state; the OS is already in that state at all times.

**Table 1: EndlessOS Filesystem Layout**

| Traditional Linux Directory | EndlessOS Status | EndlessOS Path | Explanation |
|----------------------------|------------------|----------------|-------------|
| / | Read-only | / | The immutable, versioned core of the operating system. |
| /usr | Read-only | /usr | Contains immutable binaries, libraries, and other shared data. |
| /etc | Writable | /etc | Stores system-wide configuration files. |
| /var | Writable | /var | Contains variable data such as logs, caches, and runtime state. |
| /home | Writable (via symlink) | /var/home | User home directories and personal files. |
| /opt | Writable (via symlink) | /var/opt | Optional application data. |
| /usr/local | Writable (via symlink) | /var/usrlocal | Writable storage for locally installed software. |

### 1.3. The Role of OSTree

At the heart of EndlessOS is libostree, a system often described as "Git for operating system binaries". OSTree provides a method for versioning and deploying entire filesystem trees. Its core operational model is analogous to a version control system: it uses a content-addressed object store to store files based on their cryptographic checksums.

The genius of this approach lies in its efficiency. When a new version of the OS is committed, only the changed files are stored in the repository. The new "deployment" is then created by hard-linking to the existing, unchanged files in the object store. This dramatically reduces the space required for storing multiple OS versions and makes updates remarkably fast, as only the deltas need to be downloaded and written to disk. The use of hardlinks also ensures that the checked-out files are immutable, as modifying a hardlink would corrupt the original file in the object store, a core concept that protects the integrity of the system.

Furthermore, OSTree is not just a repository for binaries; it is also a powerful deployment tool that manages the bootloader configuration. It creates and manages the bootloader entries for each OS version, allowing the system to switch seamlessly between different deployments on reboot. This built-in bootloader management is critical to the atomic update and rollback functionality of EndlessOS.

## 2. The Build Process: From Debian Packages to an OSTree Commit

### 2.1. The deb-ostree-builder Toolchain

The deb-ostree-builder is the primary tool used to create the base EndlessOS ostree commits. It is a powerful yet simple wrapper written in Bash, designed to be transparent and easy to understand by calling lower-level tools directly, such as debootstrap, which creates a minimal Debian system. The tool's design favors a clear, modular, and scripted approach to the build process. A build host running a Debian-compatible distribution is required to execute the tool, and it must be run as the root user to perform the necessary system-level operations.

### 2.2. The Treefile: The OS Blueprint

A Treefile is a fundamental component of the EndlessOS build system. It is a JSON-formatted file that serves as the definitive blueprint for an ostree commit. The use of a Treefile centralizes all build parameters in a single, human-readable document, which can be version-controlled in Git, enabling a reproducible and auditable build process.

The Treefile contains crucial information that guides the deb-ostree-builder through the build process. Key parameters include:

- **ref**: The name of the ostree branch for the content.
- **repos**: An array of strings specifying the names of yum repositories to be used.
- **etc-group-members**: A list of Unix groups that need to be stored in the writable /etc/group file. This is a subtle but important detail that demonstrates the need to manually manage the immutable boundary.
- **install-langs**: Sets the languages to be installed, influencing the RPM transaction flags.
- **boot-location**: Defines the location for kernel data.
- **mutate-os-release**: A string that injects the ostree version into the VERSION and PRETTY_NAME fields of the os-release file, providing a consistent way to query the OS version.

This centralized configuration provides a single source of truth for the entire OS composition, which can be automated and managed by a Continuous Integration (CI) system like Jenkins to ensure consistency across builds.

### 2.3. The Staged Build Flow

The deb-ostree-builder pipeline is executed in a series of logical stages, with each stage performing a specific set of tasks to build the final OSTree commit.

- **check_update stage**: The initial stage, which determines if a new build is necessary. It compares the current build's facts against a cache of previous build facts. If no changes are detected, the process exits early, saving time and resources.

- **OS stage**: This is the core build stage. It begins by creating a clean directory and then uses debootstrap to populate it with the specified Debian packages. The hooks directory contains customization scripts that are run at various points during this stage. Scripts ending in .chroot are particularly significant as they are executed within a chroot environment, allowing them to install packages and perform configuration changes as if they were running on the final system.

- **ostree stage**: This stage takes the filesystem tree generated in the OS stage and commits it to a local ostree repository. The ostree commit command is used with a critical flag, --bootable, which injects the standard metadata required for a bootable Linux filesystem tree.

- **publish stage**: The final stage, which takes the local ostree repository and publishes it to a remote server, making the new OS commit available for deployment and updates.

**Table 2: deb-ostree-builder Stage Summary**

| Stage Name | Primary Tools Used | Key Actions | Relevant Hooks/Input |
|-------------|-------------------|-------------|---------------------|
| check_update | Bash scripts | Compares current and cached build facts to determine if a build is needed. | hooks/check_update scripts, cached build facts. |
| OS | debootstrap, Bash scripts | Creates a clean directory, bootstraps a Debian environment, installs packages, and applies configuration. | hooks/os scripts, hooks/os .chroot scripts. |
| ostree | ostree commit | Commits the prepared filesystem tree from the OS stage to a local OSTree repository. | Treefile, ostree commit options like --bootable. |
| publish | rsync, http | Publishes the local OSTree repository to a remote server for public access. | hooks/publish scripts. |

## 3. Creating a Bootable Image: The eos-image-builder

### 3.1. Purpose and Workflow

The eos-image-builder (EIB) serves as the final tool in the build pipeline, responsible for taking an ostree commit and transforming it into a complete, bootable disk image (e.g., ISO, IMG) that can be used for installation. This process is distinct from the OS commit itself, which only creates a versioned filesystem tree. The EIB adds the necessary components to make that tree bootable and installable on a physical disk.

### 3.2. Integration of Core Components

The "image stage" within the EIB's pipeline is where all the separate elements are brought together into a cohesive product. The process begins with a checkout of the latest OSTree commit into a new directory, which will serve as the foundation for the final image.

Following this, the EIB adds crucial content and configurations. The bootloader is configured to correctly boot the checked-out ostree deployment. This configuration is a critical step, as the bootloader must be aware of the ostree structure to find the correct kernel and root filesystem. Additional content, such as pre-installed Flatpak applications, is also integrated into the image at this point. This is done using configuration settings from the flatpak-remote-* sections, reinforcing the "pure ostree" model where applications are delivered separately from the base OS. The EIB, like the deb-ostree-builder, favors using low-level tools directly, such as calling mke2fs to create the filesystem, ensuring a deep understanding of the build process and providing maximum flexibility and speed.

## 4. Disk Partitioning and Bootloader Configuration

### 4.1. The EndlessOS Partitioning Scheme

EndlessOS uses a specific, non-negotiable disk partitioning layout that is integral to its design. The standard installer is built to reformat the entire hard drive, and manual partitioning or installation into a different partition is officially unsupported. This is because the OS requires a very specific setup to function correctly. The disk image is a GPT disk image with three partitions:

- An EFI System Partition (ESP) for UEFI-based systems, which contains the EFI bootloaders.
- A BIOS boot partition for legacy BIOS systems.
- The main root partition, which houses both the live ostree deployment and the ostree repository itself.

The installer's exclusive control over the disk ensures that this precise layout is created and that the first-boot scripts, such as the one that resizes the root partition to fill the available disk space, can function as expected.

### 4.2. The Custom GRUB Bootloader

A major reason for the partitioning and installation inflexibility is that EndlessOS relies on a customized version of GRUB that is not interchangeable with the standard bootloaders used by other Linux distributions. This custom GRUB contains a specific module that implements the BootLoaderSpec (BLS), an essential component for booting an ostree-based system.

The standard practice in other Linux distributions is to regenerate a static grub.cfg file whenever a new kernel is installed. In contrast, EndlessOS's GRUB is designed to dynamically read BLS entries from the main partition. This tight integration ensures that the bootloader always has the correct, up-to-date information about the available OS deployments, including the specific kernel and initramfs to use. This design choice enhances stability and update reliability but, as a consequence, makes official multi-booting with other Linuxes unsupported.

### 4.3. BootLoaderSpec and OSTree Integration

The BootLoaderSpec (BLS) is the intricate mechanism that links the bootloader to the ostree repository. Instead of a static configuration file, the BLS entries, located on the main partition, contain all the necessary details for booting the OS.

**Dynamic Configuration**: Each BLS entry specifies the precise kernel, initramfs image, and kernel parameters needed for a particular ostree deployment. These details are generated and updated automatically by the ostree tooling when a new OS version is deployed.

**Seamless Updates**: When a new ostree commit is downloaded, new BLS entries are created to point to the new deployment. The bootloader is then automatically configured to use these new entries as the default. This means that a system update doesn't require a manual bootloader configuration change; the system simply reboots into the new deployment as if it were a new, complete installation. This is a powerful and reliable design that eliminates a common point of failure for end-users.

### 4.4. The Boot Chain

The entire boot process is a carefully choreographed sequence of events that relies on this tight integration.

1. **Firmware Handoff**: Upon power-on, the computer's UEFI or BIOS firmware loads the GRUB bootloader from the EFI or BIOS boot partition.

2. **GRUB Execution**: The custom EndlessOS GRUB loads and, using its internal ostree-aware logic, reads the BLS entries from the root partition.

3. **Deployment Selection**: The GRUB menu presents the available ostree deployments based on these BLS entries. The default entry points to the latest OS version, but the user can select an older, retained version if needed.

4. **Kernel and Initramfs**: GRUB loads the specified kernel and initramfs from the ostree deployment.

5. **Root Filesystem Mounting**: The initramfs uses ostree-prepare-root to mount the correct ostree deployment as the root filesystem, setting up the necessary bind mounts for writable directories like /var.

6. **System Initialization**: The kernel and initramfs hand off control to systemd, which completes the boot process into the read-only OS.

## 5. Post-Installation Services and the Update Mechanism

### 5.1. The eos-boot-helper and Its Services

The eos-boot-helper project contains a collection of scripts and dracut modules that perform critical, boot-time services that cannot be handled by the core ostree model. While the provided documentation is limited, a review of the repository's files reveals its purpose: to manage the boot-related "glue" that makes the OS function correctly on various hardware and in different states.

The dracut directory, for example, contains modules that assist in creating the initramfs. Commit messages within the repository indicate that these modules are responsible for updating EFI load option partition UUIDs and managing partition-related operations. The repository also includes scripts for critical first-boot tasks, such as eos-firstboot, which runs a series of one-time setup procedures, and eos-repartition-mbr, which handles repartitioning logic for older systems. Other services handle tasks such as migrating user profiles to Flatpak, enabling the BFQ I/O scheduler, and managing hardware-specific configurations for devices like NVIDIA GPUs.

### 5.2. eos-firstboot: The First-Run Experience

The eos-firstboot script is a crucial part of the post-installation lifecycle. It performs non-atomic, one-time operations that are necessary to finalize the system for the end-user. A key task of this script is to automatically resize the root partition to consume the remaining free space on the hard disk. This is an essential step, as the base image is a fixed size, and the first-boot process customizes it to fit the user's hardware.

The script also handles post-upgrade tasks, such as the one-time splitting of the Flatpak repository from the main ostree repository, a process that was introduced in Endless OS 4. This multi-stage approach, where a reproducible image is finalized by a first-boot script, is a sophisticated engineering pattern that allows the EndlessOS team to produce a single image that is both universal and adaptable to diverse hardware and installation sizes.

### 5.3. The Atomic Update Process via eos-updater

The eos-updater is a background daemon that manages the entire post-installation update cycle. It is orchestrated by a systemd timer, which periodically triggers the update process automatically.

The update mechanism is a pure implementation of the atomic ostree model. When a new OS commit is available on the remote repository, the eos-updater downloads the new content. It then commits this new filesystem tree to the local ostree repository and prepares a new deployment. This process occurs in the background without interrupting the user's current session. Once the new deployment is prepared, the user interface prompts the user to reboot the system. Upon reboot, the system seamlessly switches to the new, updated deployment, which is a complete, fresh version of the OS.

### 5.4. Rollback Functionality

A significant benefit of the atomic update model is the built-in ability to roll back to a previous, known-good OS version. Because the ostree system retains the previous deployment, a user can access the bootloader menu (GRUB) on startup and select the older OS version to boot. This provides a robust fail-safe mechanism, allowing a user to recover from a bad update without needing to perform a full reinstallation.

## 6. Architectural Insights and Comparative Analysis

### 6.1. The "Pure OSTree" Approach vs. Package Layering

EndlessOS employs a "pure ostree" model, which means it does not support "package layering" on top of the base OS. This is a fundamental divergence from systems like Fedora Silverblue. While the underlying Debian packaging tools (apt-get, dpkg) are present for internal build processes, they are non-functional for users who wish to install new packages. All new applications must be installed as Flatpak containers, which are self-contained and run independently of the core OS.

This design choice has clear implications. On one hand, it guarantees system integrity and robustness, as there is no risk of a package conflict or a broken update leaving the base OS in an unbootable state. The update mechanism is also simpler and more predictable. On the other hand, it severely limits flexibility for advanced users who rely on command-line tools or specialized software not available on Flatpak. This limitation is a deliberate trade-off that aligns with the target audience of non-technical users who prioritize stability and security over customizability.

### 6.2. EndlessOS vs. Fedora Silverblue

EndlessOS and Fedora Silverblue are two of the most prominent examples of immutable Linux distributions using ostree and Flatpak. While they share core technologies, their architectures and design philosophies are distinct.

- **Base Distribution**: EndlessOS is a derivative of Debian, leveraging a mature and stable ecosystem. Fedora Silverblue is built on Fedora, a cutting-edge, community-driven distribution.

- **Package Management**: The most significant difference is in package management. EndlessOS follows a "pure ostree" approach, with no package layering, which prioritizes a simple, unbreakable system. Silverblue, by contrast, uses rpm-ostree, a "hybrid ostree" system that allows users to layer RPM packages on top of the base image. This provides more flexibility for developers and power users.

- **Target Audience**: EndlessOS is designed for non-technical users in emerging markets and focuses on a streamlined, appliance-like experience. Fedora Silverblue is aimed at a more technical audience, offering a robust but customizable developer workstation.

- **User Interface**: EndlessOS uses a heavily customized GNOME Shell called "EOS Shell" that emulates a smartphone or tablet interface. Silverblue uses a vanilla GNOME Shell.

### 6.3. EndlessOS vs. SteamOS

SteamOS, another immutable Linux distribution, offers an interesting point of comparison. While EndlessOS is built for general-purpose computing in challenging environments, and SteamOS is built for gaming, both share the ostree and immutable OS paradigm. The key similarity is that both systems are designed to be resilient and appliance-like, with a read-only root filesystem and updates delivered atomically. However, their implementation details differ, as SteamOS is based on Arch Linux and is tailored for gaming hardware, including special window managers like Gamescope and support for specific AMD/NVIDIA components.

**Table 3: Immutable OS Comparison**

| Feature | EndlessOS | Fedora Silverblue | SteamOS |
|---------|------------|-------------------|---------|
| Base OS | Debian | Fedora/Red Hat | Arch Linux |
| Update System | OSTree + eos-updater | rpm-ostree | OSTree + custom updater |
| Package Management | Pure OSTree (Flatpak-only) | Hybrid OSTree (RPM layering) | Hybrid OSTree (Pacman/Flatpak) |
| Primary Target Audience | Non-technical users, emerging markets | Developers, enthusiasts | PC gamers, handheld gaming |

## 7. Conclusion and Recommendations

### 7.1. Summary of the EndlessOS Build Process

The build and deployment of EndlessOS is a sophisticated, multi-stage pipeline designed to produce a highly stable and secure immutable operating system. The process begins with deb-ostree-builder using a Treefile blueprint to create a root filesystem from Debian packages. This tree is then committed to an OSTree repository. The eos-image-builder then takes this commit, partitions a disk image, and installs a custom, OSTree-aware GRUB bootloader. Finally, eos-updater handles background, atomic updates, while eos-boot-helper manages critical one-time tasks on first boot, completing the full lifecycle.

### 7.2. Recommendations for Developers and System Integrators

For developers, system administrators, and integrators seeking to understand or build upon the EndlessOS architecture, several key recommendations are apparent from a detailed analysis of its build system and post-installation services.

First, the entire system is centered on the build pipeline. The OS is not a collection of packages but a single, version-controlled artifact. Therefore, mastery of the deb-ostree-builder and eos-image-builder tools and their respective customization hooks is paramount. Customization and configuration are handled at the build stage, and changes must be committed and redeployed to be applied.

Second, the system's design priorities necessitate a different approach to application and system management. The immutability model offers unparalleled stability but sacrifices traditional flexibility. Developers should be prepared to package all applications as Flatpaks, as this is the only supported method of application delivery. Attempts to bypass this by layering packages, as might be done on other immutable systems, are not supported and risk breaking the core system's update mechanism.

Finally, the eos-boot-helper is a critical component for understanding how the OS transitions from a generic, built image to a functional installation. This is where non-atomic operations, such as partition resizing and hardware-specific configurations, are handled. An in-depth study of these scripts reveals how the build team has addressed the challenges of a one-size-fits-all image. By offloading these one-time tasks to a dedicated boot-time service, the core ostree build can remain pristine and atomic, a sophisticated engineering solution that underpins the reliability of the entire system.