debian-atomic/docs/report-on-creating-bootc-images-for-fedora.md

The Fedora Bootc Base Image Build Process: An Expert Report and Strategic Analysis

Executive Summary: A Paradigm Shift in Operating System Delivery

The provided summary of the Fedora bootc base image build process is a concise and accurate overview of a profoundly significant development in modern operating system distribution. The process is not merely an incremental update to an existing method but a fundamental architectural shift that merges the established rigor of Fedora's release engineering with the agility and accessibility of cloud-native container technologies. By producing bootable operating system images that are OCI (Open Container Initiative) compliant, the Fedora Project has created a robust and strategically important foundation for immutable, GitOps-ready systems.

This report validates the claims of the summary and provides a comprehensive, multi-layered analysis of the build process, its technical underpinnings, key distinctions from traditional methods, known limitations, and its pivotal role in the future of the Fedora ecosystem.

Contextual Foundation: The Evolution from OSTree to Bootable Containers

The genesis of the Fedora bootc project is deeply rooted in the architectural principles of immutable, atomic operating systems pioneered by projects like Fedora Silverblue and Fedora CoreOS. These systems, built on the foundation of OSTree, revolutionized how operating system updates were handled by treating the entire root filesystem as a single, version-controlled artifact. This approach ensured that updates were transactional—they either succeeded completely or reverted cleanly, leaving no intermediate, broken state. This capability provided a level of reliability and predictability that was unattainable with traditional package-by-package updates.

While the OSTree model proved its technical merit, its distribution and management tooling were distinct from the burgeoning container ecosystem. The strategic pivot to bootc represents a deliberate effort to bridge this gap, delivering the proven benefits of OSTree-based systems through the universal, familiar medium of an OCI container image. The key driver behind this evolution is to make immutable operating systems more accessible to a broader audience of developers and operations teams who are already proficient with container technologies, such as Podman or Docker.

This unified approach allows organizations to manage their entire OS stack—including the kernel, bootloader, and core user space—with the same tooling and practices used for their application containers. This alignment with container-native workflows enables a powerful, declarative GitOps model for managing fleets of systems. Instead of relying on imperative scripts and manual configuration, the desired state of a system can be codified in a manifest, built into a bootable container image, and deployed as a single, consistent unit. This drastically reduces configuration drift and simplifies management at scale, particularly for edge and IoT deployments.

The significance of this strategic direction was formally recognized with the acceptance of the bootc project into the CNCF (Cloud Native Computing Foundation) Sandbox in January 2025. This provides an "explicitly vendor-neutral home" for the technology, ensuring its long-term viability and fostering a broader, more diverse contributor base beyond its original Red Hat sponsorship. This strategic move builds profound confidence in the project's future, positioning it not as a single vendor's product but as a foundational open-source technology for the entire ecosystem.

The project effectively serves as a sophisticated bridge between two distinct engineering cultures: the traditional Linux distribution model, built on auditable package management and release engineering, and the modern cloud-native paradigm, which emphasizes immutable artifacts and GitOps workflows. The process's reliance on established Fedora infrastructure, like Koji and Pungi, while producing an OCI-compliant container image, is the very essence of this fusion. It means that an organization can consume a system with the operational simplicity of a container, while benefiting from the security, reproducibility, and rigorous quality control of a major Linux distribution's official build process. This dual-world approach simplifies a fundamental challenge for enterprises seeking to adopt cloud-native best practices without abandoning the reliability of a traditional OS stack.

Core Components of the Fedora Build Infrastructure

The Fedora bootc base image build process is deeply integrated into and relies upon the existing, mature Fedora infrastructure. This is a critical distinction, as it confirms that the resulting artifacts are official, release-grade components rather than simple, community-built containers. The process leverages three core components to achieve this level of quality, reproducibility, and security.

Fedora Release Engineering (RelEng)

Fedora Release Engineering acts as the central orchestrator for the entire build pipeline. Its responsibilities are broad and foundational, including the coordination of the pipeline, managing the authoritative package sets and composition definitions, and overseeing quality assurance and release coordination. This human and technical layer ensures that the final bootc images adhere to the high standards of the Fedora Project.

Koji Build System: The Reproducible Foundation

Koji is Fedora's primary build infrastructure for RPM packages. A core function of Koji is to provide isolated build environments using a tool called Mock. This isolation ensures that every build is reproducible, free from external influence or environmental variances that could lead to non-identical results. The auditable nature of Koji, where every build is meticulously logged and traceable, is a cornerstone of the supply chain security that underpins the Fedora bootc images.

Pungi Compose System: The Package Composer

Pungi is the sophisticated compose system that orchestrates the creation of final release deliverables, including ISOs, installer trees, and, in this case, container images. Pungi's role is to act as the "glue" that takes the package set definitions and delegates the actual work of building and composing them into the final artifact. A key aspect of Pungi's design is its ability to ensure that all artifacts in a given release, including the bootc image, are built from the exact same package set. This consistency is crucial for maintaining a clean, predictable system state. While Pungi orchestrates the process, it delegates the heavy lifting—the "actual work in an auditable way"—to Koji, ensuring the entire pipeline is transparent and secure.

The deep reliance on this established and battle-tested infrastructure confirms that the Fedora bootc base images are not ad-hoc containers but are official, release-grade artifacts. This rigorous process, which is the same one used to build Fedora Workstation or Server, provides a high degree of confidence in the provenance and integrity of the images, addressing modern supply chain security concerns from the ground up.

The Multi-Stage Fedora Bootc Build Process: A Technical Deep Dive

The creation of a Fedora bootc base image is a sophisticated, multi-stage process that leverages the core Fedora infrastructure to produce a container artifact. This process stands in stark contrast to traditional Dockerfile builds and is a testament to the project's unique "package-first" approach.

Stage 1: Package Selection and Manifest Definition

The process begins not with a FROM statement but with a declarative manifest that specifies the desired final state of the operating system. This is accomplished through two key files:

Comps Files: These are XML-based files that define logical groupings of packages, providing a high-level way to select a coherent set of software (e.g., a "workstation" or "server" group).

Treefile: A JSON-formatted file that provides the detailed specification for the rpm-ostree compose tree command. This manifest is the declarative core of the build, defining which packages to include, which to exclude, and other system-level configurations. The packages key, for example, lists the mandatory RPMs for the image. Unlike the imperative RUN dnf install commands in a traditional Dockerfile, the treefile simply declares the desired end state, and rpm-ostree handles the complex dependency resolution in a clean environment to achieve it.

Stage 2: RPM-to-OSTree Conversion

The rpm-ostree compose tree command is the engine that drives this stage. It takes the declarative treefile manifest, downloads the specified RPM packages, and installs them into a temporary, isolated environment. The output of this process is an OSTree commit—a single, cryptographically-checksummed representation of the entire filesystem tree. This commit is the foundational artifact for an immutable system, with all the necessary metadata for atomic updates and rollbacks already embedded. The process handles file deduplication and compression to optimize the storage of the resulting OSTree repository.

Stage 3: Container-Native Transformation and Optimization

Once the OSTree commit is created, it must be converted into an OCI-compliant container image. This is accomplished using commands such as rpm-ostree compose container-encapsulate or rpm-ostree compose build-chunked-oci. The core innovation here is the creation of a "chunked" image format. Instead of simply packaging the entire filesystem into a single, massive layer, the chunking process breaks the content into multiple, smaller layers. This is critically important for efficient distribution, as a small update—for example, a single patched package—only requires the client to download a small new layer and a metadata layer, rather than the entire image. This mechanism is particularly optimized for OS content, which significantly reduces bandwidth requirements for updates in deployed fleets.

A crucial nuance in this stage is the choice of output format version: --format-version=1 (the default) and --format-version=2. Version 1 creates "sparse" layers, which are minimally sized by omitting parent directory metadata. This reduces the image size but can break reproducible builds, as the implicit creation of directories results in unpredictable metadata. In contrast, version 2 explicitly defines all parent directories, increasing the layer size but ensuring bit-for-bit reproducibility of the build. The default choice of v1 prioritizes efficient distribution and backwards compatibility, which is a strategic trade-off for over-the-air updates to edge or IoT devices. For those who require the most stringent build security and reproducibility, the option to use v2 is a deliberate design choice that demonstrates an understanding of different user needs.

Stage 4: Registry Publication and Verification

The final stage involves pushing the multi-architecture images to the Quay.io container registry under the quay.io/fedora/ namespace. This process is coordinated by the Koji infrastructure to ensure builds for x86_64, aarch64, ppc64le, and s390x are all generated correctly. Automated quality assurance and security scanning are performed on the generated images to verify bootability and assess for vulnerabilities. The images are tagged with Fedora release numbers (e.g., :42) as well as rolling tags like :latest and :rawhide, providing a clear versioning scheme. Image signing and verification metadata are included, providing a robust, cryptographically-secured artifact for deployment.

Comparative Analysis: Bootable Containers vs. Traditional Builds

The Fedora bootc build process introduces several fundamental architectural distinctions when compared to traditional Dockerfile-based application builds or classic kickstart-based operating system installations. These differences are at the core of the project's value proposition.

No Base Image Dependency: A traditional Containerfile begins with a FROM statement, pulling an existing base image from a registry. This introduces a dependency on the provenance and build process of that base image. In contrast, the bootc base image is built from scratch, directly from the RPM packages. This provides complete control over the base system's composition and ensures the consistent OSTree structure from the very foundation.

Package-First Composition: The bootc process composes the entire filesystem declaratively from individual RPMs, rather than imperatively installing packages into an existing system. This leads to more efficient space utilization, better dependency resolution, and a cleaner system state without the ephemeral artifacts of an installation process.

Native OSTree Integration: The most significant distinction is that the bootc image is not simply an application container; it is a full, bootable operating system with native OSTree compatibility baked in. This includes proper handling of extended attributes, embedded metadata for atomic updates, and pre-configured bootloader integration. The container image is essentially a transport mechanism for a read-only, atomic operating system.

The following table provides a direct, technical comparison of these different workflows:

Feature	Fedora bootc Build	Traditional Dockerfile Build	Kickstart OS Installation
Primary Build Artifact	OCI Container Image	OCI Container Image	Disk Image (ISO, QCOW2, etc.)
Build Source	Declarative treefile and RPMs	Imperative Containerfile and base image	Declarative kickstart file and RPMs
Build Process	Declarative, multi-stage, package-first composition via rpm-ostree	Imperative, layered, install-on-top approach via RUN commands	Scripted, single-pass installation process
System State	Immutable, read-only root filesystem with rpm-ostree backend	Writable, layered filesystem that is ephemeral at runtime	Writable root filesystem (/)
Update Mechanism	Atomic, image-based update of the entire OS via bootc and rpm-ostree	Individual package updates via dnf/apt on a running system	Individual package updates via dnf/apt on a running system
Rollback Capability	Atomic, single-command rollback to a previous working deployment	Not natively supported; requires manual effort or tooling	Not natively supported; requires manual effort or tooling
Primary Benefit	Operational consistency, atomic rollbacks, and scalable fleet management	Application portability and environmental isolation	Broad configuration flexibility for initial provisioning

Customization and Consumption: Building on the Base Image

The Fedora bootc base images are not designed to be the final product for every user; rather, they serve as a robust, immutable foundation upon which custom systems can be built.

Derived Images and Containerfiles

Once a Fedora bootc base image is available on a registry like Quay.io, users can easily extend it using a standard Containerfile. This is a familiar workflow for anyone in the container space. A user can start with a FROM quay.io/fedora/fedora-bootc:42 statement and then add additional software, configuration, or applications using RUN dnf install, COPY, or other standard instructions. The flexibility of this approach allows for the creation of specialized, purpose-built images for specific applications or environments.

Specialized Tooling for Production

While Containerfiles are used to customize the images, a set of specialized tools is necessary for their lifecycle management and deployment.

bootc-image-builder: This tool is designed to convert a bootable container image into a physical disk image (e.g., ISO, QCOW2). This is the standard method for provisioning new physical or virtual machines from a bootc image. The tool can inject users, SSH keys, and define partition layouts.

podman-bootc: This command-line interface provides a convenient way to run a bootable container image locally in a virtual machine managed by Podman, facilitating development and testing workflows.

Variant Management

The Fedora Project builds and maintains multiple official bootc variants from shared base configurations, catering to different use cases.

Variant	Description	Primary Use Case
fedora-bootc	A minimal, server-oriented base image.	Server deployments and CI/CD runners where a minimal footprint is desired.
fedora-bootc-desktop	Includes a desktop environment and core applications.	Immutable desktop environments for developers or managed corporate workstations.
fedora-bootc-minimal	An extremely minimal base, stripped down to the essentials.	Micro-service hosts, embedded systems, and other specialized deployments where every byte matters.

Technical Challenges and Mitigation Strategies

While bootc represents a significant leap forward, it is a maturing technology with known limitations and ongoing development challenges. An expert-level understanding of the project requires a candid assessment of these issues.

Known Limitations and Day 2 Pain Points

A significant limitation is the fact that any update or package installation on a bootc system currently requires a reboot to take effect. This is a direct consequence of the atomic update model. Additionally, there is still the potential for configuration drift within the /etc directory, which remains a writable part of the system. The initial installation process can also be challenging, and it is not easy to add extra, non-root disks via a Containerfile.

Supply Chain Security and Visibility

A consequence of the declarative, image-based model is the absence of traditional dnf or rpm changelogs on a deployed system. This makes it difficult to see what packages have changed between image versions. The solution to this problem lies in a modern approach to software supply chain security: creating and comparing SBOMs (Software Bill of Materials) of the bootc images. This allows for a detailed, programmatic understanding of package version changes and dependencies.

Performance and Storage Challenges

The chunking process, while efficient for distribution, introduces its own set of performance and storage overhead for the build pipeline itself. Community discussions reveal that the rechunk process, which creates these highly optimized images, can add a significant amount of time and disk space to a build. For example, the process has been observed to add 5-6 minutes to a build and potentially require 4x the storage of the resulting image. This is a crucial detail that demonstrates a key engineering trade-off: the optimization for deployment comes at a cost to the build process.

The open discussion of these performance and storage challenges in public forums, such as GitHub issues, is a strong indicator of a healthy, transparent open-source project. This transparency provides a clear roadmap for what to expect and what the community is actively working on. It allows a technical professional to set realistic expectations for their CI/CD pipeline and to understand the specific areas where the technology is still maturing.

The following table provides a concise summary of these challenges and the proposed solutions or workarounds:

Challenge	Mitigation Strategy
Updates require a system reboot	A direct consequence of the atomic update model; ensures transactional integrity.
Configuration drift in /etc directory	Codify configuration changes using a GitOps model and apply them consistently via Containerfile or other manifests.
Initial installation can be difficult	Use bootc-image-builder for provisioning, or bootstrap from a Fedora CoreOS system.
No traditional changelog for package changes	Create and compare SBOMs (Software Bill of Materials) for each image version.
Performance and storage overhead of chunking	Be aware of build pipeline requirements and consider using --format-version=2 for reproducible builds.

Strategic Vision and Future Evolution

The Fedora bootc project is not a fleeting trend but a core component of the Fedora Project's long-term strategic vision. Its future evolution is guided by clear principles and goals.

CNCF Sandbox Acceptance and Vendor Neutrality

The acceptance of bootc into the CNCF Sandbox is a landmark event that signals the project's long-term commitment to open source principles. This move provides a vendor-neutral home, which will be essential for fostering a broader community and ensuring the project's viability and continued development beyond the significant contributions from Red Hat. This strategic move builds profound confidence in the project's future, positioning it not as a single vendor's product but as a foundational open-source technology for the entire ecosystem.

Fedora's "Upstream First" Principle

The Fedora Project operates on an "upstream first" principle, meaning innovations and improvements are contributed directly to the original source software, benefiting all users. bootc is a prime example of this model. The project serves as a proving ground for the core technologies that will eventually form the foundation for Red Hat's enterprise products, such as RHEL Image Mode. The strategic goal is for Fedora CoreOS, the community-driven upstream for RHEL CoreOS, to eventually build directly on bootc, aligning the technical stack and creating a powerful feedback loop.

Fedora Strategy 2028

The bootc project is a key pillar of Fedora's overarching "Strategy 2028". The strategy identifies "Atomic Desktops and Image Mode" as a core focus area for technical innovation, demonstrating a long-term commitment to image-based operating systems. The ambition to unify multiple base images—such as for CoreOS, IoT, and Atomic Desktops—into a single base image is a central tenet of this strategy. This long-term vision positions bootc as a foundational technology that will shape the future of OS deployment.

Conclusion: A Foundational Shift

The analysis confirms that the Fedora bootc base image build process is a sophisticated and strategically significant evolution in operating system distribution. By masterfully integrating the declarative, auditable build practices of traditional Linux release engineering with the portable, GitOps-friendly nature of OCI containers, the project delivers a powerful and robust foundation for immutable operating systems.

The process provides a high degree of operational consistency, simplifies updates and rollbacks through its atomic nature, and seamlessly integrates with a vast ecosystem of container tooling. While the technology is still maturing and faces known challenges, such as the need for reboots and performance overhead in the build pipeline, the project's transparency and open development model provide a clear path forward. The acceptance into the CNCF Sandbox and its central role in Fedora's long-term strategy underscore its importance as a foundational technology for the future of Linux.

For organizations seeking to adopt a more reliable, scalable, and manageable approach to their OS fleets, Fedora bootc presents a compelling and strategically sound solution.