debian-ostree-systems-notes/apt2ostree.md

apt2ostree Project Breakdown

Project Overview

apt2ostree is a sophisticated build system developed by stb-tester for creating Debian/Ubuntu-based OSTree images. Unlike traditional tools like debootstrap or multistrap that create filesystem directories, apt2ostree directly outputs OSTree repositories, focusing on speed, reproducibility, and space efficiency.

Repository Structure

Based on the project analysis, here's the repository structure:

apt2ostree/
├── README.md                           # Project documentation
├── setup.py                           # Python package setup
├── apt2ostree/                        # Main Python package
│   ├── __init__.py                    # Package initialization
│   ├── apt.py                         # APT integration and dependency resolution
│   ├── ninja.py                       # Ninja build file generation
│   ├── ostree.py                      # OSTree operations and management
│   ├── lockfile.py                    # Lockfile generation and parsing
│   └── keyrings/                      # APT keyrings for verification
│       ├── debian/                    # Debian repository keys
│       │   ├── jessie/
│       │   ├── stretch/
│       │   └── buster/
│       └── ubuntu/                    # Ubuntu repository keys
│           ├── trusty/
│           ├── xenial/
│           ├── bionic/
│           └── focal/
├── examples/                          # Example configurations
│   ├── nginx/                         # Nginx server image example
│   │   ├── configure.py               # Build configuration script
│   │   ├── Packages                   # High-level package list
│   │   └── Packages.lock              # Generated lockfile
│   └── multistrap/                    # Multistrap compatibility
│       ├── multistrap.py              # Multistrap config parser
│       └── example.conf               # Sample multistrap config
├── tests/                             # Unit tests
│   ├── test_apt.py
│   ├── test_lockfile.py
│   └── test_ninja.py
└── scripts/                           # Utility scripts
    └── update-keyrings.py             # Keyring maintenance

How It Builds a Debian OSTree System

1. Lockfile-Based Reproducibility

The core innovation of apt2ostree is its lockfile system, inspired by modern package managers like Cargo and npm:

Package Definition Process

# High-level packages (Packages file)
nginx
systemd
openssh-server

# Dependency resolution creates lockfile (Packages.lock)
Package: nginx
Version: 1.18.0-6ubuntu14.4
SHA256: 8a3b2f4c5d6e7f8a9b0c1d2e3f4g5h6i7j8k9l0m1n2o3p4q5r6s7t8u9v0w1x2y3z
Depends: libc6 (>= 2.27), libpcre3, libssl1.1
...

Package: libc6
Version: 2.31-0ubuntu9.9
SHA256: 9b8c7d6e5f4a3b2c1d0e9f8g7h6i5j4k3l2m1n0o9p8q7r6s5t4u3v2w1x0y9z8a
...

Benefits of Lockfiles

• Reproducible Builds: Exact package versions recorded in source control
• Security Tracking: Package updates visible in git history
• Rollback Capability: Can rebuild any historical state
• CI Integration: Automated lockfile updates via CI/CD pipelines

2. Two-Stage Build Process

apt2ostree implements a two-stage approach similar to multistrap:

Stage 1: Package Unpacking (Fast)

# Downloads and unpacks .deb files directly to OSTree
# Output: ostree ref deb/$lockfile_name/unpacked
ninja                                  # Build unpacked image

Stage 2: Package Configuration (Slower)

# Runs dpkg --configure -a in chroot environment
# Output: ostree ref deb/$lockfile_name/configured
ninja stage2                           # Configure packages

3. Ninja Build System Integration

apt2ostree generates ninja build files for parallel, incremental builds:

Build File Generation

# configure.py example
import apt2ostree

# Define package list
packages = ['nginx', 'systemd', 'openssh-server']

# Generate ninja build rules
builder = apt2ostree.NinjaBuilder()
builder.add_image(
    name='nginx-server',
    packages=packages,
    suite='focal',
    arch='amd64'
)
builder.write('build.ninja')

Ninja Build Rules

# Generated build.ninja
rule download_deb
  command = wget -O $out $url
  description = Download $name

rule unpack_deb
  command = apt2ostree unpack $in $out
  description = Unpack $name

rule create_ostree_ref
  command = ostree commit --tree=dir=$in -b $branch
  description = Commit $branch

build nginx.deb: download_deb
  url = http://archive.ubuntu.com/ubuntu/pool/main/n/nginx/nginx_1.18.0-6ubuntu14.4_amd64.deb
  name = nginx

build nginx-unpacked: unpack_deb nginx.deb
build ostree-ref: create_ostree_ref nginx-unpacked
  branch = deb/nginx-server/unpacked

4. Space and Speed Optimizations

Deduplication Strategy

Build Process:
Package A ──┐
            ├── OSTree Repository (shared storage)
Package B ──┤    ├── Blob 1 (shared)
            │    ├── Blob 2 (unique to A)
Package C ──┘    └── Blob 3 (unique to B/C)

Traditional Approach:
Package A ── Copy ── Image A (full copy)
Package B ── Copy ── Image B (full copy) 
Package C ── Copy ── Image C (full copy)

Performance Benefits

• Incremental Downloads: Only download changed packages
• Parallel Processing: Multiple operations via ninja parallelism
• Smart Caching: OSTree deduplication at file level
• No Redundant Work: Ninja tracks what needs rebuilding

5. APT Integration Without Host Dependencies

apt2ostree cleverly avoids requiring apt/dpkg on the build host:

Dependency Resolution

# Uses aptly for dependency resolution (Go binary)
# No apt/dpkg required on build host
import apt2ostree.apt

resolver = apt.DependencyResolver()
resolver.add_repository('http://archive.ubuntu.com/ubuntu', 'focal')
resolver.add_packages(['nginx', 'systemd'])
resolved = resolver.resolve()  # Complete dependency tree

Package Processing

# Direct .deb manipulation without dpkg
for package in resolved:
    deb_file = download(package.url)
    contents = extract_deb_contents(deb_file)
    ostree_commit(contents, package.ref)

6. Lockfile Management System

The lockfile system provides sophisticated package version control:

Lockfile Format (Debian Package Index)

Package: nginx
Version: 1.18.0-6ubuntu14.4
Architecture: amd64
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Installed-Size: 3588
Depends: libc6 (>= 2.27), libpcre3, libssl1.1, lsb-base (>= 3.0-6)
Filename: pool/main/n/nginx/nginx_1.18.0-6ubuntu14.4_amd64.deb
Size: 1432568
MD5sum: 4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d
SHA1: 9e8d7c6b5a4f3e2d1c0b9a8f7e6d5c4b3a2f1e0d
SHA256: 8a3b2f4c5d6e7f8a9b0c1d2e3f4g5h6i7j8k9l0m1n2o3p4q5r6s7t8u9v0w1x2y3z

Update Workflow

# Update lockfiles (equivalent to apt update + apt list --upgradable)
ninja update-lockfiles

# Review changes in git
git diff Packages.lock

# Commit updates
git add Packages.lock
git commit -m "Security updates: nginx 1.18.0-6ubuntu14.3 → 1.18.0-6ubuntu14.4"

# Build updated image
ninja

Architecture Deep Dive

1. Python Library Design

apt2ostree is primarily a Python library with several key modules:

Core Modules

# apt.py - APT repository handling
class Repository:
    def __init__(self, url, suite, components):
        self.packages_index = download_packages_index(url, suite)
    
    def resolve_dependencies(self, packages):
        return dependency_resolver.solve(packages, self.packages_index)

# ostree.py - OSTree operations  
class OSTreeRepo:
    def commit_packages(self, packages, ref_name):
        for pkg in packages:
            self.import_package_contents(pkg)
        self.create_ref(ref_name)

# ninja.py - Build file generation
class NinjaBuilder:
    def add_deb_download_rule(self, package):
        self.rules.append(f"build {package.filename}: download_deb")
    
    def add_ostree_commit_rule(self, ref_name, dependencies):
        self.rules.append(f"build {ref_name}: ostree_commit {dependencies}")

2. Build Orchestration Flow

graph TD
    A[configure.py] --> B[Load Package Lists]
    B --> C[Generate/Update Lockfiles]
    C --> D[Create ninja build.ninja]
    D --> E[ninja Execution]
    E --> F[Download .deb Files]
    F --> G[Extract Package Contents]
    G --> H[Commit to OSTree]
    H --> I[Create OSTree Refs]
    I --> J[Optional: Configure Packages]
    J --> K[Final OSTree Image]
    
    subgraph "Parallel Operations"
    F1[Download Package 1]
    F2[Download Package 2]  
    F3[Download Package 3]
    G1[Extract Package 1]
    G2[Extract Package 2]
    G3[Extract Package 3]
    end
    
    F --> F1
    F --> F2
    F --> F3
    F1 --> G1
    F2 --> G2
    F3 --> G3

3. Configuration and Usage Examples

Basic Configuration (configure.py)

#!/usr/bin/env python3
import apt2ostree

# Define base system
packages = [
    'systemd',
    'openssh-server',
    'nginx',
    'curl'
]

# Create build configuration
config = apt2ostree.Config()
config.add_repository('http://archive.ubuntu.com/ubuntu', 'focal', 'main universe')
config.add_image(
    name='nginx-server',
    packages=packages,
    architecture='amd64'
)

# Generate ninja build file
builder = apt2ostree.NinjaBuilder(config)
builder.write_build_file('build.ninja')

Usage Workflow

# Initialize OSTree repository
mkdir -p _build/ostree
ostree init --mode=bare-user --repo=_build/ostree

# Generate build configuration
./configure.py

# Build image (stage 1 - unpacked)
ninja

# Update packages to latest versions
ninja update-lockfiles

# Commit changes to source control
git add Packages.lock
git commit -m "Package updates $(date --iso-8601)"

# Build configured image (stage 2)
ninja stage2

# Create versioned branch
ostree commit --tree=ref=deb/images/Packages.lock/configured \
              --repo=_build/ostree \
              -b production/v1.0 \
              -s "Production release v1.0"

Key Technical Innovations

1. Direct OSTree Integration

Unlike deb-ostree-builder which creates filesystem trees then commits them, apt2ostree works directly with OSTree:

# Traditional approach (deb-ostree-builder)
filesystem_tree = create_filesystem()
populate_with_packages(filesystem_tree)
ostree_commit(filesystem_tree)

# apt2ostree approach
for package in packages:
    deb_contents = extract_deb(package)
    ostree_import_package(deb_contents)  # Direct import
ostree_create_ref(package_refs)          # Combine refs

2. Ninja-Based Parallelism

The use of ninja enables sophisticated build optimizations:

Dependency Tracking

# Ninja tracks file dependencies automatically
build nginx-configured: stage2 nginx-unpacked systemd-unpacked
build final-image: combine nginx-configured base-system

Parallel Execution

• Multiple package downloads simultaneously
• Concurrent package extraction operations
• Parallel OSTree operations where possible
• Intelligent rebuild detection

3. Advanced Caching Strategy

Multi-Level Caching

Cache Levels:
1. Downloaded .deb files (shared across builds)
2. Extracted package contents in OSTree (deduplicated)
3. Built image refs (incremental updates only)
4. Ninja build state (tracks what needs rebuilding)

Space Efficiency

• No Duplicate Downloads: Same .deb used across multiple images
• OSTree Deduplication: File-level sharing between images
• Incremental Builds: Only rebuild changed components

Comparison with deb-ostree-builder

Feature	apt2ostree	deb-ostree-builder
Speed	Very fast (ninja, caching)	Moderate (bash, sequential)
Reproducibility	Lockfiles in git	Build-time package resolution
Dependencies	Python, ninja, aptly	Bash, debootstrap, multistrap
Complexity	Medium (Python library)	Low (bash scripts)
Customization	Python API, ninja rules	Hook scripts
Scope	Image building focus	Full system management

Real-World Usage Example

Embedded System Deployment

For stb-tester's HDMI testing devices:

# configure.py for test device
packages = [
    'systemd',
    'openssh-server', 
    'gstreamer1.0-tools',
    'v4l-utils',
    'python3-opencv',
    'stb-tester'  # Custom package
]

# Multiple image variants
config.add_image('test-runner', packages + ['nodejs', 'chromium'])
config.add_image('production', packages)
config.add_image('debug', packages + ['gdb', 'strace', 'tcpdump'])

CI/CD Integration

# .github/workflows/update-packages.yml
name: Update Package Lockfiles
on:
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Update lockfiles
        run: |
          ./configure.py
          ninja update-lockfiles
          git add *.lock
          git commit -m "Automated package updates $(date --iso-8601)"
          git push origin update-lockfiles

Technical Advantages

1. Build System Philosophy

apt2ostree embodies several modern build system principles:

• Declarative Configuration: Specify what you want, not how to build it
• Incremental Builds: Only rebuild what changed
• Parallelism: Maximum utilization of build resources
• Reproducibility: Deterministic outputs from same inputs
• Caching: Aggressive reuse of previous work

2. OSTree-Native Design

Instead of adapting traditional packaging to OSTree, apt2ostree was designed from the ground up for OSTree:

# Native OSTree operations
ostree_repo.import_package_contents(deb_file)  # Direct import
ostree_repo.create_ref_from_packages(packages) # Efficient combining
ostree_repo.commit_with_metadata(metadata)     # Rich commit info

3. Modern Package Management

The lockfile approach brings modern package management concepts to system building:

• Explicit Updates: Security updates are deliberate, tracked operations
• Dependency Transparency: Complete dependency tree in source control
• Version Pinning: Exact versions specified, no surprise updates
• Change Tracking: Git history shows exactly what packages changed

Use Cases and Applications

1. Embedded Systems

• IoT devices requiring reliable updates
• Set-top boxes and media players
• Industrial control systems

2. Cloud Infrastructure

• Container base images with OSTree benefits
• Immutable server deployments
• Edge computing nodes

3. Development Environments

• Reproducible development containers
• Testing multiple package combinations
• Continuous integration environments

Limitations and Considerations

Current Limitations

• Single Repository Support: Cannot combine packages from multiple repos (e.g., main + updates)
• Stage 2 Performance: Package configuration stage needs optimization
• Root Privileges: Stage 2 currently requires sudo access
• Foreign Architecture: Limited testing with qemu binfmt-misc

Future Improvements

• Multi-repository Support: High priority missing feature
• Rootless Stage 2: Using fakeroot or user namespaces
• Performance Optimization: overlayfs and rofiles-fuse integration
• Enhanced Caching: More aggressive deduplication strategies

apt2ostree represents a modern approach to system image building that combines the reliability of OSTree with the speed and reproducibility requirements of contemporary development workflows. Its focus on lockfile-based reproducibility and ninja-powered parallelism makes it particularly well-suited for CI/CD environments and embedded system development where predictable, fast builds are essential.