added apt2ostree.md
This commit is contained in:
parent
7fcfe927c9
commit
ebe5e82af9
1 changed files with 520 additions and 0 deletions
520
apt2ostree.md
Normal file
520
apt2ostree.md
Normal file
|
|
@ -0,0 +1,520 @@
|
|||
# apt2ostree Project Breakdown
|
||||
|
||||
## Project Overview
|
||||
|
||||
apt2ostree is a sophisticated build system developed by stb-tester for creating Debian/Ubuntu-based OSTree images. Unlike traditional tools like debootstrap or multistrap that create filesystem directories, apt2ostree directly outputs OSTree repositories, focusing on speed, reproducibility, and space efficiency.
|
||||
|
||||
## Repository Structure
|
||||
|
||||
Based on the project analysis, here's the repository structure:
|
||||
|
||||
```
|
||||
apt2ostree/
|
||||
├── README.md # Project documentation
|
||||
├── setup.py # Python package setup
|
||||
├── apt2ostree/ # Main Python package
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── apt.py # APT integration and dependency resolution
|
||||
│ ├── ninja.py # Ninja build file generation
|
||||
│ ├── ostree.py # OSTree operations and management
|
||||
│ ├── lockfile.py # Lockfile generation and parsing
|
||||
│ └── keyrings/ # APT keyrings for verification
|
||||
│ ├── debian/ # Debian repository keys
|
||||
│ │ ├── jessie/
|
||||
│ │ ├── stretch/
|
||||
│ │ └── buster/
|
||||
│ └── ubuntu/ # Ubuntu repository keys
|
||||
│ ├── trusty/
|
||||
│ ├── xenial/
|
||||
│ ├── bionic/
|
||||
│ └── focal/
|
||||
├── examples/ # Example configurations
|
||||
│ ├── nginx/ # Nginx server image example
|
||||
│ │ ├── configure.py # Build configuration script
|
||||
│ │ ├── Packages # High-level package list
|
||||
│ │ └── Packages.lock # Generated lockfile
|
||||
│ └── multistrap/ # Multistrap compatibility
|
||||
│ ├── multistrap.py # Multistrap config parser
|
||||
│ └── example.conf # Sample multistrap config
|
||||
├── tests/ # Unit tests
|
||||
│ ├── test_apt.py
|
||||
│ ├── test_lockfile.py
|
||||
│ └── test_ninja.py
|
||||
└── scripts/ # Utility scripts
|
||||
└── update-keyrings.py # Keyring maintenance
|
||||
```
|
||||
|
||||
## How It Builds a Debian OSTree System
|
||||
|
||||
### 1. **Lockfile-Based Reproducibility**
|
||||
|
||||
The core innovation of apt2ostree is its lockfile system, inspired by modern package managers like Cargo and npm:
|
||||
|
||||
#### Package Definition Process
|
||||
```python
|
||||
# High-level packages (Packages file)
|
||||
nginx
|
||||
systemd
|
||||
openssh-server
|
||||
|
||||
# Dependency resolution creates lockfile (Packages.lock)
|
||||
Package: nginx
|
||||
Version: 1.18.0-6ubuntu14.4
|
||||
SHA256: 8a3b2f4c5d6e7f8a9b0c1d2e3f4g5h6i7j8k9l0m1n2o3p4q5r6s7t8u9v0w1x2y3z
|
||||
Depends: libc6 (>= 2.27), libpcre3, libssl1.1
|
||||
...
|
||||
|
||||
Package: libc6
|
||||
Version: 2.31-0ubuntu9.9
|
||||
SHA256: 9b8c7d6e5f4a3b2c1d0e9f8g7h6i5j4k3l2m1n0o9p8q7r6s5t4u3v2w1x0y9z8a
|
||||
...
|
||||
```
|
||||
|
||||
#### Benefits of Lockfiles
|
||||
- **Reproducible Builds**: Exact package versions recorded in source control
|
||||
- **Security Tracking**: Package updates visible in git history
|
||||
- **Rollback Capability**: Can rebuild any historical state
|
||||
- **CI Integration**: Automated lockfile updates via CI/CD pipelines
|
||||
|
||||
### 2. **Two-Stage Build Process**
|
||||
|
||||
apt2ostree implements a two-stage approach similar to multistrap:
|
||||
|
||||
#### Stage 1: Package Unpacking (Fast)
|
||||
```bash
|
||||
# Downloads and unpacks .deb files directly to OSTree
|
||||
# Output: ostree ref deb/$lockfile_name/unpacked
|
||||
ninja # Build unpacked image
|
||||
```
|
||||
|
||||
#### Stage 2: Package Configuration (Slower)
|
||||
```bash
|
||||
# Runs dpkg --configure -a in chroot environment
|
||||
# Output: ostree ref deb/$lockfile_name/configured
|
||||
ninja stage2 # Configure packages
|
||||
```
|
||||
|
||||
### 3. **Ninja Build System Integration**
|
||||
|
||||
apt2ostree generates ninja build files for parallel, incremental builds:
|
||||
|
||||
#### Build File Generation
|
||||
```python
|
||||
# configure.py example
|
||||
import apt2ostree
|
||||
|
||||
# Define package list
|
||||
packages = ['nginx', 'systemd', 'openssh-server']
|
||||
|
||||
# Generate ninja build rules
|
||||
builder = apt2ostree.NinjaBuilder()
|
||||
builder.add_image(
|
||||
name='nginx-server',
|
||||
packages=packages,
|
||||
suite='focal',
|
||||
arch='amd64'
|
||||
)
|
||||
builder.write('build.ninja')
|
||||
```
|
||||
|
||||
#### Ninja Build Rules
|
||||
```ninja
|
||||
# Generated build.ninja
|
||||
rule download_deb
|
||||
command = wget -O $out $url
|
||||
description = Download $name
|
||||
|
||||
rule unpack_deb
|
||||
command = apt2ostree unpack $in $out
|
||||
description = Unpack $name
|
||||
|
||||
rule create_ostree_ref
|
||||
command = ostree commit --tree=dir=$in -b $branch
|
||||
description = Commit $branch
|
||||
|
||||
build nginx.deb: download_deb
|
||||
url = http://archive.ubuntu.com/ubuntu/pool/main/n/nginx/nginx_1.18.0-6ubuntu14.4_amd64.deb
|
||||
name = nginx
|
||||
|
||||
build nginx-unpacked: unpack_deb nginx.deb
|
||||
build ostree-ref: create_ostree_ref nginx-unpacked
|
||||
branch = deb/nginx-server/unpacked
|
||||
```
|
||||
|
||||
### 4. **Space and Speed Optimizations**
|
||||
|
||||
#### Deduplication Strategy
|
||||
```
|
||||
Build Process:
|
||||
Package A ──┐
|
||||
├── OSTree Repository (shared storage)
|
||||
Package B ──┤ ├── Blob 1 (shared)
|
||||
│ ├── Blob 2 (unique to A)
|
||||
Package C ──┘ └── Blob 3 (unique to B/C)
|
||||
|
||||
Traditional Approach:
|
||||
Package A ── Copy ── Image A (full copy)
|
||||
Package B ── Copy ── Image B (full copy)
|
||||
Package C ── Copy ── Image C (full copy)
|
||||
```
|
||||
|
||||
#### Performance Benefits
|
||||
- **Incremental Downloads**: Only download changed packages
|
||||
- **Parallel Processing**: Multiple operations via ninja parallelism
|
||||
- **Smart Caching**: OSTree deduplication at file level
|
||||
- **No Redundant Work**: Ninja tracks what needs rebuilding
|
||||
|
||||
### 5. **APT Integration Without Host Dependencies**
|
||||
|
||||
apt2ostree cleverly avoids requiring apt/dpkg on the build host:
|
||||
|
||||
#### Dependency Resolution
|
||||
```python
|
||||
# Uses aptly for dependency resolution (Go binary)
|
||||
# No apt/dpkg required on build host
|
||||
import apt2ostree.apt
|
||||
|
||||
resolver = apt.DependencyResolver()
|
||||
resolver.add_repository('http://archive.ubuntu.com/ubuntu', 'focal')
|
||||
resolver.add_packages(['nginx', 'systemd'])
|
||||
resolved = resolver.resolve() # Complete dependency tree
|
||||
```
|
||||
|
||||
#### Package Processing
|
||||
```python
|
||||
# Direct .deb manipulation without dpkg
|
||||
for package in resolved:
|
||||
deb_file = download(package.url)
|
||||
contents = extract_deb_contents(deb_file)
|
||||
ostree_commit(contents, package.ref)
|
||||
```
|
||||
|
||||
### 6. **Lockfile Management System**
|
||||
|
||||
The lockfile system provides sophisticated package version control:
|
||||
|
||||
#### Lockfile Format (Debian Package Index)
|
||||
```
|
||||
Package: nginx
|
||||
Version: 1.18.0-6ubuntu14.4
|
||||
Architecture: amd64
|
||||
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
|
||||
Installed-Size: 3588
|
||||
Depends: libc6 (>= 2.27), libpcre3, libssl1.1, lsb-base (>= 3.0-6)
|
||||
Filename: pool/main/n/nginx/nginx_1.18.0-6ubuntu14.4_amd64.deb
|
||||
Size: 1432568
|
||||
MD5sum: 4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d
|
||||
SHA1: 9e8d7c6b5a4f3e2d1c0b9a8f7e6d5c4b3a2f1e0d
|
||||
SHA256: 8a3b2f4c5d6e7f8a9b0c1d2e3f4g5h6i7j8k9l0m1n2o3p4q5r6s7t8u9v0w1x2y3z
|
||||
```
|
||||
|
||||
#### Update Workflow
|
||||
```bash
|
||||
# Update lockfiles (equivalent to apt update + apt list --upgradable)
|
||||
ninja update-lockfiles
|
||||
|
||||
# Review changes in git
|
||||
git diff Packages.lock
|
||||
|
||||
# Commit updates
|
||||
git add Packages.lock
|
||||
git commit -m "Security updates: nginx 1.18.0-6ubuntu14.3 → 1.18.0-6ubuntu14.4"
|
||||
|
||||
# Build updated image
|
||||
ninja
|
||||
```
|
||||
|
||||
## Architecture Deep Dive
|
||||
|
||||
### 1. **Python Library Design**
|
||||
|
||||
apt2ostree is primarily a Python library with several key modules:
|
||||
|
||||
#### Core Modules
|
||||
```python
|
||||
# apt.py - APT repository handling
|
||||
class Repository:
|
||||
def __init__(self, url, suite, components):
|
||||
self.packages_index = download_packages_index(url, suite)
|
||||
|
||||
def resolve_dependencies(self, packages):
|
||||
return dependency_resolver.solve(packages, self.packages_index)
|
||||
|
||||
# ostree.py - OSTree operations
|
||||
class OSTreeRepo:
|
||||
def commit_packages(self, packages, ref_name):
|
||||
for pkg in packages:
|
||||
self.import_package_contents(pkg)
|
||||
self.create_ref(ref_name)
|
||||
|
||||
# ninja.py - Build file generation
|
||||
class NinjaBuilder:
|
||||
def add_deb_download_rule(self, package):
|
||||
self.rules.append(f"build {package.filename}: download_deb")
|
||||
|
||||
def add_ostree_commit_rule(self, ref_name, dependencies):
|
||||
self.rules.append(f"build {ref_name}: ostree_commit {dependencies}")
|
||||
```
|
||||
|
||||
### 2. **Build Orchestration Flow**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[configure.py] --> B[Load Package Lists]
|
||||
B --> C[Generate/Update Lockfiles]
|
||||
C --> D[Create ninja build.ninja]
|
||||
D --> E[ninja Execution]
|
||||
E --> F[Download .deb Files]
|
||||
F --> G[Extract Package Contents]
|
||||
G --> H[Commit to OSTree]
|
||||
H --> I[Create OSTree Refs]
|
||||
I --> J[Optional: Configure Packages]
|
||||
J --> K[Final OSTree Image]
|
||||
|
||||
subgraph "Parallel Operations"
|
||||
F1[Download Package 1]
|
||||
F2[Download Package 2]
|
||||
F3[Download Package 3]
|
||||
G1[Extract Package 1]
|
||||
G2[Extract Package 2]
|
||||
G3[Extract Package 3]
|
||||
end
|
||||
|
||||
F --> F1
|
||||
F --> F2
|
||||
F --> F3
|
||||
F1 --> G1
|
||||
F2 --> G2
|
||||
F3 --> G3
|
||||
```
|
||||
|
||||
### 3. **Configuration and Usage Examples**
|
||||
|
||||
#### Basic Configuration (configure.py)
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
import apt2ostree
|
||||
|
||||
# Define base system
|
||||
packages = [
|
||||
'systemd',
|
||||
'openssh-server',
|
||||
'nginx',
|
||||
'curl'
|
||||
]
|
||||
|
||||
# Create build configuration
|
||||
config = apt2ostree.Config()
|
||||
config.add_repository('http://archive.ubuntu.com/ubuntu', 'focal', 'main universe')
|
||||
config.add_image(
|
||||
name='nginx-server',
|
||||
packages=packages,
|
||||
architecture='amd64'
|
||||
)
|
||||
|
||||
# Generate ninja build file
|
||||
builder = apt2ostree.NinjaBuilder(config)
|
||||
builder.write_build_file('build.ninja')
|
||||
```
|
||||
|
||||
#### Usage Workflow
|
||||
```bash
|
||||
# Initialize OSTree repository
|
||||
mkdir -p _build/ostree
|
||||
ostree init --mode=bare-user --repo=_build/ostree
|
||||
|
||||
# Generate build configuration
|
||||
./configure.py
|
||||
|
||||
# Build image (stage 1 - unpacked)
|
||||
ninja
|
||||
|
||||
# Update packages to latest versions
|
||||
ninja update-lockfiles
|
||||
|
||||
# Commit changes to source control
|
||||
git add Packages.lock
|
||||
git commit -m "Package updates $(date --iso-8601)"
|
||||
|
||||
# Build configured image (stage 2)
|
||||
ninja stage2
|
||||
|
||||
# Create versioned branch
|
||||
ostree commit --tree=ref=deb/images/Packages.lock/configured \
|
||||
--repo=_build/ostree \
|
||||
-b production/v1.0 \
|
||||
-s "Production release v1.0"
|
||||
```
|
||||
|
||||
## Key Technical Innovations
|
||||
|
||||
### 1. **Direct OSTree Integration**
|
||||
|
||||
Unlike deb-ostree-builder which creates filesystem trees then commits them, apt2ostree works directly with OSTree:
|
||||
|
||||
```python
|
||||
# Traditional approach (deb-ostree-builder)
|
||||
filesystem_tree = create_filesystem()
|
||||
populate_with_packages(filesystem_tree)
|
||||
ostree_commit(filesystem_tree)
|
||||
|
||||
# apt2ostree approach
|
||||
for package in packages:
|
||||
deb_contents = extract_deb(package)
|
||||
ostree_import_package(deb_contents) # Direct import
|
||||
ostree_create_ref(package_refs) # Combine refs
|
||||
```
|
||||
|
||||
### 2. **Ninja-Based Parallelism**
|
||||
|
||||
The use of ninja enables sophisticated build optimizations:
|
||||
|
||||
#### Dependency Tracking
|
||||
```ninja
|
||||
# Ninja tracks file dependencies automatically
|
||||
build nginx-configured: stage2 nginx-unpacked systemd-unpacked
|
||||
build final-image: combine nginx-configured base-system
|
||||
```
|
||||
|
||||
#### Parallel Execution
|
||||
- Multiple package downloads simultaneously
|
||||
- Concurrent package extraction operations
|
||||
- Parallel OSTree operations where possible
|
||||
- Intelligent rebuild detection
|
||||
|
||||
### 3. **Advanced Caching Strategy**
|
||||
|
||||
#### Multi-Level Caching
|
||||
```
|
||||
Cache Levels:
|
||||
1. Downloaded .deb files (shared across builds)
|
||||
2. Extracted package contents in OSTree (deduplicated)
|
||||
3. Built image refs (incremental updates only)
|
||||
4. Ninja build state (tracks what needs rebuilding)
|
||||
```
|
||||
|
||||
#### Space Efficiency
|
||||
- **No Duplicate Downloads**: Same .deb used across multiple images
|
||||
- **OSTree Deduplication**: File-level sharing between images
|
||||
- **Incremental Builds**: Only rebuild changed components
|
||||
|
||||
## Comparison with deb-ostree-builder
|
||||
|
||||
| Feature | apt2ostree | deb-ostree-builder |
|
||||
|---------|------------|-------------------|
|
||||
| **Speed** | Very fast (ninja, caching) | Moderate (bash, sequential) |
|
||||
| **Reproducibility** | Lockfiles in git | Build-time package resolution |
|
||||
| **Dependencies** | Python, ninja, aptly | Bash, debootstrap, multistrap |
|
||||
| **Complexity** | Medium (Python library) | Low (bash scripts) |
|
||||
| **Customization** | Python API, ninja rules | Hook scripts |
|
||||
| **Scope** | Image building focus | Full system management |
|
||||
|
||||
## Real-World Usage Example
|
||||
|
||||
### Embedded System Deployment
|
||||
|
||||
For stb-tester's HDMI testing devices:
|
||||
|
||||
```python
|
||||
# configure.py for test device
|
||||
packages = [
|
||||
'systemd',
|
||||
'openssh-server',
|
||||
'gstreamer1.0-tools',
|
||||
'v4l-utils',
|
||||
'python3-opencv',
|
||||
'stb-tester' # Custom package
|
||||
]
|
||||
|
||||
# Multiple image variants
|
||||
config.add_image('test-runner', packages + ['nodejs', 'chromium'])
|
||||
config.add_image('production', packages)
|
||||
config.add_image('debug', packages + ['gdb', 'strace', 'tcpdump'])
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
```yaml
|
||||
# .github/workflows/update-packages.yml
|
||||
name: Update Package Lockfiles
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 2 * * *' # Daily at 2 AM
|
||||
|
||||
jobs:
|
||||
update:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
- name: Update lockfiles
|
||||
run: |
|
||||
./configure.py
|
||||
ninja update-lockfiles
|
||||
git add *.lock
|
||||
git commit -m "Automated package updates $(date --iso-8601)"
|
||||
git push origin update-lockfiles
|
||||
```
|
||||
|
||||
## Technical Advantages
|
||||
|
||||
### 1. **Build System Philosophy**
|
||||
|
||||
apt2ostree embodies several modern build system principles:
|
||||
|
||||
- **Declarative Configuration**: Specify what you want, not how to build it
|
||||
- **Incremental Builds**: Only rebuild what changed
|
||||
- **Parallelism**: Maximum utilization of build resources
|
||||
- **Reproducibility**: Deterministic outputs from same inputs
|
||||
- **Caching**: Aggressive reuse of previous work
|
||||
|
||||
### 2. **OSTree-Native Design**
|
||||
|
||||
Instead of adapting traditional packaging to OSTree, apt2ostree was designed from the ground up for OSTree:
|
||||
|
||||
```python
|
||||
# Native OSTree operations
|
||||
ostree_repo.import_package_contents(deb_file) # Direct import
|
||||
ostree_repo.create_ref_from_packages(packages) # Efficient combining
|
||||
ostree_repo.commit_with_metadata(metadata) # Rich commit info
|
||||
```
|
||||
|
||||
### 3. **Modern Package Management**
|
||||
|
||||
The lockfile approach brings modern package management concepts to system building:
|
||||
|
||||
- **Explicit Updates**: Security updates are deliberate, tracked operations
|
||||
- **Dependency Transparency**: Complete dependency tree in source control
|
||||
- **Version Pinning**: Exact versions specified, no surprise updates
|
||||
- **Change Tracking**: Git history shows exactly what packages changed
|
||||
|
||||
## Use Cases and Applications
|
||||
|
||||
### 1. **Embedded Systems**
|
||||
- IoT devices requiring reliable updates
|
||||
- Set-top boxes and media players
|
||||
- Industrial control systems
|
||||
|
||||
### 2. **Cloud Infrastructure**
|
||||
- Container base images with OSTree benefits
|
||||
- Immutable server deployments
|
||||
- Edge computing nodes
|
||||
|
||||
### 3. **Development Environments**
|
||||
- Reproducible development containers
|
||||
- Testing multiple package combinations
|
||||
- Continuous integration environments
|
||||
|
||||
## Limitations and Considerations
|
||||
|
||||
### Current Limitations
|
||||
- **Single Repository Support**: Cannot combine packages from multiple repos (e.g., main + updates)
|
||||
- **Stage 2 Performance**: Package configuration stage needs optimization
|
||||
- **Root Privileges**: Stage 2 currently requires sudo access
|
||||
- **Foreign Architecture**: Limited testing with qemu binfmt-misc
|
||||
|
||||
### Future Improvements
|
||||
- **Multi-repository Support**: High priority missing feature
|
||||
- **Rootless Stage 2**: Using fakeroot or user namespaces
|
||||
- **Performance Optimization**: overlayfs and rofiles-fuse integration
|
||||
- **Enhanced Caching**: More aggressive deduplication strategies
|
||||
|
||||
apt2ostree represents a modern approach to system image building that combines the reliability of OSTree with the speed and reproducibility requirements of contemporary development workflows. Its focus on lockfile-based reproducibility and ninja-powered parallelism makes it particularly well-suited for CI/CD environments and embedded system development where predictable, fast builds are essential.
|
||||
Loading…
Add table
Add a link
Reference in a new issue