deb-bootupd/deb-bootupd.md

# Debian Bootupd Fork Plan

## Project Overview
**Goal**: Create a Debian-compatible version of bootupd to make particle-os (Debian-based ublue-os) bootable.

**Context**: 
- **Proof-of-concept**: Test if we can create an immutable Debian using ublue-os tools
- Original bootupd is Red Hat/Fedora-centric with RPM dependencies
- Need to adapt for **Debian OSTree immutable systems** (not traditional Debian)
- Target: Debian Trixie slim container image converted to bootc format
- **Critical Insight**: This is a hybrid system - Debian packages within immutable OSTree structure
- **Success metric**: Can we boot a Debian-based immutable system?

## Key Understanding: OSTree vs Traditional Debian

### **Traditional Debian (Mutable)**
- Package-based updates via `apt`/`dpkg`
- Mutable filesystem with writable `/usr`
- Kernel in `/boot/vmlinuz-$kver`
- GRUB configured via `grub-mkconfig`/`update-grub`

### **Particle-OS (Immutable OSTree)**
- Image-based atomic updates via bootc
- Immutable `/usr` (read-only core OS)
- Kernel embedded in `/usr/lib/modules/$kver/vmlinuz`
- GRUB configured to point to OSTree deployment paths
- Configuration in `/etc` (writable, three-way merge)
- State in `/var` (shared across deployments)
- OSTree object store in `/ostree`

## Critical Context: How bootc-image-builder Uses bootupd

### **bootc-image-builder Uses bootupd as a Rust Crate Packaged into an RPM**

This is a fundamental insight that shapes our entire implementation strategy:

#### **1. Build-Time Integration (Not Runtime)**
```bash
# During image build, bootc-image-builder:
1. Installs the rust-bootupd RPM package
2. RPM package contains the compiled Rust binary
3. Binary gets placed in /usr/libexec/bootupd
4. bootupctl symlink created in /usr/bin/ (multicall binary pattern)
5. Systemd services and GRUB configs installed
6. Final bootc image contains the compiled bootupd binary
```

#### **2. Package-Based Distribution**
From the Fedora RPM spec file, bootupd is distributed as:
- **`rust-bootupd`** - The RPM package containing the compiled Rust binary
- **Source**: The RPM build process compiles the Rust crate into a binary
- **Result**: A pre-compiled binary package, not source code

The bootc-image-builder installs the pre-built RPM package, which contains the compiled Rust binary.

#### **3. Why This Matters for deb-bootupd**
This is exactly what we need to replicate for Debian:

```bash
# debian-bootc-image-builder workflow:
1. Install deb-bootupd .deb package during image build
2. Package manager automatically places bootupd binary in /usr/libexec/bootupd and creates bootupctl symlink in /usr/bin/
3. Systemd services and GRUB configs installed
4. Final Debian bootc image contains deb-bootupd
```

#### **4. Key Differences from Rust Crate Usage**

| Aspect | Rust Crate | Fedora Package | Debian Package |
|--------|------------|----------------|----------------|
| **Build Process** | `cargo install` | `dnf install rust-bootupd` | `apt install deb-bootupd` |
| **Binary Location** | `~/.cargo/bin/` | `/usr/libexec/bootupd` | `/usr/libexec/bootupd` |
| **System Integration** | Manual setup | Automatic via RPM | Automatic via .deb |
| **Dependencies** | Rust deps only | System packages | System packages |
| **Image Builder** | Source compilation | Package installation | Package installation |

#### **5. Why Package-Based Approach is Critical**
1. **Reproducible Builds**: Same binary every time, no compilation variations
2. **System Integration**: Automatic service files, paths, and dependencies
3. **Security**: Signed packages with proper dependency resolution
4. **Performance**: No compilation time during image building
5. **Maintenance**: Updates come through package management, not cargo

### **Conclusion**
bootc-image-builder uses bootupd as a **Rust crate packaged into an RPM**. This means:

- **deb-bootupd must be packaged as a Debian .deb package (using dh-cargo)**
- **debian-bootc-image-builder must install the .deb package during image build**
- **The .deb package contains the compiled Rust binary**
- **No Rust compilation happens during image building - binary is pre-compiled**

This is why we're creating the Debian packaging files (`debian/control`, `debian/rules`) with `dh-cargo` - so that debian-bootc-image-builder can install deb-bootupd as a package, just like Fedora systems do with the `rust-bootupd` RPM package.

## **Critical Analysis: How bootupd is Actually Packaged**

Based on the [official bootupd repository](https://github.com/coreos/bootupd), here's exactly how they package it:

### **1. RPM Packaging Structure (contrib/packaging/bootupd.spec)**

```rpm
Name:           rust-%{crate}          # Package name: rust-bootupd
Version:        0.2.9
BuildRequires:  cargo-rpm-macros >= 25 # Uses Fedora's Rust packaging macros
```

**Key Files Installed**:
- `%{_bindir}/bootupctl` → `/usr/bin/bootupctl` (symlink)
- `%{_libexecdir}/bootupd` → `/usr/libexec/bootupd` (main binary)
- `%{_prefix}/lib/bootupd/grub2-static/` → GRUB configuration files
- `%{_unitdir}/bootloader-update.service` → systemd service

### **2. Makefile Installation Pattern**

```makefile
install:
	mkdir -p "${DESTDIR}$(PREFIX)/bin" "${DESTDIR}$(LIBEXECDIR)"
	install -D -t "${DESTDIR}$(LIBEXECDIR)" target/${PROFILE}/bootupd
	ln -f ${DESTDIR}$(LIBEXECDIR)/bootupd ${DESTDIR}$(PREFIX)/bin/bootupctl
```

**Critical Insight**: `bootupctl` is a **symlink** to the main `bootupd` binary, not a separate binary!

### **3. Multicall Binary Architecture**

From the Makefile:
```makefile
all:
	cargo build ${CARGO_ARGS}
	ln -f target/${PROFILE}/bootupd target/${PROFILE}/bootupctl
```

**How it works**:
1. **Single binary**: `bootupd` is compiled as one Rust binary
2. **Symlink creation**: `bootupctl` is created as a symlink to `bootupd`
3. **Runtime detection**: The binary detects which name it was called as (`argv[0]`)
4. **Behavior switching**: Different behavior based on the name (daemon vs client)

### **4. Why This Matters for deb-bootupd**

**Our current debian/rules is INCORRECT**:
```makefile
# Current (wrong):
ln -sf /usr/bin/bootupd debian/deb-bootupd/usr/bin/bootupctl

# Should be (correct):
ln -sf /usr/libexec/bootupd debian/deb-bootupd/usr/bin/bootupctl
```

**Correct Debian packaging should**:
1. **Install main binary**: `/usr/libexec/bootupd` (like RPM does)
2. **Create symlink**: `/usr/bin/bootupctl` → `/usr/libexec/bootupd`
3. **Follow RPM pattern**: Mirror the exact file layout from the RPM spec

### **5. File Layout Comparison**

| Component | RPM Location | Debian Location | Type |
|-----------|--------------|-----------------|------|
| **Main binary** | `/usr/libexec/bootupd` | `/usr/libexec/bootupd` | Executable |
| **CLI interface** | `/usr/bin/bootupctl` | `/usr/bin/bootupctl` | Symlink |
| **GRUB configs** | `/usr/lib/bootupd/grub2-static/` | `/usr/lib/bootupd/grub2-static/` | Static files |
| **Systemd service** | `/usr/lib/systemd/system/bootloader-update.service` | `/etc/systemd/system/bootloader-update.service` | Service file |

**Key insight**: The binary goes in `/usr/libexec/` (not `/usr/bin/`), and `bootupctl` is a symlink to it.

### **6. Critical Correction: bootupd is NOT a Daemon**

From the [official bootupd repository](https://github.com/coreos/bootupd):

> **Is bootupd a daemon?**
> 
> It was never a daemon. The name was intended to be "bootloader-upDater" not "bootloader-updater-Daemon". The choice of a "d" suffix is in retrospect probably too confusing.

**What this means for deb-bootupd**:
- **No daemon process**: `bootupd` is a command-line tool that runs and exits
- **systemd integration**: Uses `systemd-run` for locking and sandboxing, not as a persistent service
- **Event-driven**: Triggered by systemd services or bootc hooks, not running continuously
- **Multicall binary**: Single executable that behaves differently based on how it's called

## Phase 1: Project Setup & Structure

### 1.1 Create Debian Bootupd Directory Structure
```
deb-bootupd/
├── src/                    # Source code (adapted from .Red_Hat_Version/bootupd/src/)
├── systemd/               # Systemd service files
├── tests/                 # Test suite
├── Cargo.toml            # Dependencies adapted for Debian
├── README.md             # Debian-specific documentation
├── debian/               # Debian packaging files
└── scripts/              # Build and deployment scripts
```

### 1.2 Initialize Git Repository
- Fork from original bootupd repository
- Create debian branch
- Set up proper attribution and licensing

### 1.3 Git Strategy: Fork with Upstream Remote
**Approach**: Proper Git fork with upstream remote for sustainable development

**Benefits**:
- **Clean start**: No complex git history or upstream sync complexity
- **Maintainable**: Can easily pull upstream changes, security patches, and bug fixes
- **Focus on core**: Concentrate on making Debian immutable system bootable
- **Proof-of-concept**: Perfect for testing the concept while maintaining upstream compatibility
- **Future-proof**: Simplifies long-term maintenance and reduces technical debt

**Implementation**:
```bash
# Proper fork approach
git clone https://github.com/coreos/bootupd.git deb-bootupd
cd deb-bootupd
git remote rename origin upstream
git remote add origin <your-fork-url>
git checkout -b debian-adaptation
```

**Alternative (if proper fork is too complex initially)**:
```bash
# Simple copy approach (higher maintenance risk)
cp -r .Red_Hat_Version/bootupd deb-bootupd/
cd deb-bootupd
git init
git add .
git commit -m "Initial Debian fork of bootupd for immutable Debian proof-of-concept"
```

**Recommendation**: Use the proper Git fork approach. While slightly more complex initially, it significantly reduces the long-term maintenance burden and makes it easier to incorporate upstream security patches and improvements.

## Phase 2: Core Code Adaptation

### 2.1 Package System Integration (`src/packagesystem.rs`)
**Current Issue**: Hard dependency on RPM commands
**Solution**: Rewrite for DPKG/APT

**OSTree-Specific Considerations**:
- **Not traditional package management**: Particle-OS uses image-based updates, not `apt`/`dpkg`
- **Package queries**: Still need DPKG integration for querying existing package metadata
- **Update mechanism**: Updates come from new bootc images, not package repositories
- **Version tracking**: Need to track Debian package versions within the immutable OS tree

**Kernel Detection Challenge**:
- **File location**: `/usr/lib/modules/*/vmlinuz` (same as Red Hat)
- **Filename parsing**: Must handle Debian kernel naming conventions
- **Critical files**: `src/efi.rs` and `src/bios.rs` kernel detection logic
- **Example difference**: `vmlinuz-6.1.0-13-amd64` vs `vmlinuz-6.1.0-1.fc38.x86_64`

**Changes Required**:
```rust
// Replace RPM-specific code:
// OLD: rpm -q --queryformat "%{nevra},%{buildtime}" -f <file>
// NEW: dpkg -S <file> && dpkg -s <package> | grep "Installed-Time"

// Replace RPM database paths:
// OLD: usr/lib/sysimage/rpm, usr/share/rpm
// NEW: var/lib/dpkg, usr/share/doc
```

**Implementation**:
- Create `dpkg_parse_metadata()` function
- Parse DPKG output format instead of RPM
- Handle Debian package versioning conventions
- Support both `dpkg -S` and `dpkg -s` queries

### 2.2 OSTree Utilities (`src/ostreeutil.rs`)
**Current Issue**: RPM command construction
**Solution**: Adapt for Debian package database structure

**Changes Required**:
- Replace `rpm_cmd()` with `dpkg_cmd()`
- Update database path detection logic
- Maintain OSTree integration compatibility

### 2.3 CoreOS Integration (`src/coreos.rs`)
**Current Issue**: Fedora CoreOS-specific Aleph version parsing
**Solution**: Adapt for Debian/particle-os versioning

**Changes Required**:
- Replace `.coreos-aleph-version.json` with Debian-specific version file
- Update version parsing logic for Debian conventions
- Remove Fedora-specific test data references

### 2.4 Hardcoded Paths
**Current Issues**:
- `/usr/lib/bootupd/updates` (hardcoded in `model.rs`)
- `/boot` state directory (hardcoded in `statefile.rs`)
- `/run/bootupd-lock` lock file path

**Solution**: Make paths configurable or follow Debian conventions
- Consider using `/var/lib/bootupd/updates` (FHS compliant)
- Keep `/boot` for state (standard across distributions)
- Keep `/run` for locks (standard across distributions)

### 2.5 OSTree-Specific Adaptations
**Critical Changes for Debian OSTree**:

**Kernel Path Adaptation**:
```rust
// Current (Red Hat): /usr/lib/modules/*/vmlinuz
// Debian OSTree: /ostree/deploy/debian/deploy/$checksum.0/usr/lib/modules/*/vmlinuz
// Need to ensure this works with Debian kernel naming conventions
// Critical: bootupd must correctly resolve the nested OSTree deployment path
```

**Kernel Filename Parsing** (Key Challenge):
- **Red Hat convention**: `vmlinuz-6.1.0-1.fc38.x86_64`
- **Debian convention**: `vmlinuz-6.1.0-13-amd64`
- **Critical**: Filename parsing logic must handle Debian version format
- **Location**: This affects `src/efi.rs` and `src/bios.rs` kernel detection

**Bootloader Configuration**:
- **Current**: GRUB configs from coreos-assembler
- **Debian OSTree**: GRUB configs pointing to OSTree deployment paths
- **Path format**: `/ostree/deploy/debian/deploy/$checksum.0/vmlinuz`

**State Management**:
- **Current**: State in `/boot/bootupd-state.json`
- **Debian OSTree**: State must persist across OSTree deployments
- **Location**: Consider `/var/lib/bootupd/` for persistent state

## Phase 3: System Integration

### 3.1 System Commands
**Current Dependencies**:
- `efibootmgr` (EFI boot management)
- `mount`/`umount` (filesystem operations)
- `grub-common` (GRUB configuration tools)

**Debian Compatibility**:
- ✅ `efibootmgr`: Available in Debian repositories
- ✅ `mount`/`umount`: Standard Linux tools
- ✅ `grub-common`: Available in Debian repositories

**Action**: Ensure these packages are available in particle-os base image

**Note**: `bootupd`'s primary role is updating bootloader configuration files and entries, not running a full `grub-install`. In an immutable OSTree system, the core GRUB files are part of the image itself. `bootupd` uses tools like `efibootmgr` and GRUB configuration manipulation to point to the correct deployment.

### 3.2 Systemd Integration
**Current**: Hard dependency on `libsystemd`
**Debian**: ✅ Fully supports systemd
**Action**: No changes needed, maintain compatibility

### 3.3 OS Detection
**Current**: Red Hat-specific logic
**Solution**: Enhance OS detection for Debian

**Changes Required**:
```rust
// Current logic in efi.rs:
// 1. Try /etc/system-release (Red Hat specific)
// 2. Fall back to /etc/os-release

// Enhanced logic:
// 1. Try /etc/os-release first (standard)
// 2. Parse for Debian family distributions
// 3. Fall back to /etc/system-release for Red Hat
// 4. Handle Debian-specific version formats
```

## Phase 4: Debian-Specific Features

### 4.1 Package Manager Integration
**New Features**:
- APT package discovery integration
- Debian package version comparison
- Support for Debian backports and testing repositories

**OSTree-Specific Features**:
- **Image-based update detection**: Detect new bootc images vs package updates
- **Deployment coordination**: Work with OSTree deployment system
- **Rollback support**: Leverage OSTree's built-in rollback capabilities
- **State persistence**: Ensure bootupd state survives across deployments

**Trigger Mechanism**: `bootupd` is not a standalone daemon that runs on a schedule. It's triggered by systemd services (like `bootupd-post-upgrade.service`) or `bootc` hooks that run after a new OSTree deployment is committed. This event-driven approach ensures bootloader updates happen at the right time in the deployment lifecycle.

### 4.2 Debian OSTree Integration
**Unique Challenges**:
- **Hybrid approach**: Debian packages within immutable OSTree system
- **Update workflow**: New images contain updated Debian packages
- **Version compatibility**: Ensure Debian package versions work with OSTree structure
- **Bootloader updates**: Coordinate with OSTree deployment changes

### 4.3 Distribution-Specific Bootloader Configs
**Current**: Static GRUB configs from coreos-assembler
**Solution**: Create Debian-specific GRUB configurations
- Support for Debian kernel naming conventions
- Debian-specific boot parameters
- Integration with Debian initramfs tools
- **OSTree path integration**: GRUB configs pointing to OSTree deployment paths

### 4.4 Debian Package Dependencies
**Update Cargo.toml**:
- Keep core dependencies (anyhow, clap, serde, etc.)
- Maintain systemd integration
- Add Debian-specific optional features

## Phase 5: Testing & Validation

### 5.1 Test Suite Adaptation
**Current**: Fedora/RPM-centric tests
**Solution**: Create Debian-compatible test suite

**Test Categories**:
- Unit tests for DPKG parsing functions
- Integration tests with Debian package databases
- End-to-end tests in Debian container environments
- Bootloader update simulation tests

### 5.2 Particle-OS Integration Testing
**Test Scenarios**:
- Fresh particle-os installation
- Bootloader adoption from existing Debian systems
- Update process validation
- Rollback functionality testing

**OSTree-Specific Testing**:
- **Deployment switching**: Test bootloader updates across OSTree deployments
- **State persistence**: Verify bootupd state survives deployment switches
- **Image updates**: Test with new bootc images containing updated Debian packages
- **Rollback scenarios**: Test bootloader rollback with OSTree deployment rollback
- **Hybrid updates**: Test scenarios where Debian packages and OSTree images are updated

## Phase 6: Packaging & Deployment

### 6.1 Debian Package Creation
**Files Required**:
- `debian/control` (package metadata)
- `debian/rules` (build instructions)
- `debian/changelog` (version history)
- `debian/postinst` (post-installation scripts)

**Build System Integration**:
- **dh-cargo**: Use Debian's Rust packaging helper
- **Benefits**: Automates cargo build, cargo install, and dependency handling
- **debian/rules**: Simplified with dh-cargo integration
- **Cargo.lock**: Include for reproducible builds
- **Vendor directory**: Consider including for offline builds

**Important**: The Rust compilation happens when the `.deb` package is built (using `dh_cargo build --release`), not during the `debian-bootc-image-builder` process. The image builder simply installs the resulting pre-compiled binary via the `.deb` package.

### 6.2 Container Integration
**Integration Points**:
- particle-os base image requirements
- Bootc image builder integration
- Container runtime compatibility

### 6.3 Bootc Image Builder Integration
**Critical Integration Points**:

**Build Process Integration**:
- **Include deb-bootupd**: Binary must be installed as a .deb package during image build
- **Build timing**: deb-bootupd package installed during image build, not compiled from source
- **Dependencies**: Ensure all required tools (`efibootmgr`, `grub-install`) are in base image

**Image Builder Workflow**:
1. **Build phase**: Install deb-bootupd .deb package using debian-bootc-image-builder
2. **Installation**: Package manager automatically places binary in `/usr/libexec/bootupd` and creates symlinks
3. **Configuration**: Set up initial bootloader configuration
4. **First boot**: deb-bootupd runs to adopt existing bootloader or install new one

**Key Insight**: This mirrors exactly how Fedora bootc-image-builder works - it installs the `rust-bootupd` RPM package, which contains the compiled Rust binary. We must replicate this pattern with Debian packaging using `dh-cargo`.

**debian/control Integration**:
- **Build dependencies**: Use `dh-cargo` for Rust build system integration
- **Runtime dependencies**: Ensure `efibootmgr`, `grub-common` are available
- **Package naming**: Consider `deb-bootupd` vs `bootupd` package name

## Phase 7: Documentation & Maintenance

### 7.1 Documentation Updates
**Required Documents**:
- Debian-specific README
- Installation guide for particle-os
- Troubleshooting guide
- Migration guide from Red Hat systems

### 7.2 Maintenance Strategy
**Ongoing Tasks**:
- Track upstream bootupd changes
- Adapt new features for Debian
- Maintain Debian package compatibility
- Community support and bug fixes

## Implementation Priority

### High Priority (Phase 1-2) - Proof-of-Concept Core
1. **Package system adaptation** (RPM → DPKG) - Essential for Debian compatibility
2. **Debian packaging creation** - Create .deb package for debian-bootc-image-builder integration
3. **Fix binary layout** - Correct file locations to match RPM pattern (`/usr/libexec/bootupd`, symlink for `bootupctl`)
4. **Core path and dependency fixes** - Make it compile and run
5. **Basic Debian compatibility** - Get it working on Debian system

### Medium Priority (Phase 3-4) - Basic Functionality
1. **Enhanced OS detection** - Proper Debian identification
2. **Debian-specific features** - Basic bootloader management
3. **Testing infrastructure** - Verify it works

### Low Priority (Phase 5-7) - Polish & Production
1. **Advanced features** - Only if proof-of-concept succeeds
2. **Documentation** - Only if proof-of-concept succeeds
3. **Long-term maintenance** - Only if proof-of-concept succeeds

## Success Criteria

### Proof-of-Concept Success (Phases 1-2)
- [ ] **Debian bootupd compiles successfully** - Basic Rust compilation
- [ ] **Package system queries work with DPKG** - Can read Debian package info
- [ ] **Basic bootloader detection works** - Can identify EFI/BIOS systems

### Core Functionality Success (Phase 3-4)
- [ ] **Bootloader management works on Debian** - Can install/update GRUB
- [ ] **OSTree integration functions** - Works with Debian OSTree deployment
- [ ] **State management works** - Can track bootloader state

### Full Success (Phase 5+)
- [ ] **Particle-OS image boots successfully** - Proof-of-concept achieved!
- [ ] **Bootloader updates work end-to-end** - Can update from new images
- [ ] **Rollback functionality works** - Can rollback to previous state

**Ultimate Goal**: Boot a Debian-based immutable system using ublue-os tools

## Risk Mitigation

### Technical Risks
1. **Package format differences**: RPM vs DPKG complexity
2. **Path conventions**: Red Hat vs Debian filesystem standards
3. **Tool availability**: Ensuring required tools exist in Debian
4. **OSTree integration complexity**: Coordinating with Debian OSTree deployment system
5. **Hybrid system challenges**: Debian packages within immutable OSTree structure

### Mitigation Strategies
1. **Incremental development**: Test each component individually
2. **Fallback mechanisms**: Graceful degradation when tools unavailable
3. **Comprehensive testing**: Multiple Debian versions and environments
4. **Upstream independence**: Build Debian-specific features without upstream dependencies
5. **Maintenance planning**: Regular subtree pulls to stay current with upstream fixes

## Next Steps

1. **Immediate**: Set up deb-bootupd directory structure
2. **Week 1**: Adapt package system integration (RPM → DPKG)
3. **Week 2**: Create Debian packaging files (.deb package)
4. **Week 3**: Fix hardcoded paths and dependencies
5. **Week 4**: Basic testing and validation
6. **Week 5**: Particle-OS integration testing with debian-bootc-image-builder

**Critical**: Focus on creating a working .deb package first, as this is how debian-bootc-image-builder will integrate with deb-bootupd (mirroring the Fedora RPM approach).

## Resources & References

- **Original bootupd**: `.Red_Hat_Version/bootupd/`
- **Debian packaging**: https://www.debian.org/doc/manuals/debmake-doc/
- **FHS standards**: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/
- **Debian bootc tools**: `debian-bootc-image-builder/`