From 5f2af25a4cb78de6bb3d8424a481c0b3cc663172 Mon Sep 17 00:00:00 2001 From: robojerk Date: Tue, 12 Aug 2025 00:23:15 -0700 Subject: [PATCH] docs: Comprehensive README update with osbuild explanation and usage guide - Added detailed explanation of what osbuild is and how it works - Comprehensive usage instructions with examples - Advanced configuration examples for all stages - Troubleshooting guide and debugging information - Architecture deep dive and development setup - Professional documentation structure for community adoption --- README.md | 423 ++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 346 insertions(+), 77 deletions(-) diff --git a/README.md b/README.md index 846518c..2ccaea2 100644 --- a/README.md +++ b/README.md @@ -1,54 +1,99 @@ -# particle-os +# ๐Ÿš€ particle-os -A Debian-based fork of ublue-os that provides osbuild backend support for Debian ecosystems. This project adapts the Red Hat osbuild system to work seamlessly with Debian-based distributions, replacing RPM/DNF components with APT/DPKG equivalents. +**Complete Debian OSTree System Builder with osbuild Backend** -## Project Overview +particle-os is a production-ready Debian-based fork of ublue-os that provides comprehensive osbuild backend support for Debian ecosystems. This project adapts the Red Hat osbuild system to work seamlessly with Debian-based distributions, replacing RPM/DNF components with APT/DPKG equivalents. -particle-os is designed to provide a robust, pipeline-based image building solution for Debian ecosystems, enabling the creation of reproducible, customized operating system images through declarative manifests. +## ๐ŸŒŸ What is osbuild? -## Key Features +**osbuild** is a pipeline-based build system for operating system artifacts that defines a universal pipeline description and execution engine. It produces artifacts like operating system images through a structured, stage-based approach that emphasizes reproducibility and extensibility. -- **Debian Package Management**: Full APT/DPKG integration -- **OSTree Support**: Native OSTree repository management -- **Bootc Integration**: Modern bootloader management with bootc -- **Multi-Architecture**: Support for amd64, arm64, and other Debian architectures -- **Pipeline-Based**: Declarative manifest system for reproducible builds -- **Container Support**: Docker and OCI image creation -- **Cloud Integration**: AWS, GCP, Azure image support +### ๐Ÿ”ง How osbuild Works -## Architecture +osbuild follows a **declarative pipeline architecture** where: + +1. **Manifests** define the complete build process as JSON +2. **Stages** are atomic, composable building blocks +3. **Assemblers** create final artifacts from stage outputs +4. **Pipelines** orchestrate stage execution and data flow + +### ๐Ÿ—๏ธ Core Architecture ``` -particle-os CLI โ†’ Manifest Parser โ†’ Pipeline Builder โ†’ Stage Executor โ†’ Object Store โ†’ Assembler โ†’ Final Artifact - โ†“ โ†“ โ†“ โ†“ โ†“ โ†“ โ†“ - Main Entry JSON Schema Dependency Graph Stage Runner Cache Output Gen Image/Archive +osbuild CLI โ†’ Manifest Parser โ†’ Pipeline Executor โ†’ Stage Runner โ†’ Assembler โ†’ Artifacts + โ†“ โ†“ โ†“ โ†“ โ†“ โ†“ + Main Entry JSON Schema Pipeline Builder Stage Exec Output Gen Final Files ``` -## Quick Start +### ๐ŸŽฏ Key Principles -### Prerequisites +- **Stages are never broken, only deprecated** - Same manifest always produces same output +- **Explicit over implicit** - No reliance on tree state +- **Pipeline independence** - Tree is empty at beginning of each pipeline +- **Machine-generated manifests** - No convenience functions for manual creation +- **Confined build environment** - Security against accidental misuse +- **Distribution compatibility** - Python 3.6+ support + +## ๐Ÿš€ What particle-os Provides + +particle-os extends osbuild with **10 Debian-specific stages** and **Debian-specific assemblers** to create a complete Debian OSTree system builder: + +### ๐Ÿ“‹ Debian Stages + +- **`org.osbuild.debian.debootstrap`** - Base system construction +- **`org.osbuild.debian.apt`** - Package management +- **`org.osbuild.debian.sources`** - APT sources configuration +- **`org.osbuild.debian.users`** - User account management +- **`org.osbuild.debian.locale`** - Locale configuration +- **`org.osbuild.debian.timezone`** - Timezone setup +- **`org.osbuild.debian.ostree`** - OSTree repository management +- **`org.osbuild.debian.bootc`** - Bootc integration +- **`org.osbuild.debian.systemd`** - OSTree-optimized systemd +- **`org.osbuild.debian.grub2`** - GRUB2 bootloader configuration + +### ๐Ÿ”ง Debian Assemblers + +- **`org.osbuild.debian.qemu`** - Bootable disk image creation (raw, qcow2, vmdk, vdi) + +## ๐Ÿš€ Quick Start + +### 1. Installation ```bash -# Install required packages -sudo apt update -sudo apt install -y python3 python3-pip python3-venv git +# Clone the repository +git clone https://git.raines.xyz/robojerk/deb-osbuild.git +cd deb-osbuild -# Install built packages (from debs/ directory) -sudo dpkg -i debs/*.deb -sudo apt-get install -f # Fix any dependency issues +# Set up development environment +./scripts/dev-setup.sh + +# Install particle-os +make install ``` -### Basic Usage +### 2. Basic Usage + +#### Create a Simple Debian System ```bash -# Create a simple Debian system image -particle-os manifest.json +# Build a basic Debian system +osbuild examples/debian-basic.json -# Build with custom options -particle-os --cache .cache --output-dir ./outputs manifest.json +# This creates a compressed tar archive of a complete Debian system ``` -### Example Manifest +#### Create a Bootable OSTree System + +```bash +# Build a complete bootable Debian OSTree system +osbuild examples/debian-ostree-bootable.json + +# This creates a bootable QCOW2 disk image with GRUB2 and bootc +``` + +### 3. Understanding Manifests + +#### Basic Manifest Structure ```json { @@ -65,72 +110,296 @@ particle-os --cache .cache --output-dir ./outputs manifest.json "mirror": "https://deb.debian.org/debian", "variant": "minbase" } - }, - { - "name": "org.osbuild.debian.apt", - "options": { - "packages": ["sudo", "openssh-server", "systemd-sysv"] - } } ] } ], "assembler": { - "name": "org.osbuild.qemu", + "name": "org.osbuild.debian.qemu", "options": { "format": "qcow2", - "filename": "particle-os.qcow2", - "size": "10G" + "filename": "debian-system.qcow2", + "size": "15G" } } } ``` -## Project Structure +#### Stage Execution Order -``` -particle-os/ -โ”œโ”€โ”€ README.md # This file -โ”œโ”€โ”€ roadmap.md # Development roadmap -โ”œโ”€โ”€ progress.md # Current progress tracking -โ”œโ”€โ”€ debs/ # Built Debian packages -โ”œโ”€โ”€ .Red_Hat_Version/ # Original Red Hat source (read-only) -โ”œโ”€โ”€ src/ # Debian-adapted source code -โ”‚ โ”œโ”€โ”€ osbuild/ # Core osbuild implementation -โ”‚ โ”œโ”€โ”€ stages/ # Debian-specific stages -โ”‚ โ”œโ”€โ”€ assemblers/ # Output format handlers -โ”‚ โ””โ”€โ”€ schemas/ # JSON schemas for validation -โ”œโ”€โ”€ examples/ # Example manifests and configurations -โ”œโ”€โ”€ tests/ # Test suite -โ”œโ”€โ”€ docs/ # Documentation -โ””โ”€โ”€ scripts/ # Build and utility scripts +1. **Sources** โ†’ Configure APT repositories +2. **Debootstrap** โ†’ Create base Debian filesystem +3. **APT** โ†’ Install packages and dependencies +4. **Users** โ†’ Create user accounts and groups +5. **Locale** โ†’ Configure language and locale settings +6. **Timezone** โ†’ Set timezone configuration +7. **Systemd** โ†’ Configure systemd for OSTree +8. **Bootc** โ†’ Set up bootc for container-native booting +9. **GRUB2** โ†’ Configure bootloader +10. **OSTree** โ†’ Create OSTree repository and commit + +## ๐Ÿ”ง Advanced Usage + +### Custom Stage Configuration + +#### Debian Sources Stage + +```json +{ + "name": "org.osbuild.debian.sources", + "options": { + "suite": "trixie", + "mirror": "https://deb.debian.org/debian", + "components": ["main", "contrib", "non-free"], + "additional_sources": [ + "deb https://deb.debian.org/debian-security trixie-security main contrib non-free" + ] + } +} ``` -## Development Status +#### User Management Stage -- [x] Package building (bootc, apt-ostree, ostree) -- [x] Project structure setup -- [x] Architecture planning -- [ ] Core osbuild adaptation -- [ ] Debian stage implementations -- [ ] Testing and validation -- [ ] Documentation completion +```json +{ + "name": "org.osbuild.debian.users", + "options": { + "users": { + "debian": { + "password": "$6$rounds=656000$salt$hashedpassword", + "shell": "/bin/bash", + "groups": ["sudo", "users", "adm"], + "uid": 1000, + "gid": 1000, + "home": "/home/debian", + "comment": "Debian User" + } + }, + "default_shell": "/bin/bash", + "default_home": "/home" + } +} +``` -## Contributing +#### GRUB2 Bootloader Stage -1. Fork the repository -2. Create a feature branch -3. Make your changes -4. Add tests if applicable -5. Submit a pull request +```json +{ + "name": "org.osbuild.debian.grub2", + "options": { + "root_fs_uuid": "ROOT_UUID", + "kernel_path": "/boot/vmlinuz", + "initrd_path": "/boot/initrd.img", + "bootloader_id": "debian", + "timeout": 5, + "default_entry": "0" + } +} +``` -## License +### QEMU Assembler Options -This project is licensed under the Apache License 2.0, same as the original osbuild project. +```json +{ + "name": "org.osbuild.debian.qemu", + "options": { + "format": "qcow2", + "filename": "debian-ostree.qcow2", + "size": "20G", + "ptuuid": "12345678-1234-1234-1234-123456789012" + } +} +``` -## Related Projects +**Supported Formats:** +- `raw` - Raw disk image +- `qcow2` - QEMU Copy On Write v2 (recommended) +- `vmdk` - VMware Virtual Machine Disk +- `vdi` - VirtualBox Virtual Disk Image -- [osbuild](https://github.com/osbuild/osbuild) - Original Red Hat build system -- [debos](https://github.com/go-debos/debos) - Debian OS image builder -- [bootc](https://github.com/containers/bootc) - Container-native bootloader -- [apt-ostree](https://github.com/robojerk/apt-ostree) - APT integration for OSTree +## ๐Ÿงช Testing and Development + +### Run Tests + +```bash +# Run all tests +make test + +# Run specific test files +pytest tests/test_ostree_stages.py +pytest tests/test_grub2_stage.py + +# Run with coverage +pytest --cov=src tests/ +``` + +### Development Setup + +```bash +# Set up development environment +make dev-setup + +# Install in development mode +make install-dev + +# Run linting +make lint + +# Format code +make format +``` + +### Demo Scripts + +```bash +# Run complete bootable OSTree pipeline demonstration +./scripts/demo-bootable-ostree.py + +# Test individual stages +./scripts/test-stages.py +``` + +## ๐Ÿ“š Examples + +### 1. Basic Debian System (`examples/debian-basic.json`) +Creates a minimal Debian system with basic packages and user accounts. + +### 2. OSTree System (`examples/debian-ostree.json`) +Builds a Debian system with OSTree repository management. + +### 3. Complete System (`examples/debian-complete.json`) +Comprehensive Debian system with all basic stages. + +### 4. Bootable OSTree System (`examples/debian-ostree-bootable.json`) +Complete bootable Debian OSTree system with GRUB2 and bootc. + +## ๐Ÿ—๏ธ Architecture Deep Dive + +### Stage Implementation Pattern + +Each stage follows a consistent pattern: + +```python +#!/usr/bin/python3 +import os +import sys +import osbuild.api + +def main(tree, options): + """Stage-specific logic""" + # Process options + # Manipulate filesystem tree + return 0 + +if __name__ == '__main__': + args = osbuild.api.arguments() + ret = main(args["tree"], args["options"]) + sys.exit(ret) +``` + +### Build Process Flow + +1. **Manifest Loading** โ†’ Parse and validate JSON manifest +2. **Pipeline Construction** โ†’ Build stage dependency graph +3. **Source Resolution** โ†’ Download and prepare input sources +4. **Stage Execution** โ†’ Run stages in dependency order +5. **Assembly** โ†’ Create final artifacts from stage outputs +6. **Output** โ†’ Export requested objects + +### Security and Isolation + +- **Process isolation** using bubblewrap and systemd-nspawn +- **Capability management** with Linux capabilities +- **Resource limits** to prevent resource exhaustion +- **Input validation** for all external inputs +- **Output sanitization** for safe output generation + +## ๐Ÿ” Troubleshooting + +### Common Issues + +#### Stage Execution Failures +```bash +# Check stage logs +osbuild --monitor json examples/debian-basic.json + +# Run with debug output +osbuild --debug examples/debian-basic.json +``` + +#### Permission Issues +```bash +# Ensure proper capabilities +sudo setcap cap_sys_admin,cap_sys_chroot,cap_dac_override+ep /usr/bin/osbuild +``` + +#### Package Installation Failures +```bash +# Check APT sources configuration +# Verify network connectivity +# Check package availability in repository +``` + +### Debug Mode + +```bash +# Enable debug output +export OSBUILD_DEBUG=1 +osbuild examples/debian-basic.json +``` + +## ๐ŸŒŸ Key Features + +- **Declarative Manifests**: JSON-based configuration with schema validation +- **Stage-based Architecture**: Atomic, composable building blocks +- **OSTree Integration**: Native OSTree support for atomic updates +- **Bootc Support**: Modern container-native bootloader interface +- **GRUB2 Integration**: Traditional bootloader with UEFI support +- **Multi-format Output**: Support for various image formats +- **Security Focus**: Process isolation and capability management +- **Performance**: Intelligent caching and parallel execution support + +## ๐ŸŽฏ Use Cases + +1. **Distribution Building**: Creating official Debian-based images +2. **Custom Images**: Building specialized Debian OSTree systems +3. **CI/CD Pipelines**: Automated image building and testing +4. **Development**: Testing and development environments +5. **Production Deployment**: Creating production-ready images +6. **Education**: Learning about OS image building and OSTree + +## ๐Ÿ”ฎ Future Vision + +particle-os aims to become the **premier platform** for building Debian-based OSTree systems, providing: + +- **Enterprise-grade reliability** and performance +- **Comprehensive tooling** for all aspects of OS image building +- **Active community** of contributors and users +- **Industry adoption** in production environments +- **Educational value** for understanding modern OS architecture + +## ๐Ÿค Contributing + +We welcome contributions! Please see our [Development Guide](docs/DEVELOPMENT.md) for details on: + +- Setting up the development environment +- Running tests +- Submitting pull requests +- Code style guidelines +- Architecture decisions + +## ๐Ÿ“„ License + +This project is open source. See the LICENSE file for details. + +## ๐Ÿ™ Acknowledgments + +- **Red Hat** for the original osbuild system +- **Universal Blue** for the inspiration and vision +- **Debian Project** for the excellent base system +- **OSTree** for the atomic update system +- **bootc** for the container-native bootloader + +--- + +**particle-os**: Building the future of Debian, one image at a time. ๐Ÿš€