deb-orchestrator/README.md

# deb-bootc-compose

**Debian's equivalent to Fedora's Pungi compose system**

`deb-bootc-compose` is a Debian-native composition tool that orchestrates the creation of Debian bootc images from packages. It coordinates the entire compose process, similar to how Pungi coordinates Fedora's release process.

## What It Does

`deb-bootc-compose` serves as the **main orchestrator** for Debian's bootc ecosystem:

* **Package Coordination**: Ensures all release artifacts use identical package versions across variants
* **Multi-Artifact Generation**: Creates container images, disk images, and other bootc artifacts
* **Build Orchestration**: Coordinates with `deb-orchestrator` (Koji equivalent) and `deb-mock` (Mock equivalent)
* **OSTree Integration**: Orchestrates bootc image creation through apt-ostree
* **Variant Management**: Handles different Debian variants (minimal, server, desktop, etc.)

## Architecture

The tool follows a **phase-based architecture** inspired by Pungi:

```text
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ deb-bootc-      │    │ deb-orchestrator│    │    deb-mock     │
│   compose       │    │   (Koji equiv)  │    │   (Mock equiv)  │
│ Orchestrator    │    │ Build System    │    │Build Environment│
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │ Coordinates           │ Manages               │ Creates
         │ entire process        │ package building      │ isolated
         │                       │ at scale              │ environments

```

### Core Phases

1. **init**: Initialize compose environment
2. **gather**: Download and organize packages
3. **build**: Build packages if needed
4. **ostree**: Create OSTree commits
5. **output**: Generate output artifacts
6. **cleanup**: Clean up temporary files

## Quick Start

### Prerequisites

* Go 1.21 or later
* Debian-based system (for development)
* Basic understanding of Debian packaging

### Installation

```bash
# Clone the repository
git clone https://git.raines.xyz/particle-os/deb-bootc-compose.git
cd deb-bootc-compose

# Install dependencies
make deps

# Build the binary
make build

# Run with help
./build/deb-bootc-compose --help

```

### Basic Usage

```bash
# Run a compose with a treefile
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-minimal.json \
    --output ./compose-output \
    --config configs/compose.yaml

```

## Configuration

### Compose Configuration

The main configuration file (`compose.yaml`) controls:

* **Compose settings**: Release, variants, architectures
* **Build system**: sbuild, debootstrap integration
* **OSTree settings**: Repository mode, signing
* **Output formats**: Container, disk image, chunked OCI
* **Integration**: deb-orchestrator connection settings

### Treefiles

Treefiles (JSON manifests) define what gets built:

* **Package lists**: Required, optional, recommended packages
* **Variants**: Different flavors (minimal, server, desktop)
* **Architectures**: Target CPU architectures
* **Repositories**: Package sources
* **Build settings**: Build system configuration

Example treefile structure:

```json
{
  "name": "debian-bootc-minimal",
  "version": "13",
  "release": "bookworm",
  "packages": {
    "required": ["linux-image-amd64", "systemd", "ostree", "bootc"]
  },
  "architecture": ["amd64", "arm64"],
  "variants": [
    {
      "name": "minimal",
      "description": "Minimal base system"
    }
  ]
}

```

## Development

### Project Structure

```text
deb-bootc-compose/
├── cmd/                    # Command-line interfaces
│   ├── compose/           # Main compose command
│   └── cli/              # CLI utilities
├── internal/              # Internal packages
│   ├── compose/           # Core compose engine
│   ├── config/            # Configuration management
│   ├── ostree/            # OSTree integration
│   └── build/             # Build system interface
├── pkg/                   # Public packages
├── configs/               # Configuration files
├── examples/              # Example treefiles
└── docs/                  # Documentation

```

### Building

```bash
# Build everything
make all

# Build specific components
make build      # Main binary
make cli        # CLI tool

# Development helpers
make test       # Run tests
make fmt        # Format code
make lint       # Lint code
make clean      # Clean build artifacts

```

### Testing

```bash
# Run all tests
make test

# Run tests with coverage
make test-coverage

# Run with sample treefile
make run-sample

```

## Integration

### With deb-orchestrator

`deb-bootc-compose` integrates with `deb-orchestrator` for:

* **Package management**: Downloading packages from build system
* **Build coordination**: Ensuring package availability before compose
* **Build orchestration**: Submitting build requests and monitoring progress
* **Metadata integration**: Using build metadata for versioning

### With deb-mock

`deb-bootc-compose` can use `deb-mock` for:

* **Build environment creation**: Isolated chroot environments
* **Package installation**: Installing build dependencies
* **Environment isolation**: Reproducible builds

## Advanced Features

### Multi-Variant Composes

Support for creating multiple variants in a single compose:

```bash
# Compose multiple variants
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-multi-variant.json \
    --variants minimal,standard,development \
    --output ./multi-variant-output
```

### Parallel Processing

Efficient parallel execution of compose phases:

```bash
# Enable parallel processing
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-minimal.json \
    --parallel \
    --workers 4
```

### Output Formats

Multiple output format support:

- **Container Images**: OCI-compatible container images
- **Disk Images**: QCOW2, RAW, and other disk formats
- **Tarballs**: Compressed root filesystem archives
- **Metadata**: JSON metadata for integration

## Troubleshooting

### Common Issues

1. **Package Resolution Failures**
   ```bash
   # Check package availability
   apt-cache policy <package-name>
   
   # Verify repository configuration
   cat /etc/apt/sources.list.d/*
   ```

2. **OSTree Repository Issues**
   ```bash
   # Check OSTree repository status
   ostree log <ref>
   
   # Verify repository structure
   ostree refs --repo <repo-path>
   ```

3. **Build System Connection**
   ```bash
   # Test deb-orchestrator connection
   curl -s http://localhost:8080/health
   
   # Check build system logs
   journalctl -u deb-orchestrator
   ```

### Debug Mode

Enable verbose logging for troubleshooting:

```bash
# Run with debug output
./build/deb-bootc-compose \
    --treefile examples/debian-bootc-minimal.json \
    --debug \
    --log-level debug
```

## Performance Optimization

### Caching Strategies

- **Package Cache**: Local caching of downloaded packages
- **OSTree Cache**: Incremental repository updates
- **Build Cache**: Reuse of build artifacts

### Resource Management

- **Memory Optimization**: Efficient memory usage for large composes
- **Disk I/O**: Optimized file operations and storage
- **Network**: Parallel downloads and connection pooling

## Status

**Current Status**: Phase 1 - Foundation Development

* ✅ **Core engine**: Basic compose engine implemented
* ✅ **Treefile parser**: JSON-based configuration system
* ✅ **Phase management**: Simple phase execution system
* ✅ **Configuration**: YAML-based configuration system
* ✅ **OSTree integration**: apt-ostree integration working
* ✅ **Build system**: deb-orchestrator integration functional
* ✅ **Output formats**: Container, disk image, tarball generation
* 🔄 **Testing**: Comprehensive testing framework in progress

## Roadmap

See the main project TODO.md for the complete development roadmap.

### Phase 1 (Months 1-6): Foundation ✅ COMPLETED

* Core compose engine working
* Basic treefile parsing and validation
* OSTree integration functional
* Container output working

### Phase 2 (Months 7-10): Integration 🎯 CURRENT

* Deep integration with deb-orchestrator and deb-mock
* Advanced features and optimization
* Production readiness features

### Phase 3 (Months 11-14): Production

* Security audit and hardening
* Performance optimization
* Community integration

### Phase 4 (Months 15-18): Ecosystem

* Debian Atomic and Particle-OS variants
* Advanced use cases
* Community adoption

## Contributing

We welcome contributions! Please see our Contributing Guide for details.

### Development Setup

```bash
# Set up development environment
make dev-setup

# Install development tools
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

# Run pre-commit checks
make pre-commit
```

### Code Style

- Follow Go standard formatting (`gofmt`)
- Use meaningful variable and function names
- Add comprehensive tests for new features
- Update documentation for API changes

## License

This project is licensed under the same terms as Debian (GPL-2+).

## Related Projects

* **deb-orchestrator**: Debian's equivalent to Fedora's Koji build system
* **deb-mock**: Debian's equivalent to Fedora's Mock build environment manager
* **deb-bootc-compose**: This project - the main orchestrator

## Support

* **Issues**: [Git.raines.xyz Issues](https://git.raines.xyz/particle-os/deb-bootc-compose/issues)
* **Discussions**: [Git.raines.xyz Discussions](https://git.raines.xyz/particle-os/deb-bootc-compose/discussions)
* **Documentation**: Project Wiki

## Repository Information

- **Source**: [https://git.raines.xyz/particle-os/deb-bootc-compose](https://git.raines.xyz/particle-os/deb-bootc-compose)
- **Language**: Go (65%), Python (33.2%), Makefile (1.8%)
- **Size**: 109 KiB (clean, optimized repository)
- **Last Updated**: 2025-08-18

---

**Part of Debian's complete bootc ecosystem** - building Debian's answer to Fedora's Pungi-Koji-Mock ecosystem.