deb-osbuild/docs/debos.md

# debos Comprehensive Top-to-Bottom Analysis

## Overview

debos is a Debian OS image builder that creates reproducible, customized operating system images through a YAML-based recipe system. It operates by executing a series of actions in a controlled virtual machine environment, ensuring consistency across different host systems. This document provides a complete top-to-bottom analysis of the entire debos process, from recipe parsing to final artifact generation.

## Table of Contents

1. [Complete Process Flow](#complete-process-flow)
2. [Core Architecture](#core-architecture)
3. [Recipe Processing Pipeline](#recipe-processing-pipeline)
4. [Action System Deep Dive](#action-system-deep-dive)
5. [Fakemachine Integration](#fakemachine-integration)
6. [Command Execution System](#command-execution-system)
7. [Filesystem Operations](#filesystem-operations)
8. [Archive Management](#archive-management)
9. [External Tools and Dependencies](#external-tools-and-dependencies)
10. [Context Management](#context-management)
11. [Error Handling and Debugging](#error-handling-and-debugging)
12. [Performance Characteristics](#performance-characteristics)
13. [Integration Points](#integration-points)
14. [Complete Workflow Examples](#complete-workflow-examples)

## Complete Process Flow

### End-to-End Workflow

```
1. Recipe Loading → 2. Template Processing → 3. YAML Parsing → 4. Action Validation → 5. Fakemachine Setup → 6. Action Execution → 7. Artifact Generation → 8. Cleanup
```

### Detailed Process Steps

#### **Phase 1: Recipe Preparation**
1. **File Loading**: Read YAML recipe file from disk
2. **Template Processing**: Apply Go templates with variables
3. **YAML Parsing**: Parse processed YAML into Recipe struct
4. **Action Mapping**: Map YAML actions to concrete Action implementations
5. **Validation**: Verify all action parameters and dependencies

#### **Phase 2: Environment Setup**
1. **Fakemachine Creation**: Initialize virtual machine backend
2. **Resource Allocation**: Set CPU, memory, scratch space
3. **Volume Mounting**: Mount recipe directory, artifact directory
4. **Environment Variables**: Propagate proxy and custom variables
5. **Architecture Setup**: Configure cross-architecture support

#### **Phase 3: Action Execution**
1. **Pre-Machine Setup**: Run PreMachine hooks for all actions
2. **Sequential Execution**: Execute actions in recipe order
3. **Chroot Management**: Handle filesystem isolation
4. **Command Execution**: Run external tools and scripts
5. **State Management**: Maintain context across actions

#### **Phase 4: Artifact Generation**
1. **Output Collection**: Gather results from all actions
2. **Post-Processing**: Run PostMachine hooks
3. **File Generation**: Create final artifacts (images, archives)
4. **Cleanup**: Remove temporary files and mounts

## Core Architecture

### Design Philosophy

debos follows a **pipeline-based architecture** where:
- **Recipes** define the build process as a sequence of actions
- **Actions** are self-contained, independent modules
- **Fakemachine** provides isolated execution environment
- **Context** maintains state across the entire build process

### Key Components

```
debos CLI → Recipe Parser → Action Executor → Fakemachine → Output
     ↓              ↓              ↓              ↓         ↓
  Main Entry   YAML/YAML+Go   Action Runner   VM Backend  Artifacts
```

## Recipe Processing Pipeline

### Complete Recipe Processing Flow

#### **1. Template Processing Engine**
```go
func (r *Recipe) Parse(file string, printRecipe bool, dump bool, templateVars ...map[string]string) error {
    t := template.New(path.Base(file))
    funcs := template.FuncMap{
        "sector": sector,      // Add 's' suffix for sector calculations
        "escape": escape,      // Shell escape variables
        "uuid5": uuid5,        // Generate deterministic UUIDs
    }
    t.Funcs(funcs)
    
    // Add slim-sprig functions for advanced templating
    t.Funcs(sprig.FuncMap())
    
    // Parse and execute template
    if _, err := t.ParseFiles(file); err != nil {
        return err
    }
    
    data := new(bytes.Buffer)
    if err := t.Execute(data, templateVars[0]); err != nil {
        return err
    }
    
    // Unmarshal processed YAML
    if err := yaml.Unmarshal(data.Bytes(), &r); err != nil {
        return err
    }
    
    return nil
}
```

#### **2. Action Factory System**
```go
func (y *YamlAction) UnmarshalYAML(unmarshal func(interface{}) error) error {
    var aux debos.BaseAction
    
    err := unmarshal(&aux)
    if err != nil {
        return err
    }
    
    // Factory pattern for action creation
    switch aux.Action {
    case "debootstrap":
        y.Action = NewDebootstrapAction()
    case "mmdebstrap":
        y.Action = NewMmdebstrapAction()
    case "apt":
        y.Action = NewAptAction()
    case "run":
        y.Action = &RunAction{}
    case "image-partition":
        y.Action = &ImagePartitionAction{}
    case "overlay":
        y.Action = &OverlayAction{}
    case "pack":
        y.Action = NewPackAction()
    // ... more actions
    default:
        return fmt.Errorf("Unknown action: %v", aux.Action)
    }
    
    // Unmarshal action-specific options
    err = unmarshal(y.Action)
    return err
}
```

#### **3. Template Functions Available**
- **`sector(s int) string`**: Adds 's' suffix for sector calculations
- **`escape(s string) string`**: Shell-escapes variables for safe command execution
- **`uuid5(namespace, data) string`**: Generates deterministic UUIDs using SHA1
- **slim-sprig functions**: Advanced string manipulation, math, crypto functions

#### **4. Recipe Validation Process**
```go
func (r *Recipe) Parse(file string, printRecipe bool, dump bool, templateVars ...map[string]string) error {
    // ... template processing ...
    
    // Mandatory field validation
    if len(r.Architecture) == 0 {
        return fmt.Errorf("Recipe file must have 'architecture' property")
    }
    
    if len(r.Actions) == 0 {
        return fmt.Errorf("Recipe file must have at least one action")
    }
    
    // Set defaults
    if r.SectorSize == 0 {
        r.SectorSize = 512
    }
    
    return nil
}
```

### Recipe Structure and Syntax

#### **Basic Recipe Format**
```yaml
{{- $image := or .image "debian.tgz" -}}
{{- $suite := or .suite "bookworm" -}}

architecture: {{ .architecture }}
sectorsize: 512

actions:
  - action: debootstrap
    suite: {{ $suite }}
    mirror: https://deb.debian.org/debian
    variant: minbase
    
  - action: apt
    packages: [ sudo, openssh-server, systemd-sysv ]
    
  - action: run
    chroot: true
    script: scripts/setup-system.sh
    
  - action: pack
    file: {{ $image }}
    compression: gz
```

#### **Template Variable Sources**
1. **Command Line**: `debos -t variable:value recipe.yaml`
2. **Environment**: System environment variables
3. **Built-in**: Architecture, suite defaults
4. **Recipe**: Local variable definitions with `{{- $var := "value" -}}`

## Execution Flow

### 1. CLI Entry Point (`cmd/debos/debos.go`)

The main entry point handles:
- Command-line argument parsing
- Environment variable propagation
- Fakemachine backend selection
- Recipe file validation

```go
func main() {
    context := debos.DebosContext{...}
    
    // Parse command line options
    parser := flags.NewParser(&options, flags.Default)
    
    // Create fakemachine or run on host
    if !options.DisableFakeMachine {
        m, err = fakemachine.NewMachineWithBackend(options.Backend)
        // Execute in VM
    } else {
        // Execute on host
    }
}
```

### 2. Recipe Parsing (`actions/recipe.go`)

Recipes are parsed with Go template support:
- **YAML parsing** with `gopkg.in/yaml.v2`
- **Go template expansion** for dynamic values
- **Variable substitution** with `-t` command line options
- **Validation** of action parameters

### 3. Action Execution Pipeline

Actions are executed sequentially with lifecycle hooks:

```go
type Action interface {
    Verify(context *DebosContext) error           // Pre-execution validation
    PreMachine(context *DebosContext, m *fakemachine.Machine, args *[]string) error
    PreNoMachine(context *DebosContext) error     // Host-side preparation
    Run(context *DebosContext) error              // Main execution
    Cleanup(context *DebosContext) error          // Per-action cleanup
    PostMachine(context *DebosContext) error      // Post-execution processing
    PostMachineCleanup(context *DebosContext) error // Host-side cleanup
}
```

## Action System Deep Dive

### Action Types

debos provides 20+ built-in actions:

#### **System Construction Actions**
- `debootstrap`: Create base filesystem using debootstrap
- `mmdebstrap`: Alternative to debootstrap with better performance
- `pacstrap`: Arch Linux system construction
- `apt`: Package installation and management

#### **Filesystem Actions**
- `overlay`: Copy files/directories to target filesystem
- `filesystem-deploy`: Deploy rootfs to image partitions
- `unpack`: Extract archives to filesystem

#### **Image Creation Actions**
- `image-partition`: Create partitioned disk images
- `raw`: Write raw data to specific offsets
- `ostree-commit`: Create OSTree commits
- `ostree-deploy`: Deploy OSTree branches

#### **Utility Actions**
- `run`: Execute commands/scripts (chroot or host)
- `download`: Download files from network
- `pack`: Create compressed archives

### Action Implementation Pattern

Each action follows a consistent pattern:

```go
type ExampleAction struct {
    debos.BaseAction `yaml:",inline"`
    // Action-specific fields
    Parameter1 string `yaml:"parameter1"`
    Parameter2 bool   `yaml:"parameter2"`
}

func (a *ExampleAction) Verify(context *debos.DebosContext) error {
    // Validate parameters
    return nil
}

func (a *ExampleAction) PreMachine(context *debos.DebosContext, m *fakemachine.Machine, args *[]string) error {
    // Prepare fakemachine environment
    return nil
}

func (a *ExampleAction) PreNoMachine(context *debos.DebosContext) error {
    // Prepare host environment
    return nil
}

func (a *ExampleAction) Run(context *debos.DebosContext) error {
    // Execute action logic
    return nil
}

func (a *ExampleAction) Cleanup(context *debos.DebosContext) error {
    // Clean up action resources
    return nil
}

func (a *ExampleAction) PostMachine(context *debos.DebosContext) error {
    // Post-execution processing
    return nil
}

func (a *ExampleAction) PostMachineCleanup(context *debos.DebosContext) error {
    // Host-side cleanup
    return nil
}
```

### Action Lifecycle Management

#### **Execution Phases**
1. **Verify**: Validate action parameters and dependencies
2. **PreMachine/PreNoMachine**: Prepare execution environment
3. **Run**: Execute main action logic
4. **Cleanup**: Clean up action-specific resources
5. **PostMachine**: Post-execution processing
6. **PostMachineCleanup**: Host-side cleanup

#### **Resource Management**
- **Deferred Cleanup**: Ensures cleanup runs even on errors
- **Stacked Cleanup**: Multiple cleanup methods execute in reverse order
- **Context Persistence**: State maintained across action boundaries

### Key Actions Deep Dive

#### **debootstrap Action** (`actions/debootstrap_action.go`)

**Purpose**: Create base Debian filesystem
**Implementation**: Wraps `debootstrap` command with additional features

```go
type DebootstrapAction struct {
    Suite            string
    Mirror           string
    Variant          string
    Components       []string
    MergedUsr        bool `yaml:"merged-usr"`
    CheckGpg         bool `yaml:"check-gpg"`
}

func (d *DebootstrapAction) Run(context *debos.DebosContext) error {
    // Build debootstrap command
    cmd := debos.Command{}
    cmd.AddEnvKey("DEBOOTSTRAP_DIR", "/usr/share/debootstrap")
    
    // Execute with proper environment
    return cmd.Run("debootstrap", args...)
}
```

**External Tools Used**:
- `debootstrap`: Core system construction tool
- `gpg`: GPG signature verification
- `wget/curl`: Package download

#### **image-partition Action** (`actions/image_partition_action.go`)

**Purpose**: Create partitioned disk images
**Implementation**: Comprehensive disk partitioning and filesystem creation

```go
type ImagePartitionAction struct {
    ImageName    string
    ImageSize    string
    PartitionType string
    Partitions   []Partition
    MountPoints  []MountPoint
}

func (i *ImagePartitionAction) Run(context *debos.DebosContext) error {
    // Create image file
    // Partition using parted/sfdisk
    // Format filesystems
    // Mount partitions
    // Update context with partition info
}
```

**External Tools Used**:
- `parted`: GPT/MBR partition table creation
- `sfdisk`: Alternative partitioning tool
- `mkfs.*`: Filesystem formatting tools
- `mount/umount`: Partition mounting
- `losetup`: Loop device management

#### **run Action** (`actions/run_action.go`)

**Purpose**: Execute custom commands/scripts
**Implementation**: Flexible execution with chroot support

```go
type RunAction struct {
    Chroot      bool
    PostProcess bool
    Script      string
    Command     string
    Label       string
}

func (run *RunAction) doRun(context debos.DebosContext) error {
    var cmd debos.Command
    
    if run.Chroot {
        cmd = debos.NewChrootCommandForContext(context)
    }
    
    // Execute with proper environment variables
    return cmd.Run(label, cmdline...)
}
```

**External Tools Used**:
- `chroot`: Filesystem isolation
- `bash/sh`: Script execution
- `mount`: Bind mounts for script access

## Command Execution System

### Complete Command Execution Architecture

#### **Command Structure and Configuration**
```go
type Command struct {
    Architecture string            // Target architecture for cross-compilation
    Dir          string            // Working directory for command execution
    Chroot       string            // Chroot path for filesystem isolation
    ChrootMethod ChrootEnterMethod // Method to enter chroot environment
    
    bindMounts []string // Items to bind mount into chroot
    extraEnv   []string // Extra environment variables to set
}

type ChrootEnterMethod int

const (
    CHROOT_METHOD_NONE   = iota // No chroot isolation
    CHROOT_METHOD_NSPAWN        // Use systemd-nspawn for containerization
    CHROOT_METHOD_CHROOT        // Use traditional chroot
)
```

#### **Command Execution Flow**
```go
func (cmd Command) Run(label string, cmdline ...string) error {
    // 1. Setup QEMU helper for cross-architecture support
    q, err := newQemuHelper(cmd)
    if err != nil {
        return err
    }
    
    q.Setup()
    defer q.Cleanup()
    
    // 2. Build command options based on chroot method
    var options []string
    switch cmd.ChrootMethod {
    case CHROOT_METHOD_NONE:
        options = cmdline
    case CHROOT_METHOD_CHROOT:
        options = append(options, "chroot", cmd.Chroot)
        options = append(options, cmdline...)
    case CHROOT_METHOD_NSPAWN:
        options = buildNspawnCommand(cmd, cmdline)
    }
    
    // 3. Execute command with proper environment
    exe := exec.Command(options[0], options[1:]...)
    w := newCommandWrapper(label)
    
    exe.Stdout = w
    exe.Stderr = w
    
    // 4. Handle resolv.conf management
    resolvsum, err := cmd.saveResolvConf()
    if err != nil {
        return err
    }
    
    // 5. Execute and cleanup
    if err = exe.Run(); err != nil {
        return err
    }
    
    return cmd.restoreResolvConf(resolvsum)
}
```

#### **Chroot Method Implementation**

**Traditional chroot**:
```go
case CHROOT_METHOD_CHROOT:
    options = append(options, "chroot")
    options = append(options, cmd.Chroot)
    options = append(options, cmdline...)
```

**systemd-nspawn**:
```go
case CHROOT_METHOD_NSPAWN:
    options = append(options, "systemd-nspawn", "-q")
    options = append(options, "--resolv-conf=off")
    options = append(options, "--timezone=off")
    options = append(options, "--register=no")
    options = append(options, "--keep-unit")
    options = append(options, "--console=pipe")
    
    // Add environment variables
    for _, e := range cmd.extraEnv {
        options = append(options, "--setenv", e)
    }
    
    // Add bind mounts
    for _, b := range cmd.bindMounts {
        options = append(options, "--bind", b)
    }
    
    options = append(options, "-D", cmd.Chroot)
    options = append(options, cmdline...)
```

#### **Cross-Architecture Support**
```go
func newQemuHelper(c Command) (*qemuHelper, error) {
    q := qemuHelper{}
    
    if c.Chroot == "" || c.Architecture == "" {
        return &q, nil
    }
    
    // Map architectures to QEMU binaries
    switch c.Architecture {
    case "armhf", "armel", "arm":
        if runtime.GOARCH != "arm64" && runtime.GOARCH != "arm" {
            q.qemusrc = "/usr/bin/qemu-arm-static"
        }
    case "arm64":
        if runtime.GOARCH != "arm64" {
            q.qemusrc = "/usr/bin/qemu-aarch64-static"
        }
    case "i386":
        if runtime.GOARCH != "amd64" && runtime.GOARCH != "386" {
            q.qemusrc = "/usr/bin/qemu-i386-static"
        }
    case "amd64":
        if runtime.GOARCH != "amd64" {
            q.qemusrc = "/usr/bin/qemu-x86_64-static"
        }
    // ... more architectures
    }
    
    if q.qemusrc != "" {
        q.qemutarget = path.Join(c.Chroot, q.qemusrc)
    }
    
    return &q, nil
}
```

#### **Environment Variable Management**
```go
func (cmd *Command) AddEnv(env string) {
    cmd.extraEnv = append(cmd.extraEnv, env)
}

func (cmd *Command) AddEnvKey(key, value string) {
    cmd.extraEnv = append(cmd.extraEnv, fmt.Sprintf("%s=%s", key, value))
}

func (cmd *Command) AddBindMount(source, target string) {
    var mount string
    if target != "" {
        mount = fmt.Sprintf("%s:%s", source, target)
    } else {
        mount = source
    }
    cmd.bindMounts = append(cmd.bindMounts, mount)
}
```

#### **Service Management in Chroot**
```go
// Disable services start/stop for commands running in chroot
if cmd.ChrootMethod != CHROOT_METHOD_NONE {
    services := ServiceHelper{cmd.Chroot}
    services.Deny()
    defer services.Allow()
}
```

#### **resolv.conf Handling**
```go
func (cmd *Command) saveResolvConf() (*[sha256.Size]byte, error) {
    hostconf := "/etc/resolv.conf"
    chrootedconf := path.Join(cmd.Chroot, hostconf)
    savedconf := chrootedconf + ".debos"
    
    // Save original resolv.conf
    if _, err := os.Lstat(chrootedconf); !os.IsNotExist(err) {
        if err = os.Rename(chrootedconf, savedconf); err != nil {
            return nil, err
        }
    }
    
    // Copy host resolv.conf to chroot
    data, err := ioutil.ReadFile(hostconf)
    if err != nil {
        return nil, err
    }
    
    out := []byte("# Automatically generated by Debos\n")
    out = append(out, data...)
    
    sum := sha256.Sum256(out)
    err = ioutil.WriteFile(chrootedconf, out, 0644)
    
    return &sum, err
}
```

### Command Output Handling

#### **Output Wrapper System**
```go
type commandWrapper struct {
    label  string
    buffer *bytes.Buffer
}

func newCommandWrapper(label string) *commandWrapper {
    b := bytes.Buffer{}
    return &commandWrapper{label, &b}
}

func (w commandWrapper) out(atEOF bool) {
    for {
        s, err := w.buffer.ReadString('\n')
        if err == nil {
            log.Printf("%s | %v", w.label, s)
        } else {
            if len(s) > 0 {
                if atEOF && err == io.EOF {
                    log.Printf("%s | %v\n", w.label, s)
                } else {
                    w.buffer.WriteString(s)
                }
            }
            break
        }
    }
}

func (w *commandWrapper) Write(p []byte) (n int, err error) {
    n, err = w.buffer.Write(p)
    w.out(false)
    return
}
```

## Fakemachine Integration

### Purpose and Benefits

Fakemachine provides:
- **Isolation**: Build environment independent of host
- **Reproducibility**: Consistent results across different hosts
- **Safety**: Prevents host system contamination
- **Performance**: Optimized for build workloads

### Backend Selection

```go
// Automatic backend selection
m, err = fakemachine.NewMachineWithBackend("auto")

// Manual backend selection
m, err = fakemachine.NewMachineWithBackend("kvm")  // Fastest
m, err = fakemachine.NewMachineWithBackend("uml")  // Medium
m, err = fakemachine.NewMachineWithBackend("qemu") // Slowest
```

### Backend Performance Comparison

| Backend | Performance | Requirements | Use Case |
|---------|-------------|--------------|----------|
| `kvm` | ⭐⭐⭐⭐⭐ | `/dev/kvm` access | Production builds |
| `uml` | ⭐⭐⭐ | `user-mode-linux` package | Development/testing |
| `qemu` | ⭐⭐ | None | Fallback, compatibility |
| `--disable-fakemachine` | ⭐⭐⭐⭐ | Root permissions | Debugging, simple builds |

### Machine Configuration

```go
// Memory configuration
memsize, err := units.RAMInBytes(options.Memory)
m.SetMemory(int(memsize / 1024 / 1024))

// CPU configuration
m.SetNumCPUs(options.CPUs)

// Scratch space
m.SetScratch(size, "")

// Environment variables
m.SetEnviron(EnvironString)
```

## Filesystem Operations

### Complete Filesystem Management System

#### **Path Management Functions**
```go
func CleanPathAt(path, at string) string {
    if filepath.IsAbs(path) {
        return filepath.Clean(path)
    }
    return filepath.Join(at, path)
}

func CleanPath(path string) string {
    cwd, _ := os.Getwd()
    return CleanPathAt(path, cwd)
}

func RealPath(path string) (string, error) {
    p, err := filepath.EvalSymlinks(path)
    if err != nil {
        return "", err
    }
    return filepath.Abs(p)
}

func RestrictedPath(prefix, dest string) (string, error) {
    var err error
    destination := path.Join(prefix, dest)
    destination, err = filepath.Abs(destination)
    if err != nil {
        return "", err
    }
    if !strings.HasPrefix(destination, prefix) {
        return "", fmt.Errorf("The resulting path points outside of prefix '%s': '%s'\n", prefix, destination)
    }
    return destination, nil
}
```

#### **File Copying Operations**
```go
func CopyFile(src, dst string, mode os.FileMode) error {
    in, err := os.Open(src)
    if err != nil {
        return err
    }
    defer in.Close()
    
    // Create temporary file
    tmp, err := ioutil.TempFile(filepath.Dir(dst), "")
    if err != nil {
        return err
    }
    
    // Copy content
    _, err = io.Copy(tmp, in)
    if err != nil {
        tmp.Close()
        os.Remove(tmp.Name())
        return err
    }
    
    // Set permissions and close
    if err = tmp.Close(); err != nil {
        os.Remove(tmp.Name())
        return err
    }
    if err = os.Chmod(tmp.Name(), mode); err != nil {
        os.Remove(tmp.Name())
        return err
    }
    
    // Atomic rename
    if err = os.Rename(tmp.Name(), dst); err != nil {
        os.Remove(tmp.Name())
        return err
    }
    
    return nil
}
```

#### **Tree Copying Operations**
```go
func CopyTree(sourcetree, desttree string) error {
    walker := func(p string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }
        
        suffix, _ := filepath.Rel(sourcetree, p)
        target := path.Join(desttree, suffix)
        
        switch info.Mode() & os.ModeType {
        case 0: // Regular file
            err := CopyFile(p, target, info.Mode())
            if err != nil {
                return fmt.Errorf("Failed to copy file %s: %w", p, err)
            }
        case os.ModeDir: // Directory
            os.Mkdir(target, info.Mode())
        case os.ModeSymlink: // Symlink
            link, err := os.Readlink(p)
            if err != nil {
                return fmt.Errorf("Failed to read symlink %s: %w", suffix, err)
            }
            os.Symlink(link, target)
        default:
            return fmt.Errorf("File %s with mode %v not handled", p, info.Mode())
        }
        
        return nil
    }
    
    return filepath.Walk(sourcetree, walker)
}
```

#### **Filesystem Safety Features**
- **Atomic Operations**: Use temporary files and atomic renames
- **Path Validation**: Prevent directory traversal attacks
- **Permission Preservation**: Maintain original file modes
- **Symlink Handling**: Properly handle symbolic links
- **Error Recovery**: Clean up temporary files on failure

## Archive Management

### Complete Archive Processing System

#### **Archive Type Support**
```go
type ArchiveType int

const (
    _ ArchiveType = iota // Guess from file extension
    Tar
    Zip
    Deb
)

type ArchiveBase struct {
    file    string // Path to archive file
    atype   ArchiveType
    options map[interface{}]interface{} // Archiver-specific options
}
```

#### **Archive Unpacking Operations**
```go
func (tar *ArchiveTar) Unpack(destination string) error {
    command := []string{"tar"}
    usePigz := false
    
    // Determine compression type and options
    if compression, ok := tar.options["compression"].(string); ok {
        if compression == "gz" {
            // Check if pigz is available for parallel decompression
            if _, err := exec.LookPath("pigz"); err == nil {
                usePigz = true
                command = append(command, "--use-compress-program=pigz")
            } else {
                command = append(command, "--gzip")
            }
        } else if opts := tarOptions(compression); opts != "" {
            command = append(command, opts)
        }
    }
    
    command = append(command, "-xf", tar.file, "-C", destination)
    
    return unpack(command, destination)
}
```

#### **Compression Support**
```go
func tarOptions(compression string) string {
    unpackTarOpts := map[string]string{
        "bzip2": "--bzip2",
        "gz":    "--gzip",
        "lzip":  "--lzip",
        "lzma":  "--lzma",
        "lzop":  "--lzop",
        "xz":    "--xz",
        "zstd":  "--zstd",
    }
    return unpackTarOpts[compression]
}
```

#### **Archive Management Features**
- **Multiple Formats**: Support for tar, zip, and deb packages
- **Compression Detection**: Automatic compression type detection
- **Parallel Processing**: Use pigz for faster gzip decompression
- **Error Handling**: Robust error handling and cleanup
- **Option Support**: Configurable unpacking options

## External Tools and Dependencies

### Core System Tools

#### **Filesystem Management**
- `debootstrap`: Base system construction
- `mmdebstrap`: Alternative system construction
- `parted`: Partition table management
- `sfdisk`: Scriptable partitioning
- `mkfs.ext4`, `mkfs.xfs`, `mkfs.fat`: Filesystem creation
- `mount`, `umount`: Filesystem mounting
- `losetup`: Loop device management

#### **Package Management**
- `apt-get`, `apt`: Debian package management
- `pacman`: Arch Linux package management
- `pacstrap`: Arch Linux system construction

#### **Compression and Archiving**
- `tar`: Archive creation and extraction
- `gzip`, `bzip2`, `xz`: Compression
- `cpio`: Archive format support

#### **Network Tools**
- `wget`, `curl`: File download
- `rsync`: Synchronization

### Build System Dependencies

#### **Go Dependencies** (`go.mod`)
```go
require (
    github.com/go-debos/fakemachine v0.0.11    // VM backend
    github.com/docker/go-units v0.5.0           // Human-readable sizes
    github.com/freddierice/go-losetup/v2 v2.0.1 // Loop device management
    github.com/sjoerdsimons/ostree-go v0.0.0    // OSTree support
    gopkg.in/yaml.v2 v2.4.0                     // YAML parsing
)
```

#### **System Dependencies**
- `golang`: Go runtime and compiler
- `libglib2.0-dev`: GLib development files
- `libostree-dev`: OSTree development files
- `qemu-system-x86`: QEMU emulation
- `qemu-user-static`: User-mode QEMU
- `debootstrap`: Base system construction
- `systemd-container`: Container support

### External Tool Integration Points

#### **Command Execution**
```go
type Command struct {
    // Command configuration
}

func (c *Command) Run(label string, args ...string) error {
    // Execute with proper environment
    // Handle output and errors
    // Support chroot execution
}
```

#### **Chroot Support**
```go
func NewChrootCommandForContext(context DebosContext) Command {
    cmd := Command{}
    // Add bind mounts
    // Set working directory
    // Configure environment
    return cmd
}
```

## Recipe Processing

### YAML + Go Template System

Recipes support dynamic content through Go templates:

```yaml
{{- $image := or .image "debian.tgz" -}}
{{- $suite := or .suite "bookworm" -}}

architecture: {{ .architecture }}

actions:
  - action: debootstrap
    suite: {{ $suite }}
    mirror: https://deb.debian.org/debian
```

### Template Variable Sources

1. **Command Line**: `-t variable:value`
2. **Environment**: System environment variables
3. **Built-in**: Architecture, suite defaults
4. **Recipe**: Local variable definitions

### Recipe Validation

```go
func (r *Recipe) Parse(file string, printRecipe, verbose bool, templateVars map[string]string) error {
    // Load and parse YAML
    // Apply Go templates
    // Validate action parameters
    // Check dependencies
}
```

## Context Management

### DebosContext Structure

```go
type DebosContext struct {
    *CommonContext
    RecipeDir       string
    Architecture    string
    SectorSize      int
}

type CommonContext struct {
    Scratchdir      string        // Temporary working directory
    Rootdir         string        // Target filesystem root
    Artifactdir     string        // Output directory
    Downloaddir     string        // Download cache
    Image           string        // Current image file
    ImagePartitions []Partition   // Partition information
    ImageMntDir     string        // Image mount point
    ImageFSTab      bytes.Buffer  // Generated fstab
    ImageKernelRoot string        // Kernel root parameter
    DebugShell      string        // Debug shell path
    Origins         map[string]string // Path mappings
    State           DebosState    // Build state
    EnvironVars     map[string]string // Environment variables
    PrintRecipe     bool          // Print final recipe
    Verbose         bool          // Verbose output
}
```

### Origin System

The origin system provides path mapping for actions:

```go
context.Origins = map[string]string{
    "artifacts":   context.Artifactdir,    // Output directory
    "filesystem":  context.Rootdir,        // Target filesystem
    "recipe":      context.RecipeDir,      // Recipe directory
}
```

Actions can reference these origins:
```yaml
- action: overlay
  source: overlays/config
  destination: {{ .origin.filesystem }}/etc/config
```

## Error Handling and Debugging

### Error States

```go
type DebosState int

const (
    Success DebosState = iota
    Failed
)
```

### Debug Shell Integration

```go
if options.DebugShell {
    context.DebugShell = options.Shell
    // Fall into interactive shell on error
}
```

### Error Handling Flow

```go
func handleError(context *debos.DebosContext, err error, a debos.Action, stage string) bool {
    if err == nil {
        return false
    }
    
    context.State = debos.Failed
    log.Printf("Action `%s` failed at stage %s, error: %s", a, stage, err)
    
    if context.DebugShell != "" {
        debos.DebugShell(*context)
    }
    
    return true
}
```

### Cleanup Mechanisms

Actions implement cleanup hooks:
- **Cleanup**: Per-action cleanup in same environment
- **PostMachineCleanup**: Host-side cleanup for all actions
- **Deferred execution**: Ensures cleanup runs even on errors

## Performance Characteristics

### Build Time Benchmarks

Based on [pine-a64-plus/debian.yaml](https://github.com/go-debos/debos-recipes/blob/9a25b4be6c9136f4a27e542f39ab7e419fc852c9/pine-a64-plus/debian.yaml) on Intel Pentium G4560T with SSD:

| Backend | Wall Time | Prerequisites |
|---------|-----------|---------------|
| `--disable-fakemachine` | 8 min | Root permissions |
| `-b kvm` | 9 min | Access to `/dev/kvm` |
| `-b uml` | 18 min | `user-mode-linux` package |
| `-b qemu` | 166 min | None |

### Performance Optimization

#### **Scratch Space Management**
```go
if options.ScratchSize != "" {
    size, err := units.FromHumanSize(options.ScratchSize)
    scratchsizeMB := int(size / 1000 / 1000)
    m.SetScratch(size, "")
}
```

#### **Memory Configuration**
```go
if memsizeMB < 256 {
    log.Printf("WARNING: Memory size of %dMB is less than recommended minimum 256MB\n", memsizeMB)
}
```

#### **CPU Allocation**
```go
if options.CPUs == 0 {
    options.CPUs = 2  // Default to 2 CPUs
}
m.SetNumCPUs(options.CPUs)
```

## Integration Points

### Container Integration

#### **Docker Support**
```bash
# Official container
docker pull godebos/debos

# Usage
docker run --rm --privileged -v $(pwd):/workspace godebos/debos recipe.yaml
```

#### **Systemd-nspawn**
Fakemachine can use systemd-nspawn for containerization.

### CI/CD Integration

#### **Environment Variables**
```bash
# Proxy support
http_proxy, https_proxy, ftp_proxy, rsync_proxy, all_proxy, no_proxy

# Custom variables
debos -e BUILD_ID:123 -e BRANCH:main recipe.yaml
```

#### **Artifact Management**
```bash
# Output directory
debos --artifactdir ./outputs recipe.yaml

# Export specific artifacts
debos --export image --export archive recipe.yaml
```

### External Tool Integration

#### **OSTree Support**
```yaml
- action: ostree-commit
  repository: {{ .origin.artifacts }}/ostree
  parent: {{ .parent_commit }}

- action: ostree-deploy
  repository: {{ .origin.artifacts }}/ostree
  branch: {{ .branch }}
```

#### **Bootloader Integration**
debos doesn't have built-in bootloader actions but provides infrastructure:
```yaml
- action: run
  chroot: true
  script: scripts/install-grub.sh

- action: run
  chroot: true
  script: scripts/configure-bootloader.sh
```

## Advanced Features

### Multi-Architecture Support

```yaml
architecture: arm64

actions:
  - action: debootstrap
    suite: bookworm
    # Architecture-specific packages
    packages: [firmware-linux]
```

### Custom Action Development

#### **Action Interface Implementation**
```go
type CustomAction struct {
    debos.BaseAction `yaml:",inline"`
    CustomField string `yaml:"custom-field"`
}

func (a *CustomAction) Verify(context *debos.DebosContext) error {
    // Validation logic
    return nil
}

func (a *CustomAction) Run(context *debos.DebosContext) error {
    // Execution logic
    return nil
}
```

#### **Integration**
```yaml
- action: custom
  custom-field: value
```

### Network and Proxy Support

#### **Automatic Proxy Detection**
```go
var environ_vars = [...]string {
    "http_proxy", "https_proxy", "ftp_proxy", 
    "rsync_proxy", "all_proxy", "no_proxy",
}
```

#### **Proxy Validation**
```go
func warnLocalhost(variable string, value string) {
    if strings.Contains(value, "localhost") ||
       strings.Contains(value, "127.0.0.1") ||
       strings.Contains(value, "::1") {
        log.Printf("WARNING: Environment variable %s contains localhost", variable)
    }
}
```

## Conclusion

debos provides a powerful, flexible framework for creating Debian-based operating system images. Its architecture emphasizes:

1. **Reproducibility**: Consistent results through isolated execution
2. **Flexibility**: Customizable through actions and scripts
3. **Performance**: Optimized backends for different use cases
4. **Integration**: Easy integration with existing toolchains
5. **Maintainability**: Clear separation of concerns and lifecycle hooks

The system's strength lies in its ability to combine declarative recipe definitions with imperative script execution, making it suitable for both simple image creation and complex, multi-stage build processes.

### Key Strengths

- **Isolation**: Fakemachine ensures build reproducibility
- **Extensibility**: Action system allows custom functionality
- **Performance**: Multiple backend options for different scenarios
- **Integration**: Works with existing Debian ecosystem tools
- **Cross-Architecture**: Full support for ARM, MIPS, RISC-V, etc.
- **Template System**: Advanced Go templating with custom functions
- **Command Execution**: Robust chroot and cross-architecture support
- **Filesystem Safety**: Atomic operations and path validation

### Areas for Enhancement

- **Bootloader Support**: No built-in bootloader management
- **Package Management**: Limited to Debian/Arch package managers
- **Image Formats**: Focus on filesystem and archive outputs
- **Validation**: Limited manifest validation capabilities
- **Parallel Execution**: Actions execute sequentially
- **Dependency Resolution**: No automatic dependency ordering

### Complete Process Summary

debos implements a **complete end-to-end image building pipeline** that:

1. **Processes Recipes**: YAML + Go templates with validation
2. **Manages Actions**: 20+ built-in actions with lifecycle hooks
3. **Executes Commands**: Robust chroot and cross-architecture support
4. **Handles Filesystems**: Safe file operations with atomic guarantees
5. **Manages Archives**: Multiple format support with compression
6. **Provides Isolation**: Fakemachine VM backends for reproducibility
7. **Supports Templates**: Advanced variable substitution and functions
8. **Generates Artifacts**: Images, archives, and OSTree repositories

The system's architecture emphasizes **reproducibility**, **flexibility**, and **integration** while maintaining **performance** and **security** through isolated execution environments.