deb-osbuild/docs/debos.md

debos Comprehensive Top-to-Bottom Analysis

Overview

debos is a Debian OS image builder that creates reproducible, customized operating system images through a YAML-based recipe system. It operates by executing a series of actions in a controlled virtual machine environment, ensuring consistency across different host systems. This document provides a complete top-to-bottom analysis of the entire debos process, from recipe parsing to final artifact generation.

Table of Contents

1. Complete Process Flow
2. Core Architecture
3. Recipe Processing Pipeline
4. Action System Deep Dive
5. Fakemachine Integration
6. Command Execution System
7. Filesystem Operations
8. Archive Management
9. External Tools and Dependencies
10. Context Management
11. Error Handling and Debugging
12. Performance Characteristics
13. Integration Points
14. Complete Workflow Examples

Complete Process Flow

End-to-End Workflow

1. Recipe Loading → 2. Template Processing → 3. YAML Parsing → 4. Action Validation → 5. Fakemachine Setup → 6. Action Execution → 7. Artifact Generation → 8. Cleanup

Detailed Process Steps

Phase 1: Recipe Preparation

1. File Loading: Read YAML recipe file from disk
2. Template Processing: Apply Go templates with variables
3. YAML Parsing: Parse processed YAML into Recipe struct
4. Action Mapping: Map YAML actions to concrete Action implementations
5. Validation: Verify all action parameters and dependencies

Phase 2: Environment Setup

1. Fakemachine Creation: Initialize virtual machine backend
2. Resource Allocation: Set CPU, memory, scratch space
3. Volume Mounting: Mount recipe directory, artifact directory
4. Environment Variables: Propagate proxy and custom variables
5. Architecture Setup: Configure cross-architecture support

Phase 3: Action Execution

1. Pre-Machine Setup: Run PreMachine hooks for all actions
2. Sequential Execution: Execute actions in recipe order
3. Chroot Management: Handle filesystem isolation
4. Command Execution: Run external tools and scripts
5. State Management: Maintain context across actions

Phase 4: Artifact Generation

1. Output Collection: Gather results from all actions
2. Post-Processing: Run PostMachine hooks
3. File Generation: Create final artifacts (images, archives)
4. Cleanup: Remove temporary files and mounts

Core Architecture

Design Philosophy

debos follows a pipeline-based architecture where:

• Recipes define the build process as a sequence of actions
• Actions are self-contained, independent modules
• Fakemachine provides isolated execution environment
• Context maintains state across the entire build process

Key Components

debos CLI → Recipe Parser → Action Executor → Fakemachine → Output
     ↓              ↓              ↓              ↓         ↓
  Main Entry   YAML/YAML+Go   Action Runner   VM Backend  Artifacts

Recipe Processing Pipeline

Complete Recipe Processing Flow

1. Template Processing Engine

func (r *Recipe) Parse(file string, printRecipe bool, dump bool, templateVars ...map[string]string) error {
    t := template.New(path.Base(file))
    funcs := template.FuncMap{
        "sector": sector,      // Add 's' suffix for sector calculations
        "escape": escape,      // Shell escape variables
        "uuid5": uuid5,        // Generate deterministic UUIDs
    }
    t.Funcs(funcs)
    
    // Add slim-sprig functions for advanced templating
    t.Funcs(sprig.FuncMap())
    
    // Parse and execute template
    if _, err := t.ParseFiles(file); err != nil {
        return err
    }
    
    data := new(bytes.Buffer)
    if err := t.Execute(data, templateVars[0]); err != nil {
        return err
    }
    
    // Unmarshal processed YAML
    if err := yaml.Unmarshal(data.Bytes(), &r); err != nil {
        return err
    }
    
    return nil
}

2. Action Factory System

func (y *YamlAction) UnmarshalYAML(unmarshal func(interface{}) error) error {
    var aux debos.BaseAction
    
    err := unmarshal(&aux)
    if err != nil {
        return err
    }
    
    // Factory pattern for action creation
    switch aux.Action {
    case "debootstrap":
        y.Action = NewDebootstrapAction()
    case "mmdebstrap":
        y.Action = NewMmdebstrapAction()
    case "apt":
        y.Action = NewAptAction()
    case "run":
        y.Action = &RunAction{}
    case "image-partition":
        y.Action = &ImagePartitionAction{}
    case "overlay":
        y.Action = &OverlayAction{}
    case "pack":
        y.Action = NewPackAction()
    // ... more actions
    default:
        return fmt.Errorf("Unknown action: %v", aux.Action)
    }
    
    // Unmarshal action-specific options
    err = unmarshal(y.Action)
    return err
}

3. Template Functions Available

• sector(s int) string: Adds 's' suffix for sector calculations
• escape(s string) string: Shell-escapes variables for safe command execution
• uuid5(namespace, data) string: Generates deterministic UUIDs using SHA1
• slim-sprig functions: Advanced string manipulation, math, crypto functions

4. Recipe Validation Process

func (r *Recipe) Parse(file string, printRecipe bool, dump bool, templateVars ...map[string]string) error {
    // ... template processing ...
    
    // Mandatory field validation
    if len(r.Architecture) == 0 {
        return fmt.Errorf("Recipe file must have 'architecture' property")
    }
    
    if len(r.Actions) == 0 {
        return fmt.Errorf("Recipe file must have at least one action")
    }
    
    // Set defaults
    if r.SectorSize == 0 {
        r.SectorSize = 512
    }
    
    return nil
}

Recipe Structure and Syntax

Basic Recipe Format

{{- $image := or .image "debian.tgz" -}}
{{- $suite := or .suite "bookworm" -}}

architecture: {{ .architecture }}
sectorsize: 512

actions:
  - action: debootstrap
    suite: {{ $suite }}
    mirror: https://deb.debian.org/debian
    variant: minbase
    
  - action: apt
    packages: [ sudo, openssh-server, systemd-sysv ]
    
  - action: run
    chroot: true
    script: scripts/setup-system.sh
    
  - action: pack
    file: {{ $image }}
    compression: gz

Template Variable Sources

1. Command Line: debos -t variable:value recipe.yaml
2. Environment: System environment variables
3. Built-in: Architecture, suite defaults
4. Recipe: Local variable definitions with {{- $var := "value" -}}

Execution Flow

1. CLI Entry Point (cmd/debos/debos.go)

The main entry point handles:

• Command-line argument parsing
• Environment variable propagation
• Fakemachine backend selection
• Recipe file validation

func main() {
    context := debos.DebosContext{...}
    
    // Parse command line options
    parser := flags.NewParser(&options, flags.Default)
    
    // Create fakemachine or run on host
    if !options.DisableFakeMachine {
        m, err = fakemachine.NewMachineWithBackend(options.Backend)
        // Execute in VM
    } else {
        // Execute on host
    }
}

2. Recipe Parsing (actions/recipe.go)

Recipes are parsed with Go template support:

• YAML parsing with gopkg.in/yaml.v2
• Go template expansion for dynamic values
• Variable substitution with -t command line options
• Validation of action parameters

3. Action Execution Pipeline

Actions are executed sequentially with lifecycle hooks:

type Action interface {
    Verify(context *DebosContext) error           // Pre-execution validation
    PreMachine(context *DebosContext, m *fakemachine.Machine, args *[]string) error
    PreNoMachine(context *DebosContext) error     // Host-side preparation
    Run(context *DebosContext) error              // Main execution
    Cleanup(context *DebosContext) error          // Per-action cleanup
    PostMachine(context *DebosContext) error      // Post-execution processing
    PostMachineCleanup(context *DebosContext) error // Host-side cleanup
}

Action System Deep Dive

Action Types

debos provides 20+ built-in actions:

System Construction Actions

• debootstrap: Create base filesystem using debootstrap
• mmdebstrap: Alternative to debootstrap with better performance
• pacstrap: Arch Linux system construction
• apt: Package installation and management

Filesystem Actions

• overlay: Copy files/directories to target filesystem
• filesystem-deploy: Deploy rootfs to image partitions
• unpack: Extract archives to filesystem

Image Creation Actions

• image-partition: Create partitioned disk images
• raw: Write raw data to specific offsets
• ostree-commit: Create OSTree commits
• ostree-deploy: Deploy OSTree branches

Utility Actions

• run: Execute commands/scripts (chroot or host)
• download: Download files from network
• pack: Create compressed archives

Action Implementation Pattern

Each action follows a consistent pattern:

type ExampleAction struct {
    debos.BaseAction `yaml:",inline"`
    // Action-specific fields
    Parameter1 string `yaml:"parameter1"`
    Parameter2 bool   `yaml:"parameter2"`
}

func (a *ExampleAction) Verify(context *debos.DebosContext) error {
    // Validate parameters
    return nil
}

func (a *ExampleAction) PreMachine(context *debos.DebosContext, m *fakemachine.Machine, args *[]string) error {
    // Prepare fakemachine environment
    return nil
}

func (a *ExampleAction) PreNoMachine(context *debos.DebosContext) error {
    // Prepare host environment
    return nil
}

func (a *ExampleAction) Run(context *debos.DebosContext) error {
    // Execute action logic
    return nil
}

func (a *ExampleAction) Cleanup(context *debos.DebosContext) error {
    // Clean up action resources
    return nil
}

func (a *ExampleAction) PostMachine(context *debos.DebosContext) error {
    // Post-execution processing
    return nil
}

func (a *ExampleAction) PostMachineCleanup(context *debos.DebosContext) error {
    // Host-side cleanup
    return nil
}

Action Lifecycle Management

Execution Phases

1. Verify: Validate action parameters and dependencies
2. PreMachine/PreNoMachine: Prepare execution environment
3. Run: Execute main action logic
4. Cleanup: Clean up action-specific resources
5. PostMachine: Post-execution processing
6. PostMachineCleanup: Host-side cleanup

Resource Management

• Deferred Cleanup: Ensures cleanup runs even on errors
• Stacked Cleanup: Multiple cleanup methods execute in reverse order
• Context Persistence: State maintained across action boundaries

Key Actions Deep Dive

debootstrap Action (actions/debootstrap_action.go)

Purpose: Create base Debian filesystem Implementation: Wraps debootstrap command with additional features

type DebootstrapAction struct {
    Suite            string
    Mirror           string
    Variant          string
    Components       []string
    MergedUsr        bool `yaml:"merged-usr"`
    CheckGpg         bool `yaml:"check-gpg"`
}

func (d *DebootstrapAction) Run(context *debos.DebosContext) error {
    // Build debootstrap command
    cmd := debos.Command{}
    cmd.AddEnvKey("DEBOOTSTRAP_DIR", "/usr/share/debootstrap")
    
    // Execute with proper environment
    return cmd.Run("debootstrap", args...)
}

External Tools Used:

• debootstrap: Core system construction tool
• gpg: GPG signature verification
• wget/curl: Package download

image-partition Action (actions/image_partition_action.go)

Purpose: Create partitioned disk images Implementation: Comprehensive disk partitioning and filesystem creation

type ImagePartitionAction struct {
    ImageName    string
    ImageSize    string
    PartitionType string
    Partitions   []Partition
    MountPoints  []MountPoint
}

func (i *ImagePartitionAction) Run(context *debos.DebosContext) error {
    // Create image file
    // Partition using parted/sfdisk
    // Format filesystems
    // Mount partitions
    // Update context with partition info
}

External Tools Used:

• parted: GPT/MBR partition table creation
• sfdisk: Alternative partitioning tool
• mkfs.*: Filesystem formatting tools
• mount/umount: Partition mounting
• losetup: Loop device management

run Action (actions/run_action.go)

Purpose: Execute custom commands/scripts Implementation: Flexible execution with chroot support

type RunAction struct {
    Chroot      bool
    PostProcess bool
    Script      string
    Command     string
    Label       string
}

func (run *RunAction) doRun(context debos.DebosContext) error {
    var cmd debos.Command
    
    if run.Chroot {
        cmd = debos.NewChrootCommandForContext(context)
    }
    
    // Execute with proper environment variables
    return cmd.Run(label, cmdline...)
}

External Tools Used:

• chroot: Filesystem isolation
• bash/sh: Script execution
• mount: Bind mounts for script access

Command Execution System

Complete Command Execution Architecture

Command Structure and Configuration

type Command struct {
    Architecture string            // Target architecture for cross-compilation
    Dir          string            // Working directory for command execution
    Chroot       string            // Chroot path for filesystem isolation
    ChrootMethod ChrootEnterMethod // Method to enter chroot environment
    
    bindMounts []string // Items to bind mount into chroot
    extraEnv   []string // Extra environment variables to set
}

type ChrootEnterMethod int

const (
    CHROOT_METHOD_NONE   = iota // No chroot isolation
    CHROOT_METHOD_NSPAWN        // Use systemd-nspawn for containerization
    CHROOT_METHOD_CHROOT        // Use traditional chroot
)

Command Execution Flow

func (cmd Command) Run(label string, cmdline ...string) error {
    // 1. Setup QEMU helper for cross-architecture support
    q, err := newQemuHelper(cmd)
    if err != nil {
        return err
    }
    
    q.Setup()
    defer q.Cleanup()
    
    // 2. Build command options based on chroot method
    var options []string
    switch cmd.ChrootMethod {
    case CHROOT_METHOD_NONE:
        options = cmdline
    case CHROOT_METHOD_CHROOT:
        options = append(options, "chroot", cmd.Chroot)
        options = append(options, cmdline...)
    case CHROOT_METHOD_NSPAWN:
        options = buildNspawnCommand(cmd, cmdline)
    }
    
    // 3. Execute command with proper environment
    exe := exec.Command(options[0], options[1:]...)
    w := newCommandWrapper(label)
    
    exe.Stdout = w
    exe.Stderr = w
    
    // 4. Handle resolv.conf management
    resolvsum, err := cmd.saveResolvConf()
    if err != nil {
        return err
    }
    
    // 5. Execute and cleanup
    if err = exe.Run(); err != nil {
        return err
    }
    
    return cmd.restoreResolvConf(resolvsum)
}

Chroot Method Implementation

Traditional chroot:

case CHROOT_METHOD_CHROOT:
    options = append(options, "chroot")
    options = append(options, cmd.Chroot)
    options = append(options, cmdline...)

systemd-nspawn:

case CHROOT_METHOD_NSPAWN:
    options = append(options, "systemd-nspawn", "-q")
    options = append(options, "--resolv-conf=off")
    options = append(options, "--timezone=off")
    options = append(options, "--register=no")
    options = append(options, "--keep-unit")
    options = append(options, "--console=pipe")
    
    // Add environment variables
    for _, e := range cmd.extraEnv {
        options = append(options, "--setenv", e)
    }
    
    // Add bind mounts
    for _, b := range cmd.bindMounts {
        options = append(options, "--bind", b)
    }
    
    options = append(options, "-D", cmd.Chroot)
    options = append(options, cmdline...)

Cross-Architecture Support

func newQemuHelper(c Command) (*qemuHelper, error) {
    q := qemuHelper{}
    
    if c.Chroot == "" || c.Architecture == "" {
        return &q, nil
    }
    
    // Map architectures to QEMU binaries
    switch c.Architecture {
    case "armhf", "armel", "arm":
        if runtime.GOARCH != "arm64" && runtime.GOARCH != "arm" {
            q.qemusrc = "/usr/bin/qemu-arm-static"
        }
    case "arm64":
        if runtime.GOARCH != "arm64" {
            q.qemusrc = "/usr/bin/qemu-aarch64-static"
        }
    case "i386":
        if runtime.GOARCH != "amd64" && runtime.GOARCH != "386" {
            q.qemusrc = "/usr/bin/qemu-i386-static"
        }
    case "amd64":
        if runtime.GOARCH != "amd64" {
            q.qemusrc = "/usr/bin/qemu-x86_64-static"
        }
    // ... more architectures
    }
    
    if q.qemusrc != "" {
        q.qemutarget = path.Join(c.Chroot, q.qemusrc)
    }
    
    return &q, nil
}

Environment Variable Management

func (cmd *Command) AddEnv(env string) {
    cmd.extraEnv = append(cmd.extraEnv, env)
}

func (cmd *Command) AddEnvKey(key, value string) {
    cmd.extraEnv = append(cmd.extraEnv, fmt.Sprintf("%s=%s", key, value))
}

func (cmd *Command) AddBindMount(source, target string) {
    var mount string
    if target != "" {
        mount = fmt.Sprintf("%s:%s", source, target)
    } else {
        mount = source
    }
    cmd.bindMounts = append(cmd.bindMounts, mount)
}

Service Management in Chroot

// Disable services start/stop for commands running in chroot
if cmd.ChrootMethod != CHROOT_METHOD_NONE {
    services := ServiceHelper{cmd.Chroot}
    services.Deny()
    defer services.Allow()
}

resolv.conf Handling

func (cmd *Command) saveResolvConf() (*[sha256.Size]byte, error) {
    hostconf := "/etc/resolv.conf"
    chrootedconf := path.Join(cmd.Chroot, hostconf)
    savedconf := chrootedconf + ".debos"
    
    // Save original resolv.conf
    if _, err := os.Lstat(chrootedconf); !os.IsNotExist(err) {
        if err = os.Rename(chrootedconf, savedconf); err != nil {
            return nil, err
        }
    }
    
    // Copy host resolv.conf to chroot
    data, err := ioutil.ReadFile(hostconf)
    if err != nil {
        return nil, err
    }
    
    out := []byte("# Automatically generated by Debos\n")
    out = append(out, data...)
    
    sum := sha256.Sum256(out)
    err = ioutil.WriteFile(chrootedconf, out, 0644)
    
    return &sum, err
}

Command Output Handling

Output Wrapper System

type commandWrapper struct {
    label  string
    buffer *bytes.Buffer
}

func newCommandWrapper(label string) *commandWrapper {
    b := bytes.Buffer{}
    return &commandWrapper{label, &b}
}

func (w commandWrapper) out(atEOF bool) {
    for {
        s, err := w.buffer.ReadString('\n')
        if err == nil {
            log.Printf("%s | %v", w.label, s)
        } else {
            if len(s) > 0 {
                if atEOF && err == io.EOF {
                    log.Printf("%s | %v\n", w.label, s)
                } else {
                    w.buffer.WriteString(s)
                }
            }
            break
        }
    }
}

func (w *commandWrapper) Write(p []byte) (n int, err error) {
    n, err = w.buffer.Write(p)
    w.out(false)
    return
}

Fakemachine Integration

Purpose and Benefits

Fakemachine provides:

• Isolation: Build environment independent of host
• Reproducibility: Consistent results across different hosts
• Safety: Prevents host system contamination
• Performance: Optimized for build workloads

Backend Selection

// Automatic backend selection
m, err = fakemachine.NewMachineWithBackend("auto")

// Manual backend selection
m, err = fakemachine.NewMachineWithBackend("kvm")  // Fastest
m, err = fakemachine.NewMachineWithBackend("uml")  // Medium
m, err = fakemachine.NewMachineWithBackend("qemu") // Slowest

Backend Performance Comparison

Backend	Performance	Requirements	Use Case
kvm	⭐⭐⭐⭐⭐	/dev/kvm access	Production builds
uml	⭐⭐⭐	user-mode-linux package	Development/testing
qemu	⭐⭐	None	Fallback, compatibility
--disable-fakemachine	⭐⭐⭐⭐	Root permissions	Debugging, simple builds

Machine Configuration

// Memory configuration
memsize, err := units.RAMInBytes(options.Memory)
m.SetMemory(int(memsize / 1024 / 1024))

// CPU configuration
m.SetNumCPUs(options.CPUs)

// Scratch space
m.SetScratch(size, "")

// Environment variables
m.SetEnviron(EnvironString)

Filesystem Operations

Complete Filesystem Management System

Path Management Functions

func CleanPathAt(path, at string) string {
    if filepath.IsAbs(path) {
        return filepath.Clean(path)
    }
    return filepath.Join(at, path)
}

func CleanPath(path string) string {
    cwd, _ := os.Getwd()
    return CleanPathAt(path, cwd)
}

func RealPath(path string) (string, error) {
    p, err := filepath.EvalSymlinks(path)
    if err != nil {
        return "", err
    }
    return filepath.Abs(p)
}

func RestrictedPath(prefix, dest string) (string, error) {
    var err error
    destination := path.Join(prefix, dest)
    destination, err = filepath.Abs(destination)
    if err != nil {
        return "", err
    }
    if !strings.HasPrefix(destination, prefix) {
        return "", fmt.Errorf("The resulting path points outside of prefix '%s': '%s'\n", prefix, destination)
    }
    return destination, nil
}

File Copying Operations

func CopyFile(src, dst string, mode os.FileMode) error {
    in, err := os.Open(src)
    if err != nil {
        return err
    }
    defer in.Close()
    
    // Create temporary file
    tmp, err := ioutil.TempFile(filepath.Dir(dst), "")
    if err != nil {
        return err
    }
    
    // Copy content
    _, err = io.Copy(tmp, in)
    if err != nil {
        tmp.Close()
        os.Remove(tmp.Name())
        return err
    }
    
    // Set permissions and close
    if err = tmp.Close(); err != nil {
        os.Remove(tmp.Name())
        return err
    }
    if err = os.Chmod(tmp.Name(), mode); err != nil {
        os.Remove(tmp.Name())
        return err
    }
    
    // Atomic rename
    if err = os.Rename(tmp.Name(), dst); err != nil {
        os.Remove(tmp.Name())
        return err
    }
    
    return nil
}

Tree Copying Operations

func CopyTree(sourcetree, desttree string) error {
    walker := func(p string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }
        
        suffix, _ := filepath.Rel(sourcetree, p)
        target := path.Join(desttree, suffix)
        
        switch info.Mode() & os.ModeType {
        case 0: // Regular file
            err := CopyFile(p, target, info.Mode())
            if err != nil {
                return fmt.Errorf("Failed to copy file %s: %w", p, err)
            }
        case os.ModeDir: // Directory
            os.Mkdir(target, info.Mode())
        case os.ModeSymlink: // Symlink
            link, err := os.Readlink(p)
            if err != nil {
                return fmt.Errorf("Failed to read symlink %s: %w", suffix, err)
            }
            os.Symlink(link, target)
        default:
            return fmt.Errorf("File %s with mode %v not handled", p, info.Mode())
        }
        
        return nil
    }
    
    return filepath.Walk(sourcetree, walker)
}

Filesystem Safety Features

• Atomic Operations: Use temporary files and atomic renames
• Path Validation: Prevent directory traversal attacks
• Permission Preservation: Maintain original file modes
• Symlink Handling: Properly handle symbolic links
• Error Recovery: Clean up temporary files on failure

Archive Management

Complete Archive Processing System

Archive Type Support

type ArchiveType int

const (
    _ ArchiveType = iota // Guess from file extension
    Tar
    Zip
    Deb
)

type ArchiveBase struct {
    file    string // Path to archive file
    atype   ArchiveType
    options map[interface{}]interface{} // Archiver-specific options
}

Archive Unpacking Operations

func (tar *ArchiveTar) Unpack(destination string) error {
    command := []string{"tar"}
    usePigz := false
    
    // Determine compression type and options
    if compression, ok := tar.options["compression"].(string); ok {
        if compression == "gz" {
            // Check if pigz is available for parallel decompression
            if _, err := exec.LookPath("pigz"); err == nil {
                usePigz = true
                command = append(command, "--use-compress-program=pigz")
            } else {
                command = append(command, "--gzip")
            }
        } else if opts := tarOptions(compression); opts != "" {
            command = append(command, opts)
        }
    }
    
    command = append(command, "-xf", tar.file, "-C", destination)
    
    return unpack(command, destination)
}

Compression Support

func tarOptions(compression string) string {
    unpackTarOpts := map[string]string{
        "bzip2": "--bzip2",
        "gz":    "--gzip",
        "lzip":  "--lzip",
        "lzma":  "--lzma",
        "lzop":  "--lzop",
        "xz":    "--xz",
        "zstd":  "--zstd",
    }
    return unpackTarOpts[compression]
}

Archive Management Features

• Multiple Formats: Support for tar, zip, and deb packages
• Compression Detection: Automatic compression type detection
• Parallel Processing: Use pigz for faster gzip decompression
• Error Handling: Robust error handling and cleanup
• Option Support: Configurable unpacking options

External Tools and Dependencies

Core System Tools

Filesystem Management

• debootstrap: Base system construction
• mmdebstrap: Alternative system construction
• parted: Partition table management
• sfdisk: Scriptable partitioning
• mkfs.ext4, mkfs.xfs, mkfs.fat: Filesystem creation
• mount, umount: Filesystem mounting
• losetup: Loop device management

Package Management

• apt-get, apt: Debian package management
• pacman: Arch Linux package management
• pacstrap: Arch Linux system construction

Compression and Archiving

• tar: Archive creation and extraction
• gzip, bzip2, xz: Compression
• cpio: Archive format support

Network Tools

• wget, curl: File download
• rsync: Synchronization

Build System Dependencies

Go Dependencies (go.mod)

require (
    github.com/go-debos/fakemachine v0.0.11    // VM backend
    github.com/docker/go-units v0.5.0           // Human-readable sizes
    github.com/freddierice/go-losetup/v2 v2.0.1 // Loop device management
    github.com/sjoerdsimons/ostree-go v0.0.0    // OSTree support
    gopkg.in/yaml.v2 v2.4.0                     // YAML parsing
)

System Dependencies

• golang: Go runtime and compiler
• libglib2.0-dev: GLib development files
• libostree-dev: OSTree development files
• qemu-system-x86: QEMU emulation
• qemu-user-static: User-mode QEMU
• debootstrap: Base system construction
• systemd-container: Container support

External Tool Integration Points

Command Execution

type Command struct {
    // Command configuration
}

func (c *Command) Run(label string, args ...string) error {
    // Execute with proper environment
    // Handle output and errors
    // Support chroot execution
}

Chroot Support

func NewChrootCommandForContext(context DebosContext) Command {
    cmd := Command{}
    // Add bind mounts
    // Set working directory
    // Configure environment
    return cmd
}

Recipe Processing

YAML + Go Template System

Recipes support dynamic content through Go templates:

{{- $image := or .image "debian.tgz" -}}
{{- $suite := or .suite "bookworm" -}}

architecture: {{ .architecture }}

actions:
  - action: debootstrap
    suite: {{ $suite }}
    mirror: https://deb.debian.org/debian

Template Variable Sources

1. Command Line: -t variable:value
2. Environment: System environment variables
3. Built-in: Architecture, suite defaults
4. Recipe: Local variable definitions

Recipe Validation

func (r *Recipe) Parse(file string, printRecipe, verbose bool, templateVars map[string]string) error {
    // Load and parse YAML
    // Apply Go templates
    // Validate action parameters
    // Check dependencies
}

Context Management

DebosContext Structure

type DebosContext struct {
    *CommonContext
    RecipeDir       string
    Architecture    string
    SectorSize      int
}

type CommonContext struct {
    Scratchdir      string        // Temporary working directory
    Rootdir         string        // Target filesystem root
    Artifactdir     string        // Output directory
    Downloaddir     string        // Download cache
    Image           string        // Current image file
    ImagePartitions []Partition   // Partition information
    ImageMntDir     string        // Image mount point
    ImageFSTab      bytes.Buffer  // Generated fstab
    ImageKernelRoot string        // Kernel root parameter
    DebugShell      string        // Debug shell path
    Origins         map[string]string // Path mappings
    State           DebosState    // Build state
    EnvironVars     map[string]string // Environment variables
    PrintRecipe     bool          // Print final recipe
    Verbose         bool          // Verbose output
}

Origin System

The origin system provides path mapping for actions:

context.Origins = map[string]string{
    "artifacts":   context.Artifactdir,    // Output directory
    "filesystem":  context.Rootdir,        // Target filesystem
    "recipe":      context.RecipeDir,      // Recipe directory
}

Actions can reference these origins:

- action: overlay
  source: overlays/config
  destination: {{ .origin.filesystem }}/etc/config

Error Handling and Debugging

Error States

type DebosState int

const (
    Success DebosState = iota
    Failed
)

Debug Shell Integration

if options.DebugShell {
    context.DebugShell = options.Shell
    // Fall into interactive shell on error
}

Error Handling Flow

func handleError(context *debos.DebosContext, err error, a debos.Action, stage string) bool {
    if err == nil {
        return false
    }
    
    context.State = debos.Failed
    log.Printf("Action `%s` failed at stage %s, error: %s", a, stage, err)
    
    if context.DebugShell != "" {
        debos.DebugShell(*context)
    }
    
    return true
}

Cleanup Mechanisms

Actions implement cleanup hooks:

• Cleanup: Per-action cleanup in same environment
• PostMachineCleanup: Host-side cleanup for all actions
• Deferred execution: Ensures cleanup runs even on errors

Performance Characteristics

Build Time Benchmarks

Based on pine-a64-plus/debian.yaml on Intel Pentium G4560T with SSD:

Backend	Wall Time	Prerequisites
--disable-fakemachine	8 min	Root permissions
-b kvm	9 min	Access to /dev/kvm
-b uml	18 min	user-mode-linux package
-b qemu	166 min	None

Performance Optimization

Scratch Space Management

if options.ScratchSize != "" {
    size, err := units.FromHumanSize(options.ScratchSize)
    scratchsizeMB := int(size / 1000 / 1000)
    m.SetScratch(size, "")
}

Memory Configuration

if memsizeMB < 256 {
    log.Printf("WARNING: Memory size of %dMB is less than recommended minimum 256MB\n", memsizeMB)
}

CPU Allocation

if options.CPUs == 0 {
    options.CPUs = 2  // Default to 2 CPUs
}
m.SetNumCPUs(options.CPUs)

Integration Points

Container Integration

Docker Support

# Official container
docker pull godebos/debos

# Usage
docker run --rm --privileged -v $(pwd):/workspace godebos/debos recipe.yaml

Systemd-nspawn

Fakemachine can use systemd-nspawn for containerization.

CI/CD Integration

Environment Variables

# Proxy support
http_proxy, https_proxy, ftp_proxy, rsync_proxy, all_proxy, no_proxy

# Custom variables
debos -e BUILD_ID:123 -e BRANCH:main recipe.yaml

Artifact Management

# Output directory
debos --artifactdir ./outputs recipe.yaml

# Export specific artifacts
debos --export image --export archive recipe.yaml

External Tool Integration

OSTree Support

- action: ostree-commit
  repository: {{ .origin.artifacts }}/ostree
  parent: {{ .parent_commit }}

- action: ostree-deploy
  repository: {{ .origin.artifacts }}/ostree
  branch: {{ .branch }}

Bootloader Integration

debos doesn't have built-in bootloader actions but provides infrastructure:

- action: run
  chroot: true
  script: scripts/install-grub.sh

- action: run
  chroot: true
  script: scripts/configure-bootloader.sh

Advanced Features

Multi-Architecture Support

architecture: arm64

actions:
  - action: debootstrap
    suite: bookworm
    # Architecture-specific packages
    packages: [firmware-linux]

Custom Action Development

Action Interface Implementation

type CustomAction struct {
    debos.BaseAction `yaml:",inline"`
    CustomField string `yaml:"custom-field"`
}

func (a *CustomAction) Verify(context *debos.DebosContext) error {
    // Validation logic
    return nil
}

func (a *CustomAction) Run(context *debos.DebosContext) error {
    // Execution logic
    return nil
}

Integration

- action: custom
  custom-field: value

Network and Proxy Support

Automatic Proxy Detection

var environ_vars = [...]string {
    "http_proxy", "https_proxy", "ftp_proxy", 
    "rsync_proxy", "all_proxy", "no_proxy",
}

Proxy Validation

func warnLocalhost(variable string, value string) {
    if strings.Contains(value, "localhost") ||
       strings.Contains(value, "127.0.0.1") ||
       strings.Contains(value, "::1") {
        log.Printf("WARNING: Environment variable %s contains localhost", variable)
    }
}

Conclusion

debos provides a powerful, flexible framework for creating Debian-based operating system images. Its architecture emphasizes:

1. Reproducibility: Consistent results through isolated execution
2. Flexibility: Customizable through actions and scripts
3. Performance: Optimized backends for different use cases
4. Integration: Easy integration with existing toolchains
5. Maintainability: Clear separation of concerns and lifecycle hooks

The system's strength lies in its ability to combine declarative recipe definitions with imperative script execution, making it suitable for both simple image creation and complex, multi-stage build processes.

Key Strengths

• Isolation: Fakemachine ensures build reproducibility
• Extensibility: Action system allows custom functionality
• Performance: Multiple backend options for different scenarios
• Integration: Works with existing Debian ecosystem tools
• Cross-Architecture: Full support for ARM, MIPS, RISC-V, etc.
• Template System: Advanced Go templating with custom functions
• Command Execution: Robust chroot and cross-architecture support
• Filesystem Safety: Atomic operations and path validation

Areas for Enhancement

• Bootloader Support: No built-in bootloader management
• Package Management: Limited to Debian/Arch package managers
• Image Formats: Focus on filesystem and archive outputs
• Validation: Limited manifest validation capabilities
• Parallel Execution: Actions execute sequentially
• Dependency Resolution: No automatic dependency ordering

Complete Process Summary

debos implements a complete end-to-end image building pipeline that:

1. Processes Recipes: YAML + Go templates with validation
2. Manages Actions: 20+ built-in actions with lifecycle hooks
3. Executes Commands: Robust chroot and cross-architecture support
4. Handles Filesystems: Safe file operations with atomic guarantees
5. Manages Archives: Multiple format support with compression
6. Provides Isolation: Fakemachine VM backends for reproducibility
7. Supports Templates: Advanced variable substitution and functions
8. Generates Artifacts: Images, archives, and OSTree repositories

The system's architecture emphasizes reproducibility, flexibility, and integration while maintaining performance and security through isolated execution environments.