deb-osbuild/docs/ublue-os-guide.md

# Building Universal Blue Systems on Your Own Infrastructure: Complete Guide

This guide walks you through building Universal Blue operating systems (Bazzite, Aurora, Bluefin, uCore, and custom images) on your own git servers and infrastructure, including the complete ecosystem of tools needed to create bootable operating system images.

## Overview

Universal Blue systems use a container-native approach where the OS is built as a container image, then converted to bootable formats using a comprehensive ecosystem of tools. The key components are:

### Core Infrastructure Tools
- **Container Image**: The OS filesystem and packages in OCI format
- **bootc**: Container bootloader interface for updates and management
- **bootupd**: Handles A/B partition updates for atomic upgrades
- **bootc-image-builder**: Converts container images to bootable drive formats
- **osbuilder**: Red Hat's advanced OS building tool for complex image customization and multi-arch builds
- **rpm-ostree**: Package layering and system management

### Universal Blue Ecosystem Tools
- **BlueBuild**: Declarative image building framework using `recipe.yml` files
- **startingpoint**: Template repository for creating custom Universal Blue images
- **forge**: On-premise Universal Blue infrastructure for self-hosting
- **ujust**: Justfile-based system management and automation
- **upgrade-tools**: Migration utilities for switching between Universal Blue images
- **ublue-os packages**: Package management and application installation system

### Target Systems
- **Bazzite**: Gaming-focused desktop and handheld OS
- **Aurora**: KDE desktop environment variant
- **Bluefin**: GNOME-based developer workstation
- **uCore**: Fedora CoreOS with batteries included
- **Custom Images**: Your own Universal Blue derivatives

## Prerequisites

### Infrastructure Requirements
- Git server (GitLab, Gitea, etc.) with webhook capabilities
- Container registry (Harbor, GitLab Registry, Docker Registry, etc.)
- CI/CD system with privileged container support
- Sufficient storage (100GB+ for builds, 20GB+ per bootable image)
- x86_64 build environment with adequate RAM (16GB+ recommended for multiple variants)
- For osbuilder: Additional storage for composer workspaces (100GB+ recommended)

### Software Dependencies
- Podman or Docker with buildah support
- bootc-image-builder container access
- osbuilder (osbuild, composer) for advanced builds
- BlueBuild CLI for declarative builds
- Registry authentication configured in CI
- just command runner for automation

## Step 1: Repository Setup and Tool Installation

### 1.1 Install Universal Blue Tools

Create `scripts/setup-ublue-tools.sh`:

```bash
#!/bin/bash
set -euo pipefail

echo "Setting up Universal Blue ecosystem tools..."

# Install just command runner
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin

# Install BlueBuild CLI
cargo install bluebuild

# Install bootc tools
sudo rpm-ostree install bootc bootupd

# Install osbuilder/osbuild
sudo dnf install -y osbuild osbuild-composer composer-cli

# Start and enable osbuild-composer
sudo systemctl enable --now osbuild-composer.socket

# Add current user to weldr group for composer access
sudo usermod -a -G weldr $(whoami)

echo "Universal Blue tools installation complete"
```

### 1.2 Clone Universal Blue Repositories

```bash
# Create workspace for all Universal Blue systems
mkdir -p ublue-workspace
cd ublue-workspace

# Clone main Universal Blue repositories
git clone https://github.com/ublue-os/main.git
git clone https://github.com/ublue-os/bazzite.git
git clone https://github.com/ublue-os/bluefin.git
git clone https://github.com/ublue-os/aurora.git
git clone https://github.com/ublue-os/ucore.git
git clone https://github.com/ublue-os/startingpoint.git
git clone https://github.com/ublue-os/forge.git

# Clone ecosystem tools
git clone https://github.com/ublue-os/packages.git
git clone https://github.com/ublue-os/upgrade-tools.git

# Update remotes to point to your infrastructure
for repo in main bazzite bluefin aurora ucore startingpoint forge packages upgrade-tools; do
  cd $repo
  git remote set-url origin https://your-git-server.com/ublue-os/$repo.git
  git push origin main
  cd ..
done
```

### 1.3 Set Up forge for On-Premise Infrastructure

```bash
cd forge

# Configure forge for your environment
cp config/config.yaml.example config/config.yaml

# Edit config.yaml with your settings
cat > config/config.yaml << EOF
forge:
  domain: "forge.yourdomain.com"
  registry: "your-registry.com"
  git_server: "your-git-server.com"
  
builds:
  parallel_jobs: 4
  storage_path: "/var/lib/forge/builds"
  
images:
  - name: "bazzite"
    variants: ["desktop", "deck", "ally", "legion"]
  - name: "bluefin" 
    variants: ["base", "dx", "nvidia"]
  - name: "aurora"
    variants: ["base", "dx", "nvidia"]
  - name: "ucore"
    variants: ["base", "minimal"]
EOF

# Deploy forge
just setup-forge
```

## Step 2: BlueBuild Integration for Declarative Builds

### 2.1 Understanding BlueBuild

BlueBuild enables building full images by only editing a recipe file, with no need to delve into Containerfiles or GitHub Actions. This makes it much easier to maintain Universal Blue systems.

### 2.2 Convert Existing Images to BlueBuild Format

Create `recipes/bazzite.yml`:

```yaml
name: bazzite
description: Bazzite Gaming OS
base-image: ghcr.io/ublue-os/silverblue-main
image-version: 40

stages:
  - type: default-flatpaks
    notify: true
    system:
      repo-url: https://dl.flathub.org/repo/flathub.flatpakrepo
      install:
        - com.valvesoftware.Steam
        - org.lutris.Lutris
        - com.heroicgameslauncher.hgl

modules:
  - type: rpm-ostree
    repos:
      - https://copr.fedorainfracloud.org/coprs/kylegospo/bazzite/repo/fedora-%OS_VERSION%/kylegospo-bazzite-fedora-%OS_VERSION%.repo
    install:
      - gamemode
      - gamescope
      - jupiter-fan-control
      - steamdeck-kde-presets
    remove:
      - firefox
      - firefox-langpacks

  - type: signing
    cosign-private-key: /etc/pki/containers/cosign.key

  - type: script
    scripts:
      - gaming-optimizations.sh
      - steam-deck-support.sh

files:
  - source: config/
    destination: /etc/bazzite/
```

### 2.3 Create BlueBuild Recipes for All Systems

**recipes/bluefin.yml:**
```yaml
name: bluefin
description: Bluefin Developer Workstation
base-image: ghcr.io/ublue-os/silverblue-main
image-version: 40

modules:
  - type: rpm-ostree
    repos:
      - https://copr.fedorainfracloud.org/coprs/projectbluefin/bluefin/repo/fedora-%OS_VERSION%/projectbluefin-bluefin-fedora-%OS_VERSION%.repo
    install:
      - distrobox
      - toolbox
      - code
      - podman-compose
      - docker-compose

  - type: containerfiles
    containerfiles:
      - containerfiles/bluefin/Containerfile.dx

stages:
  - type: default-flatpaks
    system:
      install:
        - com.visualstudio.code
        - com.docker.Docker
        - io.podman_desktop.PodmanDesktop
```

**recipes/aurora.yml:**
```yaml
name: aurora
description: Aurora KDE Desktop
base-image: ghcr.io/ublue-os/kinoite-main
image-version: 40

modules:
  - type: rpm-ostree
    install:
      - kde-connect
      - krita
      - kdenlive
    remove:
      - konversation

stages:
  - type: default-flatpaks
    system:
      install:
        - org.kde.krita
        - org.kde.kdenlive
        - org.telegram.desktop
```

**recipes/ucore.yml:**
```yaml
name: ucore
description: Fedora CoreOS with batteries included
base-image: quay.io/fedora/fedora-coreos
image-version: stable

modules:
  - type: rpm-ostree
    install:
      - tailscale
      - podman-compose
      - docker-compose
      - btop
      - micro

  - type: systemd
    system:
      enabled:
        - tailscaled
        - podman.socket
```

## Step 3: Comprehensive CI/CD Pipeline

### 3.1 GitLab CI Configuration

Create `.gitlab-ci.yml`:

```yaml
stages:
  - setup
  - build-base
  - build-variants
  - bootc-validate
  - generate-bootable-images
  - publish
  - deploy-forge

variables:
  REGISTRY: "your-registry.com"
  FORGE_URL: "https://forge.yourdomain.com"

# Build matrix for all Universal Blue systems
.build_template: &build_template
  stage: build-variants
  script:
    - |
      if [ -f "recipes/${SYSTEM}.yml" ]; then
        bluebuild build recipes/${SYSTEM}.yml --registry ${REGISTRY}
      else
        podman build -f Containerfile.${SYSTEM} -t ${REGISTRY}/ublue-os/${SYSTEM}:${CI_COMMIT_SHA} .
      fi
    - podman push ${REGISTRY}/ublue-os/${SYSTEM}:${CI_COMMIT_SHA}
  tags:
    - privileged

setup-tools:
  stage: setup
  script:
    - ./scripts/setup-ublue-tools.sh
  artifacts:
    paths:
      - /usr/local/bin/just
    expire_in: 1 hour

build-base-images:
  stage: build-base
  script:
    - cd main
    - podman build -f Containerfile.silverblue -t ${REGISTRY}/ublue-os/silverblue-main:latest
    - podman build -f Containerfile.kinoite -t ${REGISTRY}/ublue-os/kinoite-main:latest
    - podman push ${REGISTRY}/ublue-os/silverblue-main:latest
    - podman push ${REGISTRY}/ublue-os/kinoite-main:latest
  dependencies:
    - setup-tools

# Build all Universal Blue systems
bazzite:
  <<: *build_template
  variables:
    SYSTEM: "bazzite"
  dependencies:
    - build-base-images

bluefin:
  <<: *build_template
  variables:
    SYSTEM: "bluefin"
  dependencies:
    - build-base-images

aurora:
  <<: *build_template
  variables:
    SYSTEM: "aurora"
  dependencies:
    - build-base-images

ucore:
  <<: *build_template
  variables:
    SYSTEM: "ucore"
  dependencies:
    - build-base-images

# Validation stage
validate-images:
  stage: bootc-validate
  parallel:
    matrix:
      - SYSTEM: ["bazzite", "bluefin", "aurora", "ucore"]
  script:
    - podman run --rm ${REGISTRY}/ublue-os/${SYSTEM}:${CI_COMMIT_SHA} bootc container lint
  dependencies:
    - bazzite
    - bluefin
    - aurora
    - ucore

# Bootable image generation
generate-bootable:
  stage: generate-bootable-images
  parallel:
    matrix:
      - SYSTEM: ["bazzite", "bluefin", "aurora", "ucore"]
        BUILDER: ["bootc-image-builder", "osbuilder"]
  script:
    - mkdir -p output/${SYSTEM}
    - |
      if [ "$BUILDER" == "bootc-image-builder" ]; then
        podman run --rm --privileged \
          -v $PWD/output/${SYSTEM}:/output \
          -v /var/lib/containers/storage:/var/lib/containers/storage \
          quay.io/centos-bootc/bootc-image-builder:latest \
          --type iso,raw \
          --output /output \
          ${REGISTRY}/ublue-os/${SYSTEM}:${CI_COMMIT_SHA}
      else
        BUILD_METHOD=osbuilder-only OUTPUT_DIR=output/${SYSTEM} \
          ./scripts/build-with-osbuilder.sh ${REGISTRY}/ublue-os/${SYSTEM}:${CI_COMMIT_SHA}
      fi
  artifacts:
    paths:
      - output/
    expire_in: 1 week
  tags:
    - privileged
  dependencies:
    - validate-images

# Deploy to forge
deploy-to-forge:
  stage: deploy-forge
  script:
    - |
      curl -X POST ${FORGE_URL}/api/builds \
        -H "Authorization: Bearer ${FORGE_TOKEN}" \
        -d '{
          "images": ["bazzite", "bluefin", "aurora", "ucore"],
          "tag": "'${CI_COMMIT_SHA}'"
        }'
  dependencies:
    - generate-bootable
```

## Step 4: ujust Integration for System Management

### 4.1 Create Comprehensive Justfile

Create `justfile` for system management:

```just
#!/usr/bin/env just --justfile

# Default recipe to display help
default:
    @just --list

# Build all Universal Blue systems
build-all:
    #!/usr/bin/env bash
    for system in bazzite bluefin aurora ucore; do
        echo "Building $system..."
        if [ -f "recipes/$system.yml" ]; then
            bluebuild build "recipes/$system.yml"
        else
            podman build -f "Containerfile.$system" -t "localhost/ublue-os/$system:latest" .
        fi
    done

# Build specific system
build system:
    #!/usr/bin/env bash
    if [ -f "recipes/{{system}}.yml" ]; then
        bluebuild build "recipes/{{system}}.yml"
    else
        podman build -f "Containerfile.{{system}}" -t "localhost/ublue-os/{{system}}:latest" .
    fi

# Generate bootable images for a system
generate-bootable system builder="both":
    #!/usr/bin/env bash
    mkdir -p output/{{system}}
    BUILD_METHOD={{builder}} OUTPUT_DIR=output/{{system}} \
        ./scripts/build-bootable.sh localhost/ublue-os/{{system}}:latest

# Set up development environment
setup-dev:
    #!/usr/bin/env bash
    ./scripts/setup-ublue-tools.sh
    echo "Development environment ready"

# Clean build artifacts
clean:
    #!/usr/bin/env bash
    rm -rf output/
    podman image prune -f
    podman system prune -f

# Install ujust recipes for end users
install-ujust-recipes:
    #!/usr/bin/env bash
    sudo mkdir -p /usr/share/ublue-os/just
    sudo cp -r just/* /usr/share/ublue-os/just/
    echo "ujust recipes installed system-wide"

# Update all base images
update-bases:
    #!/usr/bin/env bash
    podman pull ghcr.io/ublue-os/silverblue-main:latest
    podman pull ghcr.io/ublue-os/kinoite-main:latest
    podman pull quay.io/fedora/fedora-coreos:stable
    echo "Base images updated"

# Validate all images
validate-all:
    #!/usr/bin/env bash
    for system in bazzite bluefin aurora ucore; do
        echo "Validating $system..."
        podman run --rm localhost/ublue-os/$system:latest bootc container lint
    done

# Deploy to forge
deploy-forge tag="latest":
    #!/usr/bin/env bash
    curl -X POST ${FORGE_URL}/api/builds \
        -H "Authorization: Bearer ${FORGE_TOKEN}" \
        -d '{
            "images": ["bazzite", "bluefin", "aurora", "ucore"],
            "tag": "{{tag}}"
        }'
```

### 4.2 End-User ujust Recipes

Create `just/` directory with user-facing commands:

**just/bazzite.just:**
```just
# Gaming-specific ujust commands for Bazzite

# Install Steam
install-steam:
    flatpak install -y flathub com.valvesoftware.Steam

# Install Lutris
install-lutris:
    flatpak install -y flathub org.lutris.Lutris

# Enable Gamemode optimizations
enable-gamemode:
    systemctl --user enable gamemoded

# Install OpenTabletDriver
install-opentabletdriver:
    rpm-ostree install opentabletdriver

# Toggle Wayland session
toggle-wayland:
    #!/usr/bin/env bash
    if grep -q "WaylandEnable=false" /etc/gdm/custom.conf; then
        sudo sed -i 's/WaylandEnable=false/WaylandEnable=true/' /etc/gdm/custom.conf
        echo "Wayland enabled. Restart required."
    else
        sudo sed -i 's/WaylandEnable=true/WaylandEnable=false/' /etc/gdm/custom.conf
        echo "Wayland disabled. Restart required."
    fi
```

**just/bluefin.just:**
```just
# Developer-focused ujust commands for Bluefin

# Set up development environment
setup-dev-env:
    #!/usr/bin/env bash
    distrobox create --name dev --image registry.fedoraproject.org/fedora:latest
    distrobox enter dev -- sudo dnf install -y nodejs npm python3-pip golang rust cargo
    echo "Development environment created in 'dev' distrobox"

# Install VS Code via Flatpak
install-vscode:
    flatpak install -y flathub com.visualstudio.code

# Enable Docker/Podman services
enable-containers:
    systemctl --user enable podman.socket
    sudo systemctl enable docker

# Install development Flatpaks
install-dev-flatpaks:
    #!/usr/bin/env bash
    flatpaks=(
        "com.docker.Docker"
        "io.podman_desktop.PodmanDesktop"
        "com.github.tchx84.Flatseal"
        "org.gnome.Builder"
    )
    for app in "${flatpaks[@]}"; do
        flatpak install -y flathub $app
    done
```

## Step 5: Package Management Integration

### 5.1 Universal Blue Packages System

Create `packages/packages.json`:

```json
{
  "packages": {
    "gaming": {
      "flatpaks": [
        "com.valvesoftware.Steam",
        "org.lutris.Lutris",
        "com.heroicgameslauncher.hgl",
        "net.davidotek.pupgui2"
      ],
      "rpms": [
        "gamemode",
        "gamescope"
      ]
    },
    "development": {
      "flatpaks": [
        "com.visualstudio.code",
        "com.docker.Docker",
        "io.podman_desktop.PodmanDesktop"
      ],
      "rpms": [
        "distrobox",
        "toolbox"
      ]
    },
    "multimedia": {
      "flatpaks": [
        "org.kde.krita",
        "org.blender.Blender",
        "org.audacityteam.Audacity"
      ]
    }
  }
}
```

### 5.2 Package Installation Scripts

Create `scripts/install-packages.sh`:

```bash
#!/bin/bash
set -euo pipefail

PACKAGE_GROUP="$1"
PACKAGES_FILE="packages/packages.json"

echo "Installing package group: $PACKAGE_GROUP"

# Install Flatpaks
if jq -e ".packages.$PACKAGE_GROUP.flatpaks" "$PACKAGES_FILE" > /dev/null; then
    flatpaks=$(jq -r ".packages.$PACKAGE_GROUP.flatpaks[]" "$PACKAGES_FILE")
    for app in $flatpaks; do
        echo "Installing Flatpak: $app"
        flatpak install -y flathub "$app"
    done
fi

# Install RPMs
if jq -e ".packages.$PACKAGE_GROUP.rpms" "$PACKAGES_FILE" > /dev/null; then
    rpms=$(jq -r ".packages.$PACKAGE_GROUP.rpms[]" "$PACKAGES_FILE")
    for pkg in $rpms; do
        echo "Installing RPM: $pkg"
        rpm-ostree install "$pkg"
    done
fi

echo "Package group $PACKAGE_GROUP installation complete"
```

## Step 6: Upgrade Tools Integration

### 6.1 Migration Between Universal Blue Systems

Create `scripts/ublue-migration.sh`:

```bash
#!/bin/bash
set -euo pipefail

SOURCE_SYSTEM="$1"
TARGET_SYSTEM="$2"
REGISTRY="${REGISTRY:-your-registry.com}"

echo "Migrating from $SOURCE_SYSTEM to $TARGET_SYSTEM"

# Use upgrade-tools for safe migration
if command -v ublue-upgrade >/dev/null 2>&1; then
    ublue-upgrade "$REGISTRY/ublue-os/$TARGET_SYSTEM:latest"
else
    # Fallback to bootc switch
    sudo bootc switch "$REGISTRY/ublue-os/$TARGET_SYSTEM:latest"
fi

echo "Migration initiated. Reboot to complete."
```

## Step 7: Advanced osbuilder Configurations

### 7.1 Multi-System osbuilder Manifests

Create `scripts/generate-multi-system-manifest.sh`:

```bash
#!/bin/bash
set -euo pipefail

SYSTEMS=("bazzite" "bluefin" "aurora" "ucore")
OUTPUT_DIR="osbuilder-workspace"

mkdir -p "$OUTPUT_DIR"

for system in "${SYSTEMS[@]}"; do
    cat > "$OUTPUT_DIR/$system-manifest.json" << EOF
{
  "version": "2",
  "pipelines": [
    {
      "name": "$system-container-tree",
      "source": {
        "org.osbuild.containers": {
          "images": {
            "$system": {
              "source": "your-registry.com/ublue-os/$system:latest",
              "local": false
            }
          }
        }
      },
      "stages": [
        {
          "type": "org.osbuild.container-deploy",
          "options": {
            "image": "$system"
          }
        }
      ]
    },
    {
      "name": "$system-image",
      "build": "$system-container-tree",
      "stages": [
        {
          "type": "org.osbuild.truncate",
          "options": {
            "filename": "$system-disk.img",
            "size": "10737418240"
          }
        },
        {
          "type": "org.osbuild.bootc.install-to-filesystem",
          "options": {
            "deployment": {
              "container-image-reference": "your-registry.com/ublue-os/$system:latest"
            },
            "root_filesystem": "xfs"
          }
        }
      ]
    }
  ]
}
EOF
done

echo "Generated osbuilder manifests for all systems"
```

## Step 8: Security and Signing

### 8.1 Comprehensive Signing Pipeline

Create `scripts/sign-all-images.sh`:

```bash
#!/bin/bash
set -euo pipefail

SYSTEMS=("bazzite" "bluefin" "aurora" "ucore")
REGISTRY="${REGISTRY:-your-registry.com}"
COSIGN_KEY="${COSIGN_KEY:-cosign.key}"

for system in "${SYSTEMS[@]}"; do
    echo "Signing $system..."
    cosign sign --key "$COSIGN_KEY" "$REGISTRY/ublue-os/$system:latest"
    
    # Generate SBOM
    syft "$REGISTRY/ublue-os/$system:latest" -o spdx-json > "$system-sbom.spdx.json"
    cosign attest --key "$COSIGN_KEY" --predicate "$system-sbom.spdx.json" \
        "$REGISTRY/ublue-os/$system:latest"
done

echo "All images signed and attested"
```

## Step 9: forge Deployment and Management

### 9.1 forge Configuration for All Systems

Update your forge configuration:

```yaml
# forge/config/systems.yaml
systems:
  bazzite:
    description: "Gaming-focused desktop and handheld OS"
    variants:
      - desktop
      - deck
      - ally
      - legion
    base_image: "silverblue-main"
    
  bluefin:
    description: "GNOME-based developer workstation"
    variants:
      - base
      - dx
      - nvidia
    base_image: "silverblue-main"
    
  aurora:
    description: "KDE desktop environment variant"
    variants:
      - base
      - dx
      - nvidia
    base_image: "kinoite-main"
    
  ucore:
    description: "Fedora CoreOS with batteries included"
    variants:
      - base
      - minimal
    base_image: "fedora-coreos"
    
build_matrix:
  architectures:
    - x86_64
    - aarch64
  formats:
    - iso
    - raw
    - qcow2
  builders:
    - bootc-image-builder
    - osbuilder
```

### 9.2 forge API Integration

Create `scripts/forge-api.sh`:

```bash
#!/bin/bash
set -euo pipefail

FORGE_URL="${FORGE_URL:-https://forge.yourdomain.com}"
FORGE_TOKEN="${FORGE_TOKEN}"
ACTION="$1"
SYSTEM="${2:-all}"

case "$ACTION" in
    "build")
        curl -X POST "$FORGE_URL/api/builds" \
            -H "Authorization: Bearer $FORGE_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"system\": \"$SYSTEM\"}"
        ;;
    "status")
        curl -X GET "$FORGE_URL/api/builds/status" \
            -H "Authorization: Bearer $FORGE_TOKEN"
        ;;
    "artifacts")
        curl -X GET "$FORGE_URL/api/artifacts/$SYSTEM" \
            -H "Authorization: Bearer $FORGE_TOKEN"
        ;;
    *)
        echo "Usage: $0 {build|status|artifacts} [system]"
        exit 1
        ;;
esac
```

## Step 10: Testing and Validation

### 10.1 Comprehensive Testing Pipeline

Create `scripts/test-all-systems.sh`:

```bash
#!/bin/bash
set -euo pipefail

SYSTEMS=("bazzite" "bluefin" "aurora" "ucore")
OUTPUT_DIR="output"

for system in "${SYSTEMS[@]}"; do
    echo "Testing $system..."
    
    # Test container lint
    podman run --rm "your-registry.com/ublue-os/$system:latest" bootc container lint
    
    # Test bootable image
    if [ -f "$OUTPUT_DIR/$system/disk.raw" ]; then
        echo "Testing boot for $system..."
        timeout 300 qemu-system-x86_64 \
            -m 2G \
            -drive file="$OUTPUT_DIR/$system/disk.raw",format=raw \
            -nographic \
            -serial stdio \
            -monitor none || echo "$system boot test completed"
    fi
    
    # Test specific system features
    case "$system" in
        "bazzite")
            echo "Testing gaming features..."
            podman run --rm "your-registry.com/ublue-os/$system:latest" \
                rpm -q gamemode gamescope
            ;;
        "bluefin")
            echo "Testing development tools..."
            podman run --rm "your-registry.com/ublue-os/$system:latest" \
                rpm -q distrobox toolbox
            ;;
    esac
done

echo "All system tests completed"
```

## Step 11: Documentation and User Guides

### 11.1 Generate System Documentation

Create `scripts/generate-docs.sh`:

```bash
#!/bin/bash
set -euo pipefail

SYSTEMS=("bazzite" "bluefin" "aurora" "ucore")
DOCS_DIR="docs"

mkdir -p "$DOCS_DIR"

for system in "${SYSTEMS[@]}"; do
    cat > "$DOCS_DIR/$system.md" << EOF
# $system Installation and Usage Guide

## Installation

### From ISO
1. Download the $system ISO from your forge: https://forge.yourdomain.com
2. Write to USB: \`dd if=$system.iso of=/dev/sdX bs=4M status=progress\`
3. Boot and follow the installer

### From Existing System
\`\`\`bash
sudo bootc switch your-registry.com/ublue-os/$system:latest
sudo systemctl reboot
\`\`\`

## Available ujust Commands
\`\`\`bash
ujust --list
\`\`\`

## System-Specific Features
EOF

    # Add system-specific documentation
    case "$system" in
        "bazzite")
            cat >> "$DOCS_DIR/$system.md" << EOF

### Gaming Features
- Steam pre-installed and optimized
- Gamemode for performance optimization  
- Handheld-specific optimizations for Steam Deck, ROG Ally, Legion Go
- \`ujust install-opentabletdriver\` for drawing tablet support

### Gaming Commands
- \`ujust install-steam\` - Install Steam
- \`ujust install-lutris\` - Install Lutris
- \`ujust enable-gamemode\` - Enable Gamemode optimizations
EOF
            ;;
        "bluefin")
            cat >> "$DOCS_DIR/$system.md" << EOF

### Developer Features
- Distrobox and Toolbox pre-configured
- Container development tools
- VS Code and development Flatpaks available

### Development Commands
- \`ujust setup-dev-env\` - Create development distrobox
- \`ujust install-vscode\` - Install VS Code
- \`ujust enable-containers\` - Enable Docker/Podman services
- \`ujust install-dev-flatpaks\` - Install development Flatpaks
EOF
            ;;
        "aurora")
            cat >> "$DOCS_DIR/$system.md" << EOF

### KDE Features
- KDE Plasma desktop environment
- KDE applications suite
- Multimedia tools pre-installed

### KDE Commands
- Standard ujust commands available
- KDE-specific configurations in System Settings
EOF
            ;;
        "ucore")
            cat >> "$DOCS_DIR/$system.md" << EOF

### Container Platform Features
- Fedora CoreOS base with additional tools
- Container orchestration ready
- Optimized for server/edge deployments

### Server Commands
- Standard container management tools
- Tailscale VPN integration
- Optimized for headless operation
EOF
            ;;
    esac

    echo "Generated documentation for $system"
done

echo "Documentation generation complete"
```

## Step 12: Maintenance and Monitoring

### 12.1 Automated Health Checks

Create `scripts/health-check.sh`:

```bash
#!/bin/bash
set -euo pipefail

SYSTEMS=("bazzite" "bluefin" "aurora" "ucore")
REGISTRY="${REGISTRY:-your-registry.com}"
HEALTH_LOG="/var/log/ublue-health.log"

echo "=== Universal Blue Health Check $(date) ===" >> "$HEALTH_LOG"

for system in "${SYSTEMS[@]}"; do
    echo "Checking $system..." | tee -a "$HEALTH_LOG"
    
    # Check if image exists and is pullable
    if podman pull "$REGISTRY/ublue-os/$system:latest" &>/dev/null; then
        echo "✓ $system image available" | tee -a "$HEALTH_LOG"
    else
        echo "✗ $system image unavailable" | tee -a "$HEALTH_LOG"
        continue
    fi
    
    # Check container lint
    if podman run --rm "$REGISTRY/ublue-os/$system:latest" bootc container lint &>/dev/null; then
        echo "✓ $system passes bootc lint" | tee -a "$HEALTH_LOG"
    else
        echo "✗ $system fails bootc lint" | tee -a "$HEALTH_LOG"
    fi
    
    # Check signature
    if cosign verify "$REGISTRY/ublue-os/$system:latest" &>/dev/null; then
        echo "✓ $system signature valid" | tee -a "$HEALTH_LOG"
    else
        echo "✗ $system signature invalid" | tee -a "$HEALTH_LOG"
    fi
done

echo "Health check complete" | tee -a "$HEALTH_LOG"
```

### 12.2 Update Monitoring

Create `scripts/monitor-updates.sh`:

```bash
#!/bin/bash
set -euo pipefail

SYSTEMS=("bazzite" "bluefin" "aurora" "ucore")
REGISTRY="${REGISTRY:-your-registry.com}"
WEBHOOK_URL="${WEBHOOK_URL:-}"

check_for_updates() {
    local system="$1"
    local current_digest=$(podman inspect "$REGISTRY/ublue-os/$system:latest" --format '{{.Digest}}' 2>/dev/null || echo "")
    local remote_digest=$(skopeo inspect "docker://$REGISTRY/ublue-os/$system:latest" --format '{{.Digest}}' 2>/dev/null || echo "")
    
    if [ "$current_digest" != "$remote_digest" ] && [ -n "$remote_digest" ]; then
        echo "Update available for $system"
        if [ -n "$WEBHOOK_URL" ]; then
            curl -X POST "$WEBHOOK_URL" \
                -H "Content-Type: application/json" \
                -d "{\"text\": \"Update available for $system: $remote_digest\"}"
        fi
        return 0
    fi
    return 1
}

echo "Monitoring for updates..."
for system in "${SYSTEMS[@]}"; do
    if check_for_updates "$system"; then
        echo "Triggering rebuild for $system"
        # Trigger CI pipeline or forge build
        ./scripts/forge-api.sh build "$system"
    fi
done
```

### 12.3 Storage Management

Create `scripts/cleanup-storage.sh`:

```bash
#!/bin/bash
set -euo pipefail

RETENTION_DAYS="${RETENTION_DAYS:-7}"
OUTPUT_DIR="output"
OSBUILDER_WORKSPACE="osbuilder-workspace"

echo "Cleaning up build artifacts older than $RETENTION_DAYS days..."

# Clean output artifacts
find "$OUTPUT_DIR" -type f -name "*.iso" -mtime +$RETENTION_DAYS -delete
find "$OUTPUT_DIR" -type f -name "*.raw" -mtime +$RETENTION_DAYS -delete
find "$OUTPUT_DIR" -type f -name "*.qcow2" -mtime +$RETENTION_DAYS -delete

# Clean osbuilder workspace
if [ -d "$OSBUILDER_WORKSPACE" ]; then
    find "$OSBUILDER_WORKSPACE" -type f -mtime +$RETENTION_DAYS -delete
    # Clean osbuilder store cache (keep recent builds)
    sudo find "$OSBUILDER_WORKSPACE/store" -type f -mtime +$RETENTION_DAYS -delete 2>/dev/null || true
fi

# Prune container images (keep last 5 versions of each system)
SYSTEMS=("bazzite" "bluefin" "aurora" "ucore")
for system in "${SYSTEMS[@]}"; do
    echo "Pruning old $system images..."
    # Keep only the 5 most recent tags
    podman images --format "table {{.Repository}}:{{.Tag}} {{.CreatedAt}}" \
        --filter "reference=*/$system" \
        --sort created \
        | tail -n +6 \
        | awk '{print $1}' \
        | xargs -r podman rmi 2>/dev/null || true
done

# General container cleanup
podman system prune -f --volumes

echo "Storage cleanup complete"
```

## Step 13: Advanced forge Features

### 13.1 forge Web Interface Configuration

Create `forge/web/config.yaml`:

```yaml
server:
  port: 8080
  host: "0.0.0.0"
  
database:
  type: "postgresql"
  host: "localhost"
  port: 5432
  name: "forge"
  user: "forge"
  password_file: "/run/secrets/db_password"

build_queue:
  workers: 4
  max_concurrent_builds: 2
  
storage:
  artifacts_path: "/var/lib/forge/artifacts"
  max_artifact_age: "30d"
  
systems:
  bazzite:
    display_name: "Bazzite"
    description: "The next generation of Linux gaming"
    icon: "/static/icons/bazzite.svg"
    variants:
      desktop:
        display_name: "Desktop"
        description: "Full desktop gaming experience"
      deck:
        display_name: "Steam Deck"
        description: "Optimized for Steam Deck"
      ally:
        display_name: "ROG Ally"
        description: "Optimized for ASUS ROG Ally"
      legion:
        display_name: "Legion Go"
        description: "Optimized for Lenovo Legion Go"
        
  bluefin:
    display_name: "Bluefin"
    description: "The developer experience"
    icon: "/static/icons/bluefin.svg"
    variants:
      base:
        display_name: "Base"
        description: "Standard developer workstation"
      dx:
        display_name: "Developer Experience"
        description: "Enhanced developer tools and workflows"
      nvidia:
        display_name: "NVIDIA"
        description: "With NVIDIA driver support"
        
  aurora:
    display_name: "Aurora"
    description: "KDE desktop experience"
    icon: "/static/icons/aurora.svg"
    variants:
      base:
        display_name: "Base"
        description: "Standard KDE desktop"
      dx:
        display_name: "Developer Experience"  
        description: "KDE with developer tools"
      nvidia:
        display_name: "NVIDIA"
        description: "With NVIDIA driver support"
        
  ucore:
    display_name: "uCore"
    description: "Fedora CoreOS with batteries included"
    icon: "/static/icons/ucore.svg"
    variants:
      base:
        display_name: "Base"
        description: "Standard container platform"
      minimal:
        display_name: "Minimal" 
        description: "Minimal container platform"
```

### 13.2 forge API Extensions

Create `forge/api/systems.py`:

```python
from flask import Blueprint, request, jsonify
from forge.models import BuildJob, System
from forge.queue import build_queue

systems_bp = Blueprint('systems', __name__)

@systems_bp.route('/api/systems', methods=['GET'])
def list_systems():
    """List all available systems and variants"""
    systems = System.query.all()
    return jsonify({
        'systems': [{
            'name': s.name,
            'display_name': s.display_name,
            'description': s.description,
            'variants': [v.to_dict() for v in s.variants]
        } for s in systems]
    })

@systems_bp.route('/api/builds', methods=['POST'])
def create_build():
    """Create a new build job"""
    data = request.json
    
    job = BuildJob(
        system=data['system'],
        variant=data.get('variant', 'base'),
        architecture=data.get('architecture', 'x86_64'),
        builder=data.get('builder', 'bootc-image-builder'),
        formats=data.get('formats', ['iso', 'raw'])
    )
    
    build_queue.enqueue(job)
    
    return jsonify({
        'job_id': job.id,
        'status': 'queued'
    })

@systems_bp.route('/api/builds/<job_id>/status', methods=['GET'])
def build_status(job_id):
    """Get build job status"""
    job = BuildJob.query.get_or_404(job_id)
    
    return jsonify({
        'job_id': job.id,
        'status': job.status,
        'created_at': job.created_at.isoformat(),
        'completed_at': job.completed_at.isoformat() if job.completed_at else None,
        'artifacts': [a.to_dict() for a in job.artifacts] if job.artifacts else []
    })
```

## Step 14: User Migration Tools

### 14.1 Universal Blue System Switcher

Create `scripts/ublue-switch.sh`:

```bash
#!/bin/bash
set -euo pipefail

show_help() {
    cat << EOF
Universal Blue System Switcher

Usage: $0 [OPTIONS] TARGET_SYSTEM [VARIANT]

Switch between Universal Blue systems (bazzite, bluefin, aurora, ucore)

OPTIONS:
    -h, --help          Show this help
    -r, --registry      Custom registry (default: your-registry.com)
    -f, --force         Force switch without confirmation
    -b, --backup        Create backup before switching

EXAMPLES:
    $0 bazzite          # Switch to Bazzite (base variant)
    $0 bluefin dx       # Switch to Bluefin DX variant
    $0 aurora nvidia    # Switch to Aurora with NVIDIA drivers
    $0 ucore minimal    # Switch to uCore minimal variant

EOF
}

REGISTRY="your-registry.com"
FORCE=false
BACKUP=false
TARGET_SYSTEM=""
VARIANT="base"

while [[ $# -gt 0 ]]; do
    case $1 in
        -h|--help)
            show_help
            exit 0
            ;;
        -r|--registry)
            REGISTRY="$2"
            shift 2
            ;;
        -f|--force)
            FORCE=true
            shift
            ;;
        -b|--backup)
            BACKUP=true
            shift
            ;;
        -*)
            echo "Unknown option $1"
            exit 1
            ;;
        *)
            if [ -z "$TARGET_SYSTEM" ]; then
                TARGET_SYSTEM="$1"
            else
                VARIANT="$1"
            fi
            shift
            ;;
    esac
done

if [ -z "$TARGET_SYSTEM" ]; then
    echo "Error: TARGET_SYSTEM required"
    show_help
    exit 1
fi

# Validate target system
case "$TARGET_SYSTEM" in
    bazzite|bluefin|aurora|ucore)
        ;;
    *)
        echo "Error: Invalid system '$TARGET_SYSTEM'"
        echo "Valid systems: bazzite, bluefin, aurora, ucore"
        exit 1
        ;;
esac

CURRENT_SYSTEM=$(rpm-ostree status --json | jq -r '.deployments[0]."container-image-reference"' | cut -d'/' -f3 | cut -d':' -f1 2>/dev/null || echo "unknown")
TARGET_IMAGE="$REGISTRY/ublue-os/$TARGET_SYSTEM"

if [ "$VARIANT" != "base" ]; then
    TARGET_IMAGE="$TARGET_IMAGE-$VARIANT"
fi
TARGET_IMAGE="$TARGET_IMAGE:latest"

echo "Current system: $CURRENT_SYSTEM"
echo "Target system: $TARGET_SYSTEM${VARIANT:+ ($VARIANT)}"
echo "Target image: $TARGET_IMAGE"

if [ "$FORCE" != true ]; then
    echo
    read -p "Continue with system switch? [y/N] " -n 1 -r
    echo
    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
        echo "Aborted"
        exit 1
    fi
fi

# Create backup if requested
if [ "$BACKUP" = true ]; then
    echo "Creating backup..."
    sudo ostree admin pin 0
fi

# Perform the switch
echo "Switching to $TARGET_SYSTEM..."
sudo bootc switch "$TARGET_IMAGE"

echo
echo "System switch initiated. Reboot to complete the transition."
echo "After reboot, run 'rpm-ostree status' to verify the new system."

if [ "$BACKUP" = true ]; then
    echo "Backup created. You can rollback with 'rpm-ostree rollback' if needed."
fi
```

### 14.2 Configuration Migration Tool

Create `scripts/migrate-config.sh`:

```bash
#!/bin/bash
set -euo pipefail

SOURCE_SYSTEM="$1"
TARGET_SYSTEM="$2"
CONFIG_DIR="$HOME/.config/ublue-migration"

mkdir -p "$CONFIG_DIR"

echo "Migrating configuration from $SOURCE_SYSTEM to $TARGET_SYSTEM..."

# Export current Flatpaks
echo "Backing up Flatpaks..."
flatpak list --app --columns=application > "$CONFIG_DIR/flatpaks-user.txt"
flatpak list --app --system --columns=application > "$CONFIG_DIR/flatpaks-system.txt"

# Export dconf settings
echo "Backing up GNOME/KDE settings..."
dconf dump / > "$CONFIG_DIR/dconf-settings.txt"

# Export layered packages
echo "Backing up layered packages..."
rpm-ostree status --json | jq -r '.deployments[0]."requested-packages"[]?' > "$CONFIG_DIR/layered-packages.txt"

# System-specific migrations
case "$TARGET_SYSTEM" in
    "bazzite")
        echo "Preparing gaming-specific configuration..."
        # Gaming controller configs, Steam settings, etc.
        if [ -d "$HOME/.steam" ]; then
            echo "Steam configuration found, will be preserved"
        fi
        ;;
    "bluefin")
        echo "Preparing development environment..."
        # Export distrobox containers
        if command -v distrobox >/dev/null 2>&1; then
            distrobox list | tail -n +2 > "$CONFIG_DIR/distroboxes.txt"
        fi
        ;;
    "aurora")
        echo "Preparing KDE configuration..."
        # KDE-specific settings
        if [ -d "$HOME/.config/kde" ]; then
            tar -czf "$CONFIG_DIR/kde-config.tar.gz" -C "$HOME/.config" kde
        fi
        ;;
esac

# Create restoration script
cat > "$CONFIG_DIR/restore-config.sh" << EOF
#!/bin/bash
# Configuration restoration script for $TARGET_SYSTEM

echo "Restoring configuration for $TARGET_SYSTEM..."

# Restore Flatpaks
if [ -f "$CONFIG_DIR/flatpaks-user.txt" ]; then
    while read -r app; do
        flatpak install -y --user flathub "\$app" 2>/dev/null || true
    done < "$CONFIG_DIR/flatpaks-user.txt"
fi

# Restore dconf settings (be careful with cross-desktop migrations)
if [ -f "$CONFIG_DIR/dconf-settings.txt" ] && command -v dconf >/dev/null 2>&1; then
    echo "Restoring desktop settings..."
    dconf load / < "$CONFIG_DIR/dconf-settings.txt"
fi

# Restore layered packages
if [ -f "$CONFIG_DIR/layered-packages.txt" ]; then
    echo "Restoring layered packages..."
    packages=\$(cat "$CONFIG_DIR/layered-packages.txt" | tr '\n' ' ')
    if [ -n "\$packages" ]; then
        rpm-ostree install \$packages
    fi
fi

echo "Configuration restoration complete"
echo "Some applications may need to be reconfigured manually"
EOF

chmod +x "$CONFIG_DIR/restore-config.sh"

echo "Configuration backup complete"
echo "After switching systems, run: $CONFIG_DIR/restore-config.sh"
```

## Conclusion

This comprehensive guide provides a complete Universal Blue ecosystem for self-hosted infrastructure, including:

### Core Tools Covered:
- **bootc & bootupd**: Container-native OS management
- **bootc-image-builder**: Standard bootable image generation  
- **osbuilder**: Advanced multi-architecture and cloud-optimized builds
- **rpm-ostree**: Package layering and system management
- **BlueBuild**: Declarative image building with recipe files
- **ujust**: System automation and user-friendly commands
- **forge**: On-premise Universal Blue infrastructure
- **startingpoint**: Template for custom image creation
- **upgrade-tools**: Safe system migration utilities
- **ublue-os packages**: Standardized package management

### Universal Blue Systems Supported:
- **Bazzite**: Gaming desktop and handheld optimization
- **Bluefin**: Developer workstation with containers and tools  
- **Aurora**: KDE desktop environment variant
- **uCore**: Server/edge container platform
- **Custom Images**: Your own Universal Blue derivatives

### Key Capabilities:
1. **Multi-system builds** with parallel CI/CD pipelines
2. **Declarative configuration** using BlueBuild recipes
3. **Advanced customization** with osbuilder for cloud/edge deployments
4. **User-friendly management** through ujust commands
5. **Safe system migration** between Universal Blue variants
6. **Comprehensive testing** and validation pipelines
7. **On-premise infrastructure** with forge web interface
8. **Security and signing** with cosign and SBOMs
9. **Health monitoring** and automated maintenance
10. **Complete documentation** and user guides

This ecosystem provides enterprise-grade container-native operating system distribution capabilities while maintaining the user-friendly experience that makes Universal Blue systems popular for desktop, gaming, development, and server workloads.

The combination of all these tools creates a robust, maintainable, and scalable platform for delivering modern Linux distributions entirely on your own infrastructure.