debian-ostree-systems-notes/eos-updater.md

# eos-updater Project Breakdown

## Project Overview

eos-updater is the **production-grade system update infrastructure** for Endless OS, implementing a complete OSTree-based automatic update system. Unlike the build tools we've examined (deb-ostree-builder and apt2ostree), eos-updater focuses on the **runtime update delivery and management** for deployed systems.

## Repository Structure

Based on the project analysis, here's the inferred repository structure:

```
eos-updater/
├── README.md                          # Project documentation
├── meson.build                        # Main build configuration
├── meson_options.txt                  # Build options
├── eos-autoupdater/                   # Automatic update scheduler
│   ├── eos-autoupdater.c             # Main autoupdater daemon
│   ├── meson.build                   # Autoupdater build config
│   └── docs/
│       ├── eos-autoupdater.8         # Man page
│       └── eos-autoupdater.conf.5    # Configuration man page
├── eos-updater/                       # Core update daemon
│   ├── eos-updater.c                 # Main updater daemon
│   ├── eos-updater.h                 # Header definitions
│   ├── eos-updater-dbus.c            # D-Bus interface implementation
│   ├── eos-updater-poll.c            # Update polling logic
│   ├── eos-updater-fetch.c           # Update downloading
│   ├── eos-updater-apply.c           # Update application
│   ├── meson.build                   # Updater build config
│   └── docs/
│       ├── eos-updater.8             # Man page
│       └── eos-updater.conf.5        # Configuration man page
├── eos-updater-ctl/                   # Command-line control tool
│   ├── eos-updater-ctl.c             # CLI interface
│   ├── meson.build
│   └── docs/
│       └── eos-updater-ctl.8         # Man page
├── eos-update-server/                 # Local network update server
│   ├── eos-update-server.c           # HTTP server for local updates
│   ├── eos-update-server-dbus.c      # D-Bus integration
│   ├── meson.build
│   └── docs/
│       ├── eos-update-server.8       # Man page
│       └── eos-update-server.conf.5  # Configuration man page
├── eos-updater-avahi/                 # Network discovery service
│   ├── eos-updater-avahi.c           # Avahi/mDNS integration
│   ├── meson.build
│   └── docs/
│       └── eos-updater-avahi.8       # Man page
├── eos-updater-prepare-volume/        # USB/removable media support
│   ├── eos-updater-prepare-volume.c  # Volume preparation utility
│   ├── meson.build
│   └── docs/
│       └── eos-updater-prepare-volume.8
├── libeos-updater-util/               # Shared utility library
│   ├── util.c                        # Common utility functions
│   ├── util.h                        # Utility headers
│   ├── config.c                      # Configuration management
│   ├── ostree-util.c                 # OSTree helper functions
│   └── meson.build
├── libeos-updater-flatpak-installer/  # Flatpak integration
│   ├── installer.c                   # Flatpak application installer
│   ├── installer.h                   # Installer headers
│   └── meson.build
├── data/                              # Configuration and service files
│   ├── eos-autoupdater.conf           # Default autoupdater config
│   ├── eos-updater.conf               # Default updater config
│   ├── systemd/                       # Systemd service files
│   │   ├── eos-autoupdater.service
│   │   ├── eos-autoupdater.timer
│   │   ├── eos-updater.service
│   │   └── eos-update-server.service
│   └── dbus/                          # D-Bus service definitions
│       ├── com.endlessm.Updater.service
│       ├── com.endlessm.UpdaterAvahi.service
│       └── com.endlessm.UpdateServer.service
├── tests/                             # Test suite
│   ├── test-autoupdater.c            # Autoupdater tests
│   ├── test-updater.c                # Core updater tests
│   ├── test-util.c                   # Utility function tests
│   └── installed-tests/              # Integration tests
│       ├── test-update-from-lan.py
│       ├── test-update-from-usb.py
│       └── test-update-lifecycle.py
├── docs/                              # Additional documentation
│   ├── design.md                     # System design documentation
│   ├── testing.md                    # Testing guidelines
│   └── deployment.md                 # Deployment instructions
└── scripts/                           # Utility scripts
    ├── ostree-update-generator.sh    # Update generation scripts
    └── test-server-setup.sh          # Test environment setup
```

## How eos-updater Works

### 1. **Component Architecture**

eos-updater implements a **multi-component distributed system** for OSTree updates:

#### Core Components
```mermaid
graph TB
    subgraph "Client System"
        A[eos-autoupdater] --> B[eos-updater daemon]
        B --> C[OSTree Repository]
        D[eos-updater-ctl] --> B
        E[Desktop UI] --> B
    end
    
    subgraph "Network Services"
        F[eos-update-server] --> G[Local OSTree Repo]
        H[eos-updater-avahi] --> F
    end
    
    subgraph "External"
        I[Internet Update Server]
        J[USB/Removable Media]
    end
    
    B --> I
    B --> F
    B --> J
    
    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style F fill:#e8f5e8
```

### 2. **Update Process Flow**

The system implements a sophisticated multi-phase update process:

#### Phase 1: Automatic Update Detection
```c
// eos-autoupdater (triggered by systemd timer)
1. Wake up on schedule (daily/weekly)
2. Connect to eos-updater daemon via D-Bus
3. Initiate Poll() operation
4. Exit when automatic portion complete
```

#### Phase 2: Update Discovery and Download
```c
// eos-updater daemon operations
1. Poll remote servers for updates
2. Check multiple sources:
   - Internet update servers
   - Local network servers (via Avahi)
   - USB/removable media
3. Download update metadata
4. Download OSTree objects (if automatic download enabled)
```

#### Phase 3: User Interaction and Application
```c
// User-interactive portion
1. Desktop UI notifies user of available update
2. User chooses to apply update
3. eos-updater applies OSTree update
4. System prepares for reboot into new deployment
```

### 3. **D-Bus Interface Architecture**

The system uses D-Bus for inter-process communication:

#### Main D-Bus Interface
```xml
<!-- com.endlessm.Updater -->
<interface name="com.endlessm.Updater">
  <method name="Poll">
    <arg name="flags" type="u" direction="in"/>
  </method>
  
  <method name="Fetch">
    <arg name="flags" type="u" direction="in"/>
  </method>
  
  <method name="Apply">
    <arg name="flags" type="u" direction="in"/>
  </method>
  
  <property name="State" type="u" access="read"/>
  <property name="UpdateID" type="s" access="read"/>
  <property name="UpdateRefspec" type="s" access="read"/>
  
  <signal name="StateChanged">
    <arg name="old_state" type="u"/>
    <arg name="new_state" type="u"/>
  </signal>
</interface>
```

#### Update States
```c
typedef enum {
  EOS_UPDATER_STATE_NONE = 0,           // No update available
  EOS_UPDATER_STATE_READY,              // Ready to check for updates
  EOS_UPDATER_STATE_POLLING,            // Checking for updates
  EOS_UPDATER_STATE_UPDATE_AVAILABLE,   // Update found
  EOS_UPDATER_STATE_FETCHING,           // Downloading update
  EOS_UPDATER_STATE_UPDATE_READY,       // Update downloaded
  EOS_UPDATER_STATE_APPLYING_UPDATE,    // Installing update
  EOS_UPDATER_STATE_UPDATE_APPLIED,     // Update applied, reboot needed
  EOS_UPDATER_STATE_ERROR               // Error occurred
} EosUpdaterState;
```

### 4. **Network Update Discovery**

#### Local Network Updates (Avahi/mDNS)
```c
// eos-updater-avahi discovers local update servers
1. Advertise local update server via mDNS
2. Listen for other update servers on network
3. Coordinate with eos-update-server for serving
4. Enable LAN-based updates (no internet required)
```

#### Update Server Discovery Priority
```c
// Update source priority order
1. Local USB/removable media (highest priority)
2. Local network servers (LAN)
3. Internet update servers (lowest priority)
```

### 5. **OSTree Integration**

eos-updater wraps OSTree operations in a user-friendly daemon:

#### Core OSTree Operations
```c
// Simplified OSTree update process
ostree_repo_pull(repo, remote_name, refs_to_pull, flags, &error);
ostree_sysroot_load(sysroot, &error);
ostree_sysroot_deploy_tree(sysroot, os_name, revision, origin, 
                          merge_deployment, override_kernel_argv, &error);
ostree_sysroot_simple_write_deployment(sysroot, os_name, new_deployment, 
                                       merge_deployment, flags, &error);
```

#### Metadata Management
```c
// Update metadata tracking
typedef struct {
  gchar *id;              // Update identifier
  gchar *refspec;         // OSTree refspec
  gchar *original_refspec; // Original branch
  gchar *version;         // Version string  
  gchar *checksum;        // OSTree commit checksum
  guint64 download_size;  // Size of download
  guint64 installed_size; // Installed size
} EosUpdaterData;
```

### 6. **Configuration System**

#### Main Configuration Files
```ini
# /etc/eos-updater/eos-updater.conf
[Updates]
AutoUpdate=true
AutoUpdateInterval=86400
PollingEnabled=true
MaxBandwidth=1048576

[Network]
AllowLANUpdates=true
PreferLANUpdates=true

[OSTree]
RemoteName=eos
RepositoryURL=https://ostree.endlessm.com/ostree/eos-amd64
```

```ini
# /etc/eos-updater/eos-autoupdater.conf  
[Automatic Updates]
OnlineAutomatic=true
OfflineAutomatic=false
StampFile=/var/lib/eos-autoupdater/last-update-check
```

### 7. **Update Sources and Methods**

#### Internet Updates
```c
// Standard OSTree remote pulling
struct update_source internet_source = {
  .type = UPDATE_SOURCE_INTERNET,
  .url = "https://ostree.endlessm.com/ostree/eos-amd64",
  .priority = 3  // Lowest priority
};
```

#### LAN Updates
```c
// Local network update discovery
struct update_source lan_source = {
  .type = UPDATE_SOURCE_LAN,
  .discovery_method = AVAHI_MDNS,
  .priority = 2  // Medium priority
};
```

#### USB/Removable Media Updates
```c
// Offline update support
struct update_source usb_source = {
  .type = UPDATE_SOURCE_VOLUME,
  .mount_point = "/media/eos-updates",
  .priority = 1  // Highest priority
};
```

## Key Technical Features

### 1. **Graduated Automation**

The system implements **configurable automation levels**:

```c
// Automation configuration options
typedef enum {
  AUTO_UPDATE_DISABLED = 0,    // Manual updates only
  AUTO_UPDATE_DOWNLOAD,        // Auto download, manual apply
  AUTO_UPDATE_FULL            // Fully automatic updates
} AutoUpdateLevel;
```

#### Example Automation Flow
```
Automatic Phase (eos-autoupdater):
├── Check for updates ✓
├── Download updates ✓ 
└── Exit, wait for user

User Interactive Phase:
├── Desktop notification appears
├── User chooses to apply update
└── System applies and schedules reboot
```

### 2. **Multi-Source Update Discovery**

#### Source Priority System
```c
// Update source evaluation
for (source in update_sources) {
  if (source_has_newer_update(source)) {
    if (source.priority > best_source.priority) {
      best_source = source;
      best_update = source.update;
    }
  }
}
```

#### LAN Update Optimization
- **Bandwidth Savings**: Updates from local network instead of internet
- **Speed Improvements**: Local network speeds vs. internet bandwidth
- **Offline Capability**: Updates possible without internet connectivity

### 3. **Robust Error Handling**

#### Update Failure Recovery
```c
// Error state management
switch (update_error) {
  case UPDATE_ERROR_NETWORK:
    retry_with_backoff();
    break;
  case UPDATE_ERROR_DISK_SPACE:
    notify_user_insufficient_space();
    break;
  case UPDATE_ERROR_OSTREE:
    rollback_to_previous_state();
    break;
}
```

#### State Persistence
- **Update state survives reboots**: daemon tracks progress across restarts
- **Atomic operations**: Either update completes fully or rolls back
- **Resume capability**: Can resume interrupted downloads

### 4. **Integration with System Services**

#### Systemd Integration
```ini
# eos-autoupdater.service
[Unit]
Description=Endless OS Automatic Updater
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/eos-autoupdater
User=eos-updater

# eos-autoupdater.timer  
[Unit]
Description=Endless OS Automatic Update Timer

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=3600

[Install]
WantedBy=timers.target
```

#### Network Manager Integration
```c
// Network awareness
if (network_manager_is_metered_connection()) {
  if (!allow_metered_updates) {
    defer_update_until_unmetered();
    return;
  }
}
```

## Component Deep Dive

### 1. **eos-autoupdater - Scheduled Update Initiator**

This component runs on a systemd timer and handles the automatic portion of updates:

#### Responsibilities
- **Scheduled Checks**: Runs daily/weekly based on configuration
- **Network Awareness**: Respects metered connections and user preferences
- **Battery Awareness**: Defers updates on low battery (laptops)
- **User Presence Detection**: Avoids updates during active user sessions

#### Example Operation Flow
```c
int main() {
  // Connect to eos-updater daemon
  GDBusProxy *updater = connect_to_updater_daemon();
  
  // Check configuration
  if (!is_automatic_updates_enabled()) {
    exit(0);
  }
  
  // Initiate poll for updates
  g_dbus_proxy_call_sync(updater, "Poll", 
                        g_variant_new("(u)", POLL_FLAGS_AUTOMATIC),
                        G_DBUS_CALL_FLAGS_NONE, -1, NULL, &error);
  
  // Wait for automatic phase to complete
  wait_for_state_change(EOS_UPDATER_STATE_UPDATE_AVAILABLE);
  
  // Exit - user interaction now required
  return 0;
}
```

### 2. **eos-updater - Core Update Daemon**

The main daemon that orchestrates the entire update process:

#### State Machine Implementation
```c
// Update state machine
static void handle_state_transition(EosUpdater *updater,
                                   EosUpdaterState old_state,
                                   EosUpdaterState new_state) {
  switch (new_state) {
    case EOS_UPDATER_STATE_POLLING:
      start_update_poll(updater);
      break;
    case EOS_UPDATER_STATE_FETCHING:
      start_update_download(updater);
      break;
    case EOS_UPDATER_STATE_APPLYING_UPDATE:
      start_update_application(updater);
      break;
  }
  
  emit_state_changed_signal(updater, old_state, new_state);
}
```

#### OSTree Integration
```c
// Wrapping OSTree operations in daemon
static gboolean poll_for_updates(EosUpdater *updater) {
  OstreeRepo *repo = ostree_repo_new_default();
  
  // Check each configured remote
  for (remote in configured_remotes) {
    gchar *checksum = NULL;
    ostree_repo_resolve_rev(repo, remote.refspec, FALSE, &checksum, &error);
    
    if (checksum && !g_str_equal(checksum, current_deployment_checksum)) {
      updater->update_available = TRUE;
      updater->update_checksum = g_strdup(checksum);
      break;
    }
  }
  
  set_state(updater, EOS_UPDATER_STATE_UPDATE_AVAILABLE);
  return TRUE;
}
```

### 3. **eos-update-server - LAN Update Distribution**

Provides local network update distribution to reduce bandwidth usage:

#### HTTP Server Implementation
```c
// Simple HTTP server for OSTree repositories
static void handle_ostree_request(SoupServer *server,
                                 SoupMessage *msg,
                                 const char *path,
                                 GHashTable *query,
                                 SoupClientContext *context,
                                 gpointer user_data) {
  
  EosUpdateServer *self = EOS_UPDATE_SERVER(user_data);
  
  // Serve OSTree repository files
  if (g_str_has_prefix(path, "/ostree/")) {
    serve_ostree_file(self, msg, path);
  }
}
```

#### Avahi Integration
```c
// Service advertisement via mDNS
static void advertise_update_service(EosUpdateServer *server) {
  AvahiEntryGroup *group;
  
  avahi_entry_group_add_service(group,
                               AVAHI_IF_UNSPEC,
                               AVAHI_PROTO_UNSPEC,
                               0,
                               "EOS Update Server",
                               "_eos_updater._tcp",
                               NULL, NULL,
                               server->port,
                               "version=1.0",
                               "path=/ostree",
                               NULL);
}
```

### 4. **eos-updater-ctl - Command Line Interface**

Provides administrative control over the update system:

#### CLI Commands
```bash
# Check current state
eos-updater-ctl status

# Force update check
eos-updater-ctl poll

# Download available update
eos-updater-ctl fetch

# Apply downloaded update
eos-updater-ctl apply

# Monitor update progress
eos-updater-ctl monitor
```

#### Implementation Example
```c
static int cmd_status(int argc, char **argv) {
  GDBusProxy *proxy = connect_to_updater();
  GVariant *state_variant;
  
  g_dbus_proxy_get_cached_property(proxy, "State");
  EosUpdaterState state = g_variant_get_uint32(state_variant);
  
  printf("Update Status: %s\n", state_to_string(state));
  
  if (state == EOS_UPDATER_STATE_UPDATE_AVAILABLE) {
    print_update_details(proxy);
  }
  
  return 0;
}
```

## Advanced Features

### 1. **Flatpak Application Updates**

The system includes integrated Flatpak application management:

```c
// libeos-updater-flatpak-installer
static gboolean install_flatpak_updates(EosUpdater *updater) {
  FlatpakInstallation *installation;
  GList *updates;
  
  // Get available Flatpak updates
  updates = flatpak_installation_list_installed_refs_for_update(installation, 
                                                               NULL, &error);
  
  // Apply updates in parallel with OS update
  for (update in updates) {
    flatpak_installation_update(installation, 
                               FLATPAK_UPDATE_FLAGS_NO_DEPLOY,
                               update->kind, update->name, 
                               update->arch, update->branch,
                               NULL, &error);
  }
  
  return TRUE;
}
```

### 2. **Bandwidth Management**

#### Adaptive Download Strategy
```c
// Network condition awareness
static void adjust_download_strategy(EosUpdater *updater) {
  NetworkInfo *info = get_network_info();
  
  if (info->is_metered) {
    updater->max_download_size = config->metered_max_size;
    updater->download_priority = PRIORITY_LOW;
  }
  
  if (info->bandwidth < BANDWIDTH_THRESHOLD_LOW) {
    enable_delta_updates(updater);  // Use smaller delta updates
  }
}
```

#### Delta Update Support
```c
// Efficient incremental updates
static gboolean fetch_delta_update(EosUpdater *updater) {
  // Try to fetch static delta between current and target
  gboolean delta_available = ostree_repo_pull_with_options(
    repo, remote_name,
    g_variant_new("a{sv}", "flags", 
                 g_variant_new_uint32(OSTREE_REPO_PULL_FLAGS_COMMIT_ONLY),
                 "refs", g_variant_new_strv(refs, -1),
                 NULL), NULL, &error);
  
  if (!delta_available) {
    // Fall back to full object download
    return fetch_full_update(updater);
  }
  
  return TRUE;
}
```

### 3. **Security and Verification**

#### GPG Signature Verification
```c
// Update verification
static gboolean verify_update(EosUpdater *updater, const gchar *checksum) {
  OstreeRepo *repo = updater->repo;
  
  // Verify GPG signature on commit
  gboolean valid = ostree_repo_verify_commit_ext(repo, checksum,
                                                NULL, NULL, 
                                                NULL, &error);
  if (!valid) {
    set_error_state(updater, "Update signature verification failed");
    return FALSE;
  }
  
  return TRUE;
}
```

## Deployment and Production Considerations

### 1. **System Integration**

#### Package Dependencies
```
Runtime Dependencies:
├── ostree (>= 2020.5)        # Core OSTree functionality
├── libglib2.0-0              # GLib for daemon infrastructure  
├── systemd                   # Service management
├── avahi-daemon              # Network discovery
├── libsoup2.4-1              # HTTP client/server
└── flatpak (optional)        # Application updates
```

#### Service Configuration
```systemd
# Complete service ecosystem
eos-autoupdater.timer          # Scheduled update checks
├── eos-autoupdater.service    # Update initiator
└── eos-updater.service        # Core daemon
    ├── eos-update-server.service     # LAN server
    └── eos-updater-avahi.service     # Discovery service
```

### 2. **Testing Infrastructure**

#### Comprehensive Test Suite
```c
// Unit tests for each component
test_autoupdater_scheduling()
test_updater_state_machine()
test_update_server_http_serving()
test_avahi_service_discovery()

// Integration tests
test_full_update_lifecycle()
test_lan_update_discovery()
test_usb_update_application()
test_error_recovery()
```

#### Production Validation
```bash
# Installed-only tests
gnome-desktop-testing-runner libeos-updater-util-0
gnome-desktop-testing-runner libeos-updater-flatpak-installer-0
sudo gnome-desktop-testing-runner eos-updater-0
```

## Real-World Impact

### 1. **User Experience Benefits**
- **Automatic Updates**: Users get security updates without intervention
- **Non-Disruptive**: Updates happen in background, apply on reboot
- **Rollback Safety**: Bad updates can be rolled back instantly
- **Bandwidth Optimization**: LAN updates reduce internet usage

### 2. **System Administrator Benefits**
- **Centralized Management**: Control update policies across organization
- **Bandwidth Efficiency**: LAN-based update distribution
- **Audit Trail**: Complete update history in OSTree
- **Flexible Deployment**: Support for air-gapped environments via USB

### 3. **Enterprise Features**
- **Staged Updates**: Download and prepare updates without applying
- **Update Scheduling**: Control when updates are applied
- **Network Efficiency**: Peer-to-peer update distribution
- **Compliance**: Maintain exact system state records

## Architecture Philosophy

eos-updater embodies **production-ready system design principles**:

1. **Separation of Concerns**: Each component has a single, well-defined responsibility
2. **Fault Tolerance**: Robust error handling and recovery mechanisms  
3. **User Agency**: Users maintain control over when updates are applied
4. **Network Efficiency**: Multiple strategies to minimize bandwidth usage
5. **Enterprise Ready**: Configuration and deployment suitable for organizational use

This represents a **mature, production implementation** of OSTree-based system updates, demonstrating how the theoretical benefits of atomic system management translate into practical, user-friendly update experiences. The system successfully bridges the gap between OSTree's technical capabilities and the real-world needs of desktop and embedded system users.

## Comparison Summary

| Aspect | eos-updater | deb-ostree-builder | apt2ostree |
|--------|-------------|-------------------|------------|
| **Purpose** | Runtime update delivery | Build system creation | Build optimization |
| **Focus** | User experience | System conversion | Developer workflow |  
| **Language** | C with GLib | Bash scripts | Python library |
| **Scope** | Complete update infrastructure | Build orchestration | Build performance |
| **Target** | End users | System administrators | Developers |

eos-updater completes the OSTree ecosystem by providing the production update delivery system that end users interact with daily.