apt-ostree/docs/apt-ostree-daemon-plan/implementation/deployment-guide.md

# 🚀 **apt-ostree Deployment Guide**

## 🎯 **Overview**

This document provides comprehensive guidance for deploying apt-ostree in production environments, including system requirements, installation, configuration, monitoring, and maintenance procedures. The deployment approach ensures reliability, security, and optimal performance.

## 📋 **System Requirements**

### **Hardware Requirements**

#### **Minimum Requirements**
- **CPU**: 2 cores (x86_64/amd64)
- **RAM**: 4GB
- **Storage**: 20GB available space
- **Network**: Internet connectivity for package updates

#### **Recommended Requirements**
- **CPU**: 4+ cores (x86_64/amd64)
- **RAM**: 8GB+
- **Storage**: 50GB+ available space
- **Network**: High-speed internet connection
- **SSD**: For optimal performance

### **Software Requirements**

#### **Operating System**
- **Debian**: 13 (Trixie) or later
- **Ubuntu**: 25.04 (Plucky Puffin) or later
- **Kernel**: Linux 6.12+ with OSTree support

#### **System Dependencies**
```bash
# Essential system packages
sudo apt update
sudo apt install -y \
    systemd \
    dbus \
    polkit \
    ostree \
    apt \
    dpkg \
    systemd-boot \
    dracut

# Development tools (for building from source)
sudo apt install -y \
    build-essential \
    pkg-config \
    cmake \
    git \
    curl
```

## 🏗️ **Installation Methods**

### **Method 1: Package Installation (Recommended)**

#### **Add Repository**
```bash
# Add apt-ostree repository
echo "deb [signed-by=/usr/share/keyrings/apt-ostree-archive-keyring.gpg] \
    https://apt.ostree.dev/debian trixie main" | \
    sudo tee /etc/apt/sources.list.d/apt-ostree.list

# Add repository key
curl -fsSL https://apt.ostree.dev/debian/apt-ostree-archive-keyring.gpg | \
    sudo gpg --dearmor -o /usr/share/keyrings/apt-ostree-archive-keyring.gpg

# Update package lists
sudo apt update
```

#### **Install apt-ostree**
```bash
# Install apt-ostree
sudo apt install -y apt-ostree

# Verify installation
apt-ostree --version

# Check system status
apt-ostree status
```

### **Method 2: Build from Source**

#### **Clone Repository**
```bash
# Clone the repository
git clone https://github.com/your-org/apt-ostree.git
cd apt-ostree

# Checkout stable release
git checkout v1.0.0
```

#### **Build and Install**
```bash
# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Build the project
cargo build --release

# Install system-wide
sudo cp target/release/apt-ostree /usr/local/bin/
sudo cp target/release/aptostreed /usr/local/bin/

# Install systemd services
sudo cp systemd/aptostreed.service /etc/systemd/system/
sudo cp systemd/aptostreed.socket /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload
```

## ⚙️ **Configuration**

### **Main Configuration File**

#### **Configuration Location**
```bash
# Main configuration file
sudo mkdir -p /etc/apt-ostree
sudo nano /etc/apt-ostree/config.toml
```

#### **Configuration Example**
```toml
# /etc/apt-ostree/config.toml
[system]
# System-wide settings
ostree_path = "/ostree"
sysroot_path = "/"
default_os = "debian"

[daemon]
# Daemon configuration
host = "127.0.0.1"
port = 8080
max_connections = 100
idle_timeout = 300
log_level = "info"

[security]
# Security settings
polkit_enabled = true
require_authorization = true
allowed_users = ["root", "admin"]
allowed_groups = ["sudo", "wheel"]

[packages]
# Package management settings
auto_update = false
update_interval = 86400
cache_dir = "/var/cache/apt-ostree"
max_cache_size = "10GB"

[ostree]
# OSTree settings
repo_mode = "bare-user"
compression = "gzip"
commit_timeout = 300
max_commits = 100

[logging]
# Logging configuration
log_file = "/var/log/apt-ostree/aptostreed.log"
max_log_size = "100MB"
max_log_files = 5
log_format = "json"

[monitoring]
# Monitoring settings
metrics_enabled = true
metrics_port = 9090
health_check_interval = 60
```

### **Environment-Specific Configuration**

#### **Development Environment**
```toml
# /etc/apt-ostree/config.dev.toml
[system]
ostree_path = "/tmp/ostree-dev"
sysroot_path = "/tmp/sysroot-dev"

[daemon]
log_level = "debug"
host = "0.0.0.0"

[security]
polkit_enabled = false
require_authorization = false

[packages]
auto_update = true
update_interval = 3600
```

#### **Production Environment**
```toml
# /etc/apt-ostree/config.prod.toml
[system]
ostree_path = "/ostree"
sysroot_path = "/"

[daemon]
log_level = "warn"
host = "127.0.0.1"

[security]
polkit_enabled = true
require_authorization = true

[packages]
auto_update = false
update_interval = 86400

[monitoring]
metrics_enabled = true
health_check_interval = 30
```

### **Polkit Configuration**

#### **Policy Files**
```xml
<!-- /usr/share/polkit-1/actions/org.projectatomic.aptostree.policy -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policyconfig PUBLIC
 "-//freedesktop//DTD PolicyKit Policy Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/PolicyKit/1/policyconfig.dtd">
<policyconfig>
  <vendor>Project Atomic</vendor>
  <vendor_url>https://github.com/projectatomic/apt-ostree</vendor_url>
  
  <action id="org.projectatomic.aptostree.status">
    <description>Get system status</description>
    <message>Authentication is required to get system status</message>
    <defaults>
      <allow_any>yes</allow_any>
      <allow_inactive>yes</allow_inactive>
      <allow_active>yes</allow_active>
    </defaults>
  </action>
  
  <action id="org.projectatomic.aptostree.upgrade">
    <description>Upgrade system</description>
    <message>Authentication is required to upgrade the system</message>
    <defaults>
      <allow_any>auth_admin</allow_any>
      <allow_inactive>auth_admin</allow_inactive>
      <allow_active>auth_admin</allow_active>
    </defaults>
  </action>
  
  <action id="org.projectatomic.aptostree.install">
    <description>Install packages</description>
    <message>Authentication is required to install packages</message>
    <defaults>
      <allow_any>auth_admin</allow_any>
      <allow_inactive>auth_admin</allow_inactive>
      <allow_active>auth_admin</allow_active>
    </defaults>
  </action>
</policyconfig>
```

#### **Rules Configuration**
```javascript
// /etc/polkit-1/rules.d/50-apt-ostree.rules
polkit.addRule(function(action, subject) {
    if (action.id == "org.projectatomic.aptostree.status") {
        return polkit.Result.YES;
    }
    
    if (action.id == "org.projectatomic.aptostree.upgrade" ||
        action.id == "org.projectatomic.aptostree.install") {
        if (subject.isInGroup("sudo") || subject.isInGroup("wheel")) {
            return polkit.Result.YES;
        }
    }
    
    return polkit.Result.NO;
});
```

## 🚀 **Service Management**

### **Systemd Service Configuration**

#### **Daemon Service**
```ini
# /etc/systemd/system/aptostreed.service
[Unit]
Description=apt-ostree Daemon
Documentation=man:aptostreed(8)
After=network.target ostree.service
Requires=ostree.service
Wants=polkit.service

[Service]
Type=notify
ExecStart=/usr/bin/aptostreed --config /etc/apt-ostree/config.toml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
User=root
Group=root
RuntimeDirectory=apt-ostree
RuntimeDirectoryMode=0755
StateDirectory=apt-ostree
StateDirectoryMode=0755
LogsDirectory=apt-ostree
LogsDirectoryMode=0755

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/ostree /var/lib/apt-ostree /var/cache/apt-ostree
CapabilityBoundingSet=CAP_SYS_ADMIN CAP_DAC_OVERRIDE

[Install]
WantedBy=multi-user.target
```

#### **Socket Configuration**
```ini
# /etc/systemd/system/aptostreed.socket
[Unit]
Description=apt-ostree Daemon Socket
Documentation=man:aptostreed(8)
Before=aptostreed.service

[Socket]
ListenStream=127.0.0.1:8080
SocketUser=root
SocketGroup=root
SocketMode=0660

[Install]
WantedBy=sockets.target
```

### **Service Management Commands**

#### **Start and Enable Services**
```bash
# Enable and start services
sudo systemctl enable aptostreed.socket
sudo systemctl enable aptostreed.service
sudo systemctl start aptostreed.socket
sudo systemctl start aptostreed.service

# Check service status
sudo systemctl status aptostreed.service
sudo systemctl status aptostreed.socket

# View service logs
sudo journalctl -u aptostreed.service -f
```

#### **Service Control**
```bash
# Restart services
sudo systemctl restart aptostreed.service

# Reload configuration
sudo systemctl reload aptostreed.service

# Stop services
sudo systemctl stop aptostreed.service
sudo systemctl stop aptostreed.socket

# Disable services
sudo systemctl disable aptostreed.service
sudo systemctl disable aptostreed.socket
```

## 🔒 **Security Configuration**

### **User and Group Management**

#### **Create Dedicated User**
```bash
# Create apt-ostree user
sudo useradd -r -s /bin/false -d /var/lib/apt-ostree aptostree

# Create necessary directories
sudo mkdir -p /var/lib/apt-ostree
sudo mkdir -p /var/cache/apt-ostree
sudo mkdir -p /var/log/apt-ostree

# Set ownership
sudo chown -R aptostree:aptostree /var/lib/apt-ostree
sudo chown -R aptostree:aptostree /var/cache/apt-ostree
sudo chown -R aptostree:aptostree /var/log/apt-ostree

# Set permissions
sudo chmod 750 /var/lib/apt-ostree
sudo chmod 750 /var/cache/apt-ostree
sudo chmod 750 /var/log/apt-ostree
```

#### **Configure sudo Access**
```bash
# Add user to sudo group
sudo usermod -a -G sudo aptostree

# Configure sudoers
echo "aptostree ALL=(ALL) NOPASSWD: /usr/bin/apt-ostree" | \
    sudo tee /etc/sudoers.d/apt-ostree

# Verify configuration
sudo visudo -c
```

### **Network Security**

#### **Firewall Configuration**
```bash
# Configure UFW firewall
sudo ufw allow from 127.0.0.1 to any port 8080
sudo ufw allow from 192.168.1.0/24 to any port 8080

# Enable firewall
sudo ufw enable

# Check firewall status
sudo ufw status
```

#### **Network Isolation**
```bash
# Create network namespace
sudo ip netns add aptostree

# Move interface to namespace
sudo ip link set eth0 netns aptostree

# Configure namespace networking
sudo ip netns exec aptostree ip addr add 192.168.100.1/24 dev eth0
sudo ip netns exec aptostree ip link set eth0 up
```

## 📊 **Monitoring and Logging**

### **Logging Configuration**

#### **Log Rotation**
```bash
# /etc/logrotate.d/apt-ostree
/var/log/apt-ostree/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 644 aptostree aptostree
    postrotate
        systemctl reload aptostreed.service
    endscript
}
```

#### **Structured Logging**
```json
// Example log entry
{
    "timestamp": "2024-01-15T10:30:00Z",
    "level": "info",
    "component": "daemon",
    "operation": "package_install",
    "user_id": 1000,
    "package": "vim",
    "version": "2:9.0.1378-1",
    "duration_ms": 1500,
    "success": true,
    "message": "Package vim installed successfully"
}
```

### **Metrics Collection**

#### **Prometheus Metrics**
```rust
// Metrics endpoint configuration
use prometheus::{Registry, Counter, Histogram, Gauge};

pub struct Metrics {
    pub operations_total: Counter,
    pub operation_duration: Histogram,
    pub active_connections: Gauge,
    pub package_installations: Counter,
}

impl Metrics {
    pub fn new(registry: &Registry) -> Self {
        let operations_total = Counter::new(
            "apt_ostree_operations_total",
            "Total number of operations"
        ).unwrap();
        
        let operation_duration = Histogram::new(
            "apt_ostree_operation_duration_seconds",
            "Operation duration in seconds"
        ).unwrap();
        
        let active_connections = Gauge::new(
            "apt_ostree_active_connections",
            "Number of active connections"
        ).unwrap();
        
        let package_installations = Counter::new(
            "apt_ostree_package_installations_total",
            "Total number of package installations"
        ).unwrap();
        
        registry.register(Box::new(operations_total.clone())).unwrap();
        registry.register(Box::new(operation_duration.clone())).unwrap();
        registry.register(Box::new(active_connections.clone())).unwrap();
        registry.register(Box::new(package_installations.clone())).unwrap();
        
        Self {
            operations_total,
            operation_duration,
            active_connections,
            package_installations,
        }
    }
}
```

#### **Health Check Endpoint**
```rust
// Health check implementation
use axum::{routing::get, Router, Json};
use serde_json::json;

pub async fn health_check() -> Json<serde_json::Value> {
    let health_status = check_system_health().await;
    
    Json(json!({
        "status": if health_status.is_healthy { "healthy" } else { "unhealthy" },
        "timestamp": chrono::Utc::now().to_rfc3339(),
        "version": env!("CARGO_PKG_VERSION"),
        "uptime": get_system_uptime(),
        "components": {
            "ostree": health_status.ostree_healthy,
            "apt": health_status.apt_healthy,
            "database": health_status.database_healthy,
            "filesystem": health_status.filesystem_healthy
        },
        "details": health_status.details
    }))
}

pub fn create_health_router() -> Router {
    Router::new()
        .route("/health", get(health_check))
        .route("/ready", get(health_check))
}
```

## 🔧 **Maintenance Procedures**

### **Regular Maintenance Tasks**

#### **Daily Tasks**
```bash
# Check service status
sudo systemctl status aptostreed.service

# Check log files
sudo tail -f /var/log/apt-ostree/aptostreed.log

# Check disk space
df -h /ostree /var/cache/apt-ostree

# Check system resources
htop
```

#### **Weekly Tasks**
```bash
# Clean old packages
sudo apt-ostree cleanup

# Update package lists
sudo apt-ostree refresh-md

# Check for updates
sudo apt-ostree upgrade --preview

# Rotate log files
sudo logrotate -f /etc/logrotate.d/apt-ostree
```

#### **Monthly Tasks**
```bash
# Full system health check
sudo apt-ostree status --verbose

# Check OSTree repository health
sudo ostree fsck

# Review and clean old deployments
sudo apt-ostree cleanup --old-deployments

# Update system packages
sudo apt-ostree upgrade
```

### **Backup and Recovery**

#### **Backup Procedures**
```bash
# Backup configuration
sudo tar -czf /backup/apt-ostree-config-$(date +%Y%m%d).tar.gz \
    /etc/apt-ostree /etc/systemd/system/aptostreed*

# Backup OSTree repository
sudo ostree admin backup --repo=/ostree/repo /backup/ostree-backup-$(date +%Y%m%d)

# Backup package cache
sudo tar -czf /backup/apt-cache-$(date +%Y%m%d).tar.gz /var/cache/apt-ostree

# Backup logs
sudo tar -czf /backup/apt-ostree-logs-$(date +%Y%m%d).tar.gz /var/log/apt-ostree
```

#### **Recovery Procedures**
```bash
# Restore configuration
sudo tar -xzf /backup/apt-ostree-config-$(date +%Y%m%d).tar.gz -C /

# Restore OSTree repository
sudo ostree admin restore --repo=/ostree/repo /backup/ostree-backup-$(date +%Y%m%d)

# Restart services
sudo systemctl restart aptostreed.service

# Verify recovery
sudo apt-ostree status
```

## 🚨 **Troubleshooting**

### **Common Issues**

#### **Service Won't Start**
```bash
# Check service status
sudo systemctl status aptostreed.service

# Check logs
sudo journalctl -u aptostreed.service -n 50

# Check configuration
sudo aptostreed --config /etc/apt-ostree/config.toml --validate

# Check dependencies
sudo systemctl status ostree.service
sudo systemctl status dbus.service
```

#### **Permission Issues**
```bash
# Check file permissions
ls -la /ostree /var/lib/apt-ostree /var/cache/apt-ostree

# Check user/group ownership
id aptostree
groups aptostree

# Fix permissions
sudo chown -R aptostree:aptostree /ostree
sudo chown -R aptostree:aptostree /var/lib/apt-ostree
sudo chown -R aptostree:aptostree /var/cache/apt-ostree
```

#### **Network Issues**
```bash
# Check network connectivity
ping -c 3 8.8.8.8

# Check service binding
sudo netstat -tlnp | grep 8080

# Check firewall
sudo ufw status

# Test local connectivity
curl -v http://127.0.0.1:8080/health
```

### **Debug Mode**

#### **Enable Debug Logging**
```bash
# Edit configuration
sudo nano /etc/apt-ostree/config.toml

# Set log level to debug
log_level = "debug"

# Restart service
sudo systemctl restart aptostreed.service

# Monitor logs
sudo journalctl -u aptostreed.service -f
```

#### **Command Line Debugging**
```bash
# Run daemon in foreground with debug
sudo aptostreed --config /etc/apt-ostree/config.toml --debug --foreground

# Test CLI commands with verbose output
apt-ostree --debug status --verbose

# Check system information
apt-ostree status --json
```

## 📈 **Performance Optimization**

### **System Tuning**

#### **Kernel Parameters**
```bash
# /etc/sysctl.conf
# Increase file descriptor limits
fs.file-max = 1000000

# Optimize memory management
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5

# Network tuning
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 5000

# Apply changes
sudo sysctl -p
```

#### **Resource Limits**
```bash
# /etc/security/limits.conf
aptostree soft nofile 65536
aptostree hard nofile 65536
aptostree soft nproc 32768
aptostree hard nproc 32768
```

### **Application Optimization**

#### **Connection Pooling**
```rust
// Connection pool configuration
use deadpool::managed::{Manager, Pool, PoolError};

pub struct ConnectionPool {
    pool: Pool<ConnectionManager>,
}

impl ConnectionPool {
    pub async fn new(max_connections: usize) -> Result<Self, PoolError> {
        let manager = ConnectionManager::new();
        let pool = Pool::builder(manager)
            .max_size(max_connections)
            .build()?;
        
        Ok(Self { pool })
    }
    
    pub async fn get_connection(&self) -> Result<PooledConnection, PoolError> {
        self.pool.get().await
    }
}
```

#### **Caching Strategy**
```rust
// Cache configuration
use moka::future::Cache;

pub struct CacheManager {
    package_cache: Cache<String, PackageInfo>,
    deployment_cache: Cache<String, DeploymentInfo>,
}

impl CacheManager {
    pub fn new() -> Self {
        let package_cache = Cache::builder()
            .max_capacity(10000)
            .time_to_live(Duration::from_secs(3600))
            .build();
            
        let deployment_cache = Cache::builder()
            .max_capacity(1000)
            .time_to_live(Duration::from_secs(1800))
            .build();
        
        Self {
            package_cache,
            deployment_cache,
        }
    }
}
```

## 🎯 **Next Steps**

### **Immediate Actions**
1. **Review system requirements** and ensure compatibility
2. **Choose installation method** (package vs source)
3. **Configure basic settings** and security policies
4. **Test basic functionality** and verify installation

### **Short-term Goals**
1. **Set up monitoring** and alerting systems
2. **Configure backup** and recovery procedures
3. **Implement performance** monitoring and optimization
4. **Establish maintenance** schedules and procedures

### **Long-term Vision**
1. **Automated deployment** and configuration management
2. **Multi-site deployment** and load balancing
3. **Advanced monitoring** and predictive maintenance
4. **Integration with** existing infrastructure management tools

---

*This deployment guide provides comprehensive instructions for deploying apt-ostree in production environments. For detailed architecture information, refer to the architecture documents in the `docs/apt-ostree-daemon-plan/architecture/` directory.*