ComposeSync/Docs/monitoring.md

# Monitoring and Logs

This guide covers how to monitor ComposeSync, view logs, and troubleshoot issues.

## Overview

ComposeSync provides comprehensive logging and monitoring capabilities to help you track updates, diagnose issues, and ensure your stacks are running smoothly.

## Service Control

### Check Service Status

```bash
# Check if ComposeSync is running
sudo systemctl status composesync

# Check if service is enabled
sudo systemctl is-enabled composesync
```

### Start/Stop/Restart Service

```bash
# Start the service
sudo systemctl start composesync

# Stop the service
sudo systemctl stop composesync

# Restart the service
sudo systemctl restart composesync

# Enable service to start on boot
sudo systemctl enable composesync

# Disable service from starting on boot
sudo systemctl disable composesync
```

## Viewing Logs

### Real-Time Logs

```bash
# Follow logs in real-time
sudo journalctl -u composesync -f

# Follow logs with timestamps
sudo journalctl -u composesync -f --output=short-precise
```

### Recent Logs

```bash
# View last 50 log entries
sudo journalctl -u composesync -n 50

# View logs from last hour
sudo journalctl -u composesync --since "1 hour ago"

# View logs from today
sudo journalctl -u composesync --since "today"

# View logs from specific time
sudo journalctl -u composesync --since "2024-01-15 10:00:00"
```

### Filtered Logs

```bash
# View only error messages
sudo journalctl -u composesync | grep "ERROR"

# View logs for specific stack
sudo journalctl -u composesync | grep "immich"

# View successful updates
sudo journalctl -u composesync | grep "Successfully updated"

# View rollback events
sudo journalctl -u composesync | grep "rollback"

# View webhook notifications
sudo journalctl -u composesync | grep "webhook"
```

## Log Format

ComposeSync logs follow this format:

```
[2024-01-15 10:30:00] Loading TOML configuration from /opt/composesync/config.toml
[2024-01-15 10:30:00] Processing stack: immich
[2024-01-15 10:30:01] Downloading https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
[2024-01-15 10:30:02] Changes detected for immich
[2024-01-15 10:30:02] Creating backup: compose-20240115103002.yml.bak
[2024-01-15 10:30:03] Successfully updated stack immich
[2024-01-15 10:30:03] Sending webhook notification for immich
```

### Log Levels

- **INFO**: Normal operations, successful updates
- **WARNING**: Non-critical issues, skipped stacks
- **ERROR**: Failed operations, rollbacks, configuration issues

## Configuration Monitoring

### Check Current Configuration

```bash
# View TOML configuration
sudo cat /opt/composesync/config.toml

# View .env configuration (if using legacy format)
sudo cat /opt/composesync/.env

# Check configuration syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 10
```

### Validate Configuration

```bash
# Test TOML syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 20 | grep "ERROR"

# Check for missing variables
sudo systemctl status composesync
```

## Stack Monitoring

### Check Stack Status

```bash
# List all configured stacks
ls -la /opt/composesync/stacks/

# Check if stack directories exist
for stack in /opt/composesync/stacks/*/; do
    echo "=== $(basename $stack) ==="
    ls -la "$stack"
done
```

### Monitor Stack Updates

```bash
# View recent stack updates
sudo journalctl -u composesync | grep "Processing stack"

# Check update frequency
sudo journalctl -u composesync | grep "UPDATE_INTERVAL_SECONDS"

# View stack-specific logs
sudo journalctl -u composesync | grep "immich"
```

### Check Docker Compose Status

```bash
# Check if stacks are running
for stack in /opt/composesync/stacks/*/; do
    stack_name=$(basename $stack)
    echo "=== $stack_name ==="
    if [ -f "$stack/docker-compose.yml" ]; then
        docker compose -f "$stack/docker-compose.yml" ps
    else
        echo "No docker-compose.yml found"
    fi
done
```

## Performance Monitoring

### Check Resource Usage

```bash
# Monitor system resources
htop

# Check disk usage
df -h /opt/composesync/

# Check backup storage
du -sh /opt/composesync/stacks/*/

# Count backup files
find /opt/composesync/stacks -name "*.bak" | wc -l
```

### Monitor Update Intervals

```bash
# Check update intervals in configuration
grep "UPDATE_INTERVAL_SECONDS" /opt/composesync/config.toml

# View actual update timing
sudo journalctl -u composesync | grep "Processing stack" | tail -10
```

## Webhook Monitoring

### Check Webhook Configuration

```bash
# Check if webhook is configured
grep "NOTIFICATION_WEBHOOK_URL" /opt/composesync/config.toml

# Test webhook manually
curl -X POST -H "Content-Type: application/json" \
  -d '{"event": "test", "message": "Test notification"}' \
  https://your-webhook-url.com/endpoint
```

### Monitor Webhook Delivery

```bash
# View webhook notifications
sudo journalctl -u composesync | grep "webhook"

# Check for webhook errors
sudo journalctl -u composesync | grep "ERROR.*webhook"

# View webhook payloads
sudo journalctl -u composesync | grep "payload"
```

## Error Monitoring

### Common Error Types

```bash
# Download failures
sudo journalctl -u composesync | grep "Failed to download"

# Docker Compose failures
sudo journalctl -u composesync | grep "docker compose.*failed"

# Permission errors
sudo journalctl -u composesync | grep "Permission denied"

# Configuration errors
sudo journalctl -u composesync | grep "Configuration error"

# Lock file issues
sudo journalctl -u composesync | grep "lock"
```

### Error Analysis

```bash
# View all errors from last hour
sudo journalctl -u composesync --since "1 hour ago" | grep "ERROR"

# Count errors by type
sudo journalctl -u composesync | grep "ERROR" | cut -d' ' -f4- | sort | uniq -c

# View error context
sudo journalctl -u composesync | grep -A 5 -B 5 "ERROR"
```

## Backup Monitoring

### Check Backup Status

```bash
# List all backup files
find /opt/composesync/stacks -name "*.bak" -type f

# Check backup retention
for stack in /opt/composesync/stacks/*/; do
    stack_name=$(basename $stack)
    backup_count=$(find "$stack" -name "*.bak" | wc -l)
    echo "$stack_name: $backup_count backups"
done

# Check backup sizes
find /opt/composesync/stacks -name "*.bak" -exec ls -lh {} \;
```

### Verify Backup Integrity

```bash
# Test backup file syntax
for backup in /opt/composesync/stacks/*/*.bak; do
    echo "Testing $backup"
    docker compose -f "$backup" config > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        echo "✓ Valid"
    else
        echo "✗ Invalid"
    fi
done
```

## Health Checks

### Automated Health Check Script

Create a monitoring script:

```bash
#!/bin/bash
# /opt/composesync/health-check.sh

echo "=== ComposeSync Health Check ==="
echo "Date: $(date)"
echo

# Check service status
echo "Service Status:"
if systemctl is-active --quiet composesync; then
    echo "✓ ComposeSync is running"
else
    echo "✗ ComposeSync is not running"
fi

# Check recent errors
echo
echo "Recent Errors (last hour):"
error_count=$(journalctl -u composesync --since "1 hour ago" | grep -c "ERROR")
if [ $error_count -eq 0 ]; then
    echo "✓ No errors in the last hour"
else
    echo "✗ $error_count errors in the last hour"
fi

# Check stack status
echo
echo "Stack Status:"
for stack in /opt/composesync/stacks/*/; do
    stack_name=$(basename $stack)
    if [ -f "$stack/docker-compose.yml" ]; then
        if docker compose -f "$stack/docker-compose.yml" ps --quiet | grep -q .; then
            echo "✓ $stack_name is running"
        else
            echo "✗ $stack_name is not running"
        fi
    else
        echo "? $stack_name has no docker-compose.yml"
    fi
done

# Check disk usage
echo
echo "Disk Usage:"
usage=$(df -h /opt/composesync/ | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $usage -lt 80 ]; then
    echo "✓ Disk usage: ${usage}%"
else
    echo "✗ High disk usage: ${usage}%"
fi
```

Make it executable and run:

```bash
sudo chmod +x /opt/composesync/health-check.sh
sudo /opt/composesync/health-check.sh
```

### Scheduled Health Checks

Add to crontab for regular monitoring:

```bash
# Edit crontab
sudo crontab -e

# Add health check every hour
0 * * * * /opt/composesync/health-check.sh >> /var/log/composesync-health.log 2>&1
```

## Alerting

### Set Up Email Alerts

Create an alert script:

```bash
#!/bin/bash
# /opt/composesync/alert.sh

# Check for errors in last 10 minutes
errors=$(journalctl -u composesync --since "10 minutes ago" | grep -c "ERROR")

if [ $errors -gt 0 ]; then
    echo "ComposeSync Alert: $errors errors detected" | \
    mail -s "ComposeSync Alert" your-email@example.com
fi
```

### Webhook Alerts

Configure webhook notifications in your TOML file:

```toml
[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10
DRY_RUN = false
NOTIFICATION_WEBHOOK_URL = "https://your-webhook-url.com/endpoint"

[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"
```

## Best Practices

### 1. Regular Log Review

```bash
# Daily log review
sudo journalctl -u composesync --since "yesterday" | grep "ERROR"

# Weekly summary
sudo journalctl -u composesync --since "1 week ago" | grep "Successfully updated" | wc -l
```

### 2. Monitor Key Metrics

- **Update success rate**: Track successful vs failed updates
- **Error frequency**: Monitor error patterns
- **Response time**: Track how long updates take
- **Disk usage**: Monitor backup storage

### 3. Set Up Automated Monitoring

```bash
# Create monitoring dashboard
sudo mkdir -p /opt/composesync/monitoring
sudo nano /opt/composesync/monitoring/dashboard.sh
```

### 4. Document Issues

Keep a log of issues and resolutions:

```bash
# Create issue log
sudo nano /opt/composesync/issue-log.md
```

## Troubleshooting Commands

### Quick Diagnostics

```bash
# Check service status
sudo systemctl status composesync

# View recent logs
sudo journalctl -u composesync -n 20

# Check configuration
sudo cat /opt/composesync/config.toml

# Test Docker access
docker ps

# Check file permissions
ls -la /opt/composesync/
```

### Detailed Diagnostics

```bash
# Full system check
sudo /opt/composesync/health-check.sh

# Configuration validation
sudo systemctl restart composesync
sudo journalctl -u composesync -n 50

# Stack verification
for stack in /opt/composesync/stacks/*/; do
    echo "=== $(basename $stack) ==="
    docker compose -f "$stack/docker-compose.yml" config
done
```