robojerk/ComposeSync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ComposeSync/Docs/monitoring.md

10 KiB
Raw Blame History

Monitoring and Logs

This guide covers how to monitor ComposeSync, view logs, and troubleshoot issues.

Overview

ComposeSync provides comprehensive logging and monitoring capabilities to help you track updates, diagnose issues, and ensure your stacks are running smoothly.

Service Control

Check Service Status

# Check if ComposeSync is running
sudo systemctl status composesync

# Check if service is enabled
sudo systemctl is-enabled composesync

Start/Stop/Restart Service

# Start the service
sudo systemctl start composesync

# Stop the service
sudo systemctl stop composesync

# Restart the service
sudo systemctl restart composesync

# Enable service to start on boot
sudo systemctl enable composesync

# Disable service from starting on boot
sudo systemctl disable composesync

Viewing Logs

Real-Time Logs

# Follow logs in real-time
sudo journalctl -u composesync -f

# Follow logs with timestamps
sudo journalctl -u composesync -f --output=short-precise

Recent Logs

# View last 50 log entries
sudo journalctl -u composesync -n 50

# View logs from last hour
sudo journalctl -u composesync --since "1 hour ago"

# View logs from today
sudo journalctl -u composesync --since "today"

# View logs from specific time
sudo journalctl -u composesync --since "2024-01-15 10:00:00"

Filtered Logs

# View only error messages
sudo journalctl -u composesync | grep "ERROR"

# View logs for specific stack
sudo journalctl -u composesync | grep "immich"

# View successful updates
sudo journalctl -u composesync | grep "Successfully updated"

# View rollback events
sudo journalctl -u composesync | grep "rollback"

# View webhook notifications
sudo journalctl -u composesync | grep "webhook"

Log Format

ComposeSync logs follow this format:

[2024-01-15 10:30:00] Loading TOML configuration from /opt/composesync/config.toml
[2024-01-15 10:30:00] Processing stack: immich
[2024-01-15 10:30:01] Downloading https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
[2024-01-15 10:30:02] Changes detected for immich
[2024-01-15 10:30:02] Creating backup: compose-20240115103002.yml.bak
[2024-01-15 10:30:03] Successfully updated stack immich
[2024-01-15 10:30:03] Sending webhook notification for immich

Log Levels

  • INFO: Normal operations, successful updates
  • WARNING: Non-critical issues, skipped stacks
  • ERROR: Failed operations, rollbacks, configuration issues

Configuration Monitoring

Check Current Configuration

# View TOML configuration
sudo cat /opt/composesync/config.toml

# View .env configuration (if using legacy format)
sudo cat /opt/composesync/.env

# Check configuration syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 10

Validate Configuration

# Test TOML syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 20 | grep "ERROR"

# Check for missing variables
sudo systemctl status composesync

Stack Monitoring

Check Stack Status

# List all configured stacks
ls -la /opt/composesync/stacks/

# Check if stack directories exist
for stack in /opt/composesync/stacks/*/; do
    echo "=== $(basename $stack) ==="
    ls -la "$stack"
done

Monitor Stack Updates

# View recent stack updates
sudo journalctl -u composesync | grep "Processing stack"

# Check update frequency
sudo journalctl -u composesync | grep "UPDATE_INTERVAL_SECONDS"

# View stack-specific logs
sudo journalctl -u composesync | grep "immich"

Check Docker Compose Status

# Check if stacks are running
for stack in /opt/composesync/stacks/*/; do
    stack_name=$(basename $stack)
    echo "=== $stack_name ==="
    if [ -f "$stack/docker-compose.yml" ]; then
        docker compose -f "$stack/docker-compose.yml" ps
    else
        echo "No docker-compose.yml found"
    fi
done

Performance Monitoring

Check Resource Usage

# Monitor system resources
htop

# Check disk usage
df -h /opt/composesync/

# Check backup storage
du -sh /opt/composesync/stacks/*/

# Count backup files
find /opt/composesync/stacks -name "*.bak" | wc -l

Monitor Update Intervals

# Check update intervals in configuration
grep "UPDATE_INTERVAL_SECONDS" /opt/composesync/config.toml

# View actual update timing
sudo journalctl -u composesync | grep "Processing stack" | tail -10

Webhook Monitoring

Check Webhook Configuration

# Check if webhook is configured
grep "NOTIFICATION_WEBHOOK_URL" /opt/composesync/config.toml

# Test webhook manually
curl -X POST -H "Content-Type: application/json" \
  -d '{"event": "test", "message": "Test notification"}' \
  https://your-webhook-url.com/endpoint

Monitor Webhook Delivery

# View webhook notifications
sudo journalctl -u composesync | grep "webhook"

# Check for webhook errors
sudo journalctl -u composesync | grep "ERROR.*webhook"

# View webhook payloads
sudo journalctl -u composesync | grep "payload"

Error Monitoring

Common Error Types

# Download failures
sudo journalctl -u composesync | grep "Failed to download"

# Docker Compose failures
sudo journalctl -u composesync | grep "docker compose.*failed"

# Permission errors
sudo journalctl -u composesync | grep "Permission denied"

# Configuration errors
sudo journalctl -u composesync | grep "Configuration error"

# Lock file issues
sudo journalctl -u composesync | grep "lock"

Error Analysis

# View all errors from last hour
sudo journalctl -u composesync --since "1 hour ago" | grep "ERROR"

# Count errors by type
sudo journalctl -u composesync | grep "ERROR" | cut -d' ' -f4- | sort | uniq -c

# View error context
sudo journalctl -u composesync | grep -A 5 -B 5 "ERROR"

Backup Monitoring

Check Backup Status

# List all backup files
find /opt/composesync/stacks -name "*.bak" -type f

# Check backup retention
for stack in /opt/composesync/stacks/*/; do
    stack_name=$(basename $stack)
    backup_count=$(find "$stack" -name "*.bak" | wc -l)
    echo "$stack_name: $backup_count backups"
done

# Check backup sizes
find /opt/composesync/stacks -name "*.bak" -exec ls -lh {} \;

Verify Backup Integrity

# Test backup file syntax
for backup in /opt/composesync/stacks/*/*.bak; do
    echo "Testing $backup"
    docker compose -f "$backup" config > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        echo "✓ Valid"
    else
        echo "✗ Invalid"
    fi
done

Health Checks

Automated Health Check Script

Create a monitoring script:

#!/bin/bash
# /opt/composesync/health-check.sh

echo "=== ComposeSync Health Check ==="
echo "Date: $(date)"
echo

# Check service status
echo "Service Status:"
if systemctl is-active --quiet composesync; then
    echo "✓ ComposeSync is running"
else
    echo "✗ ComposeSync is not running"
fi

# Check recent errors
echo
echo "Recent Errors (last hour):"
error_count=$(journalctl -u composesync --since "1 hour ago" | grep -c "ERROR")
if [ $error_count -eq 0 ]; then
    echo "✓ No errors in the last hour"
else
    echo "✗ $error_count errors in the last hour"
fi

# Check stack status
echo
echo "Stack Status:"
for stack in /opt/composesync/stacks/*/; do
    stack_name=$(basename $stack)
    if [ -f "$stack/docker-compose.yml" ]; then
        if docker compose -f "$stack/docker-compose.yml" ps --quiet | grep -q .; then
            echo "✓ $stack_name is running"
        else
            echo "✗ $stack_name is not running"
        fi
    else
        echo "? $stack_name has no docker-compose.yml"
    fi
done

# Check disk usage
echo
echo "Disk Usage:"
usage=$(df -h /opt/composesync/ | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $usage -lt 80 ]; then
    echo "✓ Disk usage: ${usage}%"
else
    echo "✗ High disk usage: ${usage}%"
fi

Make it executable and run:

sudo chmod +x /opt/composesync/health-check.sh
sudo /opt/composesync/health-check.sh

Scheduled Health Checks

Add to crontab for regular monitoring:

# Edit crontab
sudo crontab -e

# Add health check every hour
0 * * * * /opt/composesync/health-check.sh >> /var/log/composesync-health.log 2>&1

Alerting

Set Up Email Alerts

Create an alert script:

#!/bin/bash
# /opt/composesync/alert.sh

# Check for errors in last 10 minutes
errors=$(journalctl -u composesync --since "10 minutes ago" | grep -c "ERROR")

if [ $errors -gt 0 ]; then
    echo "ComposeSync Alert: $errors errors detected" | \
    mail -s "ComposeSync Alert" your-email@example.com
fi

Webhook Alerts

Configure webhook notifications in your TOML file:

[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10
DRY_RUN = false
NOTIFICATION_WEBHOOK_URL = "https://your-webhook-url.com/endpoint"

[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

Best Practices

1. Regular Log Review

# Daily log review
sudo journalctl -u composesync --since "yesterday" | grep "ERROR"

# Weekly summary
sudo journalctl -u composesync --since "1 week ago" | grep "Successfully updated" | wc -l

2. Monitor Key Metrics

  • Update success rate: Track successful vs failed updates
  • Error frequency: Monitor error patterns
  • Response time: Track how long updates take
  • Disk usage: Monitor backup storage

3. Set Up Automated Monitoring

# Create monitoring dashboard
sudo mkdir -p /opt/composesync/monitoring
sudo nano /opt/composesync/monitoring/dashboard.sh

4. Document Issues

Keep a log of issues and resolutions:

# Create issue log
sudo nano /opt/composesync/issue-log.md

Troubleshooting Commands

Quick Diagnostics

# Check service status
sudo systemctl status composesync

# View recent logs
sudo journalctl -u composesync -n 20

# Check configuration
sudo cat /opt/composesync/config.toml

# Test Docker access
docker ps

# Check file permissions
ls -la /opt/composesync/

Detailed Diagnostics

# Full system check
sudo /opt/composesync/health-check.sh

# Configuration validation
sudo systemctl restart composesync
sudo journalctl -u composesync -n 50

# Stack verification
for stack in /opt/composesync/stacks/*/; do
    echo "=== $(basename $stack) ==="
    docker compose -f "$stack/docker-compose.yml" config
done