ComposeSync/Docs/monitoring.md

Monitoring Guide

This guide covers how to monitor and manage your ComposeSync service.

Viewing Logs

Real-time Log Monitoring

Watch logs in real-time:

sudo journalctl -u composesync -f

Recent Logs

View recent log entries:

# Last 50 lines
sudo journalctl -u composesync -n 50

# Last hour
sudo journalctl -u composesync --since "1 hour ago"

# Today's logs
sudo journalctl -u composesync --since "today"

Log Filtering

Filter logs by specific criteria:

# Only error messages
sudo journalctl -u composesync -p err

# Only messages containing "immich"
sudo journalctl -u composesync | grep immich

# Dry-run messages only
sudo journalctl -u composesync | grep "DRY-RUN"

Log Export

Export logs to a file:

# Export today's logs
sudo journalctl -u composesync --since "today" > composesync-logs.txt

# Export all logs
sudo journalctl -u composesync > composesync-all-logs.txt

Service Control

Service Status

Check service status:

sudo systemctl status composesync

Start/Stop/Restart

Control the service:

# Start the service
sudo systemctl start composesync

# Stop the service
sudo systemctl stop composesync

# Restart the service
sudo systemctl restart composesync

# Reload configuration (if supported)
sudo systemctl reload composesync

Enable/Disable

Manage service startup:

# Enable service to start on boot
sudo systemctl enable composesync

# Disable service from starting on boot
sudo systemctl disable composesync

Service Information

Get detailed service information:

# Show service configuration
sudo systemctl show composesync

# Show service dependencies
sudo systemctl list-dependencies composesync

Stack Monitoring

Check Stack Status

Monitor individual stacks:

# Check if a specific stack is running
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml ps

# Check all stacks
for stack in /opt/composesync/stacks/*/; do
    echo "=== $(basename $stack) ==="
    docker compose -f "$stack/docker-compose.yml" ps
done

Stack Health

Check stack health and logs:

# Check stack logs
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs

# Check specific service logs
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs immich-server

# Follow logs in real-time
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs -f

Version History

Check versioned files:

# List versioned files for a stack
ls -la /opt/composesync/stacks/immich/compose-*.yml.bak

# Check backup directories
ls -la /opt/composesync/stacks/immich/backups/

Performance Monitoring

Resource Usage

Monitor system resources:

# Check CPU and memory usage
top -p $(pgrep -f update-agent.sh)

# Check disk usage
du -sh /opt/composesync/stacks/*/

# Check for large backup directories
find /opt/composesync/stacks/ -name "backups" -type d -exec du -sh {} \;

Network Monitoring

Monitor download activity:

# Check network connections
netstat -tulpn | grep wget
netstat -tulpn | grep git

# Monitor bandwidth usage
iftop -i eth0

Alerting and Notifications

Webhook Monitoring

Test webhook notifications:

# Test webhook manually
curl -X POST -H "Content-Type: application/json" \
  -d '{"event": "test", "message": "Test notification"}' \
  $NOTIFICATION_WEBHOOK_URL

Email Notifications

Set up email alerts (if your webhook supports it):

# Example: Send email via webhook
curl -X POST -H "Content-Type: application/json" \
  -d '{"to": "admin@example.com", "subject": "ComposeSync Alert", "body": "Update failed"}' \
  https://your-webhook-service.com/email

Monitoring Best Practices

Regular Health Checks

Set up regular monitoring:

# Create a monitoring script
cat > /usr/local/bin/composesync-health-check.sh << 'EOF'
#!/bin/bash

# Check if service is running
if ! systemctl is-active --quiet composesync; then
    echo "ComposeSync service is not running!"
    exit 1
fi

# Check for recent errors
if journalctl -u composesync --since "1 hour ago" | grep -q "ERROR"; then
    echo "ComposeSync has errors in the last hour"
    exit 1
fi

# Check disk usage
if [ $(df /opt/composesync | tail -1 | awk '{print $5}' | sed 's/%//') -gt 90 ]; then
    echo "ComposeSync disk usage is high"
    exit 1
fi

echo "ComposeSync is healthy"
exit 0
EOF

chmod +x /usr/local/bin/composesync-health-check.sh

Automated Monitoring

Set up automated monitoring with cron:

# Add to crontab
echo "*/15 * * * * /usr/local/bin/composesync-health-check.sh" | sudo crontab -

Log Rotation

Configure log rotation to prevent disk space issues:

# Create logrotate configuration
sudo tee /etc/logrotate.d/composesync << EOF
/var/log/composesync/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 644 composesync composesync
}
EOF

Troubleshooting Monitoring

Service Not Starting

If the service won't start:

# Check for configuration errors
sudo systemctl status composesync

# Check logs for specific errors
sudo journalctl -u composesync -n 20

# Verify file permissions
ls -la /opt/composesync/

High Resource Usage

If ComposeSync is using too many resources:

# Check what's consuming resources
ps aux | grep update-agent

# Check for stuck processes
pgrep -f update-agent

# Restart the service
sudo systemctl restart composesync

Missing Logs

If logs are missing:

# Check if journald is working
sudo journalctl --verify

# Check journald status
sudo systemctl status systemd-journald

# Check log storage
sudo journalctl --disk-usage

Integration with External Monitoring

Prometheus/Grafana

For advanced monitoring, you can integrate with Prometheus:

# Example: Export metrics via webhook
curl -X POST -H "Content-Type: application/json" \
  -d '{"metric": "composesync_updates_total", "value": 1, "labels": {"stack": "immich"}}' \
  http://prometheus:9090/api/v1/write

Nagios/Icinga

Create custom checks for monitoring systems:

# Example Nagios check
#!/bin/bash
if systemctl is-active --quiet composesync; then
    echo "OK: ComposeSync is running"
    exit 0
else
    echo "CRITICAL: ComposeSync is not running"
    exit 2
fi

Security Monitoring

Access Monitoring

Monitor who accesses ComposeSync:

# Check who modified the configuration
ls -la /opt/composesync/.env

# Check for unauthorized changes
find /opt/composesync -mtime -1 -ls

# Monitor Docker group membership
getent group docker

Audit Logging

Enable audit logging:

# Monitor file access
auditctl -w /opt/composesync/.env -p wa -k composesync_config

# Monitor Docker socket access
auditctl -w /var/run/docker.sock -p wa -k docker_access

This comprehensive monitoring setup will help you keep track of ComposeSync's health and performance.