# Monitoring Guide

This guide covers how to monitor and manage your ComposeSync service.

## Viewing Logs

### Real-time Log Monitoring

Watch logs in real-time:
```bash
sudo journalctl -u composesync -f
```

### Recent Logs

View recent log entries:
```bash
# Last 50 lines
sudo journalctl -u composesync -n 50

# Last hour
sudo journalctl -u composesync --since "1 hour ago"

# Today's logs
sudo journalctl -u composesync --since "today"
```

### Log Filtering

Filter logs by specific criteria:
```bash
# Only error messages
sudo journalctl -u composesync -p err

# Only messages containing "immich"
sudo journalctl -u composesync | grep immich

# Dry-run messages only
sudo journalctl -u composesync | grep "DRY-RUN"
```

### Log Export

Export logs to a file:
```bash
# Export today's logs
sudo journalctl -u composesync --since "today" > composesync-logs.txt

# Export all logs
sudo journalctl -u composesync > composesync-all-logs.txt
```

## Service Control

### Service Status

Check service status:
```bash
sudo systemctl status composesync
```

### Start/Stop/Restart

Control the service:
```bash
# Start the service
sudo systemctl start composesync

# Stop the service
sudo systemctl stop composesync

# Restart the service
sudo systemctl restart composesync

# Reload configuration (if supported)
sudo systemctl reload composesync
```

### Enable/Disable

Manage service startup:
```bash
# Enable service to start on boot
sudo systemctl enable composesync

# Disable service from starting on boot
sudo systemctl disable composesync
```

### Service Information

Get detailed service information:
```bash
# Show service configuration
sudo systemctl show composesync

# Show service dependencies
sudo systemctl list-dependencies composesync
```

## Stack Monitoring

### Check Stack Status

Monitor individual stacks:
```bash
# Check if a specific stack is running
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml ps

# Check all stacks
for stack in /opt/composesync/stacks/*/; do
    echo "=== $(basename $stack) ==="
    docker compose -f "$stack/docker-compose.yml" ps
done
```

### Stack Health

Check stack health and logs:
```bash
# Check stack logs
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs

# Check specific service logs
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs immich-server

# Follow logs in real-time
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs -f
```

### Version History

Check versioned files:
```bash
# List versioned files for a stack
ls -la /opt/composesync/stacks/immich/compose-*.yml.bak

# Check backup directories
ls -la /opt/composesync/stacks/immich/backups/
```

## Performance Monitoring

### Resource Usage

Monitor system resources:
```bash
# Check CPU and memory usage
top -p $(pgrep -f update-agent.sh)

# Check disk usage
du -sh /opt/composesync/stacks/*/

# Check for large backup directories
find /opt/composesync/stacks/ -name "backups" -type d -exec du -sh {} \;
```

### Network Monitoring

Monitor download activity:
```bash
# Check network connections
netstat -tulpn | grep wget
netstat -tulpn | grep git

# Monitor bandwidth usage
iftop -i eth0
```

## Alerting and Notifications

### Webhook Monitoring

Test webhook notifications:
```bash
# Test webhook manually
curl -X POST -H "Content-Type: application/json" \
  -d '{"event": "test", "message": "Test notification"}' \
  $NOTIFICATION_WEBHOOK_URL
```

### Email Notifications

Set up email alerts (if your webhook supports it):
```bash
# Example: Send email via webhook
curl -X POST -H "Content-Type: application/json" \
  -d '{"to": "admin@example.com", "subject": "ComposeSync Alert", "body": "Update failed"}' \
  https://your-webhook-service.com/email
```

## Monitoring Best Practices

### Regular Health Checks

Set up regular monitoring:
```bash
# Create a monitoring script
cat > /usr/local/bin/composesync-health-check.sh << 'EOF'
#!/bin/bash

# Check if service is running
if ! systemctl is-active --quiet composesync; then
    echo "ComposeSync service is not running!"
    exit 1
fi

# Check for recent errors
if journalctl -u composesync --since "1 hour ago" | grep -q "ERROR"; then
    echo "ComposeSync has errors in the last hour"
    exit 1
fi

# Check disk usage
if [ $(df /opt/composesync | tail -1 | awk '{print $5}' | sed 's/%//') -gt 90 ]; then
    echo "ComposeSync disk usage is high"
    exit 1
fi

echo "ComposeSync is healthy"
exit 0
EOF

chmod +x /usr/local/bin/composesync-health-check.sh
```

### Automated Monitoring

Set up automated monitoring with cron:
```bash
# Add to crontab
echo "*/15 * * * * /usr/local/bin/composesync-health-check.sh" | sudo crontab -
```

### Log Rotation

Configure log rotation to prevent disk space issues:
```bash
# Create logrotate configuration
sudo tee /etc/logrotate.d/composesync << EOF
/var/log/composesync/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 644 composesync composesync
}
EOF
```

## Troubleshooting Monitoring

### Service Not Starting

If the service won't start:
```bash
# Check for configuration errors
sudo systemctl status composesync

# Check logs for specific errors
sudo journalctl -u composesync -n 20

# Verify file permissions
ls -la /opt/composesync/
```

### High Resource Usage

If ComposeSync is using too many resources:
```bash
# Check what's consuming resources
ps aux | grep update-agent

# Check for stuck processes
pgrep -f update-agent

# Restart the service
sudo systemctl restart composesync
```

### Missing Logs

If logs are missing:
```bash
# Check if journald is working
sudo journalctl --verify

# Check journald status
sudo systemctl status systemd-journald

# Check log storage
sudo journalctl --disk-usage
```

## Integration with External Monitoring

### Prometheus/Grafana

For advanced monitoring, you can integrate with Prometheus:
```bash
# Example: Export metrics via webhook
curl -X POST -H "Content-Type: application/json" \
  -d '{"metric": "composesync_updates_total", "value": 1, "labels": {"stack": "immich"}}' \
  http://prometheus:9090/api/v1/write
```

### Nagios/Icinga

Create custom checks for monitoring systems:
```bash
# Example Nagios check
#!/bin/bash
if systemctl is-active --quiet composesync; then
    echo "OK: ComposeSync is running"
    exit 0
else
    echo "CRITICAL: ComposeSync is not running"
    exit 2
fi
```

## Security Monitoring

### Access Monitoring

Monitor who accesses ComposeSync:
```bash
# Check who modified the configuration
ls -la /opt/composesync/.env

# Check for unauthorized changes
find /opt/composesync -mtime -1 -ls

# Monitor Docker group membership
getent group docker
```

### Audit Logging

Enable audit logging:
```bash
# Monitor file access
auditctl -w /opt/composesync/.env -p wa -k composesync_config

# Monitor Docker socket access
auditctl -w /var/run/docker.sock -p wa -k docker_access
```

This comprehensive monitoring setup will help you keep track of ComposeSync's health and performance.