Major improvements: flexible install dir, configurable compose file name for git, enhanced webhook notifications, cross-platform lock, robust rollback, and updated docs.\n\n- Install dir is now user-confirmable and dynamic\n- Added COMPOSE_FILENAME for git stacks\n- Webhook payloads now include git context and rollback events\n- Lock file age check is cross-platform\n- Rollback notifications for success/failure\n- Updated TOML example and documentation\n- Many robustness and UX improvements

This commit is contained in:
robojerk 2025-06-25 15:15:40 -07:00
parent f0dba7cc0a
commit 70486907aa
18 changed files with 3788 additions and 1767 deletions

View file

@ -1,67 +1,25 @@
# Monitoring Guide
# Monitoring and Logs
This guide covers how to monitor and manage your ComposeSync service.
This guide covers how to monitor ComposeSync, view logs, and troubleshoot issues.
## Viewing Logs
## Overview
### Real-time Log Monitoring
Watch logs in real-time:
```bash
sudo journalctl -u composesync -f
```
### Recent Logs
View recent log entries:
```bash
# Last 50 lines
sudo journalctl -u composesync -n 50
# Last hour
sudo journalctl -u composesync --since "1 hour ago"
# Today's logs
sudo journalctl -u composesync --since "today"
```
### Log Filtering
Filter logs by specific criteria:
```bash
# Only error messages
sudo journalctl -u composesync -p err
# Only messages containing "immich"
sudo journalctl -u composesync | grep immich
# Dry-run messages only
sudo journalctl -u composesync | grep "DRY-RUN"
```
### Log Export
Export logs to a file:
```bash
# Export today's logs
sudo journalctl -u composesync --since "today" > composesync-logs.txt
# Export all logs
sudo journalctl -u composesync > composesync-all-logs.txt
```
ComposeSync provides comprehensive logging and monitoring capabilities to help you track updates, diagnose issues, and ensure your stacks are running smoothly.
## Service Control
### Service Status
### Check Service Status
Check service status:
```bash
# Check if ComposeSync is running
sudo systemctl status composesync
# Check if service is enabled
sudo systemctl is-enabled composesync
```
### Start/Stop/Restart
### Start/Stop/Restart Service
Control the service:
```bash
# Start the service
sudo systemctl start composesync
@ -72,14 +30,6 @@ sudo systemctl stop composesync
# Restart the service
sudo systemctl restart composesync
# Reload configuration (if supported)
sudo systemctl reload composesync
```
### Enable/Disable
Manage service startup:
```bash
# Enable service to start on boot
sudo systemctl enable composesync
@ -87,265 +37,450 @@ sudo systemctl enable composesync
sudo systemctl disable composesync
```
### Service Information
## Viewing Logs
### Real-Time Logs
Get detailed service information:
```bash
# Show service configuration
sudo systemctl show composesync
# Follow logs in real-time
sudo journalctl -u composesync -f
# Show service dependencies
sudo systemctl list-dependencies composesync
# Follow logs with timestamps
sudo journalctl -u composesync -f --output=short-precise
```
### Recent Logs
```bash
# View last 50 log entries
sudo journalctl -u composesync -n 50
# View logs from last hour
sudo journalctl -u composesync --since "1 hour ago"
# View logs from today
sudo journalctl -u composesync --since "today"
# View logs from specific time
sudo journalctl -u composesync --since "2024-01-15 10:00:00"
```
### Filtered Logs
```bash
# View only error messages
sudo journalctl -u composesync | grep "ERROR"
# View logs for specific stack
sudo journalctl -u composesync | grep "immich"
# View successful updates
sudo journalctl -u composesync | grep "Successfully updated"
# View rollback events
sudo journalctl -u composesync | grep "rollback"
# View webhook notifications
sudo journalctl -u composesync | grep "webhook"
```
## Log Format
ComposeSync logs follow this format:
```
[2024-01-15 10:30:00] Loading TOML configuration from /opt/composesync/config.toml
[2024-01-15 10:30:00] Processing stack: immich
[2024-01-15 10:30:01] Downloading https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
[2024-01-15 10:30:02] Changes detected for immich
[2024-01-15 10:30:02] Creating backup: compose-20240115103002.yml.bak
[2024-01-15 10:30:03] Successfully updated stack immich
[2024-01-15 10:30:03] Sending webhook notification for immich
```
### Log Levels
- **INFO**: Normal operations, successful updates
- **WARNING**: Non-critical issues, skipped stacks
- **ERROR**: Failed operations, rollbacks, configuration issues
## Configuration Monitoring
### Check Current Configuration
```bash
# View TOML configuration
sudo cat /opt/composesync/config.toml
# View .env configuration (if using legacy format)
sudo cat /opt/composesync/.env
# Check configuration syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 10
```
### Validate Configuration
```bash
# Test TOML syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 20 | grep "ERROR"
# Check for missing variables
sudo systemctl status composesync
```
## Stack Monitoring
### Check Stack Status
Monitor individual stacks:
```bash
# Check if a specific stack is running
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml ps
# List all configured stacks
ls -la /opt/composesync/stacks/
# Check all stacks
# Check if stack directories exist
for stack in /opt/composesync/stacks/*/; do
echo "=== $(basename $stack) ==="
docker compose -f "$stack/docker-compose.yml" ps
ls -la "$stack"
done
```
### Stack Health
### Monitor Stack Updates
Check stack health and logs:
```bash
# Check stack logs
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs
# View recent stack updates
sudo journalctl -u composesync | grep "Processing stack"
# Check specific service logs
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs immich-server
# Check update frequency
sudo journalctl -u composesync | grep "UPDATE_INTERVAL_SECONDS"
# Follow logs in real-time
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml logs -f
# View stack-specific logs
sudo journalctl -u composesync | grep "immich"
```
### Version History
### Check Docker Compose Status
Check versioned files:
```bash
# List versioned files for a stack
ls -la /opt/composesync/stacks/immich/compose-*.yml.bak
# Check backup directories
ls -la /opt/composesync/stacks/immich/backups/
# Check if stacks are running
for stack in /opt/composesync/stacks/*/; do
stack_name=$(basename $stack)
echo "=== $stack_name ==="
if [ -f "$stack/docker-compose.yml" ]; then
docker compose -f "$stack/docker-compose.yml" ps
else
echo "No docker-compose.yml found"
fi
done
```
## Performance Monitoring
### Resource Usage
### Check Resource Usage
Monitor system resources:
```bash
# Check CPU and memory usage
top -p $(pgrep -f update-agent.sh)
# Monitor system resources
htop
# Check disk usage
df -h /opt/composesync/
# Check backup storage
du -sh /opt/composesync/stacks/*/
# Check for large backup directories
find /opt/composesync/stacks/ -name "backups" -type d -exec du -sh {} \;
# Count backup files
find /opt/composesync/stacks -name "*.bak" | wc -l
```
### Network Monitoring
### Monitor Update Intervals
Monitor download activity:
```bash
# Check network connections
netstat -tulpn | grep wget
netstat -tulpn | grep git
# Check update intervals in configuration
grep "UPDATE_INTERVAL_SECONDS" /opt/composesync/config.toml
# Monitor bandwidth usage
iftop -i eth0
# View actual update timing
sudo journalctl -u composesync | grep "Processing stack" | tail -10
```
## Alerting and Notifications
## Webhook Monitoring
### Webhook Monitoring
### Check Webhook Configuration
Test webhook notifications:
```bash
# Check if webhook is configured
grep "NOTIFICATION_WEBHOOK_URL" /opt/composesync/config.toml
# Test webhook manually
curl -X POST -H "Content-Type: application/json" \
-d '{"event": "test", "message": "Test notification"}' \
$NOTIFICATION_WEBHOOK_URL
https://your-webhook-url.com/endpoint
```
### Email Notifications
### Monitor Webhook Delivery
Set up email alerts (if your webhook supports it):
```bash
# Example: Send email via webhook
curl -X POST -H "Content-Type: application/json" \
-d '{"to": "admin@example.com", "subject": "ComposeSync Alert", "body": "Update failed"}' \
https://your-webhook-service.com/email
# View webhook notifications
sudo journalctl -u composesync | grep "webhook"
# Check for webhook errors
sudo journalctl -u composesync | grep "ERROR.*webhook"
# View webhook payloads
sudo journalctl -u composesync | grep "payload"
```
## Monitoring Best Practices
## Error Monitoring
### Regular Health Checks
### Common Error Types
```bash
# Download failures
sudo journalctl -u composesync | grep "Failed to download"
# Docker Compose failures
sudo journalctl -u composesync | grep "docker compose.*failed"
# Permission errors
sudo journalctl -u composesync | grep "Permission denied"
# Configuration errors
sudo journalctl -u composesync | grep "Configuration error"
# Lock file issues
sudo journalctl -u composesync | grep "lock"
```
### Error Analysis
```bash
# View all errors from last hour
sudo journalctl -u composesync --since "1 hour ago" | grep "ERROR"
# Count errors by type
sudo journalctl -u composesync | grep "ERROR" | cut -d' ' -f4- | sort | uniq -c
# View error context
sudo journalctl -u composesync | grep -A 5 -B 5 "ERROR"
```
## Backup Monitoring
### Check Backup Status
```bash
# List all backup files
find /opt/composesync/stacks -name "*.bak" -type f
# Check backup retention
for stack in /opt/composesync/stacks/*/; do
stack_name=$(basename $stack)
backup_count=$(find "$stack" -name "*.bak" | wc -l)
echo "$stack_name: $backup_count backups"
done
# Check backup sizes
find /opt/composesync/stacks -name "*.bak" -exec ls -lh {} \;
```
### Verify Backup Integrity
```bash
# Test backup file syntax
for backup in /opt/composesync/stacks/*/*.bak; do
echo "Testing $backup"
docker compose -f "$backup" config > /dev/null 2>&1
if [ $? -eq 0 ]; then
echo "✓ Valid"
else
echo "✗ Invalid"
fi
done
```
## Health Checks
### Automated Health Check Script
Create a monitoring script:
Set up regular monitoring:
```bash
# Create a monitoring script
cat > /usr/local/bin/composesync-health-check.sh << 'EOF'
#!/bin/bash
# /opt/composesync/health-check.sh
# Check if service is running
if ! systemctl is-active --quiet composesync; then
echo "ComposeSync service is not running!"
exit 1
echo "=== ComposeSync Health Check ==="
echo "Date: $(date)"
echo
# Check service status
echo "Service Status:"
if systemctl is-active --quiet composesync; then
echo "✓ ComposeSync is running"
else
echo "✗ ComposeSync is not running"
fi
# Check for recent errors
if journalctl -u composesync --since "1 hour ago" | grep -q "ERROR"; then
echo "ComposeSync has errors in the last hour"
exit 1
# Check recent errors
echo
echo "Recent Errors (last hour):"
error_count=$(journalctl -u composesync --since "1 hour ago" | grep -c "ERROR")
if [ $error_count -eq 0 ]; then
echo "✓ No errors in the last hour"
else
echo "✗ $error_count errors in the last hour"
fi
# Check stack status
echo
echo "Stack Status:"
for stack in /opt/composesync/stacks/*/; do
stack_name=$(basename $stack)
if [ -f "$stack/docker-compose.yml" ]; then
if docker compose -f "$stack/docker-compose.yml" ps --quiet | grep -q .; then
echo "✓ $stack_name is running"
else
echo "✗ $stack_name is not running"
fi
else
echo "? $stack_name has no docker-compose.yml"
fi
done
# Check disk usage
if [ $(df /opt/composesync | tail -1 | awk '{print $5}' | sed 's/%//') -gt 90 ]; then
echo "ComposeSync disk usage is high"
exit 1
echo
echo "Disk Usage:"
usage=$(df -h /opt/composesync/ | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $usage -lt 80 ]; then
echo "✓ Disk usage: ${usage}%"
else
echo "✗ High disk usage: ${usage}%"
fi
echo "ComposeSync is healthy"
exit 0
EOF
chmod +x /usr/local/bin/composesync-health-check.sh
```
### Automated Monitoring
Make it executable and run:
Set up automated monitoring with cron:
```bash
# Add to crontab
echo "*/15 * * * * /usr/local/bin/composesync-health-check.sh" | sudo crontab -
sudo chmod +x /opt/composesync/health-check.sh
sudo /opt/composesync/health-check.sh
```
### Log Rotation
### Scheduled Health Checks
Add to crontab for regular monitoring:
Configure log rotation to prevent disk space issues:
```bash
# Create logrotate configuration
sudo tee /etc/logrotate.d/composesync << EOF
/var/log/composesync/*.log {
daily
missingok
rotate 7
compress
delaycompress
notifempty
create 644 composesync composesync
}
EOF
# Edit crontab
sudo crontab -e
# Add health check every hour
0 * * * * /opt/composesync/health-check.sh >> /var/log/composesync-health.log 2>&1
```
## Troubleshooting Monitoring
## Alerting
### Service Not Starting
### Set Up Email Alerts
Create an alert script:
If the service won't start:
```bash
# Check for configuration errors
#!/bin/bash
# /opt/composesync/alert.sh
# Check for errors in last 10 minutes
errors=$(journalctl -u composesync --since "10 minutes ago" | grep -c "ERROR")
if [ $errors -gt 0 ]; then
echo "ComposeSync Alert: $errors errors detected" | \
mail -s "ComposeSync Alert" your-email@example.com
fi
```
### Webhook Alerts
Configure webhook notifications in your TOML file:
```toml
[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10
DRY_RUN = false
NOTIFICATION_WEBHOOK_URL = "https://your-webhook-url.com/endpoint"
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"
```
## Best Practices
### 1. Regular Log Review
```bash
# Daily log review
sudo journalctl -u composesync --since "yesterday" | grep "ERROR"
# Weekly summary
sudo journalctl -u composesync --since "1 week ago" | grep "Successfully updated" | wc -l
```
### 2. Monitor Key Metrics
- **Update success rate**: Track successful vs failed updates
- **Error frequency**: Monitor error patterns
- **Response time**: Track how long updates take
- **Disk usage**: Monitor backup storage
### 3. Set Up Automated Monitoring
```bash
# Create monitoring dashboard
sudo mkdir -p /opt/composesync/monitoring
sudo nano /opt/composesync/monitoring/dashboard.sh
```
### 4. Document Issues
Keep a log of issues and resolutions:
```bash
# Create issue log
sudo nano /opt/composesync/issue-log.md
```
## Troubleshooting Commands
### Quick Diagnostics
```bash
# Check service status
sudo systemctl status composesync
# Check logs for specific errors
# View recent logs
sudo journalctl -u composesync -n 20
# Verify file permissions
# Check configuration
sudo cat /opt/composesync/config.toml
# Test Docker access
docker ps
# Check file permissions
ls -la /opt/composesync/
```
### High Resource Usage
### Detailed Diagnostics
If ComposeSync is using too many resources:
```bash
# Check what's consuming resources
ps aux | grep update-agent
# Full system check
sudo /opt/composesync/health-check.sh
# Check for stuck processes
pgrep -f update-agent
# Restart the service
# Configuration validation
sudo systemctl restart composesync
```
sudo journalctl -u composesync -n 50
### Missing Logs
If logs are missing:
```bash
# Check if journald is working
sudo journalctl --verify
# Check journald status
sudo systemctl status systemd-journald
# Check log storage
sudo journalctl --disk-usage
```
## Integration with External Monitoring
### Prometheus/Grafana
For advanced monitoring, you can integrate with Prometheus:
```bash
# Example: Export metrics via webhook
curl -X POST -H "Content-Type: application/json" \
-d '{"metric": "composesync_updates_total", "value": 1, "labels": {"stack": "immich"}}' \
http://prometheus:9090/api/v1/write
```
### Nagios/Icinga
Create custom checks for monitoring systems:
```bash
# Example Nagios check
#!/bin/bash
if systemctl is-active --quiet composesync; then
echo "OK: ComposeSync is running"
exit 0
else
echo "CRITICAL: ComposeSync is not running"
exit 2
fi
```
## Security Monitoring
### Access Monitoring
Monitor who accesses ComposeSync:
```bash
# Check who modified the configuration
ls -la /opt/composesync/.env
# Check for unauthorized changes
find /opt/composesync -mtime -1 -ls
# Monitor Docker group membership
getent group docker
```
### Audit Logging
Enable audit logging:
```bash
# Monitor file access
auditctl -w /opt/composesync/.env -p wa -k composesync_config
# Monitor Docker socket access
auditctl -w /var/run/docker.sock -p wa -k docker_access
```
This comprehensive monitoring setup will help you keep track of ComposeSync's health and performance.
# Stack verification
for stack in /opt/composesync/stacks/*/; do
echo "=== $(basename $stack) ==="
docker compose -f "$stack/docker-compose.yml" config
done
```