robojerk/ComposeSync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ComposeSync/Docs/troubleshooting.md

13 KiB
Raw Blame History

Troubleshooting Guide

This guide helps you diagnose and resolve common issues with ComposeSync.

Quick Diagnostics

Service Status Check

# Check if ComposeSync is running
sudo systemctl status composesync

# Check recent logs
sudo journalctl -u composesync -n 20

# Check configuration
sudo cat /opt/composesync/config.toml

Basic Health Check

# Check service status
if systemctl is-active --quiet composesync; then
    echo "✓ ComposeSync is running"
else
    echo "✗ ComposeSync is not running"
fi

# Check for recent errors
error_count=$(journalctl -u composesync --since "1 hour ago" | grep -c "ERROR")
echo "Errors in last hour: $error_count"

# Check disk usage
usage=$(df -h /opt/composesync/ | tail -1 | awk '{print $5}' | sed 's/%//')
echo "Disk usage: ${usage}%"

Common Issues

Service Won't Start

Symptoms:

  • systemctl status composesync shows failed status
  • Service won't start with systemctl start composesync

Diagnosis:

# Check service status
sudo systemctl status composesync

# View detailed logs
sudo journalctl -u composesync -n 50

# Check configuration syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 10

Common Causes:

  1. Configuration Error

    # Check TOML syntax
sudo cat /opt/composesync/config.toml

# Test configuration
sudo systemctl restart composesync
sudo journalctl -u composesync -n 20 | grep "ERROR"

  2. Missing Dependencies

    # Check if required tools are installed
which wget
which git
which docker
which docker-compose

# Install missing tools
sudo apt update
sudo apt install wget git docker.io docker-compose-plugin

  3. Permission Issues

    # Check file permissions
ls -la /opt/composesync/

# Fix permissions
sudo chown -R $USER:docker /opt/composesync/
sudo chmod +x /opt/composesync/update-agent.sh
sudo chmod +x /opt/composesync/config-parser.sh

Solutions:

  • Fix configuration syntax errors
  • Install missing dependencies
  • Correct file permissions
  • Check systemd service file

Configuration Issues

Symptoms:

  • Service starts but doesn't process stacks
  • Stacks are skipped with errors
  • Configuration not loaded properly

Diagnosis:

# Check configuration file
sudo cat /opt/composesync/config.toml

# Check for configuration errors
sudo journalctl -u composesync | grep "Configuration"

# Test configuration loading
sudo systemctl restart composesync
sudo journalctl -u composesync -n 20

Common Issues:

  1. TOML Syntax Error

    # Incorrect TOML syntax
[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10

[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

    Fix: Ensure proper TOML syntax with correct indentation and quotes.

  2. Missing Required Fields

    # Missing required fields
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
# Missing PATH and TOOL

    Fix: Add all required fields (URL, PATH, TOOL).

  3. Invalid Paths

    [immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/invalid/path"  # Path doesn't exist
TOOL = "wget"

    Fix: Create the directory or use a valid path.

Solutions:

  • Validate TOML syntax
  • Ensure all required fields are present
  • Create missing directories
  • Check file permissions

Download Failures

Symptoms:

  • Stacks are skipped with download errors
  • Network connectivity issues
  • Invalid URLs

Diagnosis:

# Check network connectivity
ping -c 3 github.com

# Test URL manually
wget -O /tmp/test.yml https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml

# Check for download errors
sudo journalctl -u composesync | grep "Failed to download"

Common Issues:

  1. Network Connectivity

    # Test basic connectivity
ping -c 3 8.8.8.8

# Test DNS resolution
nslookup github.com

# Check proxy settings
echo $http_proxy
echo $https_proxy

  2. Invalid URLs

    # Incorrect URL format
[immich]
URL = "https://invalid-url.com/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

  3. Authentication Required

    # Private repository without authentication
[private-app]
URL = "https://github.com/user/private-repo.git"
PATH = "/opt/composesync/stacks/private"
TOOL = "git"

Solutions:

  • Check network connectivity
  • Verify URLs are correct and accessible
  • Configure authentication for private repositories
  • Check firewall settings

Docker Compose Failures

Symptoms:

  • Updates fail with Docker Compose errors
  • Services don't start after updates
  • Rollback occurs frequently

Diagnosis:

# Check Docker Compose syntax
cd /opt/composesync/stacks/immich
docker compose config

# Check service status
docker compose ps

# View service logs
docker compose logs

Common Issues:

  1. Invalid Compose File

    # Test compose file syntax
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml config

# Check for syntax errors
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml config 2>&1 | grep "ERROR"

  2. Port Conflicts

    # Check for port conflicts
netstat -tulpn | grep :80
netstat -tulpn | grep :443

# Check which services are using ports
sudo lsof -i :80
sudo lsof -i :443

  3. Resource Issues

    # Check available disk space
df -h

# Check available memory
free -h

# Check Docker disk usage
docker system df

Solutions:

  • Fix compose file syntax errors
  • Resolve port conflicts
  • Free up disk space and memory
  • Check Docker daemon status

Permission Issues

Symptoms:

  • Cannot write to stack directories
  • Cannot access Docker socket
  • Permission denied errors

Diagnosis:

# Check file permissions
ls -la /opt/composesync/

# Check user groups
groups $USER

# Check Docker socket permissions
ls -la /var/run/docker.sock

# Test Docker access
docker ps

Common Issues:

  1. User Not in Docker Group

    # Check if user is in docker group
groups $USER | grep docker

# Add user to docker group
sudo usermod -aG docker $USER

# Log out and back in, or run:
newgrp docker

  2. Incorrect File Ownership

    # Check ownership
ls -la /opt/composesync/stacks/

# Fix ownership
sudo chown -R $USER:docker /opt/composesync/stacks/

  3. Docker Socket Permissions

    # Check socket permissions
ls -la /var/run/docker.sock

# Fix socket permissions
sudo chmod 666 /var/run/docker.sock

Solutions:

  • Add user to docker group
  • Fix file ownership and permissions
  • Restart Docker daemon if needed
  • Check systemd service user configuration

Lock File Issues

Symptoms:

  • Service appears stuck
  • Updates not running
  • Lock file errors

Diagnosis:

# Check for lock files
ls -la /opt/composesync/.lock

# Check lock file age
stat /opt/composesync/.lock

# Check for stale locks
find /opt/composesync -name "*.lock" -mmin +5

Solutions:

# Remove stale lock file
sudo rm /opt/composesync/.lock

# Restart service
sudo systemctl restart composesync

# Check if service starts properly
sudo systemctl status composesync

Backup Issues

Symptoms:

  • No backup files created
  • Backup cleanup not working
  • Disk space issues

Diagnosis:

# Check backup files
find /opt/composesync/stacks -name "*.bak"

# Check backup retention settings
grep "KEEP_VERSIONS" /opt/composesync/config.toml

# Check disk usage
df -h /opt/composesync/

Common Issues:

  1. No Backups Created

    # Check if backups are being created
ls -la /opt/composesync/stacks/immich/compose-*.bak

# Check backup creation logs
sudo journalctl -u composesync | grep "backup"

  2. Backup Cleanup Not Working

    # Check backup retention
grep "KEEP_VERSIONS" /opt/composesync/config.toml

# Count backup files
find /opt/composesync/stacks -name "*.bak" | wc -l

# Manual cleanup
find /opt/composesync/stacks -name "*.bak" -mtime +30 -delete

Solutions:

  • Check backup creation logs
  • Verify backup retention settings
  • Manually clean old backups
  • Check disk space availability

Webhook Issues

Symptoms:

  • Webhook notifications not sent
  • Webhook delivery failures
  • Missing notifications

Diagnosis:

# Check webhook configuration
grep "NOTIFICATION_WEBHOOK_URL" /opt/composesync/config.toml

# Test webhook manually
curl -X POST -H "Content-Type: application/json" \
  -d '{"event": "test", "message": "Test notification"}' \
  https://your-webhook-url.com/endpoint

# Check webhook logs
sudo journalctl -u composesync | grep "webhook"

Common Issues:

  1. Invalid Webhook URL

    # Incorrect webhook URL
[global]
NOTIFICATION_WEBHOOK_URL = "https://invalid-webhook.com/endpoint"

  2. Network Issues

    # Test webhook connectivity
curl -I https://your-webhook-url.com/endpoint

# Check DNS resolution
nslookup your-webhook-url.com

  3. Authentication Issues

    # Test webhook with authentication
curl -X POST -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"event": "test"}' \
  https://your-webhook-url.com/endpoint

Solutions:

  • Verify webhook URL is correct
  • Check network connectivity
  • Configure authentication if required
  • Test webhook endpoint manually

Advanced Troubleshooting

Debug Mode

Enable debug logging for detailed troubleshooting:

# Edit service file to enable debug
sudo systemctl edit composesync

# Add debug environment variable
[Service]
Environment=DEBUG=true

Manual Testing

Test components manually:

# Test configuration parser
sudo -u composesync /opt/composesync/config-parser.sh

# Test update script manually
sudo -u composesync /opt/composesync/update-agent.sh

# Test Docker Compose commands
cd /opt/composesync/stacks/immich
docker compose config
docker compose ps

System Information

Gather system information for troubleshooting:

# System information
uname -a
lsb_release -a

# Docker information
docker version
docker info

# Disk usage
df -h
du -sh /opt/composesync/

# Memory usage
free -h

# Network connectivity
ping -c 3 github.com
curl -I https://github.com

Recovery Procedures

Manual Rollback

If automatic rollback fails:

# List available backups
ls -la /opt/composesync/stacks/immich/compose-*.bak

# Restore from backup
sudo cp /opt/composesync/stacks/immich/compose-20240115102001.yml.bak \
        /opt/composesync/stacks/immich/docker-compose.yml

# Apply rollback
cd /opt/composesync/stacks/immich
docker compose up -d

Service Recovery

If the service is completely broken:

# Stop service
sudo systemctl stop composesync

# Backup configuration
sudo cp /opt/composesync/config.toml /opt/composesync/config.toml.backup

# Reinstall service
sudo ./install.sh

# Restore configuration
sudo cp /opt/composesync/config.toml.backup /opt/composesync/config.toml

# Start service
sudo systemctl start composesync

Complete Reset

As a last resort:

# Stop service
sudo systemctl stop composesync
sudo systemctl disable composesync

# Backup important data
sudo cp -r /opt/composesync/stacks /tmp/composesync-backup

# Remove installation
sudo rm -rf /opt/composesync
sudo rm /etc/systemd/system/composesync.service

# Reinstall
sudo ./install.sh

# Restore stacks
sudo cp -r /tmp/composesync-backup/* /opt/composesync/stacks/

# Start service
sudo systemctl start composesync

Getting Help

Log Collection

Collect logs for troubleshooting:

# Create log archive
sudo journalctl -u composesync > /tmp/composesync-logs.txt
sudo cat /opt/composesync/config.toml > /tmp/composesync-config.txt
sudo systemctl status composesync > /tmp/composesync-status.txt

# Archive logs
tar -czf composesync-debug.tar.gz /tmp/composesync-*.txt

Information to Include

When seeking help, include:

  1. System Information:

    • OS version and distribution
    • Docker version
    • ComposeSync version

  2. Configuration:

    • Relevant parts of config.toml (remove sensitive data)
    • Service status output

  3. Logs:

    • Recent error logs
    • Service status logs
    • Configuration loading logs

  4. Steps to Reproduce:

    • What you were trying to do
    • What happened
    • What you expected to happen

Common Solutions Summary

Issue Quick Fix Detailed Fix
Service won't start Check config syntax Validate TOML, check permissions
Download failures Test URL manually Check network, verify URLs
Docker failures Check compose syntax Fix compose file, resolve conflicts
Permission issues Add user to docker group Fix ownership, check socket permissions
Lock file stuck Remove .lock file Restart service, check for processes
No backups Check retention settings Verify backup creation, check disk space
Webhook failures Test URL manually Check network, verify authentication