ComposeSync/Docs/troubleshooting.md

# Troubleshooting Guide

This guide helps you diagnose and resolve common issues with ComposeSync.

## Quick Diagnostics

### Service Status Check

```bash
# Check if ComposeSync is running
sudo systemctl status composesync

# Check recent logs
sudo journalctl -u composesync -n 20

# Check configuration
sudo cat /opt/composesync/config.toml
```

### Basic Health Check

```bash
# Check service status
if systemctl is-active --quiet composesync; then
    echo "✓ ComposeSync is running"
else
    echo "✗ ComposeSync is not running"
fi

# Check for recent errors
error_count=$(journalctl -u composesync --since "1 hour ago" | grep -c "ERROR")
echo "Errors in last hour: $error_count"

# Check disk usage
usage=$(df -h /opt/composesync/ | tail -1 | awk '{print $5}' | sed 's/%//')
echo "Disk usage: ${usage}%"
```

## Common Issues

### Service Won't Start

**Symptoms:**
- `systemctl status composesync` shows failed status
- Service won't start with `systemctl start composesync`

**Diagnosis:**
```bash
# Check service status
sudo systemctl status composesync

# View detailed logs
sudo journalctl -u composesync -n 50

# Check configuration syntax
sudo systemctl restart composesync
sudo journalctl -u composesync -n 10
```

**Common Causes:**

1. **Configuration Error**
   ```bash
   # Check TOML syntax
   sudo cat /opt/composesync/config.toml
   
   # Test configuration
   sudo systemctl restart composesync
   sudo journalctl -u composesync -n 20 | grep "ERROR"
   ```

2. **Missing Dependencies**
   ```bash
   # Check if required tools are installed
   which wget
   which git
   which docker
   which docker-compose
   
   # Install missing tools
   sudo apt update
   sudo apt install wget git docker.io docker-compose-plugin
   ```

3. **Permission Issues**
   ```bash
   # Check file permissions
   ls -la /opt/composesync/
   
   # Fix permissions
   sudo chown -R $USER:docker /opt/composesync/
   sudo chmod +x /opt/composesync/update-agent.sh
   sudo chmod +x /opt/composesync/config-parser.sh
   ```

**Solutions:**
- Fix configuration syntax errors
- Install missing dependencies
- Correct file permissions
- Check systemd service file

### Configuration Issues

**Symptoms:**
- Service starts but doesn't process stacks
- Stacks are skipped with errors
- Configuration not loaded properly

**Diagnosis:**
```bash
# Check configuration file
sudo cat /opt/composesync/config.toml

# Check for configuration errors
sudo journalctl -u composesync | grep "Configuration"

# Test configuration loading
sudo systemctl restart composesync
sudo journalctl -u composesync -n 20
```

**Common Issues:**

1. **TOML Syntax Error**
   ```toml
   # Incorrect TOML syntax
   [global]
   UPDATE_INTERVAL_SECONDS = 3600
   KEEP_VERSIONS = 10
   
   [immich]
   URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
   PATH = "/opt/composesync/stacks/immich"
   TOOL = "wget"
   ```

   **Fix:** Ensure proper TOML syntax with correct indentation and quotes.

2. **Missing Required Fields**
   ```toml
   # Missing required fields
   [immich]
   URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
   # Missing PATH and TOOL
   ```

   **Fix:** Add all required fields (URL, PATH, TOOL).

3. **Invalid Paths**
   ```toml
   [immich]
   URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
   PATH = "/invalid/path"  # Path doesn't exist
   TOOL = "wget"
   ```

   **Fix:** Create the directory or use a valid path.

**Solutions:**
- Validate TOML syntax
- Ensure all required fields are present
- Create missing directories
- Check file permissions

### Download Failures

**Symptoms:**
- Stacks are skipped with download errors
- Network connectivity issues
- Invalid URLs

**Diagnosis:**
```bash
# Check network connectivity
ping -c 3 github.com

# Test URL manually
wget -O /tmp/test.yml https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml

# Check for download errors
sudo journalctl -u composesync | grep "Failed to download"
```

**Common Issues:**

1. **Network Connectivity**
   ```bash
   # Test basic connectivity
   ping -c 3 8.8.8.8
   
   # Test DNS resolution
   nslookup github.com
   
   # Check proxy settings
   echo $http_proxy
   echo $https_proxy
   ```

2. **Invalid URLs**
   ```toml
   # Incorrect URL format
   [immich]
   URL = "https://invalid-url.com/docker-compose.yml"
   PATH = "/opt/composesync/stacks/immich"
   TOOL = "wget"
   ```

3. **Authentication Required**
   ```toml
   # Private repository without authentication
   [private-app]
   URL = "https://github.com/user/private-repo.git"
   PATH = "/opt/composesync/stacks/private"
   TOOL = "git"
   ```

**Solutions:**
- Check network connectivity
- Verify URLs are correct and accessible
- Configure authentication for private repositories
- Check firewall settings

### Docker Compose Failures

**Symptoms:**
- Updates fail with Docker Compose errors
- Services don't start after updates
- Rollback occurs frequently

**Diagnosis:**
```bash
# Check Docker Compose syntax
cd /opt/composesync/stacks/immich
docker compose config

# Check service status
docker compose ps

# View service logs
docker compose logs
```

**Common Issues:**

1. **Invalid Compose File**
   ```bash
   # Test compose file syntax
   docker compose -f /opt/composesync/stacks/immich/docker-compose.yml config
   
   # Check for syntax errors
   docker compose -f /opt/composesync/stacks/immich/docker-compose.yml config 2>&1 | grep "ERROR"
   ```

2. **Port Conflicts**
   ```bash
   # Check for port conflicts
   netstat -tulpn | grep :80
   netstat -tulpn | grep :443
   
   # Check which services are using ports
   sudo lsof -i :80
   sudo lsof -i :443
   ```

3. **Resource Issues**
   ```bash
   # Check available disk space
   df -h
   
   # Check available memory
   free -h
   
   # Check Docker disk usage
   docker system df
   ```

**Solutions:**
- Fix compose file syntax errors
- Resolve port conflicts
- Free up disk space and memory
- Check Docker daemon status

### Permission Issues

**Symptoms:**
- Cannot write to stack directories
- Cannot access Docker socket
- Permission denied errors

**Diagnosis:**
```bash
# Check file permissions
ls -la /opt/composesync/

# Check user groups
groups $USER

# Check Docker socket permissions
ls -la /var/run/docker.sock

# Test Docker access
docker ps
```

**Common Issues:**

1. **User Not in Docker Group**
   ```bash
   # Check if user is in docker group
   groups $USER | grep docker
   
   # Add user to docker group
   sudo usermod -aG docker $USER
   
   # Log out and back in, or run:
   newgrp docker
   ```

2. **Incorrect File Ownership**
   ```bash
   # Check ownership
   ls -la /opt/composesync/stacks/
   
   # Fix ownership
   sudo chown -R $USER:docker /opt/composesync/stacks/
   ```

3. **Docker Socket Permissions**
   ```bash
   # Check socket permissions
   ls -la /var/run/docker.sock
   
   # Fix socket permissions
   sudo chmod 666 /var/run/docker.sock
   ```

**Solutions:**
- Add user to docker group
- Fix file ownership and permissions
- Restart Docker daemon if needed
- Check systemd service user configuration

### Lock File Issues

**Symptoms:**
- Service appears stuck
- Updates not running
- Lock file errors

**Diagnosis:**
```bash
# Check for lock files
ls -la /opt/composesync/.lock

# Check lock file age
stat /opt/composesync/.lock

# Check for stale locks
find /opt/composesync -name "*.lock" -mmin +5
```

**Solutions:**
```bash
# Remove stale lock file
sudo rm /opt/composesync/.lock

# Restart service
sudo systemctl restart composesync

# Check if service starts properly
sudo systemctl status composesync
```

### Backup Issues

**Symptoms:**
- No backup files created
- Backup cleanup not working
- Disk space issues

**Diagnosis:**
```bash
# Check backup files
find /opt/composesync/stacks -name "*.bak"

# Check backup retention settings
grep "KEEP_VERSIONS" /opt/composesync/config.toml

# Check disk usage
df -h /opt/composesync/
```

**Common Issues:**

1. **No Backups Created**
   ```bash
   # Check if backups are being created
   ls -la /opt/composesync/stacks/immich/compose-*.bak
   
   # Check backup creation logs
   sudo journalctl -u composesync | grep "backup"
   ```

2. **Backup Cleanup Not Working**
   ```bash
   # Check backup retention
   grep "KEEP_VERSIONS" /opt/composesync/config.toml
   
   # Count backup files
   find /opt/composesync/stacks -name "*.bak" | wc -l
   
   # Manual cleanup
   find /opt/composesync/stacks -name "*.bak" -mtime +30 -delete
   ```

**Solutions:**
- Check backup creation logs
- Verify backup retention settings
- Manually clean old backups
- Check disk space availability

### Webhook Issues

**Symptoms:**
- Webhook notifications not sent
- Webhook delivery failures
- Missing notifications

**Diagnosis:**
```bash
# Check webhook configuration
grep "NOTIFICATION_WEBHOOK_URL" /opt/composesync/config.toml

# Test webhook manually
curl -X POST -H "Content-Type: application/json" \
  -d '{"event": "test", "message": "Test notification"}' \
  https://your-webhook-url.com/endpoint

# Check webhook logs
sudo journalctl -u composesync | grep "webhook"
```

**Common Issues:**

1. **Invalid Webhook URL**
   ```toml
   # Incorrect webhook URL
   [global]
   NOTIFICATION_WEBHOOK_URL = "https://invalid-webhook.com/endpoint"
   ```

2. **Network Issues**
   ```bash
   # Test webhook connectivity
   curl -I https://your-webhook-url.com/endpoint
   
   # Check DNS resolution
   nslookup your-webhook-url.com
   ```

3. **Authentication Issues**
   ```bash
   # Test webhook with authentication
   curl -X POST -H "Content-Type: application/json" \
     -H "Authorization: Bearer YOUR_TOKEN" \
     -d '{"event": "test"}' \
     https://your-webhook-url.com/endpoint
   ```

**Solutions:**
- Verify webhook URL is correct
- Check network connectivity
- Configure authentication if required
- Test webhook endpoint manually

## Advanced Troubleshooting

### Debug Mode

Enable debug logging for detailed troubleshooting:

```bash
# Edit service file to enable debug
sudo systemctl edit composesync

# Add debug environment variable
[Service]
Environment=DEBUG=true
```

### Manual Testing

Test components manually:

```bash
# Test configuration parser
sudo -u composesync /opt/composesync/config-parser.sh

# Test update script manually
sudo -u composesync /opt/composesync/update-agent.sh

# Test Docker Compose commands
cd /opt/composesync/stacks/immich
docker compose config
docker compose ps
```

### System Information

Gather system information for troubleshooting:

```bash
# System information
uname -a
lsb_release -a

# Docker information
docker version
docker info

# Disk usage
df -h
du -sh /opt/composesync/

# Memory usage
free -h

# Network connectivity
ping -c 3 github.com
curl -I https://github.com
```

## Recovery Procedures

### Manual Rollback

If automatic rollback fails:

```bash
# List available backups
ls -la /opt/composesync/stacks/immich/compose-*.bak

# Restore from backup
sudo cp /opt/composesync/stacks/immich/compose-20240115102001.yml.bak \
        /opt/composesync/stacks/immich/docker-compose.yml

# Apply rollback
cd /opt/composesync/stacks/immich
docker compose up -d
```

### Service Recovery

If the service is completely broken:

```bash
# Stop service
sudo systemctl stop composesync

# Backup configuration
sudo cp /opt/composesync/config.toml /opt/composesync/config.toml.backup

# Reinstall service
sudo ./install.sh

# Restore configuration
sudo cp /opt/composesync/config.toml.backup /opt/composesync/config.toml

# Start service
sudo systemctl start composesync
```

### Complete Reset

As a last resort:

```bash
# Stop service
sudo systemctl stop composesync
sudo systemctl disable composesync

# Backup important data
sudo cp -r /opt/composesync/stacks /tmp/composesync-backup

# Remove installation
sudo rm -rf /opt/composesync
sudo rm /etc/systemd/system/composesync.service

# Reinstall
sudo ./install.sh

# Restore stacks
sudo cp -r /tmp/composesync-backup/* /opt/composesync/stacks/

# Start service
sudo systemctl start composesync
```

## Getting Help

### Log Collection

Collect logs for troubleshooting:

```bash
# Create log archive
sudo journalctl -u composesync > /tmp/composesync-logs.txt
sudo cat /opt/composesync/config.toml > /tmp/composesync-config.txt
sudo systemctl status composesync > /tmp/composesync-status.txt

# Archive logs
tar -czf composesync-debug.tar.gz /tmp/composesync-*.txt
```

### Information to Include

When seeking help, include:

1. **System Information:**
   - OS version and distribution
   - Docker version
   - ComposeSync version

2. **Configuration:**
   - Relevant parts of config.toml (remove sensitive data)
   - Service status output

3. **Logs:**
   - Recent error logs
   - Service status logs
   - Configuration loading logs

4. **Steps to Reproduce:**
   - What you were trying to do
   - What happened
   - What you expected to happen

### Common Solutions Summary

| Issue | Quick Fix | Detailed Fix |
|-------|-----------|--------------|
| Service won't start | Check config syntax | Validate TOML, check permissions |
| Download failures | Test URL manually | Check network, verify URLs |
| Docker failures | Check compose syntax | Fix compose file, resolve conflicts |
| Permission issues | Add user to docker group | Fix ownership, check socket permissions |
| Lock file stuck | Remove .lock file | Restart service, check for processes |
| No backups | Check retention settings | Verify backup creation, check disk space |
| Webhook failures | Test URL manually | Check network, verify authentication |