ComposeSync/Docs/safety-features.md

# Safety Features

ComposeSync includes several safety features to protect your Docker stacks and ensure reliable operation.

## Overview

Safety features include:
- **Automatic rollback** on failed updates
- **Versioned backups** for easy recovery
- **Lock file protection** against concurrent updates
- **Override preservation** to maintain customizations
- **Error handling** with detailed logging
- **Dry-run mode** for testing configurations

## Automatic Rollback

When a Docker Compose update fails, ComposeSync automatically rolls back to the previous working version.

### How Rollback Works

1. **Pre-update backup** - Current configuration is saved before applying changes
2. **Update attempt** - New configuration is applied using `docker compose up -d`
3. **Failure detection** - If the update fails, the error is logged
4. **Automatic rollback** - Previous configuration is restored
5. **Notification** - Webhook notification is sent (if configured)

### Rollback Process

```bash
# Example rollback log
[2024-01-15 10:30:00] Processing stack: immich
[2024-01-15 10:30:01] Creating backup: compose-20240115103001.yml.bak
[2024-01-15 10:30:02] Applying new configuration...
[2024-01-15 10:30:03] ERROR: docker compose up -d failed
[2024-01-15 10:30:03] Rolling back to previous version
[2024-01-15 10:30:04] Rollback successful
[2024-01-15 10:30:04] Sending webhook notification
```

## Versioned Backups

ComposeSync maintains a history of your Docker Compose configurations for easy recovery.

### Backup Naming

Backups are named with timestamps or Git commit hashes:

```
/opt/composesync/stacks/immich/
├── docker-compose.yml              # Current configuration
├── docker-compose.override.yml     # Your customizations
├── compose-20240115103001.yml.bak  # Timestamped backup
├── compose-20240115102001.yml.bak  # Previous backup
└── compose-20240115101001.yml.bak  # Older backup
```

### TOML Configuration

Configure backup retention in your TOML file:

```toml
# Global settings
[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10  # Keep 10 backup versions
DRY_RUN = false

# Stack configurations
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

[dev-app]
URL = "https://github.com/user/dev-app.git"
PATH = "/opt/composesync/stacks/dev-app"
TOOL = "git"
GIT_SUBPATH = "docker/docker-compose.yml"
GIT_REF = "develop"
KEEP_VERSIONS = 5  # Keep fewer versions for dev
```

### Legacy .env Configuration

```env
# Global settings
UPDATE_INTERVAL_SECONDS=3600
KEEP_VERSIONS=10
DRY_RUN=false

# Stack configurations
STACKS=2

STACK_1_NAME=immich
STACK_1_URL=https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
STACK_1_PATH=/opt/composesync/stacks/immich
STACK_1_TOOL=wget

STACK_2_NAME=dev-app
STACK_2_URL=https://github.com/user/dev-app.git
STACK_2_PATH=/opt/composesync/stacks/dev-app
STACK_2_TOOL=git
STACK_2_GIT_SUBPATH=docker/docker-compose.yml
STACK_2_GIT_REF=develop
STACK_2_KEEP_VERSIONS=5
```

### Manual Rollback

To manually rollback to a previous version:

```bash
# List available backups
ls -la /opt/composesync/stacks/immich/compose-*.yml.bak

# Restore a specific backup
sudo cp /opt/composesync/stacks/immich/compose-20240115102001.yml.bak \
        /opt/composesync/stacks/immich/docker-compose.yml

# Apply the rollback
cd /opt/composesync/stacks/immich
docker compose up -d
```

## Lock File Protection

ComposeSync uses lock files to prevent concurrent updates that could cause conflicts.

### Lock File Operation

1. **Lock acquisition** - Creates a lock file before starting updates
2. **Update process** - Performs the update while lock is held
3. **Lock release** - Removes the lock file when complete
4. **Timeout handling** - Automatically removes stale locks

### Lock File Location

```
/opt/composesync/
├── update-agent.sh
├── config.toml
├── .env
└── .lock                    # Global lock file
```

### Lock File Behavior

- **Single process** - Only one update process can run at a time
- **Automatic cleanup** - Stale locks are automatically removed
- **Error recovery** - Locks are released even if the process crashes
- **Timeout protection** - Long-running processes don't block indefinitely

## Override Preservation

Your `docker-compose.override.yml` files are always preserved during updates.

### Override File Structure

```
/opt/composesync/stacks/immich/
├── docker-compose.yml              # Updated from remote source
├── docker-compose.override.yml     # Your customizations (preserved)
├── compose-*.yml.bak               # Backup versions
└── .git/                           # Git repository (if using git)
```

### Override File Example

```yaml
# docker-compose.override.yml
version: '3.8'
services:
  immich-server:
    environment:
      - DATABASE_URL=postgresql://user:pass@localhost:5432/immich
      - REDIS_URL=redis://localhost:6379
    volumes:
      - /mnt/photos:/photos
      - /mnt/backups:/backups
    restart: unless-stopped
    networks:
      - immich-network

networks:
  immich-network:
    external: true
```

### How Overrides Work

1. **Download** - New `docker-compose.yml` is downloaded
2. **Preserve** - Your `docker-compose.override.yml` is kept unchanged
3. **Merge** - Docker Compose merges both files automatically
4. **Apply** - Combined configuration is applied

## Error Handling

ComposeSync includes comprehensive error handling and logging.

### Error Types

| Error Type | Description | Action Taken |
|------------|-------------|--------------|
| **Download failure** | Cannot download compose file | Skip stack, continue with others |
| **Parse error** | Invalid YAML in compose file | Skip stack, log error |
| **Docker failure** | `docker compose up -d` fails | Rollback to previous version |
| **Permission error** | Cannot write to stack directory | Skip stack, log error |
| **Network error** | Cannot reach remote source | Skip stack, retry later |

### Error Logging

```bash
# View error logs
sudo journalctl -u composesync | grep "ERROR"

# View recent errors
sudo journalctl -u composesync -n 100 | grep "ERROR"

# View specific stack errors
sudo journalctl -u composesync | grep "immich.*ERROR"
```

### Error Recovery

ComposeSync automatically recovers from most errors:

- **Temporary failures** - Retried on next update cycle
- **Permanent failures** - Logged and skipped until resolved
- **Partial failures** - Other stacks continue to update normally

## Dry-Run Mode

Test your configuration safely without applying changes.

### TOML Configuration

```toml
# Global dry-run mode
[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10
DRY_RUN = true  # Enable dry-run mode

# Stack configurations
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"
```

### Per-Stack Dry-Run Mode

```toml
# Global settings
[global]
DRY_RUN = false  # Default: apply changes

# Production stack (apply changes)
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

# Development stack (dry-run only)
[dev-app]
URL = "https://github.com/user/dev-app.git"
PATH = "/opt/composesync/stacks/dev-app"
TOOL = "git"
GIT_SUBPATH = "docker/docker-compose.yml"
GIT_REF = "develop"
DRY_RUN = true  # Override global setting
```

### Dry-Run Benefits

- **Safe testing** - No actual changes are applied
- **Configuration validation** - Verify URLs and settings work
- **Change preview** - See what would be updated
- **Webhook testing** - Test notifications without affecting stacks

## Best Practices

### 1. Use Appropriate Backup Retention

```toml
# Production stacks (keep more versions)
[production]
URL = "https://..."
PATH = "/opt/composesync/stacks/production"
TOOL = "wget"
KEEP_VERSIONS = 15

# Development stacks (keep fewer versions)
[development]
URL = "https://..."
PATH = "/opt/composesync/stacks/development"
TOOL = "git"
KEEP_VERSIONS = 5
```

### 2. Test with Dry-Run Mode

```toml
# Always test new configurations
[global]
DRY_RUN = true

[new-stack]
URL = "https://..."
PATH = "/opt/composesync/stacks/new-stack"
TOOL = "wget"
```

### 3. Monitor Error Logs

```bash
# Set up log monitoring
sudo journalctl -u composesync -f

# Check for errors regularly
sudo journalctl -u composesync --since "1 hour ago" | grep "ERROR"
```

### 4. Use Webhook Notifications

```toml
# Get notified of issues
[global]
NOTIFICATION_WEBHOOK_URL = "https://your-webhook-url.com/endpoint"

[immich]
URL = "https://..."
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"
```

### 5. Regular Backup Verification

```bash
# Verify backups are being created
ls -la /opt/composesync/stacks/*/compose-*.yml.bak

# Test backup restoration
sudo cp /opt/composesync/stacks/immich/compose-*.yml.bak /tmp/test.yml
docker compose -f /tmp/test.yml config
```

## Troubleshooting

### Rollback Not Working

```bash
# Check if backups exist
ls -la /opt/composesync/stacks/immich/compose-*.yml.bak

# Check rollback logs
sudo journalctl -u composesync | grep "rollback"

# Verify Docker Compose works
cd /opt/composesync/stacks/immich
docker compose config
```

### Lock File Issues

```bash
# Check for stale lock
ls -la /opt/composesync/.lock

# Remove stale lock (if needed)
sudo rm /opt/composesync/.lock

# Restart service
sudo systemctl restart composesync
```

### Override File Issues

```bash
# Check override file syntax
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml \
               -f /opt/composesync/stacks/immich/docker-compose.override.yml config

# Verify override file permissions
ls -la /opt/composesync/stacks/immich/docker-compose.override.yml
```

### Backup Cleanup Issues

```bash
# Check backup retention settings
grep "KEEP_VERSIONS" /opt/composesync/config.toml

# Manually clean old backups
find /opt/composesync/stacks -name "compose-*.yml.bak" -mtime +30 -delete
```