robojerk/ComposeSync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ComposeSync/Docs/safety-features.md

9.9 KiB
Raw Blame History

Safety Features

ComposeSync includes several safety features to protect your Docker stacks and ensure reliable operation.

Overview

Safety features include:

  • Automatic rollback on failed updates
  • Versioned backups for easy recovery
  • Lock file protection against concurrent updates
  • Override preservation to maintain customizations
  • Error handling with detailed logging
  • Dry-run mode for testing configurations

Automatic Rollback

When a Docker Compose update fails, ComposeSync automatically rolls back to the previous working version.

How Rollback Works

  1. Pre-update backup - Current configuration is saved before applying changes
  2. Update attempt - New configuration is applied using docker compose up -d
  3. Failure detection - If the update fails, the error is logged
  4. Automatic rollback - Previous configuration is restored
  5. Notification - Webhook notification is sent (if configured)

Rollback Process

# Example rollback log
[2024-01-15 10:30:00] Processing stack: immich
[2024-01-15 10:30:01] Creating backup: compose-20240115103001.yml.bak
[2024-01-15 10:30:02] Applying new configuration...
[2024-01-15 10:30:03] ERROR: docker compose up -d failed
[2024-01-15 10:30:03] Rolling back to previous version
[2024-01-15 10:30:04] Rollback successful
[2024-01-15 10:30:04] Sending webhook notification

Versioned Backups

ComposeSync maintains a history of your Docker Compose configurations for easy recovery.

Backup Naming

Backups are named with timestamps or Git commit hashes:

/opt/composesync/stacks/immich/
├── docker-compose.yml              # Current configuration
├── docker-compose.override.yml     # Your customizations
├── compose-20240115103001.yml.bak  # Timestamped backup
├── compose-20240115102001.yml.bak  # Previous backup
└── compose-20240115101001.yml.bak  # Older backup

TOML Configuration

Configure backup retention in your TOML file:

# Global settings
[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10  # Keep 10 backup versions
DRY_RUN = false

# Stack configurations
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

[dev-app]
URL = "https://github.com/user/dev-app.git"
PATH = "/opt/composesync/stacks/dev-app"
TOOL = "git"
GIT_SUBPATH = "docker/docker-compose.yml"
GIT_REF = "develop"
KEEP_VERSIONS = 5  # Keep fewer versions for dev

Legacy .env Configuration

# Global settings
UPDATE_INTERVAL_SECONDS=3600
KEEP_VERSIONS=10
DRY_RUN=false

# Stack configurations
STACKS=2

STACK_1_NAME=immich
STACK_1_URL=https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
STACK_1_PATH=/opt/composesync/stacks/immich
STACK_1_TOOL=wget

STACK_2_NAME=dev-app
STACK_2_URL=https://github.com/user/dev-app.git
STACK_2_PATH=/opt/composesync/stacks/dev-app
STACK_2_TOOL=git
STACK_2_GIT_SUBPATH=docker/docker-compose.yml
STACK_2_GIT_REF=develop
STACK_2_KEEP_VERSIONS=5

Manual Rollback

To manually rollback to a previous version:

# List available backups
ls -la /opt/composesync/stacks/immich/compose-*.yml.bak

# Restore a specific backup
sudo cp /opt/composesync/stacks/immich/compose-20240115102001.yml.bak \
        /opt/composesync/stacks/immich/docker-compose.yml

# Apply the rollback
cd /opt/composesync/stacks/immich
docker compose up -d

Lock File Protection

ComposeSync uses lock files to prevent concurrent updates that could cause conflicts.

Lock File Operation

  1. Lock acquisition - Creates a lock file before starting updates
  2. Update process - Performs the update while lock is held
  3. Lock release - Removes the lock file when complete
  4. Timeout handling - Automatically removes stale locks

Lock File Location

/opt/composesync/
├── update-agent.sh
├── config.toml
├── .env
└── .lock                    # Global lock file

Lock File Behavior

  • Single process - Only one update process can run at a time
  • Automatic cleanup - Stale locks are automatically removed
  • Error recovery - Locks are released even if the process crashes
  • Timeout protection - Long-running processes don't block indefinitely

Override Preservation

Your docker-compose.override.yml files are always preserved during updates.

Override File Structure

/opt/composesync/stacks/immich/
├── docker-compose.yml              # Updated from remote source
├── docker-compose.override.yml     # Your customizations (preserved)
├── compose-*.yml.bak               # Backup versions
└── .git/                           # Git repository (if using git)

Override File Example

# docker-compose.override.yml
version: '3.8'
services:
  immich-server:
    environment:
      - DATABASE_URL=postgresql://user:pass@localhost:5432/immich
      - REDIS_URL=redis://localhost:6379
    volumes:
      - /mnt/photos:/photos
      - /mnt/backups:/backups
    restart: unless-stopped
    networks:
      - immich-network

networks:
  immich-network:
    external: true

How Overrides Work

  1. Download - New docker-compose.yml is downloaded
  2. Preserve - Your docker-compose.override.yml is kept unchanged
  3. Merge - Docker Compose merges both files automatically
  4. Apply - Combined configuration is applied

Error Handling

ComposeSync includes comprehensive error handling and logging.

Error Types

Error Type Description Action Taken
Download failure Cannot download compose file Skip stack, continue with others
Parse error Invalid YAML in compose file Skip stack, log error
Docker failure docker compose up -d fails Rollback to previous version
Permission error Cannot write to stack directory Skip stack, log error
Network error Cannot reach remote source Skip stack, retry later

Error Logging

# View error logs
sudo journalctl -u composesync | grep "ERROR"

# View recent errors
sudo journalctl -u composesync -n 100 | grep "ERROR"

# View specific stack errors
sudo journalctl -u composesync | grep "immich.*ERROR"

Error Recovery

ComposeSync automatically recovers from most errors:

  • Temporary failures - Retried on next update cycle
  • Permanent failures - Logged and skipped until resolved
  • Partial failures - Other stacks continue to update normally

Dry-Run Mode

Test your configuration safely without applying changes.

TOML Configuration

# Global dry-run mode
[global]
UPDATE_INTERVAL_SECONDS = 3600
KEEP_VERSIONS = 10
DRY_RUN = true  # Enable dry-run mode

# Stack configurations
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

Per-Stack Dry-Run Mode

# Global settings
[global]
DRY_RUN = false  # Default: apply changes

# Production stack (apply changes)
[immich]
URL = "https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml"
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

# Development stack (dry-run only)
[dev-app]
URL = "https://github.com/user/dev-app.git"
PATH = "/opt/composesync/stacks/dev-app"
TOOL = "git"
GIT_SUBPATH = "docker/docker-compose.yml"
GIT_REF = "develop"
DRY_RUN = true  # Override global setting

Dry-Run Benefits

  • Safe testing - No actual changes are applied
  • Configuration validation - Verify URLs and settings work
  • Change preview - See what would be updated
  • Webhook testing - Test notifications without affecting stacks

Best Practices

1. Use Appropriate Backup Retention

# Production stacks (keep more versions)
[production]
URL = "https://..."
PATH = "/opt/composesync/stacks/production"
TOOL = "wget"
KEEP_VERSIONS = 15

# Development stacks (keep fewer versions)
[development]
URL = "https://..."
PATH = "/opt/composesync/stacks/development"
TOOL = "git"
KEEP_VERSIONS = 5

2. Test with Dry-Run Mode

# Always test new configurations
[global]
DRY_RUN = true

[new-stack]
URL = "https://..."
PATH = "/opt/composesync/stacks/new-stack"
TOOL = "wget"

3. Monitor Error Logs

# Set up log monitoring
sudo journalctl -u composesync -f

# Check for errors regularly
sudo journalctl -u composesync --since "1 hour ago" | grep "ERROR"

4. Use Webhook Notifications

# Get notified of issues
[global]
NOTIFICATION_WEBHOOK_URL = "https://your-webhook-url.com/endpoint"

[immich]
URL = "https://..."
PATH = "/opt/composesync/stacks/immich"
TOOL = "wget"

5. Regular Backup Verification

# Verify backups are being created
ls -la /opt/composesync/stacks/*/compose-*.yml.bak

# Test backup restoration
sudo cp /opt/composesync/stacks/immich/compose-*.yml.bak /tmp/test.yml
docker compose -f /tmp/test.yml config

Troubleshooting

Rollback Not Working

# Check if backups exist
ls -la /opt/composesync/stacks/immich/compose-*.yml.bak

# Check rollback logs
sudo journalctl -u composesync | grep "rollback"

# Verify Docker Compose works
cd /opt/composesync/stacks/immich
docker compose config

Lock File Issues

# Check for stale lock
ls -la /opt/composesync/.lock

# Remove stale lock (if needed)
sudo rm /opt/composesync/.lock

# Restart service
sudo systemctl restart composesync

Override File Issues

# Check override file syntax
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml \
               -f /opt/composesync/stacks/immich/docker-compose.override.yml config

# Verify override file permissions
ls -la /opt/composesync/stacks/immich/docker-compose.override.yml

Backup Cleanup Issues

# Check backup retention settings
grep "KEEP_VERSIONS" /opt/composesync/config.toml

# Manually clean old backups
find /opt/composesync/stacks -name "compose-*.yml.bak" -mtime +30 -delete