ComposeSync/Docs/safety-features.md

Safety Features

This guide covers the safety features built into ComposeSync to protect your Docker Compose stacks.

Automatic Rollback

ComposeSync includes automatic rollback functionality to handle failed updates:

How Rollback Works

1. Pre-Update Backup: Before applying changes, all current files are backed up
2. Update Attempt: Changes are applied using docker compose up -d
3. Failure Detection: If the Docker Compose command fails, rollback is triggered
4. Automatic Restoration: All files are restored from the backup
5. Stack Restart: The stack is restarted with the original configuration

Rollback Process

When a rollback occurs, ComposeSync will:

# 1. Restore main compose file
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/docker-compose.yml /opt/composesync/stacks/myapp/docker-compose.yml

# 2. Restore extra files (if any)
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/hwaccel.ml.yml /opt/composesync/stacks/myapp/hwaccel.ml.yml

# 3. Restore override file (if it existed)
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/docker-compose.override.yml /opt/composesync/stacks/myapp/docker-compose.override.yml

# 4. Restart stack with original configuration
docker compose -f /opt/composesync/stacks/myapp/docker-compose.yml up -d

Rollback Benefits

This ensures that:

• Your stack never gets stuck in a broken state
• Failed updates don't leave your services down
• You can quickly recover from problematic updates
• The system remains stable even with network issues or invalid configurations

Lock File Protection

ComposeSync uses lock files to prevent concurrent updates to the same stack.

How Lock Files Work

Each stack has a .lock file in its directory that:

• Is created when an update starts
• Is automatically removed when the update completes (success or failure)
• Has a 5-minute timeout to handle stale locks from crashed processes
• Uses directory-based locking for atomicity

Lock File Location

/opt/composesync/stacks/myapp/
├── docker-compose.yml
├── docker-compose.override.yml
├── .lock/                    # Lock directory
└── backups/

Stale Lock Detection

If a lock file exists and is older than 5 minutes, ComposeSync will automatically remove it and proceed with the update. This handles cases where:

• The previous update process crashed
• The system was rebooted during an update
• Network issues interrupted the update process

Lock File Benefits

Lock files prevent:

• Race conditions when multiple instances run simultaneously
• Corruption of compose files during updates
• Multiple update processes running on the same stack

Versioned History

ComposeSync maintains a versioned history of your compose files for easy rollback and audit trails.

Version Identifiers

• For Git sources: Uses the Git commit hash as the version identifier
• For other sources: Uses a timestamp (YYYYMMDDHHMMSS format)

Versioned File Structure

stack/
├── docker-compose.yml           # Current active compose file
├── docker-compose.override.yml  # Your customizations
├── compose-20240315123456.yml.bak   # Versioned copy (timestamp)
├── compose-a1b2c3d.yml.bak          # Versioned copy (git hash)
└── backups/                     # Backup directory for rollbacks

Manual Rollback

To roll back to a specific version:

# Example: Roll back to a specific version
cp /opt/composesync/stacks/immich/compose-20240315123456.yml.bak /opt/composesync/stacks/immich/docker-compose.yml

# Restart the stack with the rolled back configuration
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml \
  -f /opt/composesync/stacks/immich/docker-compose.override.yml \
  up -d --remove-orphans

Version Cleanup

ComposeSync automatically maintains a configurable number of versioned files (default: 10) to prevent disk space issues. You can configure this per stack using the STACK_N_KEEP_VERSIONS environment variable.

Backup Management

ComposeSync creates comprehensive backups before applying any changes.

Backup Structure

Each update creates a timestamped backup directory:

/opt/composesync/stacks/myapp/backups/
├── backup-20240115103000/
│   ├── docker-compose.yml
│   ├── docker-compose.override.yml
│   ├── hwaccel.ml.yml
│   └── hwaccel.transcoding.yml
├── backup-20240115100000/
└── backup-20240115093000/

Backup Contents

Each backup includes:

• Main docker-compose.yml file
• docker-compose.override.yml (if it exists)
• All extra compose files
• Complete state before the update

Backup Cleanup

Backup directories are automatically cleaned up based on the KEEP_VERSIONS setting:

• Default: Keep 10 backup directories
• Configurable per stack with STACK_N_KEEP_VERSIONS
• Oldest backups are removed first

Error Handling

ComposeSync includes comprehensive error handling to ensure system stability.

Error Types Handled

1. Download Failures: Network issues, invalid URLs, authentication problems
2. File Validation: Empty files, corrupted downloads, missing files
3. Docker Compose Failures: Invalid configurations, service startup issues
4. Permission Issues: File access problems, directory creation failures
5. Lock File Issues: Stale locks, concurrent access attempts

Error Recovery

When errors occur, ComposeSync will:

• Log detailed error messages
• Attempt automatic recovery where possible
• Trigger rollback for critical failures
• Continue processing other stacks
• Send webhook notifications (if configured)

Error Logging

All errors are logged with timestamps and context:

[2024-01-15 10:30:00] ERROR: Failed to download https://example.com/compose.yml
[2024-01-15 10:30:01] ERROR: Downloaded file is empty
[2024-01-15 10:30:02] ERROR: Failed to update stack myapp, attempting rollback...
[2024-01-15 10:30:03] Successfully rolled back stack myapp

File Validation

ComposeSync validates downloaded files before processing them.

Validation Checks

• File Existence: Ensures files were downloaded successfully
• File Size: Verifies files are not empty
• File Format: Basic YAML validation
• Content Integrity: Checks for corrupted downloads

Validation Failures

If validation fails:

• The file is not used
• An error is logged
• Rollback is triggered if necessary
• The process continues with other files

Update Modes

ComposeSync supports different update modes for different safety levels.

Notify Only Mode

UPDATE_MODE=notify_only

In this mode:

• Files are downloaded and processed
• Changes are detected and logged
• No updates are applied to running stacks
• Perfect for testing and monitoring

Notify and Apply Mode

UPDATE_MODE=notify_and_apply

In this mode:

• Files are downloaded and processed
• Changes are automatically applied
• Rollback occurs on failures
• Full automation with safety features

Best Practices

1. Test with Dry-Run Mode

Always test new configurations with dry-run mode:

DRY_RUN=true

2. Use Appropriate Update Intervals

Set reasonable update intervals based on your needs:

• Production: 6-24 hours
• Development: 30 minutes - 2 hours
• Testing: Use dry-run mode

3. Monitor Logs

Regularly check the service logs:

sudo journalctl -u composesync -f

4. Configure Webhook Notifications

Set up webhook notifications to monitor updates:

NOTIFICATION_WEBHOOK_URL=https://your-webhook-url.com/endpoint

5. Regular Backup Verification

Periodically verify your backups are working:

ls -la /opt/composesync/stacks/*/backups/

6. Version Management

Keep an appropriate number of versions:

• More versions = more rollback options
• Fewer versions = less disk space usage
• Balance based on your needs and storage