ComposeSync/Docs/safety-features.md

# Safety Features

This guide covers the safety features built into ComposeSync to protect your Docker Compose stacks.

## Automatic Rollback

ComposeSync includes automatic rollback functionality to handle failed updates:

### How Rollback Works

1. **Pre-Update Backup**: Before applying changes, all current files are backed up
2. **Update Attempt**: Changes are applied using `docker compose up -d`
3. **Failure Detection**: If the Docker Compose command fails, rollback is triggered
4. **Automatic Restoration**: All files are restored from the backup
5. **Stack Restart**: The stack is restarted with the original configuration

### Rollback Process

When a rollback occurs, ComposeSync will:

```bash
# 1. Restore main compose file
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/docker-compose.yml /opt/composesync/stacks/myapp/docker-compose.yml

# 2. Restore extra files (if any)
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/hwaccel.ml.yml /opt/composesync/stacks/myapp/hwaccel.ml.yml

# 3. Restore override file (if it existed)
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/docker-compose.override.yml /opt/composesync/stacks/myapp/docker-compose.override.yml

# 4. Restart stack with original configuration
docker compose -f /opt/composesync/stacks/myapp/docker-compose.yml up -d
```

### Rollback Benefits

This ensures that:
- Your stack never gets stuck in a broken state
- Failed updates don't leave your services down
- You can quickly recover from problematic updates
- The system remains stable even with network issues or invalid configurations

## Lock File Protection

ComposeSync uses lock files to prevent concurrent updates to the same stack.

### How Lock Files Work

Each stack has a `.lock` file in its directory that:
- Is created when an update starts
- Is automatically removed when the update completes (success or failure)
- Has a 5-minute timeout to handle stale locks from crashed processes
- Uses directory-based locking for atomicity

### Lock File Location

```
/opt/composesync/stacks/myapp/
├── docker-compose.yml
├── docker-compose.override.yml
├── .lock/                    # Lock directory
└── backups/
```

### Stale Lock Detection

If a lock file exists and is older than 5 minutes, ComposeSync will automatically remove it and proceed with the update. This handles cases where:
- The previous update process crashed
- The system was rebooted during an update
- Network issues interrupted the update process

### Lock File Benefits

Lock files prevent:
- Race conditions when multiple instances run simultaneously
- Corruption of compose files during updates
- Multiple update processes running on the same stack

## Versioned History

ComposeSync maintains a versioned history of your compose files for easy rollback and audit trails.

### Version Identifiers

- **For Git sources**: Uses the Git commit hash as the version identifier
- **For other sources**: Uses a timestamp (YYYYMMDDHHMMSS format)

### Versioned File Structure

```
stack/
├── docker-compose.yml           # Current active compose file
├── docker-compose.override.yml  # Your customizations
├── compose-20240315123456.yml.bak   # Versioned copy (timestamp)
├── compose-a1b2c3d.yml.bak          # Versioned copy (git hash)
└── backups/                     # Backup directory for rollbacks
```

### Manual Rollback

To roll back to a specific version:

```bash
# Example: Roll back to a specific version
cp /opt/composesync/stacks/immich/compose-20240315123456.yml.bak /opt/composesync/stacks/immich/docker-compose.yml

# Restart the stack with the rolled back configuration
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml \
  -f /opt/composesync/stacks/immich/docker-compose.override.yml \
  up -d --remove-orphans
```

### Version Cleanup

ComposeSync automatically maintains a configurable number of versioned files (default: 10) to prevent disk space issues. You can configure this per stack using the `STACK_N_KEEP_VERSIONS` environment variable.

## Backup Management

ComposeSync creates comprehensive backups before applying any changes.

### Backup Structure

Each update creates a timestamped backup directory:

```
/opt/composesync/stacks/myapp/backups/
├── backup-20240115103000/
│   ├── docker-compose.yml
│   ├── docker-compose.override.yml
│   ├── hwaccel.ml.yml
│   └── hwaccel.transcoding.yml
├── backup-20240115100000/
└── backup-20240115093000/
```

### Backup Contents

Each backup includes:
- Main `docker-compose.yml` file
- `docker-compose.override.yml` (if it exists)
- All extra compose files
- Complete state before the update

### Backup Cleanup

Backup directories are automatically cleaned up based on the `KEEP_VERSIONS` setting:
- Default: Keep 10 backup directories
- Configurable per stack with `STACK_N_KEEP_VERSIONS`
- Oldest backups are removed first

## Error Handling

ComposeSync includes comprehensive error handling to ensure system stability.

### Error Types Handled

1. **Download Failures**: Network issues, invalid URLs, authentication problems
2. **File Validation**: Empty files, corrupted downloads, missing files
3. **Docker Compose Failures**: Invalid configurations, service startup issues
4. **Permission Issues**: File access problems, directory creation failures
5. **Lock File Issues**: Stale locks, concurrent access attempts

### Error Recovery

When errors occur, ComposeSync will:
- Log detailed error messages
- Attempt automatic recovery where possible
- Trigger rollback for critical failures
- Continue processing other stacks
- Send webhook notifications (if configured)

### Error Logging

All errors are logged with timestamps and context:

```
[2024-01-15 10:30:00] ERROR: Failed to download https://example.com/compose.yml
[2024-01-15 10:30:01] ERROR: Downloaded file is empty
[2024-01-15 10:30:02] ERROR: Failed to update stack myapp, attempting rollback...
[2024-01-15 10:30:03] Successfully rolled back stack myapp
```

## File Validation

ComposeSync validates downloaded files before processing them.

### Validation Checks

- **File Existence**: Ensures files were downloaded successfully
- **File Size**: Verifies files are not empty
- **File Format**: Basic YAML validation
- **Content Integrity**: Checks for corrupted downloads

### Validation Failures

If validation fails:
- The file is not used
- An error is logged
- Rollback is triggered if necessary
- The process continues with other files

## Update Modes

ComposeSync supports different update modes for different safety levels.

### Notify Only Mode

```env
UPDATE_MODE=notify_only
```

In this mode:
- Files are downloaded and processed
- Changes are detected and logged
- No updates are applied to running stacks
- Perfect for testing and monitoring

### Notify and Apply Mode

```env
UPDATE_MODE=notify_and_apply
```

In this mode:
- Files are downloaded and processed
- Changes are automatically applied
- Rollback occurs on failures
- Full automation with safety features

## Best Practices

### 1. Test with Dry-Run Mode

Always test new configurations with dry-run mode:
```env
DRY_RUN=true
```

### 2. Use Appropriate Update Intervals

Set reasonable update intervals based on your needs:
- Production: 6-24 hours
- Development: 30 minutes - 2 hours
- Testing: Use dry-run mode

### 3. Monitor Logs

Regularly check the service logs:
```bash
sudo journalctl -u composesync -f
```

### 4. Configure Webhook Notifications

Set up webhook notifications to monitor updates:
```env
NOTIFICATION_WEBHOOK_URL=https://your-webhook-url.com/endpoint
```

### 5. Regular Backup Verification

Periodically verify your backups are working:
```bash
ls -la /opt/composesync/stacks/*/backups/
```

### 6. Version Management

Keep an appropriate number of versions:
- More versions = more rollback options
- Fewer versions = less disk space usage
- Balance based on your needs and storage