Initial commit: ComposeSync - Automated Docker Compose update agent
This commit is contained in:
commit
f0dba7cc0a
16 changed files with 3019 additions and 0 deletions
272
Docs/safety-features.md
Normal file
272
Docs/safety-features.md
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
# Safety Features
|
||||
|
||||
This guide covers the safety features built into ComposeSync to protect your Docker Compose stacks.
|
||||
|
||||
## Automatic Rollback
|
||||
|
||||
ComposeSync includes automatic rollback functionality to handle failed updates:
|
||||
|
||||
### How Rollback Works
|
||||
|
||||
1. **Pre-Update Backup**: Before applying changes, all current files are backed up
|
||||
2. **Update Attempt**: Changes are applied using `docker compose up -d`
|
||||
3. **Failure Detection**: If the Docker Compose command fails, rollback is triggered
|
||||
4. **Automatic Restoration**: All files are restored from the backup
|
||||
5. **Stack Restart**: The stack is restarted with the original configuration
|
||||
|
||||
### Rollback Process
|
||||
|
||||
When a rollback occurs, ComposeSync will:
|
||||
|
||||
```bash
|
||||
# 1. Restore main compose file
|
||||
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/docker-compose.yml /opt/composesync/stacks/myapp/docker-compose.yml
|
||||
|
||||
# 2. Restore extra files (if any)
|
||||
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/hwaccel.ml.yml /opt/composesync/stacks/myapp/hwaccel.ml.yml
|
||||
|
||||
# 3. Restore override file (if it existed)
|
||||
cp /opt/composesync/stacks/myapp/backups/backup-20240115103000/docker-compose.override.yml /opt/composesync/stacks/myapp/docker-compose.override.yml
|
||||
|
||||
# 4. Restart stack with original configuration
|
||||
docker compose -f /opt/composesync/stacks/myapp/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
### Rollback Benefits
|
||||
|
||||
This ensures that:
|
||||
- Your stack never gets stuck in a broken state
|
||||
- Failed updates don't leave your services down
|
||||
- You can quickly recover from problematic updates
|
||||
- The system remains stable even with network issues or invalid configurations
|
||||
|
||||
## Lock File Protection
|
||||
|
||||
ComposeSync uses lock files to prevent concurrent updates to the same stack.
|
||||
|
||||
### How Lock Files Work
|
||||
|
||||
Each stack has a `.lock` file in its directory that:
|
||||
- Is created when an update starts
|
||||
- Is automatically removed when the update completes (success or failure)
|
||||
- Has a 5-minute timeout to handle stale locks from crashed processes
|
||||
- Uses directory-based locking for atomicity
|
||||
|
||||
### Lock File Location
|
||||
|
||||
```
|
||||
/opt/composesync/stacks/myapp/
|
||||
├── docker-compose.yml
|
||||
├── docker-compose.override.yml
|
||||
├── .lock/ # Lock directory
|
||||
└── backups/
|
||||
```
|
||||
|
||||
### Stale Lock Detection
|
||||
|
||||
If a lock file exists and is older than 5 minutes, ComposeSync will automatically remove it and proceed with the update. This handles cases where:
|
||||
- The previous update process crashed
|
||||
- The system was rebooted during an update
|
||||
- Network issues interrupted the update process
|
||||
|
||||
### Lock File Benefits
|
||||
|
||||
Lock files prevent:
|
||||
- Race conditions when multiple instances run simultaneously
|
||||
- Corruption of compose files during updates
|
||||
- Multiple update processes running on the same stack
|
||||
|
||||
## Versioned History
|
||||
|
||||
ComposeSync maintains a versioned history of your compose files for easy rollback and audit trails.
|
||||
|
||||
### Version Identifiers
|
||||
|
||||
- **For Git sources**: Uses the Git commit hash as the version identifier
|
||||
- **For other sources**: Uses a timestamp (YYYYMMDDHHMMSS format)
|
||||
|
||||
### Versioned File Structure
|
||||
|
||||
```
|
||||
stack/
|
||||
├── docker-compose.yml # Current active compose file
|
||||
├── docker-compose.override.yml # Your customizations
|
||||
├── compose-20240315123456.yml.bak # Versioned copy (timestamp)
|
||||
├── compose-a1b2c3d.yml.bak # Versioned copy (git hash)
|
||||
└── backups/ # Backup directory for rollbacks
|
||||
```
|
||||
|
||||
### Manual Rollback
|
||||
|
||||
To roll back to a specific version:
|
||||
|
||||
```bash
|
||||
# Example: Roll back to a specific version
|
||||
cp /opt/composesync/stacks/immich/compose-20240315123456.yml.bak /opt/composesync/stacks/immich/docker-compose.yml
|
||||
|
||||
# Restart the stack with the rolled back configuration
|
||||
docker compose -f /opt/composesync/stacks/immich/docker-compose.yml \
|
||||
-f /opt/composesync/stacks/immich/docker-compose.override.yml \
|
||||
up -d --remove-orphans
|
||||
```
|
||||
|
||||
### Version Cleanup
|
||||
|
||||
ComposeSync automatically maintains a configurable number of versioned files (default: 10) to prevent disk space issues. You can configure this per stack using the `STACK_N_KEEP_VERSIONS` environment variable.
|
||||
|
||||
## Backup Management
|
||||
|
||||
ComposeSync creates comprehensive backups before applying any changes.
|
||||
|
||||
### Backup Structure
|
||||
|
||||
Each update creates a timestamped backup directory:
|
||||
|
||||
```
|
||||
/opt/composesync/stacks/myapp/backups/
|
||||
├── backup-20240115103000/
|
||||
│ ├── docker-compose.yml
|
||||
│ ├── docker-compose.override.yml
|
||||
│ ├── hwaccel.ml.yml
|
||||
│ └── hwaccel.transcoding.yml
|
||||
├── backup-20240115100000/
|
||||
└── backup-20240115093000/
|
||||
```
|
||||
|
||||
### Backup Contents
|
||||
|
||||
Each backup includes:
|
||||
- Main `docker-compose.yml` file
|
||||
- `docker-compose.override.yml` (if it exists)
|
||||
- All extra compose files
|
||||
- Complete state before the update
|
||||
|
||||
### Backup Cleanup
|
||||
|
||||
Backup directories are automatically cleaned up based on the `KEEP_VERSIONS` setting:
|
||||
- Default: Keep 10 backup directories
|
||||
- Configurable per stack with `STACK_N_KEEP_VERSIONS`
|
||||
- Oldest backups are removed first
|
||||
|
||||
## Error Handling
|
||||
|
||||
ComposeSync includes comprehensive error handling to ensure system stability.
|
||||
|
||||
### Error Types Handled
|
||||
|
||||
1. **Download Failures**: Network issues, invalid URLs, authentication problems
|
||||
2. **File Validation**: Empty files, corrupted downloads, missing files
|
||||
3. **Docker Compose Failures**: Invalid configurations, service startup issues
|
||||
4. **Permission Issues**: File access problems, directory creation failures
|
||||
5. **Lock File Issues**: Stale locks, concurrent access attempts
|
||||
|
||||
### Error Recovery
|
||||
|
||||
When errors occur, ComposeSync will:
|
||||
- Log detailed error messages
|
||||
- Attempt automatic recovery where possible
|
||||
- Trigger rollback for critical failures
|
||||
- Continue processing other stacks
|
||||
- Send webhook notifications (if configured)
|
||||
|
||||
### Error Logging
|
||||
|
||||
All errors are logged with timestamps and context:
|
||||
|
||||
```
|
||||
[2024-01-15 10:30:00] ERROR: Failed to download https://example.com/compose.yml
|
||||
[2024-01-15 10:30:01] ERROR: Downloaded file is empty
|
||||
[2024-01-15 10:30:02] ERROR: Failed to update stack myapp, attempting rollback...
|
||||
[2024-01-15 10:30:03] Successfully rolled back stack myapp
|
||||
```
|
||||
|
||||
## File Validation
|
||||
|
||||
ComposeSync validates downloaded files before processing them.
|
||||
|
||||
### Validation Checks
|
||||
|
||||
- **File Existence**: Ensures files were downloaded successfully
|
||||
- **File Size**: Verifies files are not empty
|
||||
- **File Format**: Basic YAML validation
|
||||
- **Content Integrity**: Checks for corrupted downloads
|
||||
|
||||
### Validation Failures
|
||||
|
||||
If validation fails:
|
||||
- The file is not used
|
||||
- An error is logged
|
||||
- Rollback is triggered if necessary
|
||||
- The process continues with other files
|
||||
|
||||
## Update Modes
|
||||
|
||||
ComposeSync supports different update modes for different safety levels.
|
||||
|
||||
### Notify Only Mode
|
||||
|
||||
```env
|
||||
UPDATE_MODE=notify_only
|
||||
```
|
||||
|
||||
In this mode:
|
||||
- Files are downloaded and processed
|
||||
- Changes are detected and logged
|
||||
- No updates are applied to running stacks
|
||||
- Perfect for testing and monitoring
|
||||
|
||||
### Notify and Apply Mode
|
||||
|
||||
```env
|
||||
UPDATE_MODE=notify_and_apply
|
||||
```
|
||||
|
||||
In this mode:
|
||||
- Files are downloaded and processed
|
||||
- Changes are automatically applied
|
||||
- Rollback occurs on failures
|
||||
- Full automation with safety features
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Test with Dry-Run Mode
|
||||
|
||||
Always test new configurations with dry-run mode:
|
||||
```env
|
||||
DRY_RUN=true
|
||||
```
|
||||
|
||||
### 2. Use Appropriate Update Intervals
|
||||
|
||||
Set reasonable update intervals based on your needs:
|
||||
- Production: 6-24 hours
|
||||
- Development: 30 minutes - 2 hours
|
||||
- Testing: Use dry-run mode
|
||||
|
||||
### 3. Monitor Logs
|
||||
|
||||
Regularly check the service logs:
|
||||
```bash
|
||||
sudo journalctl -u composesync -f
|
||||
```
|
||||
|
||||
### 4. Configure Webhook Notifications
|
||||
|
||||
Set up webhook notifications to monitor updates:
|
||||
```env
|
||||
NOTIFICATION_WEBHOOK_URL=https://your-webhook-url.com/endpoint
|
||||
```
|
||||
|
||||
### 5. Regular Backup Verification
|
||||
|
||||
Periodically verify your backups are working:
|
||||
```bash
|
||||
ls -la /opt/composesync/stacks/*/backups/
|
||||
```
|
||||
|
||||
### 6. Version Management
|
||||
|
||||
Keep an appropriate number of versions:
|
||||
- More versions = more rollback options
|
||||
- Fewer versions = less disk space usage
|
||||
- Balance based on your needs and storage
|
||||
Loading…
Add table
Add a link
Reference in a new issue