feat: Add comprehensive testing suite for apt-ostree
- Create justfile-based testing framework with 20+ test targets - Support different deployment scenarios: container, booted system, disk-based, fresh install - Environment detection for OSTree, container, boot type, and system info - CLI functionality testing for all apt-ostree commands - Error handling and performance testing - Test report generation and cleanup utilities - Comprehensive documentation in TESTING.md - Mindful of system state: booted, container, installed, or disk image
This commit is contained in:
parent
35a22c366a
commit
7a631f95ef
2 changed files with 503 additions and 0 deletions
316
TESTING.md
Normal file
316
TESTING.md
Normal file
|
|
@ -0,0 +1,316 @@
|
|||
# apt-ostree Testing Suite
|
||||
|
||||
A comprehensive testing framework for `apt-ostree` that handles different deployment scenarios in Debian-based OSTree systems.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```bash
|
||||
# Install just (if not already installed)
|
||||
sudo apt install just
|
||||
|
||||
# Run comprehensive system test
|
||||
just test-system
|
||||
|
||||
# Check system environment
|
||||
just detect-env
|
||||
|
||||
# Test CLI functionality only
|
||||
just test-cli
|
||||
```
|
||||
|
||||
## 🔍 Environment Detection
|
||||
|
||||
The testing suite automatically detects your system environment:
|
||||
|
||||
- **🐳 Container Environment**: Podman, Docker, LXC
|
||||
- **🚀 Booted OSTree System**: Live deployment with write access
|
||||
- **💾 Disk-based System**: QCOW2, VMDK, or mounted images
|
||||
- **🆕 Fresh Installation**: New system without OSTree
|
||||
|
||||
## 📋 Available Test Targets
|
||||
|
||||
### Environment Detection
|
||||
- `detect-env` - Detect system environment and show status
|
||||
- `test-environments` - Test apt-ostree in different environments
|
||||
|
||||
### Specific Tests
|
||||
- `test-cli` - Test CLI functionality
|
||||
- `test-ostree` - Test OSTree integration
|
||||
- `test-container` - Test in container environment
|
||||
- `test-booted-system` - Test on booted OSTree system
|
||||
- `test-disk-system` - Test on disk-based OSTree system
|
||||
- `test-fresh-install` - Test on fresh installation
|
||||
|
||||
### Comprehensive Testing
|
||||
- `test-system` - Run comprehensive system test
|
||||
- `test-commands` - Test specific commands
|
||||
- `test-errors` - Test error handling
|
||||
- `test-performance` - Test performance
|
||||
|
||||
### Utilities
|
||||
- `test-report` - Generate test report
|
||||
- `clean` - Clean up test artifacts
|
||||
- `help` - Show help message
|
||||
|
||||
## 🐳 Container Testing
|
||||
|
||||
### Test in Existing Container
|
||||
```bash
|
||||
# If already running in container
|
||||
just test-container
|
||||
```
|
||||
|
||||
### Create Temporary Container
|
||||
```bash
|
||||
# Creates temporary container for testing
|
||||
just run-container-test
|
||||
```
|
||||
|
||||
This will:
|
||||
- Use Podman if available, fallback to Docker
|
||||
- Mount current directory as `/workspace`
|
||||
- Run basic CLI tests
|
||||
- Clean up automatically
|
||||
|
||||
## 🚀 Booted OSTree System Testing
|
||||
|
||||
For systems running live OSTree deployments:
|
||||
|
||||
```bash
|
||||
# Test on booted system
|
||||
just test-booted-system
|
||||
```
|
||||
|
||||
This tests:
|
||||
- OSTree deployment status
|
||||
- System-specific commands
|
||||
- Write access to OSTree directories
|
||||
|
||||
## 💾 Disk-based System Testing
|
||||
|
||||
For QCOW2, VMDK, or mounted OSTree images:
|
||||
|
||||
```bash
|
||||
# Test on disk-based system
|
||||
just test-disk-system
|
||||
```
|
||||
|
||||
This detects:
|
||||
- OSTree configuration files
|
||||
- Directory permissions
|
||||
- Read-only vs. writable status
|
||||
|
||||
## 🆕 Fresh Installation Testing
|
||||
|
||||
For new systems without OSTree:
|
||||
|
||||
```bash
|
||||
# Test on fresh installation
|
||||
just test-fresh-install
|
||||
```
|
||||
|
||||
This tests:
|
||||
- Basic CLI functionality
|
||||
- Help system
|
||||
- Version information
|
||||
- No OSTree dependencies
|
||||
|
||||
## 📊 Test Reports
|
||||
|
||||
Generate comprehensive test reports:
|
||||
|
||||
```bash
|
||||
# Generate test report
|
||||
just test-report
|
||||
|
||||
# View report
|
||||
cat test-report.txt
|
||||
```
|
||||
|
||||
Reports include:
|
||||
- System environment detection
|
||||
- CLI test results
|
||||
- OSTree integration status
|
||||
- Performance metrics
|
||||
|
||||
## 🔧 Customization
|
||||
|
||||
### Add Custom Tests
|
||||
Add new test targets to the `justfile`:
|
||||
|
||||
```makefile
|
||||
# Custom test example
|
||||
test-custom:
|
||||
@echo "🧪 Running custom test..."
|
||||
@apt-ostree custom-command
|
||||
@echo "✅ Custom test complete"
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
Set environment variables for testing:
|
||||
|
||||
```bash
|
||||
# Set test environment
|
||||
export APT_OSTREE_TEST_ENV=production
|
||||
export APT_OSTREE_LOG_LEVEL=debug
|
||||
|
||||
# Run tests
|
||||
just test-system
|
||||
```
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### OSTree Not Available
|
||||
```bash
|
||||
# Check if OSTree is installed
|
||||
which ostree
|
||||
|
||||
# Install OSTree if needed
|
||||
sudo apt install ostree
|
||||
```
|
||||
|
||||
#### Permission Denied
|
||||
```bash
|
||||
# Check directory permissions
|
||||
ls -la /ostree
|
||||
|
||||
# Run with appropriate privileges
|
||||
sudo just test-booted-system
|
||||
```
|
||||
|
||||
#### Container Issues
|
||||
```bash
|
||||
# Check container runtime
|
||||
which podman || which docker
|
||||
|
||||
# Verify container access
|
||||
ls -la /.dockerenv /run/.containerenv
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
Enable verbose output:
|
||||
|
||||
```bash
|
||||
# Run with debug output
|
||||
just --verbose test-system
|
||||
```
|
||||
|
||||
## 📈 Performance Testing
|
||||
|
||||
Test command performance:
|
||||
|
||||
```bash
|
||||
# Run performance tests
|
||||
just test-performance
|
||||
```
|
||||
|
||||
This measures:
|
||||
- Help command response time
|
||||
- Search command performance
|
||||
- Version command speed
|
||||
|
||||
## 🧹 Cleanup
|
||||
|
||||
Clean up test artifacts:
|
||||
|
||||
```bash
|
||||
# Remove test files
|
||||
just clean
|
||||
```
|
||||
|
||||
## 🔄 Continuous Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
```yaml
|
||||
name: Test apt-ostree
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-rust@v3
|
||||
- run: |
|
||||
sudo apt install just ostree
|
||||
just test-system
|
||||
just test-report
|
||||
- uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: test-report
|
||||
path: test-report.txt
|
||||
```
|
||||
|
||||
### Local CI
|
||||
```bash
|
||||
# Run tests before commit
|
||||
just test-system && just test-report
|
||||
|
||||
# Check for failures
|
||||
grep -i "❌\|FAIL" test-report.txt
|
||||
```
|
||||
|
||||
## 📚 Examples
|
||||
|
||||
### Development Workflow
|
||||
```bash
|
||||
# 1. Check environment
|
||||
just detect-env
|
||||
|
||||
# 2. Run basic tests
|
||||
just test-cli
|
||||
|
||||
# 3. Test in container
|
||||
just run-container-test
|
||||
|
||||
# 4. Full system test
|
||||
just test-system
|
||||
|
||||
# 5. Generate report
|
||||
just test-report
|
||||
```
|
||||
|
||||
### Production Testing
|
||||
```bash
|
||||
# Test on production system
|
||||
just test-booted-system
|
||||
|
||||
# Performance testing
|
||||
just test-performance
|
||||
|
||||
# Error handling
|
||||
just test-errors
|
||||
```
|
||||
|
||||
### CI/CD Pipeline
|
||||
```bash
|
||||
# Automated testing
|
||||
just test-system
|
||||
just test-performance
|
||||
just test-report
|
||||
|
||||
# Exit on failure
|
||||
if grep -q "❌\|FAIL" test-report.txt; then
|
||||
echo "Tests failed!"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
When adding new tests:
|
||||
|
||||
1. **Follow naming convention**: `test-<feature>`
|
||||
2. **Add to help section**: Update the `help` target
|
||||
3. **Include in system test**: Add to `test-system` target
|
||||
4. **Document usage**: Add examples to this README
|
||||
|
||||
## 📄 License
|
||||
|
||||
This testing suite is part of the `apt-ostree` project and follows the same license terms.
|
||||
|
||||
---
|
||||
|
||||
**Happy Testing! 🧪✨**
|
||||
187
justfile
Normal file
187
justfile
Normal file
|
|
@ -0,0 +1,187 @@
|
|||
# apt-ostree Testing Suite
|
||||
# Comprehensive testing for Debian-based OSTree systems
|
||||
|
||||
# Default target - show available commands
|
||||
default:
|
||||
@just --list
|
||||
|
||||
# Detect system environment and show status
|
||||
detect-env:
|
||||
@echo "🔍 Detecting system environment..."
|
||||
@echo "OSTree Status:"
|
||||
@ostree admin status 2>/dev/null || echo " ❌ OSTree not available"
|
||||
@echo ""
|
||||
@echo "Container Environment:"
|
||||
@bash -c 'if [ -f /.dockerenv ] || [ -f /run/.containerenv ]; then echo " 🐳 Running in container"; echo " Container type: $(cat /proc/1/cgroup | grep -o "docker\|podman\|lxc" | head -1 || echo "unknown")"; else echo " 💻 Running on bare metal/VM"; fi'
|
||||
@echo ""
|
||||
@echo "Boot Status:"
|
||||
@bash -c 'if [ -d /sys/firmware/efi ]; then echo " 🔌 UEFI boot detected"; else echo " 💾 Legacy BIOS boot detected"; fi'
|
||||
@echo ""
|
||||
@echo "System Information:"
|
||||
@echo " OS: $(cat /etc/os-release | grep PRETTY_NAME | cut -d'"' -f2 || echo 'unknown')"
|
||||
@echo " Kernel: $(uname -r)"
|
||||
@echo " Architecture: $(uname -m)"
|
||||
|
||||
# Test apt-ostree CLI functionality
|
||||
test-cli:
|
||||
@echo "🧪 Testing apt-ostree CLI functionality..."
|
||||
@echo "Testing main help..."
|
||||
@apt-ostree --help
|
||||
@echo ""
|
||||
@echo "Testing version..."
|
||||
@apt-ostree --version
|
||||
@echo ""
|
||||
@echo "Testing command help..."
|
||||
@apt-ostree status --help
|
||||
@echo ""
|
||||
@echo "Testing search functionality..."
|
||||
@apt-ostree search bash | head -5
|
||||
|
||||
# Test OSTree system status
|
||||
test-ostree:
|
||||
@echo "🌳 Testing OSTree system status..."
|
||||
@bash -c 'if command -v ostree >/dev/null 2>&1; then echo "OSTree version: $(ostree --version)"; echo ""; echo "Current deployment:"; ostree admin status 2>/dev/null || echo " ❌ No deployments found"; echo ""; echo "Repository status:"; ostree admin status --repo 2>/dev/null || echo " ❌ Repository not accessible"; else echo "❌ OSTree not installed"; fi'
|
||||
|
||||
# Test apt-ostree in different environments
|
||||
test-environments:
|
||||
@echo "🔧 Testing apt-ostree in different environments..."
|
||||
@echo ""
|
||||
@echo "1. Testing in current environment..."
|
||||
@just test-cli
|
||||
@echo ""
|
||||
@echo "2. Testing OSTree integration..."
|
||||
@just test-ostree
|
||||
@echo ""
|
||||
@echo "3. Testing package search..."
|
||||
@apt-ostree search apt | head -3
|
||||
|
||||
# Test container environment (Podman/Docker)
|
||||
test-container:
|
||||
@echo "🐳 Testing apt-ostree in container environment..."
|
||||
@bash -c 'if [ -f /.dockerenv ] || [ -f /run/.containerenv ]; then echo "✅ Running in container"; just test-cli; just test-ostree; else echo "❌ Not running in container"; echo "To test in container, run: just run-container-test"; fi'
|
||||
|
||||
# Run container test (creates temporary container)
|
||||
run-container-test:
|
||||
@echo "🐳 Creating temporary container for testing..."
|
||||
@bash -c 'if command -v podman >/dev/null 2>&1; then echo "Using Podman..."; podman run --rm -it --name apt-ostree-test -v $(pwd):/workspace debian:bookworm-slim bash -c "cd /workspace && just test-cli"; elif command -v docker >/dev/null 2>&1; then echo "Using Docker..."; docker run --rm -it --name apt-ostree-test -v $(pwd):/workspace debian:bookworm-slim bash -c "cd /workspace && just test-cli"; else echo "❌ Neither Podman nor Docker found"; fi'
|
||||
|
||||
# Test booted OSTree system
|
||||
test-booted-system:
|
||||
@echo "🚀 Testing apt-ostree on booted OSTree system..."
|
||||
@bash -c 'if [ -d /ostree ] && [ -d /sysroot ]; then echo "✅ OSTree system detected"; just test-cli; just test-ostree; echo ""; echo "Testing system-specific commands..."; apt-ostree status 2>/dev/null || echo " ❌ Status command failed"; apt-ostree search systemd | head -3; else echo "❌ Not running on booted OSTree system"; echo "This test requires a live OSTree deployment"; fi'
|
||||
|
||||
# Test disk-based OSTree system (QCOW2, etc.)
|
||||
test-disk-system:
|
||||
@echo "💾 Testing apt-ostree on disk-based OSTree system..."
|
||||
@bash -c 'if [ -f /etc/ostree/ostree.conf ] || [ -d /ostree ]; then echo "✅ OSTree configuration found"; just test-cli; echo ""; echo "Testing disk-specific operations..."; if [ -w /ostree ]; then echo "✅ OSTree directory is writable"; else echo "⚠️ OSTree directory is read-only (may be mounted image)"; fi; else echo "❌ OSTree configuration not found"; echo "This may be a fresh system or non-OSTree system"; fi'
|
||||
|
||||
# Test fresh installation
|
||||
test-fresh-install:
|
||||
@echo "🆕 Testing apt-ostree on fresh installation..."
|
||||
@bash -c 'if ! command -v ostree >/dev/null 2>&1; then echo "✅ Fresh system detected (no OSTree)"; just test-cli; echo ""; echo "Testing without OSTree dependencies..."; apt-ostree --help; apt-ostree --version; else echo "⚠️ OSTree already installed"; just test-cli; fi'
|
||||
|
||||
# Comprehensive system test
|
||||
test-system:
|
||||
@echo "🔍 Running comprehensive system test..."
|
||||
@just detect-env
|
||||
@echo ""
|
||||
@just test-environments
|
||||
@echo ""
|
||||
@echo "📊 Test Summary:"
|
||||
@echo " CLI Tests: $(just test-cli >/dev/null 2>&1 && echo '✅ PASS' || echo '❌ FAIL')"
|
||||
@echo " OSTree Tests: $(just test-ostree >/dev/null 2>&1 && echo '✅ PASS' || echo '❌ FAIL')"
|
||||
@echo " Environment Tests: $(just test-environments >/dev/null 2>&1 && echo '✅ PASS' || echo '❌ FAIL')"
|
||||
|
||||
# Test specific apt-ostree commands
|
||||
test-commands:
|
||||
@echo "📋 Testing specific apt-ostree commands..."
|
||||
@echo ""
|
||||
@echo "Testing help commands..."
|
||||
@apt-ostree --help >/dev/null && echo "✅ Main help" || echo "❌ Main help"
|
||||
@apt-ostree status --help >/dev/null && echo "✅ Status help" || echo "❌ Status help"
|
||||
@apt-ostree install --help >/dev/null && echo "✅ Install help" || echo "❌ Install help"
|
||||
@echo ""
|
||||
@echo "Testing search commands..."
|
||||
@apt-ostree search bash >/dev/null && echo "✅ Search bash" || echo "❌ Search bash"
|
||||
@apt-ostree search systemd >/dev/null && echo "✅ Search systemd" || echo "❌ Search systemd"
|
||||
@echo ""
|
||||
@echo "Testing status commands..."
|
||||
@apt-ostree status >/dev/null 2>&1 && echo "✅ Status command" || echo "❌ Status command"
|
||||
|
||||
# Test error handling
|
||||
test-errors:
|
||||
@echo "⚠️ Testing error handling..."
|
||||
@echo ""
|
||||
@echo "Testing missing arguments..."
|
||||
@apt-ostree install 2>&1 | grep -q "No package specified" && echo "✅ Install error handling" || echo "❌ Install error handling"
|
||||
@apt-ostree search 2>&1 | grep -q "No query specified" && echo "✅ Search error handling" || echo "❌ Search error handling"
|
||||
@echo ""
|
||||
@echo "Testing invalid commands..."
|
||||
@apt-ostree invalid-command 2>&1 | grep -q "Unknown command" && echo "✅ Invalid command handling" || echo "❌ Invalid command handling"
|
||||
|
||||
# Performance testing
|
||||
test-performance:
|
||||
@echo "⚡ Testing performance..."
|
||||
@echo ""
|
||||
@echo "Help command performance:"
|
||||
@bash -c 'start=$(date +%s); apt-ostree --help >/dev/null; end=$(date +%s); echo " Time: $((end - start))s"'
|
||||
@echo ""
|
||||
@echo "Search command performance:"
|
||||
@bash -c 'start=$(date +%s); apt-ostree search bash >/dev/null; end=$(date +%s); echo " Time: $((end - start))s"'
|
||||
@echo ""
|
||||
@echo "Version command performance:"
|
||||
@bash -c 'start=$(date +%s); apt-ostree --version >/dev/null; end=$(date +%s); echo " Time: $((end - start))s"'
|
||||
|
||||
# Generate test report
|
||||
test-report:
|
||||
@echo "📊 Generating test report..."
|
||||
@echo "Test Report - $(date)" > test-report.txt
|
||||
@echo "========================" >> test-report.txt
|
||||
@echo "" >> test-report.txt
|
||||
@just detect-env >> test-report.txt 2>&1
|
||||
@echo "" >> test-report.txt
|
||||
@echo "CLI Tests:" >> test-report.txt
|
||||
@just test-cli >> test-report.txt 2>&1
|
||||
@echo "" >> test-report.txt
|
||||
@echo "OSTree Tests:" >> test-report.txt
|
||||
@just test-ostree >> test-report.txt 2>&1
|
||||
@echo "Report saved to: test-report.txt"
|
||||
|
||||
# Clean up test artifacts
|
||||
clean:
|
||||
@echo "🧹 Cleaning up test artifacts..."
|
||||
@rm -f test-report.txt
|
||||
@echo "✅ Cleanup complete"
|
||||
|
||||
# Show help
|
||||
help:
|
||||
@echo "apt-ostree Testing Suite"
|
||||
@echo "========================"
|
||||
@echo ""
|
||||
@echo "Environment Detection:"
|
||||
@echo " detect-env - Detect system environment and status"
|
||||
@echo " test-environments - Test apt-ostree in different environments"
|
||||
@echo ""
|
||||
@echo "Specific Tests:"
|
||||
@echo " test-cli - Test CLI functionality"
|
||||
@echo " test-ostree - Test OSTree integration"
|
||||
@echo " test-container - Test in container environment"
|
||||
@echo " test-booted-system - Test on booted OSTree system"
|
||||
@echo " test-disk-system - Test on disk-based OSTree system"
|
||||
@echo " test-fresh-install - Test on fresh installation"
|
||||
@echo ""
|
||||
@echo "Comprehensive Testing:"
|
||||
@echo " test-system - Run comprehensive system test"
|
||||
@echo " test-commands - Test specific commands"
|
||||
@echo " test-errors - Test error handling"
|
||||
@echo " test-performance - Test performance"
|
||||
@echo ""
|
||||
@echo "Utilities:"
|
||||
@echo " test-report - Generate test report"
|
||||
@echo " clean - Clean up test artifacts"
|
||||
@echo " help - Show this help message"
|
||||
@echo ""
|
||||
@echo "Examples:"
|
||||
@echo " just test-system - Run full system test"
|
||||
@echo " just test-cli - Test CLI only"
|
||||
@echo " just detect-env - Check system status"
|
||||
Loading…
Add table
Add a link
Reference in a new issue