From 4ec9eb38f60cd9e704a9a76827c7970f9c78a417 Mon Sep 17 00:00:00 2001
From: robojerk <rob@raines.xyz>
Date: Fri, 22 Aug 2025 21:21:50 -0700
Subject: [PATCH] Complete documentation updates: architecture, user guide,
 deployment, troubleshooting

---
 debian-forge-docs/TODO.md                     | 494 ++++++++++
 debian-forge-docs/changelog.md                |  68 ++
 debian-forge-docs/debian-forge-roadmap.md     | 203 ++++
 debian-forge-docs/debos-osbuild-mapping.md    | 117 +++
 debian-forge-docs/deployment-documentation.md | 811 ++++++++++++++++
 debian-forge-docs/flowchart-debian.md         | 276 ++++++
 debian-forge-docs/flowchart.md                | 165 ++++
 debian-forge-docs/osbuild-architecture.md     | 136 +++
 debian-forge-docs/troubleshooting-guides.md   | 873 ++++++++++++++++++
 debian-forge-docs/user-documentation.md       | 524 +++++++++++
 10 files changed, 3667 insertions(+)
 create mode 100755 debian-forge-docs/TODO.md
 create mode 100755 debian-forge-docs/changelog.md
 create mode 100755 debian-forge-docs/debian-forge-roadmap.md
 create mode 100755 debian-forge-docs/debos-osbuild-mapping.md
 create mode 100644 debian-forge-docs/deployment-documentation.md
 create mode 100755 debian-forge-docs/flowchart-debian.md
 create mode 100755 debian-forge-docs/flowchart.md
 create mode 100644 debian-forge-docs/osbuild-architecture.md
 create mode 100644 debian-forge-docs/troubleshooting-guides.md
 create mode 100644 debian-forge-docs/user-documentation.md

diff --git a/debian-forge-docs/TODO.md b/debian-forge-docs/TODO.md
new file mode 100755
index 00000000..a28f66d3
--- /dev/null
+++ b/debian-forge-docs/TODO.md
@@ -0,0 +1,494 @@
+# Debian Forge TODO: Building Debian Atomic System
+
+## Project Goal
+Build a complete Debian atomic build system that combines OSBuild architecture, debos logic, and Koji-like orchestration to create Debian atomic images with bootc containers.
+
+## Phase 1: Foundation and Analysis (Weeks 1-6)
+
+### Week 1-2: Project Setup
+- [x] **Set up development environment**
+  - [x] Configure Debian-based development machine
+  - [x] Install required dependencies (Go, Python, OSTree, etc.)
+  - [x] Set up version control and branching strategy
+  - [x] Create development documentation
+
+- [x] **Fork OSBuild repository**
+  - [x] Verify all OSBuild components are present
+  - [ ] Identify Fedora/RPM-specific components to replace
+  - [ ] Document current OSBuild architecture
+
+- [x] **Download and analyze debos source code**
+  - [ ] Map debos actions to OSBuild stages
+  - [ ] Identify Debian package management logic to integrate
+  - [ ] Document debos OSTree integration points
+
+- [x] **Set up basic project structure**
+  - [x] Create integration layer between OSBuild and debos
+  - [x] Set up build environment for testing
+  - [x] Create initial project documentation
+
+### Week 3-4: Code Analysis
+- [x] **Analyze OSBuild architecture**
+  - [x] Map OSBuild pipeline components (stages, devices, mounts, inputs)
+  - [x] Identify RPM-specific components to replace
+  - [x] Document OSBuild's build environment management
+  - [x] Analyze OSBuild's object store and caching
+
+- [x] **Map debos functionality to OSBuild components**
+  - [x] Create mapping table: debos actions → OSBuild stages
+  - [x] Identify debos package management logic
+  - [x] Document debos filesystem operations
+  - [x] Analyze debos OSTree integration
+
+- [ ] **Identify integration points**
+  - [ ] Define interfaces between OSBuild and debos
+  - [ ] Plan Koji-like orchestration integration
+  - [ ] Document data flow between components
+  - [ ] Identify shared data structures
+
+- [x] **Document current architecture**
+  - [x] Create architecture diagrams
+  - [x] Document component relationships
+  - [x] Identify modification points
+  - [x] Create integration specifications
+
+### Week 5-6: Planning and Design
+- [x] **Design Debian-specific component architecture**
+  - [x] Design Debian package management layer
+  - [x] Plan OSTree composition for Debian packages
+  - [ ] Design build environment management
+  - [x] Create component interface specifications
+
+- [ ] **Plan Koji-like orchestration system**
+  - [ ] Design build scheduling system
+  - [ ] Plan build queue management
+  - [ ] Design resource allocation
+  - [ ] Plan build monitoring and logging
+
+- [ ] **Define interfaces between components**
+  - [ ] Define API contracts
+  - [ ] Design data exchange formats
+  - [ ] Plan error handling and recovery
+  - [ ] Define configuration management
+
+- [ ] **Create detailed technical specifications**
+  - [ ] Write component specifications
+  - [ ] Define data flow specifications
+  - [ ] Create API documentation
+  - [ ] Write integration test specifications
+
+**Deliverables**: Project structure, architecture documentation, technical specifications
+
+## Phase 2: Core OSBuild Modification (Weeks 7-18)
+
+### Week 7-10: Package Management Layer
+- [x] **Replace RPM package management with Debian package management**
+  - [x] Create Debian package input source
+  - [x] Implement Debian repository handling
+  - [x] Replace RPM-specific stages with Debian equivalents
+  - [x] Test basic package operations
+  - [ ] Add mmdebstrap as default alternative to debootstrap
+
+- [x] **Integrate debos package handling logic**
+  - [x] Extract debos package management code
+  - [x] Create OSBuild stages for debos actions
+  - [x] Integrate debos package installation logic
+  - [ ] Test package dependency resolution
+
+- [x] **Implement Debian repository handling**
+  - [x] Create Debian repository source
+  - [x] Implement package metadata parsing
+  - [ ] Add repository authentication support
+  - [ ] Test repository operations
+
+- [x] **Test basic package operations**
+  - [x] Test package download
+  - [x] Test package installation
+  - [x] Test dependency resolution
+  - [x] Test package removal
+
+### Week 11-14: Build Environment Integration
+- [x] **Replace Mock integration with sbuild/pbuilder integration**
+  - [x] Create sbuild build environment stage
+  - [ ] Implement pbuilder integration
+  - [x] Replace Mock-specific components
+  - [ ] Test build environment creation
+
+- [x] **Implement Debian build environment setup**
+  - [x] Create Debian chroot setup
+  - [x] Implement build dependency installation
+  - [x] Add build environment customization
+  - [x] Test environment isolation
+
+- [x] **Test build environment creation and management**
+  - [x] Test chroot creation
+  - [x] Test package building in environments
+  - [x] Test environment cleanup
+  - [x] Test environment reuse
+
+- [ ] **Validate package building in Debian environments**
+  - [ ] Test simple package builds
+  - [ ] Test complex dependency chains
+  - [ ] Test build failures and recovery
+  - [ ] Test build environment persistence
+
+### Week 15-18: OSTree Composition
+- [x] **Adapt OSTree composition for Debian packages**
+  - [x] Create Debian OSTree composition stage
+  - [x] Implement package-to-OSTree conversion
+  - [x] Add Debian-specific metadata handling
+  - [x] Test basic OSTree composition
+
+- [x] **Test OSTree commit creation from Debian packages**
+  - [x] Test single package commits
+  - [x] Test multi-package commits
+  - [x] Test commit metadata
+  - [x] Test commit verification
+
+- [ ] **Validate filesystem composition**
+  - [ ] Test filesystem structure
+  - [ ] Test file permissions and ownership
+  - [ ] Test symbolic links and hard links
+  - [ ] Test filesystem integrity
+
+- [ ] **Test basic atomic functionality**
+  - [ ] Test atomic updates
+  - [ ] Test rollback capability
+  - [ ] Test atomic rebasing
+  - [ ] Test atomic deployment
+
+**Deliverables**: Modified OSBuild with Debian package support, working OSTree composition
+
+## Phase 3: Build Orchestration (Weeks 19-30)
+
+### Week 19-22: Basic Orchestration
+- [x] **Implement build scheduling system**
+  - [x] Create build queue management
+  - [x] Implement priority-based scheduling
+  - [ ] Add resource allocation logic
+  - [ ] Test scheduling algorithms
+
+- [x] **Add build queue management**
+  - [x] Implement build submission
+  - [x] Add build status tracking
+  - [x] Implement queue prioritization
+  - [x] Test queue operations
+
+- [x] **Implement basic resource allocation**
+  - [x] Add CPU allocation
+  - [x] Add memory allocation
+  - [x] Add storage allocation
+  - [x] Test resource management
+
+- [x] **Test build submission and tracking**
+  - [x] Test build submission
+  - [x] Test build status updates
+  - [x] Test build cancellation
+  - [x] Test build prioritization
+
+### Week 23-26: OSBuild Integration
+- [x] **Integrate modified OSBuild with orchestration system**
+- [x] Connect OSBuild pipeline to orchestration
+- [x] Implement build submission interface
+- [x] Add build result handling
+- [ ] **Test osbuild-composer integration**
+- [ ] Test composer APIs with Debian stages
+- [ ] Validate blueprint system for Debian atomic images
+- [ ] Test composer orchestration with our build system
+
+### Week 23-26: Build Management
+- [x] **Add build status tracking**
+  - [x] Implement build state machine
+  - [x] Add status persistence
+  - [x] Implement status notifications
+  - [x] Test status tracking
+
+- [x] **Implement build logging and monitoring**
+  - [x] Add build log collection
+  - [x] Implement log streaming
+  - [x] Add build metrics collection
+  - [x] Test logging system
+
+- [x] **Add build history and artifact management**
+  - [x] Implement build history storage
+  - [x] Add artifact storage and retrieval
+  - [x] Implement build cleanup policies
+  - [x] Test artifact management
+
+- [x] **Test complete build lifecycle**
+  - [x] Test build submission to completion
+  - [x] Test build failure handling
+  - [x] Test build retry mechanisms
+  - [x] Test build cleanup
+
+### Week 27-30: Environment Management
+- [x] **Implement isolated build environment management**
+  - [x] Create environment isolation
+  - [x] Implement environment cleanup
+  - [x] Add environment reuse policies
+  - [x] Test environment isolation
+
+- [x] **Add build host monitoring and health checks**
+  - [x] Implement host health monitoring
+  - [x] Add resource usage tracking
+  - [x] Implement health check failures
+  - [x] Test monitoring system
+
+- [x] **Implement build environment cleanup**
+  - [x] Add automatic cleanup policies
+  - [x] Implement cleanup scheduling
+  - [x] Add cleanup verification
+  - [x] Test cleanup operations
+
+- [x] **Test environment isolation and management**
+  - [x] Test environment isolation
+  - [x] Test resource cleanup
+  - [x] Test environment reuse
+  - [x] Test failure recovery
+
+**Deliverables**: Working build orchestration system, build environment management
+
+## Phase 4: Integration and Testing (Weeks 31-42)
+
+### Week 31-34: System Integration
+- [x] **Integrate modified OSBuild with orchestration system**
+  - [x] Connect OSBuild pipeline to orchestration
+  - [x] Implement build submission interface
+  - [x] Add build result handling
+  - [x] Test complete integration
+
+- [x] **Test complete build pipeline from submission to completion**
+  - [x] Test end-to-end build process
+  - [x] Test build pipeline stages
+  - [x] Test build result generation
+  - [x] Test build artifact storage
+
+- [x] **Validate multi-stage build workflows**
+  - [x] Test complex build workflows
+  - [x] Test workflow dependencies
+  - [x] Test workflow failures
+  - [x] Test workflow recovery
+
+- [x] **Test error handling and recovery**
+  - [x] Test build failures
+  - [x] Test system failures
+  - [x] Test recovery mechanisms
+  - [x] Test error reporting
+
+### Week 35-38: Output Generation
+- [x] **Test image generation (ISO, QCOW2, RAW)**
+  - [x] Test ISO generation
+  - [x] Test QCOW2 generation
+  - [x] Test RAW image generation
+  - [x] Validate image formats
+
+- [x] **Validate bootc container creation**
+  - [x] Test container creation
+  - [x] Test container metadata
+  - [x] Test container verification
+  - [x] Test container deployment
+
+- [x] **Test multi-format output generation**
+  - [x] Test simultaneous format generation
+  - [x] Test format-specific optimizations
+  - [x] Test format validation
+  - [x] Test format distribution
+
+- [x] **Validate image bootability**
+  - [x] Test image boot process
+  - [x] Test kernel loading
+  - [x] Test filesystem mounting
+  - [x] Test system initialization
+
+### Week 39-42: Performance and Stability
+- [x] **Performance testing and optimization**
+  - [x] Test build performance
+  - [x] Identify bottlenecks
+  - [x] Implement optimizations
+  - [x] Validate performance improvements
+
+- [x] **Stress testing with multiple concurrent builds**
+  - [x] Test concurrent build limits
+  - [x] Test resource contention
+  - [x] Test system stability under load
+  - [x] Test failure scenarios
+
+- [x] **Bug fixing and stability improvements**
+  - [x] Fix identified bugs
+  - [x] Improve error handling
+  - [x] Add stability improvements
+  - [x] Test stability fixes
+
+- [x] **Documentation updates**
+  - [x] Update technical documentation
+  - [x] Create user documentation
+  - [x] Create deployment documentation
+  - [x] Create troubleshooting guides
+
+**Deliverables**: Integrated system, working image generation, performance optimization
+
+## Phase 5: Web Interface and Production Features (Weeks 43-54)
+
+### Week 43-46: Basic Web UI
+- [ ] **Implement basic web interface for build management**
+  - [ ] Create build submission forms
+  - [ ] Add build status display
+  - [ ] Implement build history view
+  - [ ] Test web interface functionality
+
+- [ ] **Add build status monitoring**
+  - [ ] Implement real-time status updates
+  - [ ] Add build progress indicators
+  - [ ] Implement status notifications
+  - [ ] Test monitoring features
+
+- [ ] **Implement build submission forms**
+  - [ ] Create build configuration forms
+  - [ ] Add validation and error handling
+  - [ ] Implement form submission
+  - [ ] Test form functionality
+
+- [ ] **Test web interface functionality**
+  - [ ] Test form submission
+  - [ ] Test status updates
+  - [ ] Test error handling
+  - [ ] Test user experience
+
+### Week 47-50: Advanced Web Features
+- [ ] **Add build history and search**
+  - [ ] Implement build history storage
+  - [ ] Add search and filtering
+  - [ ] Implement pagination
+  - [ ] Test search functionality
+
+- [ ] **Implement user management and permissions**
+  - [ ] Add user authentication
+  - [ ] Implement role-based access control
+  - [ ] Add user management interface
+  - [ ] Test permission system
+
+- [ ] **Add system administration interface**
+  - [ ] Create system configuration interface
+  - [ ] Add monitoring and health checks
+  - [ ] Implement system maintenance tools
+  - [ ] Test admin interface
+
+- [ ] **Test complete web interface**
+  - [ ] Test all web features
+  - [ ] Test user workflows
+  - [ ] Test admin workflows
+  - [ ] Test security features
+
+### Week 51-54: Production Readiness
+- [ ] **Security testing and hardening**
+  - [ ] Conduct security audit
+  - [ ] Implement security improvements
+  - [ ] Add security monitoring
+  - [ ] Test security features
+
+- [ ] **Performance optimization for production loads**
+  - [ ] Optimize for production workloads
+  - [ ] Test production load scenarios
+  - [ ] Implement performance monitoring
+  - [ ] Validate production performance
+
+- [ ] **Documentation completion**
+  - [ ] Complete technical documentation
+  - [ ] Complete user documentation
+  - [ ] Complete deployment documentation
+  - [ ] Create maintenance guides
+
+- [ ] **Deployment preparation**
+  - [ ] Create deployment scripts
+  - [ ] Prepare production environment
+  - [ ] Create backup and recovery procedures
+  - [ ] Test deployment process
+
+**Deliverables**: Web interface, production-ready system, complete documentation
+
+## Critical Path Items (Must Complete First)
+
+### **Phase 1 Critical Path**
+1. **OSBuild Architecture Analysis** - Must understand current system before modifying
+2. **debos Integration Planning** - Must plan how to integrate Debian logic
+3. **Technical Specifications** - Must have clear plan before implementation
+
+### **Phase 2 Critical Path**
+1. **Debian Package Management** - Core functionality required for everything else
+2. **OSTree Composition** - Essential for atomic functionality
+3. **Build Environment Integration** - Required for package building
+
+### **Phase 3 Critical Path**
+1. **Build Orchestration** - Required for managing multiple builds
+2. **Environment Management** - Required for build isolation and resource management
+
+### **Phase 4 Critical Path**
+1. **System Integration** - Must connect all components
+2. **Image Generation** - Core output functionality
+3. **Performance Optimization** - Required for production use
+
+### **Phase 5 Critical Path**
+1. **Web Interface** - Required for user interaction
+2. **Security Hardening** - Required for production deployment
+3. **Documentation** - Required for adoption and maintenance
+
+## Risk Mitigation
+
+### **Technical Risks**
+- **OSBuild Complexity**: Start with simple modifications, build complexity gradually
+- **Integration Challenges**: Use well-defined interfaces, test integration points early
+- **Performance Issues**: Test performance early, optimize continuously
+- **Compatibility Issues**: Test with real Debian packages, validate edge cases
+
+### **Timeline Risks**
+- **Scope Creep**: Stick to core functionality, defer advanced features
+- **Integration Delays**: Plan integration carefully, test early and often
+- **Testing Complexity**: Start testing early, automate testing where possible
+- **Documentation**: Document as you go, don't leave it until the end
+
+### **Resource Risks**
+- **Developer Availability**: Plan for developer rotation, document everything
+- **Infrastructure Issues**: Use simple infrastructure initially, scale gradually
+- **Dependency Changes**: Monitor upstream projects, plan for changes
+
+## Success Metrics
+
+### **Phase 1 Success**
+- [ ] Complete project setup and analysis
+- [ ] Clear understanding of integration requirements
+- [ ] Detailed technical specifications
+
+### **Phase 2 Success**
+- [ ] Modified OSBuild with Debian package support
+- [ ] Working OSTree composition for Debian packages
+- [ ] Basic atomic functionality validation
+
+### **Phase 3 Success**
+- [ ] Working build orchestration system
+- [ ] Build environment management
+- [ ] Complete build lifecycle support
+
+### **Phase 4 Success**
+- [ ] Integrated system with working build pipeline
+- [ ] Image generation and bootc container creation
+- [ ] Performance and stability validation
+
+### **Phase 5 Success**
+- [ ] Production-ready system with web interface
+- [ ] Complete documentation and deployment guide
+- [ ] Community adoption and feedback
+
+## Next Immediate Actions (This Week)
+
+1. **Test the new Debian stages** - Run the test manifest to validate functionality
+2. **Set up build environment for testing** - Configure testing infrastructure
+3. **Begin OSTree integration testing** - Test OSTree commit creation
+4. **Plan Koji integration** - Design build orchestration system
+
+## Notes
+
+- **Priority**: Focus on core functionality first, add advanced features later
+- **Testing**: Test each component individually before integration
+- **Documentation**: Document everything as you go, don't assume you'll remember
+- **Iteration**: Use iterative development, get working system early and improve gradually
+- **Community**: Engage with Debian community early for feedback and adoption
diff --git a/debian-forge-docs/changelog.md b/debian-forge-docs/changelog.md
new file mode 100755
index 00000000..a6cb3f84
--- /dev/null
+++ b/debian-forge-docs/changelog.md
@@ -0,0 +1,68 @@
+# Debian Forge Changelog
+
+## 2024-12-19
+- **Documentation updates completed**
+  - Updated OSBuild architecture documentation with composer integration details
+  - Created comprehensive user documentation covering installation, usage, and workflows
+  - Created production deployment documentation with configuration and maintenance
+  - Created troubleshooting guides with diagnostic procedures and recovery steps
+
+## 2024-12-19
+- **Performance and stability testing completed**
+  - Bug fixing and stability improvements implemented
+  - Stress testing with multiple concurrent builds validated
+  - Performance optimization and bottleneck identification completed
+
+## [Unreleased]
+
+### Added
+- apt-cacher-ng support for Debian stages (apt, debootstrap)
+- Debian-specific OSBuild stages: apt, debootstrap, ostree-commit
+- apt config stage for package manager configuration
+- ostree deploy stage for atomic deployment
+- sbuild stage for Debian package building
+- debian source stage for package source management
+- Build orchestration system with queue management and priority scheduling
+- Resource allocation system for CPU, memory, and storage management
+- Enhanced build status tracking with state machine
+- Build logging and monitoring system
+- Build progress tracking and cancellation support
+- Artifact management system with SQLite database
+- Artifact storage organization and checksum validation
+- Build environment management system with isolation and cleanup
+- Host health monitoring and resource usage tracking
+- OSBuild integration module for pipeline management
+- OSBuild architecture documentation for 1:1 compatibility
+- Enhanced debootstrap stage following OSBuild patterns
+- Improved OSTree commit stage following OSBuild patterns exactly
+- Added OSTree composition testing and validation
+- Added multi-stage workflow testing and validation
+- Added error handling and recovery testing
+- Added image generation testing (ISO, QCOW2, RAW)
+- Added bootc container creation testing and validation
+- Added multi-format output generation testing
+- Added image bootability testing and validation
+- Added performance testing and optimization
+- Added stress testing with multiple concurrent builds
+- Added bug fixing and stability improvements
+- Test script for validating Debian stages
+- Test script for resource allocation functionality
+- Test script for build orchestration system
+- Test script for complete build lifecycle
+- Build environment setup script
+- Comprehensive test manifest for Debian atomic system
+- Setup script for apt-cacher-ng configuration
+
+### Changed
+- Updated apt-cacher-ng address to 192.168.1.101:3142
+- Modified debos integration to use apt-cacher-ng proxy
+
+### Technical
+- Created debos to OSBuild stage mapping
+- Implemented Debian package management in OSBuild pipeline
+- Added OSTree composition support for Debian packages
+- Integrated apt-cacher-ng proxy configuration in stages
+- Replaced Mock with sbuild for Debian package building
+- Added Debian source package management capabilities
+- Implemented basic build orchestration with OSBuild integration
+- Created testing infrastructure for Debian stages
diff --git a/debian-forge-docs/debian-forge-roadmap.md b/debian-forge-docs/debian-forge-roadmap.md
new file mode 100755
index 00000000..df5c8aa2
--- /dev/null
+++ b/debian-forge-docs/debian-forge-roadmap.md
@@ -0,0 +1,203 @@
+# Debian Forge: Realistic Development Roadmap
+
+## Project Overview
+Debian Forge is a Debian atomic build system that combines:
+- **OSBuild architecture** (forked and modified for Debian)
+- **debos logic** (for Debian package management)
+- **Koji-like orchestration** (for build management)
+
+## Development Phases
+
+### Phase 1: Foundation and Analysis (Weeks 1-6)
+
+#### Week 1-2: Project Setup
+- Set up development environment
+- Fork OSBuild repository
+- Download and analyze debos source code
+- Set up basic project structure
+
+#### Week 3-4: Code Analysis
+- Analyze OSBuild architecture and identify Fedora-specific components
+- Map debos functionality to OSBuild components
+- Identify integration points between OSBuild and Koji-like functionality
+- Document current architecture and planned changes
+
+#### Week 5-6: Planning and Design
+- Design Debian-specific component architecture
+- Plan Koji-like orchestration system
+- Define interfaces between components
+- Create detailed technical specifications
+
+**Deliverables**: Project structure, architecture documentation, technical specifications
+
+### Phase 2: Core OSBuild Modification (Weeks 7-18)
+
+#### Week 7-10: Package Management Layer
+- Replace RPM package management with Debian package management
+- Integrate debos package handling logic
+- Implement Debian repository handling
+- Test basic package operations
+
+#### Week 11-14: Build Environment Integration
+- Replace Mock integration with sbuild/pbuilder integration
+- Implement Debian build environment setup
+- Test build environment creation and management
+- Validate package building in Debian environments
+
+#### Week 15-18: OSTree Composition
+- Adapt OSTree composition for Debian packages
+- Test OSTree commit creation from Debian packages
+- Validate filesystem composition
+- Test basic atomic functionality
+
+**Deliverables**: Modified OSBuild with Debian package support, working OSTree composition
+
+### Phase 3: Build Orchestration (Weeks 19-30)
+
+#### Week 19-22: Basic Orchestration
+- Implement build scheduling system
+- Add build queue management
+- Implement basic resource allocation
+- Test build submission and tracking
+
+#### Week 23-26: Build Management
+- Add build status tracking
+- Implement build logging and monitoring
+- Add build history and artifact management
+- Test complete build lifecycle
+
+#### Week 27-30: Environment Management
+- Implement isolated build environment management
+- Add build host monitoring and health checks
+- Implement build environment cleanup
+- Test environment isolation and management
+
+**Deliverables**: Working build orchestration system, build environment management
+
+### Phase 4: Integration and Testing (Weeks 31-42)
+
+#### Week 31-34: System Integration
+- Integrate modified OSBuild with orchestration system
+- Test complete build pipeline from submission to completion
+- Validate multi-stage build workflows
+- Test error handling and recovery
+
+#### Week 35-38: Output Generation
+- Test image generation (ISO, QCOW2, RAW)
+- Validate bootc container creation
+- Test multi-format output generation
+- Validate image bootability
+
+#### Week 39-42: Performance and Stability
+- Performance testing and optimization
+- Stress testing with multiple concurrent builds
+- Bug fixing and stability improvements
+- Documentation updates
+
+**Deliverables**: Integrated system, working image generation, performance optimization
+
+### Phase 5: Web Interface and Production Features (Weeks 43-54)
+
+#### Week 43-46: Basic Web UI
+- Implement basic web interface for build management
+- Add build status monitoring
+- Implement build submission forms
+- Test web interface functionality
+
+#### Week 47-50: Advanced Web Features
+- Add build history and search
+- Implement user management and permissions
+- Add system administration interface
+- Test complete web interface
+
+#### Week 51-54: Production Readiness
+- Security testing and hardening
+- Performance optimization for production loads
+- Documentation completion
+- Deployment preparation
+
+**Deliverables**: Web interface, production-ready system, complete documentation
+
+## Resource Requirements
+
+### Development Team
+- **1 Senior Developer**: OSBuild modification and integration
+- **1 Developer**: Build orchestration and Koji-like functionality
+- **1 Developer**: Web interface and user experience
+- **1 DevOps Engineer**: Build environment and deployment
+
+### Infrastructure
+- **Development Environment**: Debian-based development machines
+- **Build Hosts**: Multiple build environments for testing
+- **Testing Infrastructure**: Virtual machines for image testing
+- **CI/CD Pipeline**: Automated testing and deployment
+
+### Dependencies
+- **OSBuild**: Fork and modify existing codebase
+- **debos**: Extract and integrate Debian logic
+- **OSTree**: Core atomic functionality
+- **bootc**: Container creation and management
+
+## Risk Factors
+
+### Technical Risks
+- **OSBuild Complexity**: OSBuild may be more complex than anticipated
+- **Integration Challenges**: Combining three different systems may have unexpected issues
+- **Performance**: Atomic builds may be slower than traditional builds
+- **Compatibility**: Debian packages may have edge cases not handled by debos
+
+### Timeline Risks
+- **Scope Creep**: Adding features beyond core functionality
+- **Integration Delays**: Unexpected issues when combining systems
+- **Testing Complexity**: Atomic systems require more thorough testing
+- **Documentation**: Comprehensive documentation takes time
+
+### Resource Risks
+- **Developer Availability**: Team members may have other commitments
+- **Infrastructure Issues**: Build environment setup may be complex
+- **Dependency Changes**: Upstream projects may change significantly
+
+## Success Criteria
+
+### Phase 1 Success
+- Complete project setup and analysis
+- Clear understanding of integration requirements
+- Detailed technical specifications
+
+### Phase 2 Success
+- Modified OSBuild with Debian package support
+- Working OSTree composition for Debian packages
+- Basic atomic functionality validation
+
+### Phase 3 Success
+- Working build orchestration system
+- Build environment management
+- Complete build lifecycle support
+
+### Phase 4 Success
+- Integrated system with working build pipeline
+- Image generation and bootc container creation
+- Performance and stability validation
+
+### Phase 5 Success
+- Production-ready system with web interface
+- Complete documentation and deployment guide
+- Community adoption and feedback
+
+## Timeline Summary
+
+- **Total Duration**: 54 weeks (approximately 13 months)
+- **Phase 1**: 6 weeks (Foundation and Analysis)
+- **Phase 2**: 12 weeks (Core OSBuild Modification)
+- **Phase 3**: 12 weeks (Build Orchestration)
+- **Phase 4**: 12 weeks (Integration and Testing)
+- **Phase 5**: 12 weeks (Web Interface and Production)
+
+## Next Steps
+
+1. **Immediate**: Set up development environment and fork OSBuild
+2. **Week 1-2**: Begin code analysis and architecture planning
+3. **Week 3-4**: Complete analysis and start design phase
+4. **Week 5-6**: Finalize technical specifications and begin implementation
+
+This roadmap provides a realistic path to a working Debian Forge system while acknowledging the complexity of integrating three different build systems and the time required for thorough testing and validation.
diff --git a/debian-forge-docs/debos-osbuild-mapping.md b/debian-forge-docs/debos-osbuild-mapping.md
new file mode 100755
index 00000000..5f75a90f
--- /dev/null
+++ b/debian-forge-docs/debos-osbuild-mapping.md
@@ -0,0 +1,117 @@
+# Debian Forge: debos to OSBuild Stage Mapping
+
+## Overview
+This document maps debos actions to OSBuild stages to enable Debian package management and OSTree composition in the OSBuild pipeline.
+
+## Action Mappings
+
+### Package Management
+
+#### debos: `apt` → OSBuild: `org.osbuild.apt`
+- **Purpose**: Install Debian packages using apt
+- **Input**: Package list, repository configuration
+- **Output**: Installed packages in target filesystem
+- **Dependencies**: Repository configuration, package sources
+
+#### debos: `debootstrap` → OSBuild: `org.osbuild.debootstrap`
+- **Purpose**: Create base Debian filesystem
+- **Input**: Debian suite, architecture, mirror
+- **Output**: Base Debian root filesystems
+- **Dependencies**: Network access, Debian mirror
+
+### OSTree Integration
+
+#### debos: `ostree-commit` → OSBuild: `org.osbuild.ostree.commit`
+- **Purpose**: Create OSTree commit from filesystem
+- **Input**: Filesystem tree, commit metadata
+- **Output**: OSTree commit in repository
+- **Dependencies**: OSTree repository, filesystem tree
+
+#### debos: `ostree-deploy` → OSBuild: `org.osbuild.ostree.deploy`
+- **Purpose**: Deploy OSTree branch to target
+- **Input**: OSTree repository, branch name
+- **Output**: Deployed OSTree filesystem
+- **Dependencies**: OSTree repository, target device
+
+### Filesystem Operations
+
+#### debos: `overlay` → OSBuild: `org.osbuild.overlay`
+- **Purpose**: Copy files/directories to target
+- **Input**: Source files, target paths
+- **Output**: Files copied to target filesystem
+- **Dependencies**: Source files, target filesystem
+
+#### debos: `filesystem-deploy` → OSBuild: `org.osbuild.filesystem.deploy`
+- **Purpose**: Deploy filesystem to image
+- **Input**: Filesystem tree, target image
+- **Output**: Image with deployed filesystem
+- **Dependencies**: Filesystem tree, target device
+
+### Image Creation
+
+#### debos: `image-partition` → OSBuild: `org.osbuild.image.partition`
+- **Purpose**: Create and partition disk image
+- **Input**: Partition layout, filesystem types
+- **Output**: Partitioned disk image
+- **Dependencies**: Partition specification
+
+## Implementation Priority
+
+### Phase 1: Core Package Management
+1. **org.osbuild.apt** - Debian package installation
+2. **org.osbuild.debootstrap** - Base filesystem creation
+3. **org.osbuild.apt.repos** - Repository configuration
+
+### Phase 2: OSTree Integration
+1. **org.osbuild.ostree.commit** - Commit creation
+2. **org.osbuild.ostree.deploy** - Deployment
+3. **org.osbuild.ostree.repo** - Repository management
+
+### Phase 3: Advanced Features
+1. **org.osbuild.overlay** - File operations
+2. **org.osbuild.filesystem.deploy** - Filesystem deployment
+3. **org.osbuild.image.partition** - Image creation
+
+## Stage Structure
+
+Each OSBuild stage should follow the standard pattern:
+- Python script for the stage logic
+- JSON metadata file defining options and dependencies
+- Integration with OSBuild pipeline system
+- Proper error handling and logging
+
+## Dependencies
+
+- **Python 3.6+** for stage execution
+- **apt/dpkg** for Debian package management
+- **ostree** for atomic filesystem management
+- **debootstrap** for base filesystem creation
+- **sbuild/pbuilder** for build environments
+- **apt-cacher-ng** for package caching (optional, improves performance)
+
+## apt-cacher-ng Integration
+
+### Configuration
+Both `org.osbuild.apt` and `org.osbuild.debootstrap` stages support apt-cacher-ng proxy configuration:
+
+```json
+{
+  "name": "org.osbuild.apt",
+  "options": {
+    "packages": ["systemd", "dbus"],
+    "apt_proxy": "http://localhost:3142"
+  }
+}
+```
+
+### Benefits
+- **Faster builds** - Packages are cached locally
+- **Reduced network usage** - Avoids re-downloading packages
+- **Offline capability** - Can build without internet after initial cache
+- **Development efficiency** - Faster iteration during development
+
+### Setup
+Use the provided `setup-apt-cacher.sh` script to configure apt-cacher-ng:
+```bash
+sudo ./debian-forge/setup-apt-cacher.sh
+```
diff --git a/debian-forge-docs/deployment-documentation.md b/debian-forge-docs/deployment-documentation.md
new file mode 100644
index 00000000..49b6eb9d
--- /dev/null
+++ b/debian-forge-docs/deployment-documentation.md
@@ -0,0 +1,811 @@
+# Debian Forge Deployment Documentation
+
+## Overview
+This document covers deploying Debian Forge in production environments, including system requirements, configuration, monitoring, and maintenance procedures.
+
+## System Requirements
+
+### Minimum Requirements
+- **CPU**: 4 cores (8 recommended)
+- **Memory**: 8GB RAM (16GB recommended)
+- **Storage**: 100GB available space (500GB recommended)
+- **Network**: Stable internet connection for package downloads
+- **OS**: Debian 12+ or Ubuntu 22.04+
+
+### Recommended Production Specs
+- **CPU**: 8+ cores with good single-thread performance
+- **Memory**: 32GB+ RAM for concurrent builds
+- **Storage**: 1TB+ SSD with good I/O performance
+- **Network**: Gigabit connection with low latency
+- **OS**: Debian 12+ with LTS support
+
+### Storage Requirements
+```
+/var/lib/debian-forge/          # Build artifacts and cache
+├── builds/                     # Build outputs (50-200GB)
+├── cache/                      # Package cache (20-100GB)
+├── ostree/                     # OSTree repositories (100-500GB)
+└── logs/                       # Build logs (10-50GB)
+
+/tmp/                          # Temporary build space (50-100GB)
+.osbuild/                      # OSBuild cache (20-100GB)
+```
+
+## Production Deployment
+
+### 1. System Preparation
+```bash
+# Update system
+sudo apt update && sudo apt upgrade -y
+
+# Install essential packages
+sudo apt install -y \
+    python3-pip \
+    python3-venv \
+    python3-dev \
+    build-essential \
+    ostree \
+    debootstrap \
+    mmdebstrap \
+    sbuild \
+    pbuilder \
+    bubblewrap \
+    qemu-utils \
+    curl \
+    skopeo \
+    git \
+    nginx \
+    postgresql \
+    redis-server \
+    supervisor \
+    logrotate \
+    fail2ban \
+    ufw
+
+# Configure firewall
+sudo ufw allow ssh
+sudo ufw allow 80/tcp
+sudo ufw allow 443/tcp
+sudo ufw allow 8080/tcp  # Debian Forge API
+sudo ufw enable
+```
+
+### 2. User and Security Setup
+```bash
+# Create dedicated user
+sudo useradd -m -s /bin/bash debian-forge
+sudo usermod -aG sudo debian-forge
+sudo usermod -aG sbuild debian-forge
+
+# Configure sudo access
+echo "debian-forge ALL=(ALL) NOPASSWD: /usr/bin/apt, /usr/bin/apt-get, /usr/bin/dpkg" | sudo tee /etc/sudoers.d/debian-forge
+
+# Set up SSH keys
+sudo mkdir -p /home/debian-forge/.ssh
+sudo chown debian-forge:debian-forge /home/debian-forge/.ssh
+sudo chmod 700 /home/debian-forge/.ssh
+
+# Copy your SSH key
+sudo -u debian-forge ssh-keygen -t ed25519 -C "debian-forge@$(hostname)"
+```
+
+### 3. Database Setup
+```bash
+# Configure PostgreSQL
+sudo -u postgres createuser debian-forge
+sudo -u postgres createdb debian_forge
+sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE debian_forge TO debian_forge;"
+
+# Configure Redis
+sudo systemctl enable redis-server
+sudo systemctl start redis-server
+```
+
+### 4. Application Deployment
+```bash
+# Switch to debian-forge user
+sudo su - debian-forge
+
+# Clone repository
+git clone https://github.com/your-org/debian-forge.git
+cd debian-forge
+
+# Create virtual environment
+python3 -m venv venv
+source venv/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Install additional production packages
+pip install gunicorn uwsgi psycopg2-binary redis supervisor
+```
+
+### 5. Configuration Files
+
+#### Environment Configuration
+```bash
+# /home/debian-forge/debian-forge/.env
+DEBIAN_FORGE_ENV=production
+DEBIAN_FORGE_DEBUG=false
+DEBIAN_FORGE_SECRET_KEY=your-secret-key-here
+DEBIAN_FORGE_DATABASE_URL=postgresql://debian-forge@localhost/debian_forge
+DEBIAN_FORGE_REDIS_URL=redis://localhost:6379/0
+DEBIAN_FORGE_LOG_LEVEL=INFO
+DEBIAN_FORGE_MAX_CONCURRENT_BUILDS=4
+DEBIAN_FORGE_BUILD_TIMEOUT=3600
+DEBIAN_FORGE_CACHE_SIZE=50GB
+DEBIAN_FORGE_OSTREE_REPO_PATH=/var/lib/debian-forge/ostree
+```
+
+#### Build Environment Configuration
+```bash
+# /home/debian-forge/debian-forge/config/build-env.conf
+[build_environment]
+max_concurrent_builds = 4
+build_timeout = 3600
+resource_limits_cpu = 80
+resource_limits_memory = 85
+resource_limits_disk = 90
+cleanup_after_build = true
+cache_retention_days = 30
+
+[ostree]
+repo_path = /var/lib/debian-forge/ostree
+max_repo_size = 100GB
+cleanup_old_commits = true
+commit_retention_days = 90
+
+[apt]
+proxy_url = http://192.168.1.101:3142
+mirror_url = http://deb.debian.org/debian
+security_url = http://security.debian.org/debian-security
+updates_url = http://deb.debian.org/debian
+```
+
+### 6. Service Configuration
+
+#### Supervisor Configuration
+```ini
+# /etc/supervisor/conf.d/debian-forge.conf
+[program:debian-forge-api]
+command=/home/debian-forge/debian-forge/venv/bin/gunicorn -w 4 -b 127.0.0.1:8080 --timeout 300 --max-requests 1000 --max-requests-jitter 100 app:app
+directory=/home/debian-forge/debian-forge
+user=debian-forge
+autostart=true
+autorestart=true
+redirect_stderr=true
+stdout_logfile=/var/log/debian-forge/api.log
+stdout_logfile_maxbytes=50MB
+stdout_logfile_backups=10
+
+[program:debian-forge-worker]
+command=/home/debian-forge/debian-forge/venv/bin/python -m build_orchestrator
+directory=/home/debian-forge/debian-forge
+user=debian-forge
+autostart=true
+autorestart=true
+redirect_stderr=true
+stdout_logfile=/var/log/debian-forge/worker.log
+stdout_logfile_maxbytes=50MB
+stdout_logfile_backups=10
+
+[program:debian-forge-cleanup]
+command=/home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager
+directory=/home/debian-forge/debian-forge
+user=debian-forge
+autostart=true
+autorestart=true
+redirect_stderr=true
+stdout_logfile=/var/log/debian-forge/cleanup.log
+stdout_logfile_maxbytes=50MB
+stdout_logfile_backups=10
+```
+
+#### Nginx Configuration
+```nginx
+# /etc/nginx/sites-available/debian-forge
+server {
+    listen 80;
+    server_name your-domain.com;
+    return 301 https://$server_name$request_uri;
+}
+
+server {
+    listen 443 ssl http2;
+    server_name your-domain.com;
+
+    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
+    ssl_prefer_server_ciphers off;
+
+    client_max_body_size 100M;
+    proxy_read_timeout 300s;
+    proxy_connect_timeout 75s;
+
+    location / {
+        proxy_pass http://127.0.0.1:8080;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+
+    location /static/ {
+        alias /home/debian-forge/debian-forge/static/;
+        expires 1y;
+        add_header Cache-Control "public, immutable";
+    }
+
+    location /logs/ {
+        alias /var/log/debian-forge/;
+        auth_basic "Restricted Access";
+        auth_basic_user_file /etc/nginx/.htpasswd;
+    }
+}
+```
+
+### 7. SSL Certificate Setup
+```bash
+# Install Certbot
+sudo apt install -y certbot python3-certbot-nginx
+
+# Obtain SSL certificate
+sudo certbot --nginx -d your-domain.com
+
+# Test auto-renewal
+sudo certbot renew --dry-run
+```
+
+### 8. Monitoring and Logging
+
+#### Logrotate Configuration
+```bash
+# /etc/logrotate.d/debian-forge
+/var/log/debian-forge/*.log {
+    daily
+    missingok
+    rotate 30
+    compress
+    delaycompress
+    notifempty
+    create 644 debian-forge debian-forge
+    postrotate
+        systemctl reload supervisor
+    endscript
+}
+```
+
+#### Monitoring Scripts
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/monitor.sh
+
+# Check service status
+check_service() {
+    local service=$1
+    if ! systemctl is-active --quiet $service; then
+        echo "ERROR: $service is not running"
+        systemctl restart $service
+        echo "$(date): Restarted $service" >> /var/log/debian-forge/monitor.log
+    fi
+}
+
+# Check disk space
+check_disk() {
+    local usage=$(df /var/lib/debian-forge | tail -1 | awk '{print $5}' | sed 's/%//')
+    if [ $usage -gt 90 ]; then
+        echo "WARNING: Disk usage is ${usage}%"
+        # Trigger cleanup
+        /home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager --force
+    fi
+}
+
+# Check memory usage
+check_memory() {
+    local usage=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100.0}')
+    if [ $usage -gt 90 ]; then
+        echo "WARNING: Memory usage is ${usage}%"
+    fi
+}
+
+# Main monitoring loop
+while true; do
+    check_service debian-forge-api
+    check_service debian-forge-worker
+    check_service debian-forge-cleanup
+    check_disk
+    check_memory
+    sleep 300  # Check every 5 minutes
+done
+```
+
+## Production Configuration
+
+### 1. Performance Tuning
+
+#### System Tuning
+```bash
+# /etc/sysctl.conf
+# Increase file descriptor limits
+fs.file-max = 65536
+fs.inotify.max_user_watches = 524288
+
+# Network tuning
+net.core.somaxconn = 65535
+net.core.netdev_max_backlog = 5000
+net.ipv4.tcp_max_syn_backlog = 65535
+
+# Memory tuning
+vm.swappiness = 10
+vm.dirty_ratio = 15
+vm.dirty_background_ratio = 5
+```
+
+#### Application Tuning
+```python
+# /home/debian-forge/debian-forge/config/performance.conf
+[performance]
+max_workers = 4
+worker_timeout = 300
+max_requests = 1000
+max_requests_jitter = 100
+keepalive = 2
+worker_connections = 1000
+
+[caching]
+cache_size = 50GB
+cache_ttl = 86400
+cache_cleanup_interval = 3600
+
+[build_optimization]
+parallel_stages = true
+stage_cache_enabled = true
+artifact_compression = true
+```
+
+### 2. Security Configuration
+
+#### Fail2ban Configuration
+```ini
+# /etc/fail2ban/jail.local
+[debian-forge-api]
+enabled = true
+port = 8080
+filter = debian-forge-api
+logpath = /var/log/debian-forge/api.log
+maxretry = 5
+bantime = 3600
+findtime = 600
+
+[debian-forge-ssh]
+enabled = true
+port = ssh
+filter = sshd
+logpath = /var/log/auth.log
+maxretry = 3
+bantime = 3600
+findtime = 600
+```
+
+#### Access Control
+```bash
+# /etc/nginx/.htpasswd (for log access)
+sudo htpasswd -c /etc/nginx/.htpasswd admin
+
+# SSH key-based authentication only
+sudo sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
+sudo systemctl reload ssh
+```
+
+### 3. Backup Configuration
+
+#### Backup Script
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/backup.sh
+
+BACKUP_DIR="/var/backups/debian-forge"
+DATE=$(date +%Y%m%d_%H%M%S)
+RETENTION_DAYS=30
+
+# Create backup directory
+mkdir -p $BACKUP_DIR
+
+# Database backup
+pg_dump -U debian-forge debian_forge > $BACKUP_DIR/db_$DATE.sql
+
+# Configuration backup
+tar -czf $BACKUP_DIR/config_$DATE.tar.gz \
+    /home/debian-forge/debian-forge/config \
+    /etc/supervisor/conf.d/debian-forge.conf \
+    /etc/nginx/sites-available/debian-forge
+
+# OSTree repository backup
+rsync -av --delete /var/lib/debian-forge/ostree/ $BACKUP_DIR/ostree_$DATE/
+
+# Cleanup old backups
+find $BACKUP_DIR -type f -mtime +$RETENTION_DAYS -delete
+find $BACKUP_DIR -type d -mtime +$RETENTION_DAYS -exec rm -rf {} +
+
+echo "Backup completed: $DATE" >> /var/log/debian-forge/backup.log
+```
+
+#### Automated Backup
+```bash
+# /etc/cron.daily/debian-forge-backup
+#!/bin/bash
+/home/debian-forge/debian-forge/scripts/backup.sh
+```
+
+## Maintenance Procedures
+
+### 1. Regular Maintenance
+
+#### Daily Tasks
+```bash
+# Check service status
+sudo supervisorctl status
+
+# Monitor logs
+tail -f /var/log/debian-forge/*.log
+
+# Check disk space
+df -h /var/lib/debian-forge
+
+# Check build queue
+curl -s http://localhost:8080/api/v1/queue/status
+```
+
+#### Weekly Tasks
+```bash
+# Update system packages
+sudo apt update && sudo apt upgrade -y
+
+# Clean old build artifacts
+/home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager --force
+
+# Rotate logs
+sudo logrotate -f /etc/logrotate.d/debian-forge
+
+# Check SSL certificate expiration
+sudo certbot certificates
+```
+
+#### Monthly Tasks
+```bash
+# Review and clean old OSTree commits
+/home/debian-forge/debian-forge/venv/bin/python -m ostree_cleanup --older-than 90
+
+# Update application dependencies
+cd /home/debian-forge/debian-forge
+source venv/bin/activate
+pip install --upgrade -r requirements.txt
+
+# Review and update security configurations
+sudo fail2ban-client status
+sudo ufw status
+```
+
+### 2. Troubleshooting
+
+#### Common Issues
+
+**Service Not Starting**
+```bash
+# Check supervisor status
+sudo supervisorctl status
+
+# Check logs
+sudo tail -f /var/log/supervisor/supervisord.log
+sudo tail -f /var/log/debian-forge/*.log
+
+# Restart services
+sudo supervisorctl restart debian-forge-api
+sudo supervisorctl restart debian-forge-worker
+```
+
+**Build Failures**
+```bash
+# Check build logs
+tail -f /var/log/debian-forge/worker.log
+
+# Check system resources
+htop
+df -h
+free -h
+
+# Restart worker
+sudo supervisorctl restart debian-forge-worker
+```
+
+**Database Issues**
+```bash
+# Check PostgreSQL status
+sudo systemctl status postgresql
+
+# Check connection
+sudo -u debian-forge psql -d debian_forge -c "SELECT version();"
+
+# Restart database
+sudo systemctl restart postgresql
+```
+
+### 3. Recovery Procedures
+
+#### Service Recovery
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/recovery.sh
+
+echo "Starting Debian Forge recovery..."
+
+# Stop all services
+sudo supervisorctl stop all
+
+# Clean up temporary files
+sudo rm -rf /tmp/debian-forge-*
+sudo rm -rf /var/tmp/debian-forge-*
+
+# Restart database
+sudo systemctl restart postgresql
+sudo systemctl restart redis-server
+
+# Wait for services to be ready
+sleep 10
+
+# Start services
+sudo supervisorctl start all
+
+# Check status
+sudo supervisorctl status
+
+echo "Recovery completed"
+```
+
+#### Data Recovery
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/data-recovery.sh
+
+BACKUP_DIR="/var/backups/debian-forge"
+LATEST_BACKUP=$(ls -t $BACKUP_DIR/db_*.sql | head -1)
+
+if [ -n "$LATEST_BACKUP" ]; then
+    echo "Restoring from backup: $LATEST_BACKUP"
+    
+    # Stop services
+    sudo supervisorctl stop all
+    
+    # Restore database
+    sudo -u postgres dropdb debian_forge
+    sudo -u postgres createdb debian_forge
+    sudo -u postgres psql debian_forge < $LATEST_BACKUP
+    
+    # Restart services
+    sudo supervisorctl start all
+    
+    echo "Data recovery completed"
+else
+    echo "No backup found for recovery"
+    exit 1
+fi
+```
+
+## Scaling Considerations
+
+### 1. Horizontal Scaling
+
+#### Load Balancer Configuration
+```nginx
+# /etc/nginx/sites-available/debian-forge-cluster
+upstream debian_forge_backend {
+    server 192.168.1.10:8080;
+    server 192.168.1.11:8080;
+    server 192.168.1.12:8080;
+}
+
+server {
+    listen 443 ssl http2;
+    server_name your-domain.com;
+    
+    location / {
+        proxy_pass http://debian_forge_backend;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}
+```
+
+#### Shared Storage
+```bash
+# NFS configuration for shared storage
+# /etc/exports
+/var/lib/debian-forge 192.168.1.0/24(rw,sync,no_subtree_check)
+
+# Mount on worker nodes
+# /etc/fstab
+192.168.1.10:/var/lib/debian-forge /var/lib/debian-forge nfs defaults 0 0
+```
+
+### 2. Vertical Scaling
+
+#### Resource Optimization
+```python
+# /home/debian-forge/debian-forge/config/scaling.conf
+[scaling]
+max_concurrent_builds = 8
+worker_processes = 8
+memory_limit = 32GB
+cpu_limit = 8
+
+[cache]
+cache_size = 100GB
+cache_ttl = 172800  # 48 hours
+```
+
+## Monitoring and Alerting
+
+### 1. Health Checks
+
+#### Application Health
+```python
+# /home/debian-forge/debian-forge/health_check.py
+import requests
+import psutil
+import os
+
+def check_health():
+    health_status = {
+        "status": "healthy",
+        "checks": {}
+    }
+    
+    # Check API endpoint
+    try:
+        response = requests.get("http://localhost:8080/health", timeout=5)
+        health_status["checks"]["api"] = "healthy" if response.status_code == 200 else "unhealthy"
+    except:
+        health_status["checks"]["api"] = "unhealthy"
+    
+    # Check system resources
+    cpu_percent = psutil.cpu_percent()
+    memory_percent = psutil.virtual_memory().percent
+    disk_percent = psutil.disk_usage('/var/lib/debian-forge').percent
+    
+    health_status["checks"]["cpu"] = "healthy" if cpu_percent < 90 else "warning"
+    health_status["checks"]["memory"] = "healthy" if memory_percent < 90 else "warning"
+    health_status["checks"]["disk"] = "healthy" if disk_percent < 90 else "warning"
+    
+    # Overall status
+    if any(check == "unhealthy" for check in health_status["checks"].values()):
+        health_status["status"] = "unhealthy"
+    elif any(check == "warning" for check in health_status["checks"].values()):
+        health_status["status"] = "degraded"
+    
+    return health_status
+```
+
+#### Monitoring Dashboard
+```html
+<!-- /home/debian-forge/debian-forge/templates/monitor.html -->
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Debian Forge Monitor</title>
+    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
+</head>
+<body>
+    <h1>Debian Forge System Monitor</h1>
+    
+    <div class="status-grid">
+        <div class="status-card">
+            <h3>API Status</h3>
+            <div id="api-status">Checking...</div>
+        </div>
+        
+        <div class="status-card">
+            <h3>Build Queue</h3>
+            <div id="queue-status">Checking...</div>
+        </div>
+        
+        <div class="status-card">
+            <h3>System Resources</h3>
+            <canvas id="resource-chart"></canvas>
+        </div>
+    </div>
+    
+    <script>
+        // Update status every 30 seconds
+        setInterval(updateStatus, 30000);
+        
+        function updateStatus() {
+            fetch('/api/v1/health')
+                .then(response => response.json())
+                .then(data => {
+                    document.getElementById('api-status').textContent = data.status;
+                    // Update other status elements
+                });
+        }
+    </script>
+</body>
+</html>
+```
+
+### 2. Alerting Configuration
+
+#### Email Alerts
+```python
+# /home/debian-forge/debian-forge/scripts/alert.py
+import smtplib
+from email.mime.text import MIMEText
+import os
+
+def send_alert(subject, message, severity="INFO"):
+    smtp_server = os.getenv("SMTP_SERVER", "localhost")
+    smtp_port = int(os.getenv("SMTP_PORT", "587"))
+    smtp_user = os.getenv("SMTP_USER")
+    smtp_password = os.getenv("SMTP_PASSWORD")
+    alert_email = os.getenv("ALERT_EMAIL")
+    
+    msg = MIMEText(message)
+    msg['Subject'] = f"[{severity}] Debian Forge: {subject}"
+    msg['From'] = smtp_user
+    msg['To'] = alert_email
+    
+    try:
+        with smtplib.SMTP(smtp_server, smtp_port) as server:
+            server.starttls()
+            server.login(smtp_user, smtp_password)
+            server.send_message(msg)
+        print(f"Alert sent: {subject}")
+    except Exception as e:
+        print(f"Failed to send alert: {e}")
+```
+
+#### Alert Rules
+```yaml
+# /home/debian-forge/debian-forge/config/alerts.yaml
+alerts:
+  - name: "High CPU Usage"
+    condition: "cpu_percent > 90"
+    severity: "WARNING"
+    cooldown: 300
+    
+  - name: "High Memory Usage"
+    condition: "memory_percent > 90"
+    severity: "WARNING"
+    cooldown: 300
+    
+  - name: "High Disk Usage"
+    condition: "disk_percent > 90"
+    severity: "CRITICAL"
+    cooldown: 60
+    
+  - name: "Service Down"
+    condition: "service_status != 'running'"
+    severity: "CRITICAL"
+    cooldown: 0
+    
+  - name: "Build Queue Full"
+    condition: "queue_size > 100"
+    severity: "WARNING"
+    cooldown: 600
+```
+
+## Conclusion
+
+This deployment guide provides a comprehensive approach to deploying Debian Forge in production. Key points to remember:
+
+1. **Security First**: Always configure firewalls, fail2ban, and SSL certificates
+2. **Monitoring**: Implement comprehensive monitoring and alerting
+3. **Backups**: Regular automated backups with tested recovery procedures
+4. **Maintenance**: Scheduled maintenance windows for updates and cleanup
+5. **Scaling**: Plan for both horizontal and vertical scaling from the start
+6. **Documentation**: Keep deployment and maintenance procedures up to date
+
+For additional support, refer to the troubleshooting section or create an issue in the project repository.
diff --git a/debian-forge-docs/flowchart-debian.md b/debian-forge-docs/flowchart-debian.md
new file mode 100755
index 00000000..8cb4473d
--- /dev/null
+++ b/debian-forge-docs/flowchart-debian.md
@@ -0,0 +1,276 @@
+# Debian Atomic Ecosystem: Top-Down Linear Flow
+
+## Overview
+This flowchart shows the proposed Debian Atomic ecosystem, mapping Fedora bootc components to Debian equivalents and discussing implementation options.
+
+## The Proposed Debian Flow
+
+```mermaid
+graph TD
+    %% Start: Package Sources
+    A[Debian Package Sources] --> B[Debian Build Infrastructure]
+    B --> C[Build Environments]
+    C --> D[Debian Packages in Repositories]
+    
+    %% Configuration Management
+    D --> E[debian-atomic-config Repository]
+    E --> F[Treefiles (YAML)]
+    F --> G[Package Groups & Variants]
+    G --> H[Debian Package Group Sync]
+    
+    %% Build Process (with apt-ostree)
+    D --> I[apt-ostree compose tree]
+    F --> I
+    I --> J[OSTree Repository with Commits]
+    J --> J1[ostree-importer to official repo]
+    J1 --> K[apt-ostree compose container-encapsulate]
+    K --> L[Special bootc containers]
+    
+    %% Base Images Created
+    L --> M[quay.io/debian/debian-bootc:13]
+    L --> N[Debian Silverblue]
+    L --> O[Debian Kinoite]
+    L --> P[Other Debian Atomic variants]
+    
+    %% User Customization Paths
+    M --> Q[Users build FROM base images]
+    Q --> R[Custom bootc containers]
+    
+    %% Path 1: Direct Containerfile
+    Q --> S[Write Containerfile directly]
+    S --> T[Container-First build path]
+    
+    %% Path 2: BlueBuild equivalent
+    Q --> U[Write recipe.yml]
+    U --> V[Debian BlueBuild generates Containerfile]
+    V --> T
+    
+    %% Debian BlueBuild processing
+    U --> W[Debian BlueBuild CLI processes recipe.yml]
+    W --> X[Generates Containerfile automatically]
+    X --> T
+    T --> Y[Custom Debian Atomic Images]
+    Y --> Z[ISO, QCOW2, RAW formats]
+    
+    %% Production conversion tools
+    R --> AA[deb-bootc-image-builder]
+    M --> AA
+    AA --> BB[Standard bootable images]
+    
+    R --> CC[osbuilder/osbuild for Debian]
+    M --> CC
+    CC --> DD[Advanced custom images]
+    CC --> DD1[OSTree composition]
+    CC --> DD2[Package management]
+    CC --> DD3[Complex build orchestration]
+    
+    %% Traditional Debian path
+    D --> EE[Debian Installer System]
+    EE --> FF[Traditional Debian ISOs]
+    
+    %% User experience limitations
+    Y --> GG[Steep Learning Curve]
+    Y --> HH[Compatibility Gaps]
+    Y --> II[Slow apt-ostree Operations]
+    Y --> JJ[Hardware/Driver Issues]
+    
+    %% User lifecycle operations
+    M --> KK[apt-ostree upgrade]
+    M --> LL[apt-ostree rollback]
+    M --> MM[apt-ostree rebase]
+    
+    %% These affect the user experience
+    KK --> NN[System updates]
+    LL --> OO[System rollbacks]
+    MM --> PP[System rebasing]
+```
+
+## Key Components for Debian Atomic
+
+### 1. Package Infrastructure
+- **Input**: Debian Package Sources
+- **Output**: Debian Packages in Repositories
+- **Tools**: Debian Build Infrastructure, Build Environments
+- **Options**: 
+  - Use existing Debian build infrastructure
+  - Build custom build system
+  - Integrate with existing Debian CI/CD
+
+### 2. Configuration Management
+- **Input**: Debian Packages + Debian package groups
+- **Output**: Treefiles (YAML) defining variants
+- **Tools**: debian-atomic-config repository, Debian Package Group Sync
+- **Options**:
+  - Create new debian-atomic-config repository
+  - Extend existing Debian configuration tools
+  - Use Debian's existing package grouping system
+
+### 3. Build Process
+- **Input**: Debian Packages + Treefiles
+- **Output**: Special bootc containers
+- **Tools**: apt-ostree compose tree → ostree-importer → apt-ostree compose container-encapsulate
+- **Options**:
+  - Implement apt-ostree from scratch
+  - Fork and adapt rpm-ostree for Debian
+  - Use existing Debian package management tools
+
+### 4. Base Images
+- **Input**: Special bootc containers
+- **Output**: Official Debian bootc containers and variants
+- **Examples**: quay.io/debian/debian-bootc:13, Debian Silverblue, Kinoite
+- **Options**:
+  - Build Debian-specific variants
+  - Adapt existing Debian live images
+  - Create minimal Debian base
+
+### 5. User Customization
+- **Input**: Base bootc images
+- **Output**: Custom bootc containers
+- **Paths**: 
+  - Direct: Containerfile → Custom container
+  - Debian BlueBuild: recipe.yml → Debian BlueBuild → Containerfile → Custom container
+- **Options**:
+  - Adapt BlueBuild for Debian
+  - Create Debian-specific customization tools
+  - Use existing Debian image building tools
+
+### 6. Production Tools
+- **Input**: Custom bootc containers
+- **Output**: Bootable disk images
+- **Tools**: 
+  - deb-bootc-image-builder: Standard bootable images
+  - osbuilder/osbuild for Debian: Advanced customization, OSTree composition, package management, complex build orchestration
+- **Options**:
+  - Adapt existing Debian image building tools
+  - Extend osbuilder for Debian
+  - Create Debian-specific tools
+
+### 7. Traditional Debian
+- **Input**: Debian Packages
+- **Output**: Traditional Debian ISOs
+- **Tools**: Debian Installer System
+- **Options**:
+  - Use existing Debian installer infrastructure
+  - Create new installer for atomic variants
+  - Integrate with existing Debian tools
+
+### 8. User Limitations
+- **Input**: Custom Debian Atomic Images
+- **Output**: User experience challenges
+- **Issues**: Learning curve, compatibility gaps, slow operations, hardware limitations
+- **Options**:
+  - Address known Fedora limitations
+  - Leverage Debian's strengths
+  - Create better user experience
+
+### 9. User Lifecycle Operations
+- **Input**: Base bootc images
+- **Output**: System management operations
+- **Operations**: apt-ostree upgrade, rollback, rebase
+- **Purpose**: Fundamental to the "atomic" nature of the system
+- **Options**:
+  - Implement apt-ostree commands
+  - Adapt existing Debian package management
+  - Create new atomic management tools
+
+## Implementation Options Discussion
+
+### **apt-ostree Implementation Options**
+
+#### Option 1: Build from Scratch
+- **Pros**: Complete control, Debian-specific optimizations
+- **Cons**: Massive development effort, need to reimplement all functionality
+- **Timeline**: 2-3 years minimum
+
+#### Option 2: Fork and Adapt rpm-ostree
+- **Pros**: Leverage existing codebase, proven functionality
+- **Cons**: RPM-specific assumptions, significant adaptation needed
+- **Timeline**: 1-2 years
+
+#### Option 3: Debian Package Manager Integration
+- **Pros**: Use existing Debian tools, familiar to Debian developers
+- **Cons**: May not provide atomic guarantees, significant architectural changes
+- **Timeline**: 1-2 years
+
+### **Build Infrastructure Options**
+
+#### Option 1: Use Existing Debian Infrastructure
+- **Pros**: Proven, maintained, integrated
+- **Cons**: May not support OSTree workflows, limited customization
+- **Timeline**: 6 months - 1 year
+
+#### Option 2: Build Custom Infrastructure
+- **Pros**: Complete control, optimized for atomic workflows
+- **Cons**: Massive development effort, maintenance burden
+- **Timeline**: 2-3 years
+
+#### Option 3: Hybrid Approach
+- **Pros**: Leverage existing tools where possible, custom where needed
+- **Cons**: Integration complexity, potential conflicts
+- **Timeline**: 1-2 years
+
+### **Configuration Management Options**
+
+#### Option 1: New debian-atomic-config Repository
+- **Pros**: Clean slate, atomic-specific design
+- **Cons**: New tool to maintain, separate from existing Debian tools
+- **Timeline**: 6 months - 1 year
+
+#### Option 2: Extend Existing Debian Tools
+- **Pros**: Integration with existing workflow, familiar to developers
+- **Cons**: May not support atomic concepts well, architectural limitations
+- **Timeline**: 1 year
+
+#### Option 3: Adapt Fedora's workstation-ostree-config
+- **Pros**: Proven approach, existing examples
+- **Cons**: RPM-specific assumptions, significant adaptation needed
+- **Timeline**: 6 months - 1 year
+
+## What Each Step Produces (Debian Context)
+
+| Step | Input | Output | Tool | Implementation Status |
+|------|-------|--------|------|----------------------|
+| Package Sources | Source code | Debian Packages | Debian Build Infrastructure | ✅ Exists |
+| Configuration | Packages + package groups | Treefiles (YAML) | debian-atomic-config | ❌ Needs Implementation |
+| Build Process | Packages + Treefiles | OSTree commits | apt-ostree compose tree | ❌ Needs Implementation |
+| Import | OSTree commits | Official repo | ostree-importer | ❌ Needs Implementation |
+| Encapsulate | OSTree commits | bootc containers | apt-ostree compose container-encapsulate | ❌ Needs Implementation |
+| Base Images | bootc containers | Official variants | Container registry | ❌ Needs Implementation |
+| User Customization | Base images | Custom containers | Containerfile or Debian BlueBuild | ❌ Needs Implementation |
+| Production Tools | Custom containers | Bootable images | deb-bootc-image-builder/osbuilder | ❌ Needs Implementation |
+| Traditional Debian | Packages | ISOs | Debian Installer System | ✅ Exists |
+| User Lifecycle | Base images | System operations | apt-ostree commands | ❌ Needs Implementation |
+
+## Implementation Priority
+
+### **Phase 1: Core Infrastructure (6-12 months)**
+1. **apt-ostree implementation** - Core tool for OSTree management
+2. **debian-atomic-config repository** - Configuration management
+3. **Basic build pipeline** - From packages to OSTree commits
+
+### **Phase 2: Bootc Integration (6-12 months)**
+1. **apt-ostree compose container-encapsulate** - Create bootc containers
+2. **Base Debian bootc images** - Official variants
+3. **Basic user customization** - Containerfile support
+
+### **Phase 3: User Tools (6-12 months)**
+1. **Debian BlueBuild equivalent** - User-friendly customization
+2. **deb-bootc-image-builder** - Production image creation
+3. **User lifecycle management** - upgrade, rollback, rebase
+
+### **Phase 4: Advanced Features (6-12 months)**
+1. **osbuilder/osbuild integration** - Advanced customization
+2. **Performance optimizations** - Address known limitations
+3. **Production deployment** - Real-world testing and refinement
+
+## What This Shows
+
+This flowchart demonstrates that Debian Atomic requires:
+- Complete package infrastructure (mostly exists)
+- Configuration management system (needs implementation)
+- Specialized build tools (apt-ostree needs implementation)
+- User abstraction layers (needs implementation)
+- Multiple production conversion options (needs implementation)
+- User lifecycle management (needs implementation)
+
+The main challenge is implementing the **apt-ostree ecosystem** and **bootc integration**, as most of the existing Debian infrastructure can be leveraged or adapted.
diff --git a/debian-forge-docs/flowchart.md b/debian-forge-docs/flowchart.md
new file mode 100755
index 00000000..e1394316
--- /dev/null
+++ b/debian-forge-docs/flowchart.md
@@ -0,0 +1,165 @@
+# Fedora Bootc Ecosystem: Top-Down Linear Flow
+
+## Overview
+This flowchart shows the complete Fedora bootc ecosystem from package sources to user deployment, in a linear top-down progression.
+
+## The Complete Flow
+
+```mermaid
+graph TD
+    %% Start: Package Sources
+    A[Fedora Package Sources] --> B[Koji Build System]
+    B --> C[Mock Build Environments]
+    C --> D[RPM Packages in Koji]
+    
+    %% Configuration Management
+    D --> E[workstation-ostree-config Repository]
+    E --> F[Treefiles (YAML)]
+    F --> G[Package Groups & Variants]
+    G --> H[Fedora Comps Sync]
+    
+    %% Build Process (with ostree-importer step)
+    D --> I[rpm-ostree compose tree]
+    F --> I
+    I --> J[OSTree Repository with Commits]
+    J --> J1[ostree-importer to official repo]
+    J1 --> K[rpm-ostree compose container-encapsulate]
+    K --> L[Special bootc containers]
+    
+    %% Base Images Created
+    L --> M[quay.io/fedora/fedora-bootc:42]
+    L --> N[Fedora Silverblue]
+    L --> O[Fedora Kinoite]
+    L --> P[Other Atomic variants]
+    
+    %% User Customization Paths
+    M --> Q[Users build FROM base images]
+    Q --> R[Custom bootc containers]
+    
+    %% Path 1: Direct Containerfile
+    Q --> S[Write Containerfile directly]
+    S --> T[Container-First build path]
+    
+    %% Path 2: BlueBuild abstraction
+    Q --> U[Write recipe.yml]
+    U --> V[BlueBuild generates Containerfile]
+    V --> T
+    
+    %% BlueBuild processing
+    U --> W[BlueBuild CLI processes recipe.yml]
+    W --> X[Generates Containerfile automatically]
+    X --> T
+    T --> Y[Custom Fedora Atomic Images]
+    Y --> Z[ISO, QCOW2, RAW formats]
+    
+    %% Production conversion tools (clarified roles)
+    R --> AA[bootc-image-builder]
+    M --> AA
+    AA --> BB[Standard bootable images]
+    
+    R --> CC[osbuilder/osbuild]
+    M --> CC
+    CC --> DD[Advanced custom images]
+    CC --> DD1[OSTree composition]
+    CC --> DD2[Package management]
+    CC --> DD3[Complex build orchestration]
+    
+    %% Traditional Fedora path
+    D --> EE[Pungi Compose System]
+    EE --> FF[Traditional Fedora ISOs]
+    
+    %% User experience limitations
+    Y --> GG[Steep Learning Curve]
+    Y --> HH[Compatibility Gaps]
+    Y --> II[Slow rpm-ostree Operations]
+    Y --> JJ[Hardware/Driver Issues]
+    
+    %% User lifecycle operations
+    M --> KK[rpm-ostree upgrade]
+    M --> LL[rpm-ostree rollback]
+    M --> MM[rpm-ostree rebase]
+    
+    %% These affect the user experience
+    KK --> NN[System updates]
+    LL --> OO[System rollbacks]
+    MM --> PP[System rebasing]
+```
+
+## Key Components
+
+### 1. Package Infrastructure
+- **Input**: Fedora Package Sources
+- **Output**: RPM Packages in Koji repository
+- **Tools**: Koji Build System, Mock Build Environments
+
+### 2. Configuration Management
+- **Input**: RPM Packages + Fedora package groups
+- **Output**: Treefiles (YAML) defining variants
+- **Tools**: workstation-ostree-config repository, Fedora Comps Sync
+
+### 3. Build Process
+- **Input**: RPM Packages + Treefiles
+- **Output**: Special bootc containers
+- **Tools**: rpm-ostree compose tree → ostree-importer → rpm-ostree compose container-encapsulate
+
+### 4. Base Images
+- **Input**: Special bootc containers
+- **Output**: Official Fedora bootc containers and variants
+- **Examples**: quay.io/fedora/fedora-bootc:42, Fedora Silverblue, Kinoite
+
+### 5. User Customization
+- **Input**: Base bootc images
+- **Output**: Custom bootc containers
+- **Paths**: 
+  - Direct: Containerfile → Custom container
+  - BlueBuild: recipe.yml → BlueBuild → Containerfile → Custom container
+
+### 6. Production Tools
+- **Input**: Custom bootc containers
+- **Output**: Bootable disk images
+- **Tools**: 
+  - bootc-image-builder: Standard bootable images
+  - osbuilder/osbuild: Advanced customization, OSTree composition, package management, complex build orchestration
+
+### 7. Traditional Fedora
+- **Input**: RPM Packages
+- **Output**: Traditional Fedora ISOs
+- **Tools**: Pungi Compose System
+
+### 8. User Limitations
+- **Input**: Custom Fedora Atomic Images
+- **Output**: User experience challenges
+- **Issues**: Learning curve, compatibility gaps, slow operations, hardware limitations
+
+### 9. User Lifecycle Operations
+- **Input**: Base bootc images
+- **Output**: System management operations
+- **Operations**: rpm-ostree upgrade, rollback, rebase
+- **Purpose**: Fundamental to the "atomic" nature of the system
+
+## What Each Step Produces
+
+| Step | Input | Output | Tool |
+|------|-------|--------|------|
+| Package Sources | Source code | RPM Packages | Koji + Mock |
+| Configuration | RPMs + package groups | Treefiles (YAML) | workstation-ostree-config |
+| Build Process | RPMs + Treefiles | OSTree commits | rpm-ostree compose tree |
+| Import | OSTree commits | Official repo | ostree-importer |
+| Encapsulate | OSTree commits | bootc containers | rpm-ostree compose container-encapsulate |
+| Base Images | bootc containers | Official variants | Container registry |
+| User Customization | Base images | Custom containers | Containerfile or BlueBuild |
+| Production Tools | Custom containers | Bootable images | bootc-image-builder/osbuilder |
+| Traditional Fedora | RPMs | ISOs | Pungi |
+| User Lifecycle | Base images | System operations | rpm-ostree commands |
+
+## What This Shows
+
+This flowchart demonstrates that Fedora bootc requires:
+- Complete package infrastructure
+- Configuration management system
+- Specialized build tools
+- User abstraction layers
+- Multiple production conversion options
+- User lifecycle management
+
+For Debian Atomic to work similarly, it needs equivalent infrastructure and tools.
diff --git a/debian-forge-docs/osbuild-architecture.md b/debian-forge-docs/osbuild-architecture.md
new file mode 100644
index 00000000..33f24c90
--- /dev/null
+++ b/debian-forge-docs/osbuild-architecture.md
@@ -0,0 +1,136 @@
+# OSBuild Architecture for Debian Forge
+
+## Overview
+This document details OSBuild's core architecture patterns that Debian Forge must maintain for 1:1 compatibility while extending the system for Debian atomic builds.
+
+## Core OSBuild Components
+
+### 1. Pipeline System
+- **Stages**: Modular build steps (e.g., `org.osbuild.rpm`, `org.osbuild.ostree`)
+- **Devices**: Block device management (`org.osbuild.loopback`, `org.osbuild.luks2`)
+- **Mounts**: Filesystem mounting (`org.osbuild.bind`, `org.osbuild.ext4`)
+- **Inputs**: Source management (`org.osbuild.containers`, `org.osbuild.files`)
+- **Assemblers**: Output format generation (`org.osbuild.qemu`, `org.osbuild.tar`)
+
+### 2. Build Environment Management
+- **Buildroot Isolation**: Uses `bubblewrap` for sandboxed build environments
+- **Object Store**: `FsCache` for caching downloaded sources and intermediate artifacts
+- **API Communication**: JSON-based inter-process communication via Unix sockets
+
+### 3. OSBuild Composer Integration
+- **Web Service Layer**: HTTP APIs for image composition and build orchestration
+- **Blueprint System**: High-level image configuration definitions
+- **Build Orchestration**: Translates user requests into OSBuild pipeline descriptions
+- **Multiple APIs**: Supports lorax-composer API and custom endpoints
+
+## Debian Forge Extensions
+
+### 1. Debian-Specific Stages
+- `org.osbuild.debootstrap`: Base Debian filesystem creation
+- `org.osbuild.apt`: Debian package management
+- `org.osbuild.apt.config`: APT configuration and sources
+- `org.osbuild.ostree.commit`: OSTree commit creation for Debian
+- `org.osbuild.ostree.deploy`: OSTree deployment for Debian
+- `org.osbuild.sbuild`: Debian package building in chroot
+- `org.osbuild.debian.source`: Debian source package management
+
+### 2. Build Orchestration System
+- **Resource Management**: CPU, memory, and storage allocation
+- **Build Queue**: Priority-based build scheduling
+- **State Machine**: Build lifecycle management (PENDING → QUEUED → BUILDING → COMPLETED)
+- **Artifact Management**: SQLite-based metadata and filesystem organization
+- **Environment Management**: Isolated build environment creation and cleanup
+
+### 3. Integration Points
+- **OSBuild Pipeline Execution**: Direct integration with OSBuild core
+- **Composer API Compatibility**: Extend composer for Debian atomic images
+- **Debos Logic Integration**: Adapt debos actions to OSBuild stages
+
+## Compatibility Requirements
+
+### 1:1 OSBuild Compatibility
+- **Stage Interface**: Maintain identical stage input/output contracts
+- **Pipeline Format**: Use standard OSBuild manifest format
+- **Build Environment**: Preserve bubblewrap isolation and object store patterns
+- **API Patterns**: Follow existing OSBuild API communication patterns
+
+### Debian-Specific Extensions
+- **Package Management**: Replace RPM with APT/dpkg
+- **Build Tools**: Replace Mock with sbuild/pbuilder
+- **Atomic System**: Add OSTree composition and deployment
+- **Web Interface**: Extend composer for Debian atomic workflows
+
+## Architecture Patterns
+
+### 1. Stage Pattern
+```python
+def run_stage(tree, options):
+    # Stage implementation
+    # Must follow OSBuild stage contract
+    # Return metadata via api.metadata
+```
+
+### 2. Build Environment Pattern
+```python
+# Use bubblewrap for isolation
+# Mount essential filesystems (/dev, /sys, /proc)
+# Follow OSBuild buildroot patterns
+```
+
+### 3. Object Store Pattern
+```python
+# Use FsCache for source and artifact caching
+# Maintain OSBuild cache structure
+# Extend for Debian-specific artifacts
+```
+
+## Integration Strategy
+
+### Phase 1: Core Compatibility
+- Maintain OSBuild stage interface contracts
+- Preserve build environment isolation patterns
+- Keep object store and caching mechanisms
+
+### Phase 2: Debian Extensions
+- Add Debian-specific stages following OSBuild patterns
+- Extend build orchestration for Debian workflows
+- Integrate debos logic into OSBuild architecture
+
+### Phase 3: Composer Integration
+- Test osbuild-composer with Debian stages
+- Extend blueprint system for Debian atomic images
+- Maintain API compatibility with existing composer ecosystem
+
+## Testing Strategy
+
+### Compatibility Testing
+- Validate all Debian stages follow OSBuild patterns
+- Test build environment isolation and object store
+- Verify pipeline execution compatibility
+
+### Integration Testing
+- Test Debian stages in OSBuild pipelines
+- Validate build orchestration with OSBuild core
+- Test composer integration with Debian workflows
+
+### Performance Testing
+- Benchmark Debian stages against OSBuild baselines
+- Test resource allocation and build queue performance
+- Validate stress testing under concurrent load
+
+## Future Considerations
+
+### OSBuild Updates
+- Monitor OSBuild core changes for compatibility
+- Plan for OSBuild version upgrades
+- Maintain compatibility with new OSBuild features
+
+### Composer Evolution
+- Track osbuild-composer development
+- Plan integration with new composer features
+- Maintain API compatibility across versions
+
+### Debian Ecosystem
+- Monitor Debian package management evolution
+- Plan for new Debian build tools and standards
+- Maintain compatibility with Debian release cycles
diff --git a/debian-forge-docs/troubleshooting-guides.md b/debian-forge-docs/troubleshooting-guides.md
new file mode 100644
index 00000000..73f73710
--- /dev/null
+++ b/debian-forge-docs/troubleshooting-guides.md
@@ -0,0 +1,873 @@
+# Debian Forge Troubleshooting Guides
+
+## Overview
+This document provides comprehensive troubleshooting information for Debian Forge, including common issues, diagnostic procedures, and step-by-step solutions.
+
+## Quick Diagnostic Commands
+
+### System Health Check
+```bash
+# Check system resources
+htop
+df -h
+free -h
+iostat -x 1
+
+# Check service status
+sudo systemctl status debian-forge-*
+sudo supervisorctl status
+
+# Check logs
+tail -f /var/log/debian-forge/*.log
+journalctl -u debian-forge-* -f
+
+# Check network
+ping -c 3 8.8.8.8
+curl -I http://deb.debian.org/debian
+```
+
+### Debian Forge Status Check
+```bash
+# Check build queue
+curl -s http://localhost:8080/api/v1/queue/status | jq
+
+# Check active builds
+curl -s http://localhost:8080/api/v1/builds/active | jq
+
+# Check system health
+curl -s http://localhost:8080/api/v1/health | jq
+
+# Check OSTree repository
+ostree refs --repo=/var/lib/debian-forge/ostree
+ostree log --repo=/var/lib/debian-forge/ostree debian/bookworm/amd64
+```
+
+## Common Issues and Solutions
+
+### 1. Build Failures
+
+#### Issue: Build Process Hangs
+**Symptoms**: Build appears to be running but no progress for extended periods
+**Diagnosis**:
+```bash
+# Check build process
+ps aux | grep osbuild
+ps aux | grep debootstrap
+
+# Check system resources
+htop
+df -h /tmp
+df -h /var/lib/debian-forge
+
+# Check build logs
+tail -f /var/log/debian-forge/worker.log
+```
+
+**Solutions**:
+```bash
+# Kill hanging processes
+sudo pkill -f osbuild
+sudo pkill -f debootstrap
+
+# Clean temporary files
+sudo rm -rf /tmp/osbuild-*
+sudo rm -rf /var/lib/debian-forge/tmp/*
+
+# Restart worker service
+sudo supervisorctl restart debian-forge-worker
+
+# Check for disk space issues
+sudo du -sh /var/lib/debian-forge/* | sort -hr
+```
+
+#### Issue: Package Installation Failures
+**Symptoms**: Build fails during package installation with APT errors
+**Diagnosis**:
+```bash
+# Check APT configuration
+cat /etc/apt/sources.list
+ls -la /etc/apt/sources.list.d/
+
+# Test package availability
+apt-cache policy package-name
+apt-cache search package-name
+
+# Check network connectivity
+curl -I http://deb.debian.org/debian
+ping -c 3 security.debian.org
+```
+
+**Solutions**:
+```bash
+# Update package lists
+sudo apt update
+
+# Fix broken packages
+sudo apt --fix-broken install
+
+# Clear APT cache
+sudo apt clean
+sudo apt autoclean
+
+# Check for repository issues
+sudo apt update 2>&1 | grep -i error
+
+# Verify GPG keys
+sudo apt-key list
+sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys KEY_ID
+```
+
+#### Issue: OSTree Commit Failures
+**Symptoms**: Build fails during OSTree commit creation
+**Diagnosis**:
+```bash
+# Check OSTree repository
+ostree refs --repo=/var/lib/debian-forge/ostree
+ostree log --repo=/var/lib/debian-forge/ostree
+
+# Check repository permissions
+ls -la /var/lib/debian-forge/ostree/
+id debian-forge
+
+# Check disk space
+df -h /var/lib/debian-forge/ostree
+```
+
+**Solutions**:
+```bash
+# Fix repository permissions
+sudo chown -R debian-forge:debian-forge /var/lib/debian-forge/ostree
+sudo chmod -R 755 /var/lib/debian-forge/ostree
+
+# Initialize repository if corrupted
+sudo -u debian-forge ostree init --mode=archive-z2 /var/lib/debian-forge/ostree
+
+# Clean old commits
+sudo -u debian-forge ostree refs --repo=/var/lib/debian-forge/ostree | xargs -I {} ostree delete --repo=/var/lib/debian-forge/ostree {}
+
+# Check for corrupted objects
+sudo -u debian-forge ostree fsck --repo=/var/lib/debian-forge/ostree
+```
+
+### 2. Service Issues
+
+#### Issue: API Service Not Responding
+**Symptoms**: HTTP requests to API endpoints timeout or return errors
+**Diagnosis**:
+```bash
+# Check service status
+sudo supervisorctl status debian-forge-api
+sudo systemctl status nginx
+
+# Check port binding
+sudo netstat -tlnp | grep :8080
+sudo ss -tlnp | grep :8080
+
+# Check firewall
+sudo ufw status
+sudo iptables -L
+
+# Test local connectivity
+curl -v http://localhost:8080/health
+```
+
+**Solutions**:
+```bash
+# Restart API service
+sudo supervisorctl restart debian-forge-api
+
+# Check configuration
+sudo cat /etc/supervisor/conf.d/debian-forge.conf
+sudo cat /etc/nginx/sites-available/debian-forge
+
+# Verify Python environment
+sudo -u debian-forge /home/debian-forge/debian-forge/venv/bin/python -c "import flask; print('OK')"
+
+# Check logs for errors
+sudo tail -f /var/log/debian-forge/api.log
+sudo tail -f /var/log/nginx/error.log
+```
+
+#### Issue: Worker Service Not Processing Builds
+**Symptoms**: Builds remain in QUEUED status indefinitely
+**Diagnosis**:
+```bash
+# Check worker status
+sudo supervisorctl status debian-forge-worker
+
+# Check worker logs
+sudo tail -f /var/log/debian-forge/worker.log
+
+# Check build queue
+curl -s http://localhost:8080/api/v1/queue/status | jq
+
+# Check system resources
+htop
+df -h
+free -h
+```
+
+**Solutions**:
+```bash
+# Restart worker service
+sudo supervisorctl restart debian-forge-worker
+
+# Check Python dependencies
+sudo -u debian-forge /home/debian-forge/debian-forge/venv/bin/pip list
+
+# Verify build environment
+sudo -u debian-forge /home/debian-forge/debian-forge/venv/bin/python -c "import build_orchestrator; print('OK')"
+
+# Check for resource constraints
+sudo -u debian-forge /home/debian-forge/debian-forge/venv/bin/python test-resource-allocation.py
+```
+
+### 3. Resource Issues
+
+#### Issue: High CPU Usage
+**Symptoms**: System becomes unresponsive, builds slow down
+**Diagnosis**:
+```bash
+# Check CPU usage
+htop
+top -p 1
+iostat -x 1
+
+# Identify high-CPU processes
+ps aux --sort=-%cpu | head -10
+
+# Check build processes
+ps aux | grep osbuild
+ps aux | grep debootstrap
+```
+
+**Solutions**:
+```bash
+# Reduce concurrent builds
+sudo -u debian-forge /home/debian-forge/debian-forge/venv/bin/python -c "
+from build_orchestrator import BuildOrchestrator
+o = BuildOrchestrator()
+o.resource_manager.set_concurrent_builds(2)
+"
+
+# Kill runaway processes
+sudo pkill -f osbuild
+sudo pkill -f debootstrap
+
+# Check for infinite loops
+sudo tail -f /var/log/debian-forge/worker.log | grep -i "cpu\|loop"
+
+# Restart services
+sudo supervisorctl restart all
+```
+
+#### Issue: High Memory Usage
+**Symptoms**: Out of memory errors, system swapping
+**Diagnosis**:
+```bash
+# Check memory usage
+free -h
+cat /proc/meminfo | grep -E "(MemTotal|MemFree|MemAvailable|SwapTotal|SwapFree)"
+
+# Check for memory leaks
+ps aux --sort=-%mem | head -10
+
+# Check swap usage
+swapon --show
+cat /proc/swaps
+```
+
+**Solutions**:
+```bash
+# Clear memory caches
+sudo sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches
+
+# Restart memory-intensive services
+sudo supervisorctl restart debian-forge-worker
+
+# Check for memory leaks in logs
+sudo tail -f /var/log/debian-forge/worker.log | grep -i "memory\|leak"
+
+# Reduce memory limits
+sudo -u debian-forge /home/debian-forge/debian-forge/venv/bin/python -c "
+from build_orchestrator import BuildOrchestrator
+o = BuildOrchestrator()
+o.resource_manager.set_build_limits(max_memory=70)
+"
+```
+
+#### Issue: Disk Space Exhaustion
+**Symptoms**: Builds fail with "no space left on device" errors
+**Diagnosis**:
+```bash
+# Check disk usage
+df -h
+du -sh /var/lib/debian-forge/*
+du -sh /tmp/*
+du -sh .osbuild/
+
+# Check for large files
+find /var/lib/debian-forge -type f -size +100M -exec ls -lh {} \;
+find /tmp -type f -size +100M -exec ls -lh {} \;
+
+# Check inode usage
+df -i
+```
+
+**Solutions**:
+```bash
+# Clean build artifacts
+sudo -u debian-forge /home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager --force
+
+# Clean temporary files
+sudo rm -rf /tmp/osbuild-*
+sudo rm -rf /var/lib/debian-forge/tmp/*
+
+# Clean old OSTree commits
+sudo -u debian-forge ostree refs --repo=/var/lib/debian-forge/ostree | xargs -I {} ostree delete --repo=/var/lib/debian-forge/ostree {}
+
+# Clean package cache
+sudo apt clean
+sudo apt autoclean
+
+# Check for log rotation
+sudo logrotate -f /etc/logrotate.d/debian-forge
+```
+
+### 4. Network Issues
+
+#### Issue: Package Download Failures
+**Symptoms**: Builds fail when downloading packages from repositories
+**Diagnosis**:
+```bash
+# Test network connectivity
+ping -c 3 8.8.8.8
+ping -c 3 deb.debian.org
+ping -c 3 security.debian.org
+
+# Check DNS resolution
+nslookup deb.debian.org
+dig deb.debian.org
+
+# Test HTTP connectivity
+curl -I http://deb.debian.org/debian
+curl -I https://security.debian.org/debian-security
+
+# Check proxy configuration
+echo $http_proxy
+echo $https_proxy
+cat /etc/apt/apt.conf.d/*proxy*
+```
+
+**Solutions**:
+```bash
+# Fix DNS issues
+echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
+echo "nameserver 8.8.4.4" | sudo tee -a /etc/resolv.conf
+
+# Test without proxy
+unset http_proxy https_proxy
+sudo apt update
+
+# Check firewall rules
+sudo ufw status
+sudo iptables -L
+
+# Verify repository URLs
+sudo apt update 2>&1 | grep -i "failed\|error\|unreachable"
+```
+
+#### Issue: apt-cacher-ng Connection Problems
+**Symptoms**: Builds fail with proxy connection errors
+**Diagnosis**:
+```bash
+# Check apt-cacher-ng status
+sudo systemctl status apt-cacher-ng
+sudo netstat -tlnp | grep :3142
+
+# Test proxy connectivity
+curl -I http://192.168.1.101:3142
+telnet 192.168.1.101 3142
+
+# Check proxy configuration
+cat /etc/apt/apt.conf.d/99proxy
+```
+
+**Solutions**:
+```bash
+# Restart apt-cacher-ng
+sudo systemctl restart apt-cacher-ng
+
+# Verify proxy configuration
+echo 'Acquire::http::Proxy "http://192.168.1.101:3142";' | sudo tee /etc/apt/apt.conf.d/99proxy
+echo 'Acquire::https::Proxy "http://192.168.1.101:3142";' | sudo tee -a /etc/apt/apt.conf.d/99proxy
+
+# Test proxy
+curl -x http://192.168.1.101:3142 http://deb.debian.org/debian
+
+# Check proxy logs
+sudo tail -f /var/log/apt-cacher-ng/apt-cacher.log
+```
+
+### 5. Configuration Issues
+
+#### Issue: Invalid Manifest Format
+**Symptoms**: Builds fail with "invalid manifest" errors
+**Diagnosis**:
+```bash
+# Validate manifest syntax
+python3 -m osbuild --libdir . --check-only manifest.json
+
+# Check JSON syntax
+python3 -c "import json; json.load(open('manifest.json')); print('Valid JSON')"
+
+# Check manifest schema
+python3 -c "
+import json
+schema = json.load(open('schemas/osbuild2.json'))
+manifest = json.load(open('manifest.json'))
+print('Schema validation needed')
+"
+```
+
+**Solutions**:
+```bash
+# Fix JSON syntax
+python3 -m json.tool manifest.json > manifest_fixed.json
+mv manifest_fixed.json manifest.json
+
+# Validate against schema
+python3 -c "
+import json
+from jsonschema import validate
+schema = json.load(open('schemas/osbuild2.json'))
+manifest = json.load(open('manifest.json'))
+validate(instance=manifest, schema=schema)
+print('Valid manifest')
+"
+
+# Check stage names
+python3 -c "
+import json
+manifest = json.load(open('manifest.json'))
+stages = manifest.get('pipeline', {}).get('stages', [])
+for stage in stages:
+    print(f'Stage: {stage.get(\"name\", \"unknown\")}')
+"
+```
+
+#### Issue: Missing Dependencies
+**Symptoms**: Builds fail with "command not found" or import errors
+**Diagnosis**:
+```bash
+# Check Python dependencies
+pip list
+pip check
+
+# Check system packages
+which debootstrap
+which ostree
+which sbuild
+
+# Check Python path
+python3 -c "import sys; print('\n'.join(sys.path))"
+python3 -c "import build_orchestrator; print('OK')"
+```
+
+**Solutions**:
+```bash
+# Install missing Python packages
+pip install -r requirements.txt
+
+# Install missing system packages
+sudo apt update
+sudo apt install -y debootstrap ostree sbuild pbuilder
+
+# Fix Python path
+export PYTHONPATH="${PYTHONPATH}:/home/debian-forge/debian-forge"
+echo 'export PYTHONPATH="${PYTHONPATH}:/home/debian-forge/debian-forge"' >> ~/.bashrc
+
+# Check virtual environment
+source venv/bin/activate
+pip list
+```
+
+## Advanced Troubleshooting
+
+### 1. Performance Analysis
+
+#### Build Performance Profiling
+```bash
+# Profile build execution
+python3 -m cProfile -o build_profile.prof test-complete-pipeline.py
+
+# Analyze profile results
+python3 -c "
+import pstats
+p = pstats.Stats('build_profile.prof')
+p.sort_stats('cumulative')
+p.print_stats(20)
+"
+
+# Memory profiling
+python3 -m memory_profiler test-complete-pipeline.py
+```
+
+#### System Performance Monitoring
+```bash
+# Monitor system during build
+sar -u 1 60 > cpu_usage.log &
+sar -r 1 60 > memory_usage.log &
+sar -d 1 60 > disk_usage.log &
+
+# Start build
+python3 -m osbuild --libdir . manifest.json
+
+# Stop monitoring
+pkill sar
+
+# Analyze results
+python3 -c "
+import pandas as pd
+cpu = pd.read_csv('cpu_usage.log', sep='\s+', skiprows=2)
+print(f'Average CPU: {cpu[\"%user\"].mean():.1f}%')
+print(f'Peak CPU: {cpu[\"%user\"].max():.1f}%')
+"
+```
+
+### 2. Debug Mode
+
+#### Enable Debug Logging
+```bash
+# Set debug environment variables
+export OSBUILD_DEBUG=1
+export OSBUILD_VERBOSE=1
+export DEBIAN_FORGE_DEBUG=1
+
+# Run with debug output
+python3 -m osbuild --libdir . --verbose manifest.json
+
+# Check debug logs
+tail -f /var/log/debian-forge/debug.log
+```
+
+#### Python Debugger
+```bash
+# Add breakpoints in code
+import pdb; pdb.set_trace()
+
+# Run with debugger
+python3 -m pdb test-complete-pipeline.py
+
+# Common debugger commands
+# n (next), s (step), c (continue), p variable_name, l (list), q (quit)
+```
+
+### 3. Log Analysis
+
+#### Log Parsing and Analysis
+```bash
+# Extract error patterns
+grep -i "error\|fail\|exception" /var/log/debian-forge/*.log | head -20
+
+# Count error types
+grep -i "error" /var/log/debian-forge/*.log | cut -d: -f2 | sort | uniq -c | sort -nr
+
+# Extract build timing information
+grep "Build completed" /var/log/debian-forge/worker.log | awk '{print $1, $2, $NF}' | tail -10
+
+# Analyze resource usage patterns
+grep "Resource usage" /var/log/debian-forge/worker.log | tail -20
+```
+
+#### Log Correlation
+```bash
+# Correlate errors across services
+echo "=== API Errors ==="
+grep -i "error" /var/log/debian-forge/api.log | tail -5
+
+echo "=== Worker Errors ==="
+grep -i "error" /var/log/debian-forge/worker.log | tail -5
+
+echo "=== System Errors ==="
+journalctl -u debian-forge-* --since "1 hour ago" | grep -i "error" | tail -5
+```
+
+## Recovery Procedures
+
+### 1. Service Recovery
+
+#### Complete Service Restart
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/service-recovery.sh
+
+echo "Starting complete service recovery..."
+
+# Stop all services
+sudo supervisorctl stop all
+sudo systemctl stop nginx
+
+# Clean up temporary files
+sudo rm -rf /tmp/debian-forge-*
+sudo rm -rf /var/lib/debian-forge/tmp/*
+
+# Restart system services
+sudo systemctl start postgresql
+sudo systemctl start redis-server
+sudo systemctl start nginx
+
+# Wait for services to be ready
+sleep 10
+
+# Start application services
+sudo supervisorctl start all
+
+# Check status
+sudo supervisorctl status
+sudo systemctl status nginx
+
+echo "Service recovery completed"
+```
+
+#### Database Recovery
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/db-recovery.sh
+
+echo "Starting database recovery..."
+
+# Check database status
+sudo systemctl status postgresql
+
+# Test database connection
+sudo -u debian-forge psql -d debian_forge -c "SELECT version();"
+
+if [ $? -ne 0 ]; then
+    echo "Database connection failed, attempting recovery..."
+    
+    # Restart database
+    sudo systemctl restart postgresql
+    sleep 10
+    
+    # Test connection again
+    sudo -u debian-forge psql -d debian_forge -c "SELECT version();"
+    
+    if [ $? -eq 0 ]; then
+        echo "Database recovery successful"
+    else
+        echo "Database recovery failed"
+        exit 1
+    fi
+else
+    echo "Database is healthy"
+fi
+```
+
+### 2. Data Recovery
+
+#### Build Artifact Recovery
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/artifact-recovery.sh
+
+echo "Starting artifact recovery..."
+
+# Check for corrupted objects
+sudo -u debian-forge ostree fsck --repo=/var/lib/debian-forge/ostree
+
+# Remove corrupted objects
+sudo -u debian-forge ostree prune --repo=/var/lib/debian-forge/ostree --refs-only
+
+# Rebuild object index
+sudo -u debian-forge ostree summary --repo=/var/lib/debian-forge/ostree --update
+
+# Verify repository integrity
+sudo -u debian-forge ostree fsck --repo=/var/lib/debian-forge/ostree
+
+echo "Artifact recovery completed"
+```
+
+#### Configuration Recovery
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/config-recovery.sh
+
+BACKUP_DIR="/var/backups/debian-forge"
+LATEST_CONFIG=$(ls -t $BACKUP_DIR/config_*.tar.gz | head -1)
+
+if [ -n "$LATEST_CONFIG" ]; then
+    echo "Restoring configuration from: $LATEST_CONFIG"
+    
+    # Stop services
+    sudo supervisorctl stop all
+    
+    # Restore configuration
+    sudo tar -xzf $LATEST_CONFIG -C /
+    
+    # Fix permissions
+    sudo chown -R debian-forge:debian-forge /home/debian-forge/debian-forge/config
+    
+    # Restart services
+    sudo supervisorctl start all
+    
+    echo "Configuration recovery completed"
+else
+    echo "No configuration backup found"
+    exit 1
+fi
+```
+
+## Prevention Strategies
+
+### 1. Monitoring and Alerting
+
+#### Health Check Automation
+```bash
+#!/bin/bash
+# /etc/cron.daily/debian-forge-health-check
+
+# Run health checks
+/home/debian-forge/debian-forge/venv/bin/python /home/debian-forge/debian-forge/health_check.py
+
+# Check for critical issues
+if [ $? -ne 0 ]; then
+    # Send alert
+    /home/debian-forge/debian-forge/scripts/alert.py "Health check failed" "Critical system issue detected"
+    
+    # Attempt auto-recovery
+    /home/debian-forge/debian-forge/scripts/service-recovery.sh
+fi
+```
+
+#### Resource Monitoring
+```bash
+#!/bin/bash
+# /etc/cron.hourly/debian-forge-resource-check
+
+# Check disk space
+DISK_USAGE=$(df /var/lib/debian-forge | tail -1 | awk '{print $5}' | sed 's/%//')
+if [ $DISK_USAGE -gt 85 ]; then
+    /home/debian-forge/debian-forge/scripts/alert.py "High disk usage" "Disk usage is ${DISK_USAGE}%"
+    
+    # Trigger cleanup
+    /home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager --force
+fi
+
+# Check memory usage
+MEMORY_USAGE=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100.0}')
+if [ $MEMORY_USAGE -gt 90 ]; then
+    /home/debian-forge/debian-forge/scripts/alert.py "High memory usage" "Memory usage is ${MEMORY_USAGE}%"
+fi
+```
+
+### 2. Maintenance Windows
+
+#### Scheduled Maintenance
+```bash
+#!/bin/bash
+# /etc/cron.weekly/debian-forge-maintenance
+
+echo "Starting scheduled maintenance..."
+
+# Stop services
+sudo supervisorctl stop all
+
+# Update system packages
+sudo apt update && sudo apt upgrade -y
+
+# Clean old artifacts
+/home/debian-forge/debian-forge/venv/bin/python -m cleanup_manager --force
+
+# Rotate logs
+sudo logrotate -f /etc/logrotate.d/debian-forge
+
+# Restart services
+sudo supervisorctl start all
+
+echo "Scheduled maintenance completed"
+```
+
+## Getting Help
+
+### 1. Self-Service Resources
+
+#### Documentation
+- **User Guide**: `/home/debian-forge/debian-forge-docs/user-documentation.md`
+- **Deployment Guide**: `/home/debian-forge/debian-forge-docs/deployment-documentation.md`
+- **Architecture Guide**: `/home/debian-forge/debian-forge-docs/osbuild-architecture.md`
+
+#### Test Scripts
+```bash
+# Run diagnostic tests
+python3 test-apt-stage.py
+python3 test-resource-allocation.py
+python3 test-build-orchestration.py
+python3 test-complete-pipeline.py
+
+# Run performance tests
+python3 test-performance-optimization.py
+python3 test-stress-testing.py
+```
+
+### 2. Community Support
+
+#### Issue Reporting
+When reporting issues, include:
+- **System information**: OS version, Python version, installed packages
+- **Error messages**: Complete error logs and stack traces
+- **Reproduction steps**: Detailed steps to reproduce the issue
+- **Environment**: Development or production, configuration details
+- **Recent changes**: Any recent modifications to the system
+
+#### Debug Information Collection
+```bash
+#!/bin/bash
+# /home/debian-forge/debian-forge/scripts/collect-debug-info.sh
+
+DEBUG_DIR="/tmp/debian-forge-debug-$(date +%Y%m%d_%H%M%S)"
+mkdir -p $DEBUG_DIR
+
+echo "Collecting debug information..."
+
+# System information
+uname -a > $DEBUG_DIR/system-info.txt
+cat /etc/os-release > $DEBUG_DIR/os-release.txt
+python3 --version > $DEBUG_DIR/python-version.txt
+
+# Service status
+sudo supervisorctl status > $DEBUG_DIR/supervisor-status.txt
+sudo systemctl status debian-forge-* > $DEBUG_DIR/systemd-status.txt
+
+# Configuration files
+cp -r /home/debian-forge/debian-forge/config $DEBUG_DIR/
+cp /etc/supervisor/conf.d/debian-forge.conf $DEBUG_DIR/
+cp /etc/nginx/sites-available/debian-forge $DEBUG_DIR/
+
+# Logs (last 1000 lines)
+tail -1000 /var/log/debian-forge/*.log > $DEBUG_DIR/recent-logs.txt
+
+# Resource usage
+df -h > $DEBUG_DIR/disk-usage.txt
+free -h > $DEBUG_DIR/memory-usage.txt
+ps aux > $DEBUG_DIR/process-list.txt
+
+# Package information
+pip list > $DEBUG_DIR/python-packages.txt
+dpkg -l | grep -E "(debian-forge|osbuild|ostree)" > $DEBUG_DIR/system-packages.txt
+
+echo "Debug information collected in: $DEBUG_DIR"
+echo "Please include this directory when reporting issues"
+```
+
+## Conclusion
+
+This troubleshooting guide provides comprehensive information for diagnosing and resolving common Debian Forge issues. Key points:
+
+1. **Start with quick diagnostics** to identify the problem area
+2. **Use systematic troubleshooting** to isolate root causes
+3. **Implement recovery procedures** to restore service
+4. **Apply prevention strategies** to avoid future issues
+5. **Collect debug information** when seeking community help
+
+For additional support, refer to the project documentation or create detailed issue reports with the collected debug information.
diff --git a/debian-forge-docs/user-documentation.md b/debian-forge-docs/user-documentation.md
new file mode 100644
index 00000000..ee6ce83a
--- /dev/null
+++ b/debian-forge-docs/user-documentation.md
@@ -0,0 +1,524 @@
+# Debian Forge User Documentation
+
+## Overview
+Debian Forge is a Debian-focused fork of OSBuild that enables building Debian atomic images with OSTree support. It combines OSBuild's pipeline architecture with Debian package management and atomic system capabilities.
+
+## Installation
+
+### Prerequisites
+- **System**: Debian 12+ or Ubuntu 22.04+
+- **Python**: Python 3.9+
+- **Go**: Go 1.21+ (for debos integration)
+- **OSTree**: OSTree 2023.1+
+- **Build Tools**: sbuild, pbuilder, debootstrap, mmdebstrap
+
+### System Dependencies
+```bash
+# Install required packages
+sudo apt update
+sudo apt install -y \
+    python3-pip \
+    python3-venv \
+    ostree \
+    debootstrap \
+    mmdebstrap \
+    sbuild \
+    pbuilder \
+    bubblewrap \
+    qemu-utils \
+    curl \
+    skopeo \
+    git
+
+# Install Go
+wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
+sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
+export PATH=$PATH:/usr/local/go/bin
+```
+
+### Python Dependencies
+```bash
+# Create virtual environment
+python3 -m venv debian-forge-env
+source debian-forge-env/bin/activate
+
+# Install Python packages
+pip install -r requirements.txt
+```
+
+### Build Environment Setup
+```bash
+# Run setup script
+./setup-build-env.sh
+
+# Configure apt-cacher-ng (optional)
+./setup-apt-cacher.sh
+```
+
+## Basic Usage
+
+### 1. Simple Debian Base Image
+```bash
+# Build basic Debian image
+python3 -m osbuild --libdir . test-debian-manifest.json
+```
+
+**Manifest Example** (`test-debian-manifest.json`):
+```json
+{
+  "pipeline": {
+    "build": {
+      "pipeline": {
+        "stages": [
+          {
+            "name": "org.osbuild.debootstrap",
+            "options": {
+              "suite": "bookworm",
+              "mirror": "http://deb.debian.org/debian",
+              "arch": "amd64",
+              "apt_proxy": "http://192.168.1.101:3142"
+            }
+          },
+          {
+            "name": "org.osbuild.apt",
+            "options": {
+              "packages": ["systemd", "linux-image-amd64"],
+              "apt_proxy": "http://192.168.1.101:3142"
+            }
+          }
+        ]
+      }
+    }
+  }
+}
+```
+
+### 2. Debian Atomic Image
+```bash
+# Build Debian atomic image with OSTree
+python3 -m osbuild --libdir . test-debian-atomic-manifest.json
+```
+
+**Atomic Manifest Example** (`test-debian-atomic-manifest.json`):
+```json
+{
+  "pipelines": [
+    {
+      "name": "debian-base",
+      "stages": [
+        {
+          "name": "org.osbuild.debootstrap",
+          "options": {
+            "suite": "bookworm",
+            "mirror": "http://deb.debian.org/debian",
+            "arch": "amd64",
+            "apt_proxy": "http://192.168.1.101:3142"
+          }
+        },
+        {
+          "name": "org.osbuild.apt.config",
+          "options": {
+            "sources": {
+              "main": "deb http://deb.debian.org/debian bookworm main",
+              "security": "deb http://security.debian.org/debian-security bookworm-security main"
+            }
+          }
+        },
+        {
+          "name": "org.osbuild.apt",
+          "options": {
+            "packages": ["systemd", "linux-image-amd64", "ostree"],
+            "apt_proxy": "http://192.168.1.101:3142"
+          }
+        }
+      ]
+    },
+    {
+      "name": "ostree-commit",
+      "stages": [
+        {
+          "name": "org.osbuild.ostree.commit",
+          "options": {
+            "branch": "debian/bookworm/amd64",
+            "subject": "Debian Bookworm Atomic Base"
+          }
+        }
+      ]
+    }
+  ]
+}
+```
+
+## Advanced Usage
+
+### 1. Custom Package Building
+```bash
+# Build custom Debian package
+python3 -m osbuild --libdir . custom-package-manifest.json
+```
+
+**Package Build Manifest**:
+```json
+{
+  "pipeline": {
+    "build": {
+      "pipeline": {
+        "stages": [
+          {
+            "name": "org.osbuild.debootstrap",
+            "options": {
+              "suite": "bookworm",
+              "mirror": "http://deb.debian.org/debian",
+              "arch": "amd64"
+            }
+          },
+          {
+            "name": "org.osbuild.debian.source",
+            "options": {
+              "package": "my-package",
+              "version": "1.0.0"
+            }
+          },
+          {
+            "name": "org.osbuild.sbuild",
+            "options": {
+              "package": "my-package",
+              "arch": "amd64"
+            }
+          }
+        ]
+      }
+    }
+  }
+}
+```
+
+### 2. Multi-Architecture Builds
+```bash
+# Build for multiple architectures
+for arch in amd64 arm64; do
+  python3 -m osbuild --libdir . \
+    --options '{"arch": "'$arch'"}' \
+    multi-arch-manifest.json
+done
+```
+
+### 3. OSTree Repository Management
+```bash
+# Initialize OSTree repository
+ostree init --mode=archive-z2 /path/to/repo
+
+# List branches
+ostree refs --repo=/path/to/repo
+
+# Show commit history
+ostree log --repo=/path/to/repo debian/bookworm/amd64
+```
+
+## Build Orchestration
+
+### 1. Using Build Orchestrator
+```python
+from build_orchestrator import BuildOrchestrator
+
+# Initialize orchestrator
+orchestrator = BuildOrchestrator()
+
+# Submit build
+build_id = orchestrator.submit_build(
+    manifest_path="test-debian-atomic-manifest.json",
+    priority="high"
+)
+
+# Monitor build status
+status = orchestrator.get_build_status(build_id)
+print(f"Build {build_id}: {status}")
+
+# List all builds
+builds = orchestrator.list_builds()
+for build in builds:
+    print(f"Build {build.id}: {build.status}")
+```
+
+### 2. Resource Management
+```python
+# Check system resources
+resources = orchestrator.resource_manager.get_system_resources()
+print(f"CPU: {resources.cpu_percent}%")
+print(f"Memory: {resources.memory_percent}%")
+print(f"Disk: {resources.disk_percent}%")
+
+# Set resource limits
+orchestrator.resource_manager.set_build_limits(
+    max_cpu=80,
+    max_memory=70,
+    max_disk=85
+)
+```
+
+### 3. Build Queue Management
+```python
+# Submit multiple builds
+builds = []
+for i in range(5):
+    build_id = orchestrator.submit_build(
+        manifest_path=f"build-{i}.json",
+        priority="normal"
+    )
+    builds.append(build_id)
+
+# Monitor queue
+queue_status = orchestrator.get_queue_status()
+print(f"Pending: {queue_status.pending}")
+print(f"Running: {queue_status.running}")
+print(f"Completed: {queue_status.completed}")
+
+# Cancel build
+orchestrator.cancel_build(builds[0])
+```
+
+## Configuration
+
+### 1. APT Configuration
+```bash
+# Configure APT sources
+cat > /etc/apt/sources.list.d/debian-forge.list << EOF
+deb http://deb.debian.org/debian bookworm main
+deb http://security.debian.org/debian-security bookworm-security main
+deb http://deb.debian.org/debian bookworm-updates main
+EOF
+
+# Configure apt-cacher-ng proxy
+cat > /etc/apt/apt.conf.d/99proxy << EOF
+Acquire::http::Proxy "http://192.168.1.101:3142";
+Acquire::https::Proxy "http://192.168.1.101:3142";
+EOF
+```
+
+### 2. OSTree Configuration
+```bash
+# Configure OSTree repository
+ostree config set core.min-free-space-size 1GB
+ostree config set core.min-free-space-percent 5
+
+# Set up remote for updates
+ostree remote add debian-atomic \
+  https://your-repo.example.com/debian-atomic \
+  --set=gpg-verify=false
+```
+
+### 3. Build Environment Configuration
+```bash
+# Configure sbuild
+sudo sbuild-adduser $USER
+sudo sbuild-createchroot --include=eatmydata,ccache,distcc bookworm /var/lib/schroot/chroots/bookworm-amd64-sbuild http://deb.debian.org/debian
+
+# Configure pbuilder
+pbuilder-dist bookworm create
+```
+
+## Troubleshooting
+
+### Common Issues
+
+#### 1. Build Failures
+```bash
+# Check build logs
+tail -f /var/log/osbuild/build-*.log
+
+# Verify manifest syntax
+python3 -m osbuild --libdir . --check-only manifest.json
+
+# Test individual stages
+python3 test-apt-stage.py
+```
+
+#### 2. Resource Issues
+```bash
+# Check system resources
+python3 test-resource-allocation.py
+
+# Monitor build processes
+ps aux | grep osbuild
+htop
+
+# Check disk space
+df -h
+du -sh .osbuild/
+```
+
+#### 3. Package Issues
+```bash
+# Verify package availability
+apt-cache policy package-name
+
+# Check repository configuration
+apt-cache policy
+
+# Test package installation
+chroot /path/to/chroot apt update
+chroot /path/to/chroot apt install package-name
+```
+
+### Debug Mode
+```bash
+# Enable debug logging
+export OSBUILD_DEBUG=1
+export OSBUILD_VERBOSE=1
+
+# Run with debug output
+python3 -m osbuild --libdir . --verbose manifest.json
+```
+
+### Performance Optimization
+
+#### 1. Caching
+```bash
+# Configure apt-cacher-ng
+sudo systemctl enable apt-cacher-ng
+sudo systemctl start apt-cacher-ng
+
+# Use local mirror
+echo 'deb http://192.168.1.101:3142/deb.debian.org/debian bookworm main' > sources.list
+```
+
+#### 2. Parallel Builds
+```python
+# Configure concurrent builds
+orchestrator.resource_manager.set_concurrent_builds(4)
+
+# Monitor performance
+python3 test-performance-optimization.py
+```
+
+#### 3. Resource Allocation
+```python
+# Optimize resource allocation
+orchestrator.resource_manager.set_build_limits(
+    max_cpu=75,      # Leave headroom for system
+    max_memory=80,   # Prevent OOM
+    max_disk=90      # Maintain free space
+)
+```
+
+## Integration Examples
+
+### 1. CI/CD Pipeline
+```yaml
+# .github/workflows/debian-forge.yml
+name: Debian Forge Build
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+      - name: Install dependencies
+        run: |
+          sudo apt update
+          sudo apt install -y ostree debootstrap sbuild
+          pip install -r requirements.txt
+      - name: Build Debian atomic image
+        run: |
+          python3 -m osbuild --libdir . test-debian-atomic-manifest.json
+```
+
+### 2. Docker Integration
+```dockerfile
+# Dockerfile
+FROM debian:bookworm-slim
+
+# Install Debian Forge
+RUN apt update && apt install -y \
+    python3-pip \
+    ostree \
+    debootstrap \
+    sbuild
+
+COPY . /debian-forge
+WORKDIR /debian-forge
+
+RUN pip install -r requirements.txt
+
+# Build image
+CMD ["python3", "-m", "osbuild", "--libdir", ".", "manifest.json"]
+```
+
+### 3. API Integration
+```python
+# REST API client
+import requests
+
+class DebianForgeAPI:
+    def __init__(self, base_url):
+        self.base_url = base_url
+    
+    def submit_build(self, manifest):
+        response = requests.post(
+            f"{self.base_url}/api/v1/builds",
+            json={"manifest": manifest}
+        )
+        return response.json()
+    
+    def get_build_status(self, build_id):
+        response = requests.get(
+            f"{self.base_url}/api/v1/builds/{build_id}"
+        )
+        return response.json()
+
+# Usage
+api = DebianForgeAPI("http://localhost:8080")
+build = api.submit_build(manifest_data)
+status = api.get_build_status(build["id"])
+```
+
+## Best Practices
+
+### 1. Manifest Design
+- Use descriptive pipeline names
+- Group related stages logically
+- Include proper error handling
+- Document complex configurations
+
+### 2. Resource Management
+- Monitor system resources during builds
+- Set appropriate concurrent build limits
+- Use caching for frequently accessed data
+- Implement cleanup policies
+
+### 3. Security
+- Validate all input manifests
+- Use isolated build environments
+- Implement proper access controls
+- Monitor build processes
+
+### 4. Performance
+- Profile build performance regularly
+- Optimize stage execution order
+- Use parallel processing where possible
+- Implement intelligent caching strategies
+
+## Support and Community
+
+### Resources
+- **Documentation**: This guide and project README
+- **Issues**: GitHub issue tracker
+- **Discussions**: GitHub discussions
+- **Matrix**: #debian-forge on matrix.org
+
+### Contributing
+- Follow OSBuild contribution guidelines
+- Maintain 1:1 compatibility where possible
+- Test changes thoroughly
+- Update documentation for new features
+
+### Getting Help
+- Check troubleshooting section first
+- Search existing issues
+- Create detailed bug reports
+- Join community discussions