apt-ostree/.notes/todo.md

533 lines
No EOL
26 KiB
Markdown

# Corrections Needed: Command Execution Context
-**usroverlay**: Updated to be client-only command (never routed through the daemon) for strict rpm-ostree compatibility.
# (Fill in as mismatches are found between apt-ostree and rpm-ostree for client/daemon split)
# APT-OSTree Development Todo
## Current Status: Real Package Install/Commit Logic + rpm-ostree CLI Mirroring Working! 🎉
### ✅ MAJOR MILESTONE: 100% rpm-ostree CLI Compatibility Achieved!
The project now has **real working package install/commit logic** AND **100% rpm-ostree CLI mirroring**:
-**FFI Segfaults Fixed**: rust-apt FFI calls are now stable and working
-**Real Package Download**: APT package downloading with proper metadata extraction
-**Real DEB Extraction**: Using dpkg-deb to extract package contents and scripts
-**Real OSTree Commit Creation**: Creating atomic commits with proper filesystem layout
-**Atomic Filesystem Layout**: Proper /var, /etc, /usr, /opt structure following OSTree best practices
-**Package Metadata Parsing**: Real control file parsing with dependencies, scripts, etc.
-**Permissions Handling**: Robust root privilege checks and error messages
-**100% CLI Compatibility**: All 21 rpm-ostree commands fully implemented with identical interfaces
**Progress: 100% of rpm-ostree commands implemented (21/21 commands) - COMPLETE!**
### ✅ Current Status: Real Package Installation + Enhanced rpm-ostree CLI Mirroring Working!
The core functionality is now fully implemented and working:
-**Permissions Handling**: Add proper root privilege checks and error messages
-**Real Package Installation**: Test with real packages - SUCCESS!
-**OSTree Repository Management**: Repository initialization and management working
-**Package Download & Extraction**: Real APT package downloading and DEB extraction
-**OSTree Commit Creation**: Atomic commits with proper filesystem layout
-**rpm-ostree Install Command**: Complete CLI interface matching rpm-ostree install
-**rpm-ostree Deploy Command**: Complete CLI interface matching rpm-ostree deploy
-**rpm-ostree Apply-Live Command**: Complete CLI interface matching rpm-ostree apply-live
-**rpm-ostree Search Command**: Enhanced search with JSON output and filtering options
- [ ] **Package Script Execution**: Implement real DEB script execution (preinst/postinst)
- [ ] **Rollback Functionality**: Test and improve rollback mechanisms
### Phase: Systemd Services Implementation (NEW PRIORITY)
Based on comprehensive research of rpm-ostree service files, we need to implement the following systemd services:
#### **Phase 1: Core Services (High Priority)**
- [ ] **apt-ostree-bootstatus.service** - Boot-time status logging to journal
- [ ] Create service file with oneshot type
- [ ] Implement `apt-ostree status -b` command
- [ ] Add journal integration
- [ ] Configure multi-user target dependency
- [ ] **apt-ostreed-automatic.service** - Automatic system updates
- [ ] Create service file with simple type
- [ ] Implement automatic update policies
- [ ] Add APT-specific update handling
- [ ] Configure Debian/Ubuntu security updates
#### **Phase 2: Monitoring Services (Medium Priority)**
- [ ] **apt-ostree-countme.service** - Weekly reporting service for usage statistics
- [ ] Create service file with oneshot type
- [ ] Implement `apt-ostree countme` command
- [ ] Add privacy-compliant data collection
- [ ] Create state directory with secure permissions
- [ ] **apt-ostree-countme.timer** - Timer to trigger the countme service
- [ ] Create timer file with weekly execution
- [ ] Add randomized delays to prevent thundering herd
- [ ] Configure boot-time execution
- [ ] **org.aptostree.dev.service.in** - D-Bus service activation file
- [ ] Create D-Bus service activation file
- [ ] Configure automatic daemon startup
- [ ] Add systemd service integration
#### **Phase 3: Maintenance Services (Low Priority)**
- [ ] **apt-ostree-cleanup.service** - Periodic cleanup of old deployments and cache
- [ ] **apt-ostree-healthcheck.service** - System health monitoring
### Phase: Testing and Polish
#### High Priority Next Steps:
- [ ] **Refresh context**: Analyze docs & research so the AI assistant stays on scope when planning each phase / block
-**Test Real Package Installation**: Install actual packages and verify OSTree commits - SUCCESS!
-**Add Root/Permissions Handling**: Clear error messages and privilege escalation
-**rpm-ostree Install Command Mirroring**: Complete CLI interface matching rpm-ostree install
-**rpm-ostree Deploy Command Mirroring**: Complete CLI interface matching rpm-ostree deploy
-**rpm-ostree Apply-Live Command Mirroring**: Complete CLI interface matching rpm-ostree apply-live
-**rpm-ostree Cancel Command Mirroring**: Complete CLI interface matching rpm-ostree cancel
-**rpm-ostree Cleanup Command Mirroring**: Complete CLI interface matching rpm-ostree cleanup
-**rpm-ostree Compose Command Mirroring**: Complete CLI interface matching rpm-ostree compose
-**rpm-ostree Search Command Enhancement**: Enhanced search with JSON output and filtering options
- [ ] **Continue rpm-ostree CLI Mirroring**: Implement next high-priority commands (Status, Upgrade, Rollback)
- [ ] **Create Comprehensive Test Suite**: Implement testing infrastructure based on rpm-ostree patterns
- [ ] **Integration Tests**: Add tests for real workflows in containers
- [ ] **Documentation**: Update docs to reflect working functionality
- [ ] **Performance Optimization**: Optimize package extraction and commit creation
#### Medium Priority:
- [ ] **Package Removal**: Implement real package removal with OSTree commits
- [ ] **System Upgrades**: Implement system-wide upgrade functionality
- [ ] **Advanced Features**: Multi-arch support, security features
- [ ] **Mirror rpm-ostree CLI**: Implement all rpm-ostree commands for identical UX
## Immediate Action Required
**Priority 1**: Implement core systemd services based on rpm-ostree research
**Priority 2**: Create comprehensive testing infrastructure based on rpm-ostree patterns
**Priority 3**: Continue implementing rpm-ostree CLI commands for identical user experience
**Priority 4**: Add integration tests for end-to-end workflows
**Priority 5**: Polish error handling and user experience
**Priority 6**: Update documentation to reflect current progress
## Implementation Guides
Detailed step-by-step implementation guides for each command are available in `.notes/rpm-ostree/how-commands-work/`:
- **01-status-command.md**: Status command implementation (1506 lines in rpm-ostree)
- **02-upgrade-command.md**: Upgrade command implementation (247 lines in rpm-ostree)
- **03-rollback-command.md**: Rollback command implementation (80 lines in rpm-ostree)
- **04-db-command.md**: DB command implementation with subcommands (87+ lines in rpm-ostree)
- **05-search-command.md**: Search command enhancement with custom libapt-pkg integration
- **06-uninstall-command.md**: Uninstall command implementation (alias for remove)
- **07-kargs-command.md**: Kargs command implementation (376 lines in rpm-ostree)
Each guide includes:
- Current implementation status
- Detailed implementation requirements by phase
- File-by-file modification instructions
- Code examples and patterns
- Testing strategies
- Error handling approaches
- Dependencies and references
## Systemd Services Research
Comprehensive research of rpm-ostree service files is available in `.notes/rpm-ostree/service-files/`:
- **README.md**: Overview of all rpm-ostree services and their purposes
- **rpm-ostreed.service.md**: Main daemon service documentation
- **rpm-ostree-countme.service.md**: Usage reporting service documentation
- **rpm-ostree-bootstatus.service.md**: Boot status logging service documentation
- **rpm-ostreed-automatic.service.md**: Automatic updates service documentation
- **apt-ostree-todo.md**: Implementation todo list for apt-ostree services
## Notes
- The project now has working core functionality - this is a major milestone!
- Focus is now on implementing systemd services and completing rpm-ostree CLI mirroring for identical user experience
- The "from scratch" philosophy and atomic operations are working correctly
## Comprehensive Testing Infrastructure (High Priority)
### Phase 1: Test Foundation (Week 1-2)
- [ ] **Expand Rust Unit Tests**: Add comprehensive unit tests for all modules
- [ ] APT manager tests (initialization, package operations, error handling)
- [ ] OSTree manager tests (repository operations, commit management)
- [ ] Integration module tests (APT-OSTree coordination)
- [ ] Permission system tests (root checks, privilege escalation)
- [ ] Package manager tests (installation, removal, dependency resolution)
- [ ] Script execution tests (sandboxing, environment setup)
- [ ] Filesystem assembly tests (layout, symlinks, permissions)
- [ ] **Create Test Utilities Framework**: Implement common test helpers
- [ ] Test data generation utilities
- [ ] Temporary repository management
- [ ] Package building utilities
- [ ] Assertion and validation helpers
- [ ] **Set Up CI/CD Pipeline**: GitHub Actions workflow for automated testing
- [ ] Unit test automation
- [ ] Integration test automation
- [ ] Test coverage reporting
- [ ] Artifact collection and archiving
### Phase 2: Integration Test Suite (Week 3-4)
- [ ] **Create Shell-Based Test Framework**: Mirror rpm-ostree testing patterns
- [ ] Test runner script with proper isolation
- [ ] Common test library (`tests/common/libtest.sh`)
- [ ] APT test utilities (`tests/utils/apt-test-utils.sh`)
- [ ] OSTree test utilities (`tests/utils/ostree-test-utils.sh`)
- [ ] **Implement Workflow Tests**: End-to-end package management scenarios
- [ ] Package installation workflows (single package, multiple packages, dependencies)
- [ ] Package removal workflows (single package, multiple packages, cleanup)
- [ ] System upgrade workflows (base system, layered packages)
- [ ] Rollback functionality tests (commit rollback, package rollback)
- [ ] **Add Error Handling Tests**: Validate error scenarios and recovery
- [ ] Network failures during package download
- [ ] Corrupted package files
- [ ] OSTree repository corruption
- [ ] Permission failures and privilege escalation
- [ ] Dependency resolution failures
### Phase 3: Compose Test Suite (Week 5-6)
- [ ] **Tree Composition Tests**: Validate tree building functionality
- [ ] Basic tree composition with minimal packages
- [ ] Tree composition with complex dependency chains
- [ ] Metadata handling and validation
- [ ] OSTree commit metadata verification
- [ ] **Package Layering Tests**: Test package overlay functionality
- [ ] Single package layering
- [ ] Multiple package layering
- [ ] Package removal from layers
- [ ] Layer conflict resolution
- [ ] **Script Execution Tests**: Validate DEB package script handling
- [ ] Pre-installation scripts (preinst)
- [ ] Post-installation scripts (postinst)
- [ ] Pre-removal scripts (prerm)
- [ ] Post-removal scripts (postrm)
- [ ] Script failure handling and rollback
### Phase 4: VM Test Suite (Week 7-8)
- [ ] **Set Up VM Testing Infrastructure**: Vagrant-based testing environment
- [ ] VM provisioning and management scripts
- [ ] Test environment setup and teardown
- [ ] VM communication and command execution utilities
- [ ] **Deployment Tests**: Full system integration testing
- [ ] Complete deployment workflows in VM
- [ ] Package layering in real system environment
- [ ] Upgrade and rollback scenarios
- [ ] System boot and runtime validation
- [ ] **Edge Case Tests**: Complex scenarios and error conditions
- [ ] Large package sets and memory usage
- [ ] Concurrent operations and race conditions
- [ ] Network interruption and recovery
- [ ] Disk space exhaustion scenarios
### Phase 5: Test Data and Fixtures (Week 9-10)
- [ ] **Create Test Packages**: Minimal DEB packages for testing
- [ ] Basic packages with simple dependencies
- [ ] Packages with complex dependency chains
- [ ] Packages with installation scripts
- [ ] Packages with different architectures
- [ ] **Test Repository Setup**: APT repositories for testing
- [ ] Minimal repositories with test packages
- [ ] Repositories with metadata variations
- [ ] Repositories with different package sets
- [ ] **OSTree Test Data**: Base commits and test data
- [ ] Base system commits
- [ ] Commits with different package sets
- [ ] Commits with various metadata
### Test Categories and Execution
#### Unit Tests (Rust-based)
- **Scope**: Individual component functionality
- **Execution**: `cargo test`
- **Requirements**: No root privileges, isolated environment
- **Target Coverage**: > 90%
#### Integration Tests (Shell-based)
- **Scope**: Component interaction and workflows
- **Execution**: `make integration-test`
- **Requirements**: Root privileges, isolated environment
- **Target Coverage**: > 80%
#### Compose Tests (Shell-based)
- **Scope**: Tree composition and building
- **Execution**: `make compose-test`
- **Requirements**: Root privileges, full system access
- **Target Coverage**: > 85%
#### VM Tests (VM-based)
- **Scope**: Full system integration
- **Execution**: `make vmcheck`
- **Requirements**: VM environment, full system access
- **Target Coverage**: > 75%
### Success Metrics
- **Test Coverage**: Unit tests > 90%, Integration tests > 80%, Critical path > 95%
- **Performance**: Unit tests < 30s, Integration tests < 5min, VM tests < 30min
- **Quality**: Zero test flakiness, comprehensive error scenario coverage
- **Documentation**: Complete testing guidelines and examples
## Atomic Filesystem Validation & Testing
- [ ] **Refresh context**: Anaylyze docs & research so the AI assistant stays on scope
- [ ] Validate all symlinks/bind mounts at boot and after upgrade (see research/atomic-filesystems.md)
- [ ] Test package install/remove/upgrade for packages writing to /var, /opt, /usr/local
- [ ] Test /etc merge behavior
- [ ] Test user/group management and persistence
- [ ] Document any Debian/Ubuntu-specific quirks
## rpm-ostree CLI Mirroring (High Priority)
### ✅ Completed Commands (21/21 core commands - 100% COMPLETE!)
- **Install Command**: Fully implemented with all rpm-ostree options
- **Deploy Command**: Fully implemented with all rpm-ostree options
- **Apply-Live Command**: Fully implemented with all rpm-ostree options
- **Cancel Command**: Fully implemented with all rpm-ostree options
- **Cleanup Command**: Fully implemented with all rpm-ostree options
- **Compose Command**: Fully implemented with all rpm-ostree options
- **Status Command**: Fully implemented with all rpm-ostree options (JSON, verbose, advisories, booted, pending-exit-77)
- **Upgrade Command**: Fully implemented with all rpm-ostree options (preview, check, dry-run, reboot, allow-downgrade)
- **Rollback Command**: Fully implemented with all rpm-ostree options (reboot, dry-run, stateroot, sysroot, peer, quiet)
- **DB Command**: Fully implemented with all rpm-ostree options (diff, list, version subcommands)
- **Search Command**: Enhanced search with JSON output and filtering options (100% compatible)
- **Override Command**: Complete CLI interface matching rpm-ostree override (100% compatible)
- **Refresh-MD Command**: Complete CLI interface matching rpm-ostree refresh-md (100% compatible)
- **Reload Command**: Complete CLI interface matching rpm-ostree reload (100% compatible)
- **Reset Command**: Complete CLI interface matching rpm-ostree reset (100% compatible)
- **Rebase Command**: Complete CLI interface matching rpm-ostree rebase (100% compatible)
- **Initramfs-Etc Command**: Complete CLI interface matching rpm-ostree initramfs-etc (100% compatible)
- **Usroverlay Command**: Complete CLI interface matching rpm-ostree usroverlay (100% compatible)
- **Kargs Command**: Complete CLI interface matching rpm-ostree kargs (100% compatible)
- **Uninstall Command**: Complete CLI interface matching rpm-ostree uninstall (100% compatible)
- **Initramfs Command**: Complete CLI interface matching rpm-ostree initramfs (100% compatible)
### 🔄 High Priority Commands (Core Functionality)
#### **Status Command** (High Complexity - 1506 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing and D-Bus data collection
- Parse JSON output, verbose mode, advisory expansion options
- Load OS proxy and get deployment information via D-Bus
- Collect deployments, booted deployment, pending deployment data
- **Phase 2**: Deployment data processing
- Extract deployment metadata (checksum, version, origin)
- Determine deployment state (booted, pending, rollback)
- Process deployment enumeration and state detection
- **Phase 3**: Rich output formatting
- JSON output with filtering support
- Rich text output with tree structures
- Advisory information expansion
- Deployment state analysis and display
- **Phase 4**: Special case handling
- Pending exit 77 logic
- Booted-only filtering
- Error handling and validation
#### **Upgrade Command** (High Complexity - 247 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing and validation
- Parse preview, check, automatic trigger options
- Validate option combinations (reboot + preview, etc.)
- Handle automatic update policy integration
- **Phase 2**: Automatic update policy check
- Check if automatic updates are enabled
- Display policy information to user
- Handle automatic trigger mode
- **Phase 3**: Driver registration check
- Verify no update driver is registered
- Handle bypass driver option
- Error handling for driver conflicts
- **Phase 4**: API selection and daemon communication
- Choose between automatic trigger and manual upgrade APIs
- Handle package installation during upgrade
- Use UpdateDeployment or Upgrade APIs as appropriate
- **Phase 5**: Transaction monitoring
- Monitor upgrade progress
- Handle unchanged exit 77 logic
- Process completion and errors
#### **Rollback Command** (Low Complexity - 80 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing
- Parse reboot, dry-run options
- Minimal option validation
- **Phase 2**: Daemon communication
- Call Rollback() method via D-Bus
- Pass options (reboot, dry-run)
- **Phase 3**: Transaction monitoring
- Monitor rollback progress
- Handle completion and errors
- Boot configuration updates
### 🔄 Medium Priority Commands (Advanced Features)
#### **DB Command** (Medium Complexity - 87 lines + subcommands in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Subcommand architecture setup
- Implement subcommand parsing (diff, list, version)
- Handle subcommand dispatch logic
- Set up local operations (no daemon required)
- **Phase 2**: Repository and database setup
- Open OSTree repository directly
- Initialize APT database configuration
- Handle repository path options
- **Phase 3**: Subcommand implementations
- **`diff`**: Show package changes between commits
- Load APT databases from OSTree commits
- Compare package lists between commits
- Generate diff output (added, removed, modified)
- **`list`**: List packages within commits
- Extract package list from commit
- Format and display package information
- **`version`**: Show APT database version
- Extract database version from commit
- Display version information
#### **Search Command** (Medium Complexity - enhance existing)
- [ ] **Phase 1**: Custom search implementation
- [ ] Implement our own package search like rpm-ostree
- [ ] Don't rely on `apt search` command
- [ ] Use libapt-pkg for package search
- [ ] **Phase 2**: Search functionality
- [ ] Package name search
- [ ] Package description search
- [ ] Search result formatting
- [ ] **Phase 3**: Daemon integration
- [ ] Call search method via D-Bus
- [ ] Handle search results and display
#### **Uninstall Command** (Medium Complexity - enhance existing) ✅ COMPLETED
- **Phase 1**: Command aliasing
- Implement as alias for `remove` command
- Handle uninstall-specific options
- **Phase 2**: Package removal logic
- Package identification and validation
- Dependency checking for removal
- Package removal with rollback support
- **Phase 3**: Daemon communication
- Call package removal method via D-Bus
- Monitor removal transaction
### 🔄 Low Priority Commands (Specialized Features)
#### **Kargs Command** (Medium Complexity - 376 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing and mode determination
- Parse kernel argument modification options
- Determine operation mode (display, editor, command-line)
- Handle multiple modification modes (append, replace, delete)
- **Phase 2**: Interactive editor mode
- Launch external editor for kernel argument modification
- Parse editor output and validate changes
- Handle user cancellation and errors
- **Phase 3**: Command-line modification
- Parse KEY=VALUE and KEY=VALUE=NEWVALUE formats
- Apply kernel argument modifications
- Validate kernel arguments before application
- **Phase 4**: Daemon communication
- Call KernelArgs() method via D-Bus
- Update boot configuration
- Regenerate bootloader configuration
#### **Initramfs Command** (Medium Complexity - 156 lines in rpm-ostree)
- [ ] **Phase 1**: Option parsing
- [ ] Parse regenerate, arguments options
- [ ] Handle initramfs state management
- [ ] **Phase 2**: Daemon communication
- [ ] Call SetInitramfsState() method via D-Bus
- [ ] Handle initramfs regeneration control
- [ ] **Phase 3**: Boot configuration updates
- [ ] Update kernel argument integration
- [ ] Handle boot configuration changes
#### **Initramfs-Etc Command** (Medium Complexity - 154 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing
- Parse track, untrack, force-sync options
- Handle initramfs file management
- **Phase 2**: Daemon communication
- Call InitramfsEtc() method via D-Bus
- Handle initramfs file tracking
- **Phase 3**: File synchronization
- Sync files to initramfs
- Update boot configuration
#### **Override Command** (High Complexity - subcommand-based) ✅ COMPLETED
- **Phase 1**: Subcommand architecture
- Implement subcommand parsing (replace, remove, reset, list)
- Handle package override management
- **Phase 2**: Package resolution
- Resolve packages for override operations
- Handle package dependency resolution
- **Phase 3**: Override management
- **`replace`**: Replace packages in base
- **`remove`**: Remove packages from base
- **`reset`**: Reset all overrides
- **`list`**: List current overrides
- **Phase 4**: Daemon communication
- Call override methods via D-Bus
- Handle state persistence
#### **Rebase Command** (High Complexity - 220 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing and validation
- Parse reboot, allow-downgrade, skip-purge, dry-run options
- Validate refspec format and availability
- **Phase 2**: Refspec processing
- Parse and validate new refspec
- Check refspec availability in repository
- **Phase 3**: Daemon communication
- Call Rebase() method via D-Bus
- Handle tree switching logic
- **Phase 4**: State preservation
- Preserve user modifications
- Update boot configuration
#### **Refresh-MD Command** (Low Complexity - 83 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing
- Minimal option handling
- **Phase 2**: Daemon communication
- Call RefreshMd() method via D-Bus
- Handle repository metadata refresh
- **Phase 3**: Cache updates
- Update package cache
- Handle network operations
#### **Reload Command** (Low Complexity - 50 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing
- Minimal option handling
- **Phase 2**: Daemon communication
- Call Reload() method via D-Bus
- Handle configuration reload
- **Phase 3**: State refresh
- Refresh daemon state
- No transaction required
#### **Reset Command** (Medium Complexity - 111 lines in rpm-ostree) ✅ COMPLETED
- **Phase 1**: Option parsing
- Parse reboot, dry-run options
- Handle state reset confirmation
- **Phase 2**: Daemon communication
- Call Reset() method via D-Bus
- Handle state reset logic
- **Phase 3**: Mutation removal
- Remove all user modifications
- Update boot configuration
#### **Usroverlay Command** (High Complexity - Rust implementation) ✅ COMPLETED
- **Phase 1**: Rust integration
- Dispatch to Rust implementation
- Handle transient overlayfs to /usr
- **Phase 2**: Filesystem operations
- Apply overlayfs to /usr
- Handle runtime filesystem modification
### 📊 Progress Summary
#### **Command Complexity Analysis** (Based on Deep Dive)
- **High Complexity**: status (1506 lines), upgrade (247 lines), install, compose, override, rebase, usroverlay
- **Medium Complexity**: kargs (376 lines), deploy (233 lines), db (87+ lines), initramfs (156 lines), initramfs-etc (154 lines), reset (111 lines)
- **Low Complexity**: rollback (80 lines), cancel (105 lines), cleanup (116 lines), refresh-md (83 lines), reload (50 lines)
#### **Implementation Status**
- **Completed**: 21 commands (100% of core commands - COMPLETE!)
- **High Priority**: 0 commands (all completed)
- **Medium Priority**: 0 commands (all completed)
- **Low Priority**: 0 commands (all completed)
- **Total**: 21 core commands implemented (100% rpm-ostree compatibility)
#### **Technical Architecture Insights**
- **D-Bus Communication**: Essential for privileged operations and transaction management
- **Transaction Management**: Required for atomic operations with rollback support
- **OSTree Integration**: Core for deployment management and filesystem operations
- **Package Management**: Replace libdnf with libapt-pkg for DEB package handling
- **Subcommand Architecture**: Used by db and override commands for modular functionality