# Development Commands Troubleshooting Guide

## Overview
This document provides comprehensive troubleshooting information for apt-ostree development commands. It covers common issues, error messages, and solutions for the testutils, shlib-backend, and internals commands.

## Table of Contents
1. [Common Issues](#common-issues)
2. [Error Messages and Solutions](#error-messages-and-solutions)
3. [Debugging Techniques](#debugging-techniques)
4. [Performance Issues](#performance-issues)
5. [Security Issues](#security-issues)
6. [System-Specific Problems](#system-specific-problems)

## Common Issues

### Command Not Found
**Problem**: Development commands are not available or return "command not found"

**Symptoms**:
- `apt-ostree testutils --help` returns "unknown command"
- Development commands don't appear in help output

**Causes**:
- Development features not compiled in
- Binary built without development features
- Feature flags not properly configured

**Solutions**:
```bash
# 1. Verify development features are enabled
cargo build --features development

# 2. Check if binary includes development commands
cargo run --features development -- testutils --help

# 3. Rebuild with all development features
cargo build --features development,dev-full

# 4. Verify feature compilation
cargo check --features development
```

**Prevention**:
- Always build with `--features development` for development work
- Use feature flags consistently across build environments
- Document required features in build scripts

### Permission Denied
**Problem**: Commands fail with permission errors

**Symptoms**:
- "Permission denied" errors
- "Operation not permitted" messages
- Commands fail even when run as root

**Causes**:
- Insufficient user privileges
- Polkit policy restrictions
- File system permissions
- SELinux/AppArmor restrictions

**Solutions**:
```bash
# 1. Check user privileges
whoami
groups $USER

# 2. Verify Polkit policies
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy

# 3. Check file permissions
ls -la /usr/bin/apt-ostree
ls -la /var/lib/apt-ostree/

# 4. Verify daemon is running
sudo systemctl status apt-ostreed

# 5. Check SELinux/AppArmor status
getenforce 2>/dev/null || echo "SELinux not available"
aa-status 2>/dev/null || echo "AppArmor not available"
```

**Prevention**:
- Ensure proper Polkit policies are installed
- Configure appropriate user groups and permissions
- Test commands in isolated environments first

### Bubblewrap Issues
**Problem**: script-shell command fails with bubblewrap errors

**Symptoms**:
- "bubblewrap: command not found"
- "Failed to execute bubblewrap"
- Containerization failures

**Causes**:
- Bubblewrap not installed
- Insufficient bubblewrap permissions
- Kernel security restrictions
- User namespace limitations

**Solutions**:
```bash
# 1. Check bubblewrap installation
which bubblewrap
bubblewrap --version

# 2. Install bubblewrap if missing
sudo apt-get install bubblewrap

# 3. Test bubblewrap functionality
bubblewrap --dev-bind / / --proc /proc -- echo "test"

# 4. Check user namespace support
cat /proc/sys/kernel/unprivileged_userns_clone

# 5. Verify kernel capabilities
capsh --print
```

**Prevention**:
- Ensure bubblewrap is available in build environment
- Test bubblewrap functionality before deployment
- Configure appropriate kernel parameters

### OSTree Repository Issues
**Problem**: Commands fail due to OSTree repository problems

**Symptoms**:
- "Repository not found" errors
- "Invalid repository" messages
- Commit resolution failures

**Causes**:
- OSTree repository not initialized
- Repository corruption
- Permission issues
- Invalid repository path

**Solutions**:
```bash
# 1. Check repository status
sudo ostree show --repo=/ostree/repo

# 2. Verify repository integrity
sudo ostree fsck --repo=/ostree/repo

# 3. Check repository permissions
ls -la /ostree/repo/

# 4. Reinitialize repository if needed
sudo ostree init --repo=/ostree/repo --mode=bare

# 5. Check OSTree service status
sudo systemctl status ostree-remount
```

**Prevention**:
- Initialize OSTree repository during system setup
- Regular repository maintenance and integrity checks
- Proper backup and recovery procedures

## Error Messages and Solutions

### testutils Command Errors

#### inject-pkglist Errors
**Error**: "Failed to open OSTree repository"
```bash
# Solution: Check repository path and permissions
sudo ostree show --repo=/ostree/repo
sudo chown -R root:root /ostree/repo
```

**Error**: "Invalid commit reference"
```bash
# Solution: Verify commit exists and is accessible
sudo ostree log --repo=/ostree/repo
sudo ostree show --repo=/ostree/repo <commit-hash>
```

**Error**: "Failed to inject package list"
```bash
# Solution: Check metadata format and permissions
sudo ostree show --repo=/ostree/repo --print-metadata-key apt.packages <commit-hash>
```

#### script-shell Errors
**Error**: "Failed to create bubblewrap container"
```bash
# Solution: Check bubblewrap installation and permissions
sudo apt-get install bubblewrap
sudo chmod +s /usr/bin/bubblewrap
```

**Error**: "Script execution failed"
```bash
# Solution: Verify script permissions and content
chmod +x /tmp/test.sh
cat /tmp/test.sh
```

**Error**: "Invalid root path"
```bash
# Solution: Check path exists and is accessible
ls -la /mnt/ostree/deploy/
sudo mkdir -p /mnt/ostree/deploy/debian/13/amd64
```

#### generate-synthetic-upgrade Errors
**Error**: "Failed to generate upgrade"
```bash
# Solution: Check system state and dependencies
sudo apt-ostree internals diagnostics
sudo apt-ostree status
```

**Error**: "Invalid package list"
```bash
# Solution: Verify package format and availability
apt-cache search <package-name>
apt-cache show <package-name>
```

### shlib-backend Command Errors

#### get-basearch Errors
**Error**: "Failed to determine architecture"
```bash
# Solution: Check system architecture detection
dpkg --print-architecture
uname -m
arch
```

**Error**: "Invalid deployment reference"
```bash
# Solution: Verify deployment exists
sudo apt-ostree status
sudo ostree log --repo=/ostree/repo
```

#### varsubst-basearch Errors
**Error**: "Invalid variable format"
```bash
# Solution: Check variable syntax
echo "arch={{arch}}" | sudo apt-ostree shlib-backend varsubst-basearch
```

**Error**: "Variable substitution failed"
```bash
# Solution: Verify variable values and format
sudo apt-ostree shlib-backend get-basearch
```

#### packagelist-from-commit Errors
**Error**: "Commit not found"
```bash
# Solution: Verify commit exists
sudo ostree log --repo=/ostree/repo
sudo ostree show --repo=/ostree/repo <commit-hash>
```

**Error**: "Failed to extract package list"
```bash
# Solution: Check commit metadata and permissions
sudo ostree show --repo=/ostree/repo --print-metadata <commit-hash>
```

### internals Command Errors

#### diagnostics Errors
**Error**: "Diagnostics failed"
```bash
# Solution: Check system state and permissions
sudo apt-ostree internals diagnostics --verbose
sudo systemctl status apt-ostreed
```

**Error**: "Component check failed"
```bash
# Solution: Check specific component status
sudo apt-ostree internals diagnostics --category ostree
sudo apt-ostree internals diagnostics --category apt
```

#### validate-state Errors
**Error**: "State validation failed"
```bash
# Solution: Check system consistency
sudo apt-ostree internals validate-state --verbose
sudo apt-ostree status
```

**Error**: "Component validation failed"
```bash
# Solution: Check specific component state
sudo apt-ostree internals validate-state --component ostree
sudo apt-ostree internals validate-state --component apt
```

#### debug-dump Errors
**Error**: "Failed to dump debug information"
```bash
# Solution: Check permissions and output location
sudo apt-ostree internals debug-dump --output /tmp/debug.json
ls -la /tmp/debug.json
```

**Error**: "Category dump failed"
```bash
# Solution: Check specific category availability
sudo apt-ostree internals debug-dump --category system-info
```

## Debugging Techniques

### Verbose Output
```bash
# Enable verbose output for all commands
export APT_OSTREE_LOG_LEVEL=debug
export RUST_LOG=debug

# Run commands with verbose flags
sudo apt-ostree testutils script-shell /tmp/test.sh --verbose
sudo apt-ostree internals diagnostics --verbose
```

### Log Analysis
```bash
# Check daemon logs
sudo journalctl -u apt-ostreed -f

# Check system logs
sudo journalctl -f

# Check specific log files
sudo tail -f /var/log/apt-ostreed.log
```

### Step-by-Step Debugging
```bash
# 1. Check basic functionality
sudo apt-ostree testutils moo

# 2. Verify system state
sudo apt-ostree internals diagnostics

# 3. Test specific components
sudo apt-ostree shlib-backend get-basearch

# 4. Check dependencies
sudo apt-ostree internals validate-state

# 5. Generate debug information
sudo apt-ostree internals debug-dump --output /tmp/debug.json
```

### Environment Variables
```bash
# Set debugging environment variables
export APT_OSTREE_DEBUG=1
export APT_OSTREE_LOG_LEVEL=trace
export RUST_BACKTRACE=1
export RUST_LOG=trace

# Run commands with debugging
sudo -E apt-ostree testutils script-shell /tmp/test.sh
```

## Performance Issues

### Slow Command Execution
**Problem**: Commands take too long to execute

**Causes**:
- Large repository size
- Network latency
- Insufficient system resources
- Inefficient algorithms

**Solutions**:
```bash
# 1. Check system resources
htop
free -h
df -h

# 2. Monitor command performance
time sudo apt-ostree internals diagnostics

# 3. Use profiling tools
cargo install flamegraph
cargo flamegraph --bin apt-ostree -- internals diagnostics

# 4. Check repository size
du -sh /ostree/repo/
sudo ostree summary --repo=/ostree/repo
```

**Optimization Techniques**:
- Use appropriate timeouts for long-running operations
- Implement caching for frequently accessed data
- Optimize database queries and file operations
- Use parallel processing where possible

### Memory Issues
**Problem**: Commands consume excessive memory

**Causes**:
- Memory leaks
- Large data structures
- Inefficient memory usage
- Insufficient system memory

**Solutions**:
```bash
# 1. Monitor memory usage
ps aux | grep apt-ostree
cat /proc/$(pgrep apt-ostree)/status | grep VmRSS

# 2. Check for memory leaks
valgrind --tool=memcheck --leak-check=full apt-ostree internals diagnostics

# 3. Profile memory usage
cargo install cargo-valgrind
cargo valgrind test
```

**Optimization Techniques**:
- Use streaming for large data processing
- Implement proper cleanup and resource management
- Use appropriate data structures
- Monitor memory usage patterns

## Security Issues

### Authentication Failures
**Problem**: Commands fail due to authentication issues

**Causes**:
- Invalid user credentials
- Expired authentication tokens
- Polkit policy restrictions
- D-Bus authentication failures

**Solutions**:
```bash
# 1. Check user authentication
whoami
groups $USER

# 2. Verify Polkit policies
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy

# 3. Check D-Bus authentication
dbus-send --system --dest=org.freedesktop.DBus --type=method_call --print-reply /org/freedesktop/DBus org.freedesktop.DBus.ListNames

# 4. Test Polkit authentication
pkcheck --action-id org.projectatomic.aptostree1.manage --process $$ --user $USER
```

**Prevention**:
- Configure appropriate Polkit policies
- Use proper user authentication mechanisms
- Implement audit logging for security events
- Regular security policy reviews

### Permission Escalation
**Problem**: Commands gain unexpected privileges

**Causes**:
- Incorrect file permissions
- Insecure Polkit policies
- D-Bus interface vulnerabilities
- Container escape vulnerabilities

**Solutions**:
```bash
# 1. Check file permissions
ls -la /usr/bin/apt-ostree
ls -la /var/lib/apt-ostree/

# 2. Verify Polkit policy security
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy

# 3. Check container isolation
bubblewrap --dev-bind / / --proc /proc -- id

# 4. Audit privilege usage
sudo journalctl -u apt-ostreed | grep -i "privilege\|permission\|auth"
```

**Prevention**:
- Implement principle of least privilege
- Use secure containerization techniques
- Regular security audits and penetration testing
- Proper input validation and sanitization

## System-Specific Problems

### Debian/Ubuntu Issues
**Problem**: Commands fail on specific Debian/Ubuntu versions

**Causes**:
- Version-specific dependencies
- Package compatibility issues
- System configuration differences
- Kernel version limitations

**Solutions**:
```bash
# 1. Check system version
cat /etc/os-release
lsb_release -a

# 2. Verify package compatibility
apt-cache policy apt-ostree
apt-cache policy libostree-1-1

# 3. Check kernel version
uname -r

# 4. Verify system requirements
dpkg -l | grep -E "(ostree|apt|polkit)"
```

**Prevention**:
- Test on multiple system versions
- Document version-specific requirements
- Implement compatibility checks
- Use appropriate dependency versions

### Architecture-Specific Issues
**Problem**: Commands fail on specific architectures

**Causes**:
- Architecture-specific bugs
- Missing architecture support
- Binary compatibility issues
- Endianness problems

**Solutions**:
```bash
# 1. Check system architecture
dpkg --print-architecture
uname -m
arch

# 2. Verify binary compatibility
file /usr/bin/apt-ostree
ldd /usr/bin/apt-ostree

# 3. Check architecture support
apt-ostree shlib-backend get-basearch

# 4. Test cross-compilation
cargo build --target aarch64-unknown-linux-gnu
```

**Prevention**:
- Test on multiple architectures
- Implement architecture-specific code paths
- Use portable data formats
- Regular cross-architecture testing

### Container/VM Issues
**Problem**: Commands fail in containerized or virtualized environments

**Causes**:
- Limited system access
- Missing hardware support
- Resource limitations
- Isolation restrictions

**Solutions**:
```bash
# 1. Check container environment
cat /proc/1/cgroup
systemd-detect-virt

# 2. Verify system capabilities
capsh --print

# 3. Check resource limits
ulimit -a
cat /proc/self/limits

# 4. Test basic functionality
apt-ostree testutils moo
```

**Prevention**:
- Test in various container environments
- Implement graceful degradation
- Document environment requirements
- Use appropriate resource limits

## Best Practices for Troubleshooting

### Systematic Approach
1. **Identify the problem**: Understand what's failing and why
2. **Check system state**: Verify system health and configuration
3. **Test basic functionality**: Ensure core components work
4. **Isolate the issue**: Narrow down the problem scope
5. **Apply solutions**: Implement appropriate fixes
6. **Verify resolution**: Confirm the problem is solved
7. **Document solution**: Record the problem and solution

### Logging and Monitoring
```bash
# Enable comprehensive logging
export APT_OSTREE_LOG_LEVEL=debug
export RUST_LOG=debug

# Monitor system resources
htop
iotop
nethogs

# Track command execution
time sudo apt-ostree internals diagnostics
```

### Testing and Validation
```bash
# Test in isolated environment
sudo apt-ostree testutils script-shell /tmp/test.sh --read-only

# Validate system state
sudo apt-ostree internals validate-state

# Run comprehensive diagnostics
sudo apt-ostree internals diagnostics --verbose
```

### Documentation and Knowledge Base
- **Record problems**: Document all issues and solutions
- **Build knowledge base**: Create troubleshooting guides
- **Share solutions**: Contribute to community knowledge
- **Regular updates**: Keep documentation current

---

*This guide covers troubleshooting for apt-ostree development commands. For general troubleshooting, refer to the main User Guide.*