apt-ostree/docs/development-commands-troubleshooting.md

Development Commands Troubleshooting Guide

Overview

This document provides comprehensive troubleshooting information for apt-ostree development commands. It covers common issues, error messages, and solutions for the testutils, shlib-backend, and internals commands.

Table of Contents

1. Common Issues
2. Error Messages and Solutions
3. Debugging Techniques
4. Performance Issues
5. Security Issues
6. System-Specific Problems

Common Issues

Command Not Found

Problem: Development commands are not available or return "command not found"

Symptoms:

• apt-ostree testutils --help returns "unknown command"
• Development commands don't appear in help output

Causes:

• Development features not compiled in
• Binary built without development features
• Feature flags not properly configured

Solutions:

# 1. Verify development features are enabled
cargo build --features development

# 2. Check if binary includes development commands
cargo run --features development -- testutils --help

# 3. Rebuild with all development features
cargo build --features development,dev-full

# 4. Verify feature compilation
cargo check --features development

Prevention:

• Always build with --features development for development work
• Use feature flags consistently across build environments
• Document required features in build scripts

Permission Denied

Problem: Commands fail with permission errors

Symptoms:

• "Permission denied" errors
• "Operation not permitted" messages
• Commands fail even when run as root

Causes:

• Insufficient user privileges
• Polkit policy restrictions
• File system permissions
• SELinux/AppArmor restrictions

Solutions:

# 1. Check user privileges
whoami
groups $USER

# 2. Verify Polkit policies
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy

# 3. Check file permissions
ls -la /usr/bin/apt-ostree
ls -la /var/lib/apt-ostree/

# 4. Verify daemon is running
sudo systemctl status apt-ostreed

# 5. Check SELinux/AppArmor status
getenforce 2>/dev/null || echo "SELinux not available"
aa-status 2>/dev/null || echo "AppArmor not available"

Prevention:

• Ensure proper Polkit policies are installed
• Configure appropriate user groups and permissions
• Test commands in isolated environments first

Bubblewrap Issues

Problem: script-shell command fails with bubblewrap errors

Symptoms:

• "bubblewrap: command not found"
• "Failed to execute bubblewrap"
• Containerization failures

Causes:

• Bubblewrap not installed
• Insufficient bubblewrap permissions
• Kernel security restrictions
• User namespace limitations

Solutions:

# 1. Check bubblewrap installation
which bubblewrap
bubblewrap --version

# 2. Install bubblewrap if missing
sudo apt-get install bubblewrap

# 3. Test bubblewrap functionality
bubblewrap --dev-bind / / --proc /proc -- echo "test"

# 4. Check user namespace support
cat /proc/sys/kernel/unprivileged_userns_clone

# 5. Verify kernel capabilities
capsh --print

Prevention:

• Ensure bubblewrap is available in build environment
• Test bubblewrap functionality before deployment
• Configure appropriate kernel parameters

OSTree Repository Issues

Problem: Commands fail due to OSTree repository problems

Symptoms:

• "Repository not found" errors
• "Invalid repository" messages
• Commit resolution failures

Causes:

• OSTree repository not initialized
• Repository corruption
• Permission issues
• Invalid repository path

Solutions:

# 1. Check repository status
sudo ostree show --repo=/ostree/repo

# 2. Verify repository integrity
sudo ostree fsck --repo=/ostree/repo

# 3. Check repository permissions
ls -la /ostree/repo/

# 4. Reinitialize repository if needed
sudo ostree init --repo=/ostree/repo --mode=bare

# 5. Check OSTree service status
sudo systemctl status ostree-remount

Prevention:

• Initialize OSTree repository during system setup
• Regular repository maintenance and integrity checks
• Proper backup and recovery procedures

Error Messages and Solutions

testutils Command Errors

inject-pkglist Errors

Error: "Failed to open OSTree repository"

# Solution: Check repository path and permissions
sudo ostree show --repo=/ostree/repo
sudo chown -R root:root /ostree/repo

Error: "Invalid commit reference"

# Solution: Verify commit exists and is accessible
sudo ostree log --repo=/ostree/repo
sudo ostree show --repo=/ostree/repo <commit-hash>

Error: "Failed to inject package list"

# Solution: Check metadata format and permissions
sudo ostree show --repo=/ostree/repo --print-metadata-key apt.packages <commit-hash>

script-shell Errors

Error: "Failed to create bubblewrap container"

# Solution: Check bubblewrap installation and permissions
sudo apt-get install bubblewrap
sudo chmod +s /usr/bin/bubblewrap

Error: "Script execution failed"

# Solution: Verify script permissions and content
chmod +x /tmp/test.sh
cat /tmp/test.sh

Error: "Invalid root path"

# Solution: Check path exists and is accessible
ls -la /mnt/ostree/deploy/
sudo mkdir -p /mnt/ostree/deploy/debian/13/amd64

generate-synthetic-upgrade Errors

Error: "Failed to generate upgrade"

# Solution: Check system state and dependencies
sudo apt-ostree internals diagnostics
sudo apt-ostree status

Error: "Invalid package list"

# Solution: Verify package format and availability
apt-cache search <package-name>
apt-cache show <package-name>

shlib-backend Command Errors

get-basearch Errors

Error: "Failed to determine architecture"

# Solution: Check system architecture detection
dpkg --print-architecture
uname -m
arch

Error: "Invalid deployment reference"

# Solution: Verify deployment exists
sudo apt-ostree status
sudo ostree log --repo=/ostree/repo

varsubst-basearch Errors

Error: "Invalid variable format"

# Solution: Check variable syntax
echo "arch={{arch}}" | sudo apt-ostree shlib-backend varsubst-basearch

Error: "Variable substitution failed"

# Solution: Verify variable values and format
sudo apt-ostree shlib-backend get-basearch

packagelist-from-commit Errors

Error: "Commit not found"

# Solution: Verify commit exists
sudo ostree log --repo=/ostree/repo
sudo ostree show --repo=/ostree/repo <commit-hash>

Error: "Failed to extract package list"

# Solution: Check commit metadata and permissions
sudo ostree show --repo=/ostree/repo --print-metadata <commit-hash>

internals Command Errors

diagnostics Errors

Error: "Diagnostics failed"

# Solution: Check system state and permissions
sudo apt-ostree internals diagnostics --verbose
sudo systemctl status apt-ostreed

Error: "Component check failed"

# Solution: Check specific component status
sudo apt-ostree internals diagnostics --category ostree
sudo apt-ostree internals diagnostics --category apt

validate-state Errors

Error: "State validation failed"

# Solution: Check system consistency
sudo apt-ostree internals validate-state --verbose
sudo apt-ostree status

Error: "Component validation failed"

# Solution: Check specific component state
sudo apt-ostree internals validate-state --component ostree
sudo apt-ostree internals validate-state --component apt

debug-dump Errors

Error: "Failed to dump debug information"

# Solution: Check permissions and output location
sudo apt-ostree internals debug-dump --output /tmp/debug.json
ls -la /tmp/debug.json

Error: "Category dump failed"

# Solution: Check specific category availability
sudo apt-ostree internals debug-dump --category system-info

Debugging Techniques

Verbose Output

# Enable verbose output for all commands
export APT_OSTREE_LOG_LEVEL=debug
export RUST_LOG=debug

# Run commands with verbose flags
sudo apt-ostree testutils script-shell /tmp/test.sh --verbose
sudo apt-ostree internals diagnostics --verbose

Log Analysis

# Check daemon logs
sudo journalctl -u apt-ostreed -f

# Check system logs
sudo journalctl -f

# Check specific log files
sudo tail -f /var/log/apt-ostreed.log

Step-by-Step Debugging

# 1. Check basic functionality
sudo apt-ostree testutils moo

# 2. Verify system state
sudo apt-ostree internals diagnostics

# 3. Test specific components
sudo apt-ostree shlib-backend get-basearch

# 4. Check dependencies
sudo apt-ostree internals validate-state

# 5. Generate debug information
sudo apt-ostree internals debug-dump --output /tmp/debug.json

Environment Variables

# Set debugging environment variables
export APT_OSTREE_DEBUG=1
export APT_OSTREE_LOG_LEVEL=trace
export RUST_BACKTRACE=1
export RUST_LOG=trace

# Run commands with debugging
sudo -E apt-ostree testutils script-shell /tmp/test.sh

Performance Issues

Slow Command Execution

Problem: Commands take too long to execute

Causes:

• Large repository size
• Network latency
• Insufficient system resources
• Inefficient algorithms

Solutions:

# 1. Check system resources
htop
free -h
df -h

# 2. Monitor command performance
time sudo apt-ostree internals diagnostics

# 3. Use profiling tools
cargo install flamegraph
cargo flamegraph --bin apt-ostree -- internals diagnostics

# 4. Check repository size
du -sh /ostree/repo/
sudo ostree summary --repo=/ostree/repo

Optimization Techniques:

• Use appropriate timeouts for long-running operations
• Implement caching for frequently accessed data
• Optimize database queries and file operations
• Use parallel processing where possible

Memory Issues

Problem: Commands consume excessive memory

Causes:

• Memory leaks
• Large data structures
• Inefficient memory usage
• Insufficient system memory

Solutions:

# 1. Monitor memory usage
ps aux | grep apt-ostree
cat /proc/$(pgrep apt-ostree)/status | grep VmRSS

# 2. Check for memory leaks
valgrind --tool=memcheck --leak-check=full apt-ostree internals diagnostics

# 3. Profile memory usage
cargo install cargo-valgrind
cargo valgrind test

Optimization Techniques:

• Use streaming for large data processing
• Implement proper cleanup and resource management
• Use appropriate data structures
• Monitor memory usage patterns

Security Issues

Authentication Failures

Problem: Commands fail due to authentication issues

Causes:

• Invalid user credentials
• Expired authentication tokens
• Polkit policy restrictions
• D-Bus authentication failures

Solutions:

# 1. Check user authentication
whoami
groups $USER

# 2. Verify Polkit policies
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy

# 3. Check D-Bus authentication
dbus-send --system --dest=org.freedesktop.DBus --type=method_call --print-reply /org/freedesktop/DBus org.freedesktop.DBus.ListNames

# 4. Test Polkit authentication
pkcheck --action-id org.projectatomic.aptostree1.manage --process $$ --user $USER

Prevention:

• Configure appropriate Polkit policies
• Use proper user authentication mechanisms
• Implement audit logging for security events
• Regular security policy reviews

Permission Escalation

Problem: Commands gain unexpected privileges

Causes:

• Incorrect file permissions
• Insecure Polkit policies
• D-Bus interface vulnerabilities
• Container escape vulnerabilities

Solutions:

# 1. Check file permissions
ls -la /usr/bin/apt-ostree
ls -la /var/lib/apt-ostree/

# 2. Verify Polkit policy security
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy

# 3. Check container isolation
bubblewrap --dev-bind / / --proc /proc -- id

# 4. Audit privilege usage
sudo journalctl -u apt-ostreed | grep -i "privilege\|permission\|auth"

Prevention:

• Implement principle of least privilege
• Use secure containerization techniques
• Regular security audits and penetration testing
• Proper input validation and sanitization

System-Specific Problems

Debian/Ubuntu Issues

Problem: Commands fail on specific Debian/Ubuntu versions

Causes:

• Version-specific dependencies
• Package compatibility issues
• System configuration differences
• Kernel version limitations

Solutions:

# 1. Check system version
cat /etc/os-release
lsb_release -a

# 2. Verify package compatibility
apt-cache policy apt-ostree
apt-cache policy libostree-1-1

# 3. Check kernel version
uname -r

# 4. Verify system requirements
dpkg -l | grep -E "(ostree|apt|polkit)"

Prevention:

• Test on multiple system versions
• Document version-specific requirements
• Implement compatibility checks
• Use appropriate dependency versions

Architecture-Specific Issues

Problem: Commands fail on specific architectures

Causes:

• Architecture-specific bugs
• Missing architecture support
• Binary compatibility issues
• Endianness problems

Solutions:

# 1. Check system architecture
dpkg --print-architecture
uname -m
arch

# 2. Verify binary compatibility
file /usr/bin/apt-ostree
ldd /usr/bin/apt-ostree

# 3. Check architecture support
apt-ostree shlib-backend get-basearch

# 4. Test cross-compilation
cargo build --target aarch64-unknown-linux-gnu

Prevention:

• Test on multiple architectures
• Implement architecture-specific code paths
• Use portable data formats
• Regular cross-architecture testing

Container/VM Issues

Problem: Commands fail in containerized or virtualized environments

Causes:

• Limited system access
• Missing hardware support
• Resource limitations
• Isolation restrictions

Solutions:

# 1. Check container environment
cat /proc/1/cgroup
systemd-detect-virt

# 2. Verify system capabilities
capsh --print

# 3. Check resource limits
ulimit -a
cat /proc/self/limits

# 4. Test basic functionality
apt-ostree testutils moo

Prevention:

• Test in various container environments
• Implement graceful degradation
• Document environment requirements
• Use appropriate resource limits

Best Practices for Troubleshooting

Systematic Approach

1. Identify the problem: Understand what's failing and why
2. Check system state: Verify system health and configuration
3. Test basic functionality: Ensure core components work
4. Isolate the issue: Narrow down the problem scope
5. Apply solutions: Implement appropriate fixes
6. Verify resolution: Confirm the problem is solved
7. Document solution: Record the problem and solution

Logging and Monitoring

# Enable comprehensive logging
export APT_OSTREE_LOG_LEVEL=debug
export RUST_LOG=debug

# Monitor system resources
htop
iotop
nethogs

# Track command execution
time sudo apt-ostree internals diagnostics

Testing and Validation

# Test in isolated environment
sudo apt-ostree testutils script-shell /tmp/test.sh --read-only

# Validate system state
sudo apt-ostree internals validate-state

# Run comprehensive diagnostics
sudo apt-ostree internals diagnostics --verbose

Documentation and Knowledge Base

• Record problems: Document all issues and solutions
• Build knowledge base: Create troubleshooting guides
• Share solutions: Contribute to community knowledge
• Regular updates: Keep documentation current

---

This guide covers troubleshooting for apt-ostree development commands. For general troubleshooting, refer to the main User Guide.