e4337e5a2c
Some checks failed
Comprehensive CI/CD Pipeline / Build and Test (push) Successful in 7m17s
Comprehensive CI/CD Pipeline / Security Audit (push) Failing after 8s
Comprehensive CI/CD Pipeline / Package Validation (push) Successful in 54s
Comprehensive CI/CD Pipeline / Status Report (push) Has been skipped
- Fixed /sysroot directory requirement for bootc compatibility - Implemented proper composefs configuration files - Added log cleanup for reproducible builds - Created correct /ostree symlink to sysroot/ostree - Bootc lint now passes 11/11 checks with only minor warning - Full bootc compatibility achieved - images ready for production use Updated documentation and todo to reflect completed work. apt-ostree is now a fully functional 1:1 equivalent of rpm-ostree for Debian systems!
645 lines
16 KiB
Markdown
645 lines
16 KiB
Markdown
# Development Commands Troubleshooting Guide
|
|
|
|
## Overview
|
|
This document provides comprehensive troubleshooting information for apt-ostree development commands. It covers common issues, error messages, and solutions for the testutils, shlib-backend, and internals commands.
|
|
|
|
## Table of Contents
|
|
1. [Common Issues](#common-issues)
|
|
2. [Error Messages and Solutions](#error-messages-and-solutions)
|
|
3. [Debugging Techniques](#debugging-techniques)
|
|
4. [Performance Issues](#performance-issues)
|
|
5. [Security Issues](#security-issues)
|
|
6. [System-Specific Problems](#system-specific-problems)
|
|
|
|
## Common Issues
|
|
|
|
### Command Not Found
|
|
**Problem**: Development commands are not available or return "command not found"
|
|
|
|
**Symptoms**:
|
|
- `apt-ostree testutils --help` returns "unknown command"
|
|
- Development commands don't appear in help output
|
|
|
|
**Causes**:
|
|
- Development features not compiled in
|
|
- Binary built without development features
|
|
- Feature flags not properly configured
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Verify development features are enabled
|
|
cargo build --features development
|
|
|
|
# 2. Check if binary includes development commands
|
|
cargo run --features development -- testutils --help
|
|
|
|
# 3. Rebuild with all development features
|
|
cargo build --features development,dev-full
|
|
|
|
# 4. Verify feature compilation
|
|
cargo check --features development
|
|
```
|
|
|
|
**Prevention**:
|
|
- Always build with `--features development` for development work
|
|
- Use feature flags consistently across build environments
|
|
- Document required features in build scripts
|
|
|
|
### Permission Denied
|
|
**Problem**: Commands fail with permission errors
|
|
|
|
**Symptoms**:
|
|
- "Permission denied" errors
|
|
- "Operation not permitted" messages
|
|
- Commands fail even when run as root
|
|
|
|
**Causes**:
|
|
- Insufficient user privileges
|
|
- Polkit policy restrictions
|
|
- File system permissions
|
|
- SELinux/AppArmor restrictions
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check user privileges
|
|
whoami
|
|
groups $USER
|
|
|
|
# 2. Verify Polkit policies
|
|
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy
|
|
|
|
# 3. Check file permissions
|
|
ls -la /usr/bin/apt-ostree
|
|
ls -la /var/lib/apt-ostree/
|
|
|
|
# 4. Verify daemon is running
|
|
sudo systemctl status apt-ostreed
|
|
|
|
# 5. Check SELinux/AppArmor status
|
|
getenforce 2>/dev/null || echo "SELinux not available"
|
|
aa-status 2>/dev/null || echo "AppArmor not available"
|
|
```
|
|
|
|
**Prevention**:
|
|
- Ensure proper Polkit policies are installed
|
|
- Configure appropriate user groups and permissions
|
|
- Test commands in isolated environments first
|
|
|
|
### Bubblewrap Issues
|
|
**Problem**: script-shell command fails with bubblewrap errors
|
|
|
|
**Symptoms**:
|
|
- "bubblewrap: command not found"
|
|
- "Failed to execute bubblewrap"
|
|
- Containerization failures
|
|
|
|
**Causes**:
|
|
- Bubblewrap not installed
|
|
- Insufficient bubblewrap permissions
|
|
- Kernel security restrictions
|
|
- User namespace limitations
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check bubblewrap installation
|
|
which bubblewrap
|
|
bubblewrap --version
|
|
|
|
# 2. Install bubblewrap if missing
|
|
sudo apt-get install bubblewrap
|
|
|
|
# 3. Test bubblewrap functionality
|
|
bubblewrap --dev-bind / / --proc /proc -- echo "test"
|
|
|
|
# 4. Check user namespace support
|
|
cat /proc/sys/kernel/unprivileged_userns_clone
|
|
|
|
# 5. Verify kernel capabilities
|
|
capsh --print
|
|
```
|
|
|
|
**Prevention**:
|
|
- Ensure bubblewrap is available in build environment
|
|
- Test bubblewrap functionality before deployment
|
|
- Configure appropriate kernel parameters
|
|
|
|
### OSTree Repository Issues
|
|
**Problem**: Commands fail due to OSTree repository problems
|
|
|
|
**Symptoms**:
|
|
- "Repository not found" errors
|
|
- "Invalid repository" messages
|
|
- Commit resolution failures
|
|
|
|
**Causes**:
|
|
- OSTree repository not initialized
|
|
- Repository corruption
|
|
- Permission issues
|
|
- Invalid repository path
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check repository status
|
|
sudo ostree show --repo=/ostree/repo
|
|
|
|
# 2. Verify repository integrity
|
|
sudo ostree fsck --repo=/ostree/repo
|
|
|
|
# 3. Check repository permissions
|
|
ls -la /ostree/repo/
|
|
|
|
# 4. Reinitialize repository if needed
|
|
sudo ostree init --repo=/ostree/repo --mode=bare
|
|
|
|
# 5. Check OSTree service status
|
|
sudo systemctl status ostree-remount
|
|
```
|
|
|
|
**Prevention**:
|
|
- Initialize OSTree repository during system setup
|
|
- Regular repository maintenance and integrity checks
|
|
- Proper backup and recovery procedures
|
|
|
|
## Error Messages and Solutions
|
|
|
|
### testutils Command Errors
|
|
|
|
#### inject-pkglist Errors
|
|
**Error**: "Failed to open OSTree repository"
|
|
```bash
|
|
# Solution: Check repository path and permissions
|
|
sudo ostree show --repo=/ostree/repo
|
|
sudo chown -R root:root /ostree/repo
|
|
```
|
|
|
|
**Error**: "Invalid commit reference"
|
|
```bash
|
|
# Solution: Verify commit exists and is accessible
|
|
sudo ostree log --repo=/ostree/repo
|
|
sudo ostree show --repo=/ostree/repo <commit-hash>
|
|
```
|
|
|
|
**Error**: "Failed to inject package list"
|
|
```bash
|
|
# Solution: Check metadata format and permissions
|
|
sudo ostree show --repo=/ostree/repo --print-metadata-key apt.packages <commit-hash>
|
|
```
|
|
|
|
#### script-shell Errors
|
|
**Error**: "Failed to create bubblewrap container"
|
|
```bash
|
|
# Solution: Check bubblewrap installation and permissions
|
|
sudo apt-get install bubblewrap
|
|
sudo chmod +s /usr/bin/bubblewrap
|
|
```
|
|
|
|
**Error**: "Script execution failed"
|
|
```bash
|
|
# Solution: Verify script permissions and content
|
|
chmod +x /tmp/test.sh
|
|
cat /tmp/test.sh
|
|
```
|
|
|
|
**Error**: "Invalid root path"
|
|
```bash
|
|
# Solution: Check path exists and is accessible
|
|
ls -la /mnt/ostree/deploy/
|
|
sudo mkdir -p /mnt/ostree/deploy/debian/13/amd64
|
|
```
|
|
|
|
#### generate-synthetic-upgrade Errors
|
|
**Error**: "Failed to generate upgrade"
|
|
```bash
|
|
# Solution: Check system state and dependencies
|
|
sudo apt-ostree internals diagnostics
|
|
sudo apt-ostree status
|
|
```
|
|
|
|
**Error**: "Invalid package list"
|
|
```bash
|
|
# Solution: Verify package format and availability
|
|
apt-cache search <package-name>
|
|
apt-cache show <package-name>
|
|
```
|
|
|
|
### shlib-backend Command Errors
|
|
|
|
#### get-basearch Errors
|
|
**Error**: "Failed to determine architecture"
|
|
```bash
|
|
# Solution: Check system architecture detection
|
|
dpkg --print-architecture
|
|
uname -m
|
|
arch
|
|
```
|
|
|
|
**Error**: "Invalid deployment reference"
|
|
```bash
|
|
# Solution: Verify deployment exists
|
|
sudo apt-ostree status
|
|
sudo ostree log --repo=/ostree/repo
|
|
```
|
|
|
|
#### varsubst-basearch Errors
|
|
**Error**: "Invalid variable format"
|
|
```bash
|
|
# Solution: Check variable syntax
|
|
echo "arch={{arch}}" | sudo apt-ostree shlib-backend varsubst-basearch
|
|
```
|
|
|
|
**Error**: "Variable substitution failed"
|
|
```bash
|
|
# Solution: Verify variable values and format
|
|
sudo apt-ostree shlib-backend get-basearch
|
|
```
|
|
|
|
#### packagelist-from-commit Errors
|
|
**Error**: "Commit not found"
|
|
```bash
|
|
# Solution: Verify commit exists
|
|
sudo ostree log --repo=/ostree/repo
|
|
sudo ostree show --repo=/ostree/repo <commit-hash>
|
|
```
|
|
|
|
**Error**: "Failed to extract package list"
|
|
```bash
|
|
# Solution: Check commit metadata and permissions
|
|
sudo ostree show --repo=/ostree/repo --print-metadata <commit-hash>
|
|
```
|
|
|
|
### internals Command Errors
|
|
|
|
#### diagnostics Errors
|
|
**Error**: "Diagnostics failed"
|
|
```bash
|
|
# Solution: Check system state and permissions
|
|
sudo apt-ostree internals diagnostics --verbose
|
|
sudo systemctl status apt-ostreed
|
|
```
|
|
|
|
**Error**: "Component check failed"
|
|
```bash
|
|
# Solution: Check specific component status
|
|
sudo apt-ostree internals diagnostics --category ostree
|
|
sudo apt-ostree internals diagnostics --category apt
|
|
```
|
|
|
|
#### validate-state Errors
|
|
**Error**: "State validation failed"
|
|
```bash
|
|
# Solution: Check system consistency
|
|
sudo apt-ostree internals validate-state --verbose
|
|
sudo apt-ostree status
|
|
```
|
|
|
|
**Error**: "Component validation failed"
|
|
```bash
|
|
# Solution: Check specific component state
|
|
sudo apt-ostree internals validate-state --component ostree
|
|
sudo apt-ostree internals validate-state --component apt
|
|
```
|
|
|
|
#### debug-dump Errors
|
|
**Error**: "Failed to dump debug information"
|
|
```bash
|
|
# Solution: Check permissions and output location
|
|
sudo apt-ostree internals debug-dump --output /tmp/debug.json
|
|
ls -la /tmp/debug.json
|
|
```
|
|
|
|
**Error**: "Category dump failed"
|
|
```bash
|
|
# Solution: Check specific category availability
|
|
sudo apt-ostree internals debug-dump --category system-info
|
|
```
|
|
|
|
## Debugging Techniques
|
|
|
|
### Verbose Output
|
|
```bash
|
|
# Enable verbose output for all commands
|
|
export APT_OSTREE_LOG_LEVEL=debug
|
|
export RUST_LOG=debug
|
|
|
|
# Run commands with verbose flags
|
|
sudo apt-ostree testutils script-shell /tmp/test.sh --verbose
|
|
sudo apt-ostree internals diagnostics --verbose
|
|
```
|
|
|
|
### Log Analysis
|
|
```bash
|
|
# Check daemon logs
|
|
sudo journalctl -u apt-ostreed -f
|
|
|
|
# Check system logs
|
|
sudo journalctl -f
|
|
|
|
# Check specific log files
|
|
sudo tail -f /var/log/apt-ostreed.log
|
|
```
|
|
|
|
### Step-by-Step Debugging
|
|
```bash
|
|
# 1. Check basic functionality
|
|
sudo apt-ostree testutils moo
|
|
|
|
# 2. Verify system state
|
|
sudo apt-ostree internals diagnostics
|
|
|
|
# 3. Test specific components
|
|
sudo apt-ostree shlib-backend get-basearch
|
|
|
|
# 4. Check dependencies
|
|
sudo apt-ostree internals validate-state
|
|
|
|
# 5. Generate debug information
|
|
sudo apt-ostree internals debug-dump --output /tmp/debug.json
|
|
```
|
|
|
|
### Environment Variables
|
|
```bash
|
|
# Set debugging environment variables
|
|
export APT_OSTREE_DEBUG=1
|
|
export APT_OSTREE_LOG_LEVEL=trace
|
|
export RUST_BACKTRACE=1
|
|
export RUST_LOG=trace
|
|
|
|
# Run commands with debugging
|
|
sudo -E apt-ostree testutils script-shell /tmp/test.sh
|
|
```
|
|
|
|
## Performance Issues
|
|
|
|
### Slow Command Execution
|
|
**Problem**: Commands take too long to execute
|
|
|
|
**Causes**:
|
|
- Large repository size
|
|
- Network latency
|
|
- Insufficient system resources
|
|
- Inefficient algorithms
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check system resources
|
|
htop
|
|
free -h
|
|
df -h
|
|
|
|
# 2. Monitor command performance
|
|
time sudo apt-ostree internals diagnostics
|
|
|
|
# 3. Use profiling tools
|
|
cargo install flamegraph
|
|
cargo flamegraph --bin apt-ostree -- internals diagnostics
|
|
|
|
# 4. Check repository size
|
|
du -sh /ostree/repo/
|
|
sudo ostree summary --repo=/ostree/repo
|
|
```
|
|
|
|
**Optimization Techniques**:
|
|
- Use appropriate timeouts for long-running operations
|
|
- Implement caching for frequently accessed data
|
|
- Optimize database queries and file operations
|
|
- Use parallel processing where possible
|
|
|
|
### Memory Issues
|
|
**Problem**: Commands consume excessive memory
|
|
|
|
**Causes**:
|
|
- Memory leaks
|
|
- Large data structures
|
|
- Inefficient memory usage
|
|
- Insufficient system memory
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Monitor memory usage
|
|
ps aux | grep apt-ostree
|
|
cat /proc/$(pgrep apt-ostree)/status | grep VmRSS
|
|
|
|
# 2. Check for memory leaks
|
|
valgrind --tool=memcheck --leak-check=full apt-ostree internals diagnostics
|
|
|
|
# 3. Profile memory usage
|
|
cargo install cargo-valgrind
|
|
cargo valgrind test
|
|
```
|
|
|
|
**Optimization Techniques**:
|
|
- Use streaming for large data processing
|
|
- Implement proper cleanup and resource management
|
|
- Use appropriate data structures
|
|
- Monitor memory usage patterns
|
|
|
|
## Security Issues
|
|
|
|
### Authentication Failures
|
|
**Problem**: Commands fail due to authentication issues
|
|
|
|
**Causes**:
|
|
- Invalid user credentials
|
|
- Expired authentication tokens
|
|
- Polkit policy restrictions
|
|
- D-Bus authentication failures
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check user authentication
|
|
whoami
|
|
groups $USER
|
|
|
|
# 2. Verify Polkit policies
|
|
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy
|
|
|
|
# 3. Check D-Bus authentication
|
|
dbus-send --system --dest=org.freedesktop.DBus --type=method_call --print-reply /org/freedesktop/DBus org.freedesktop.DBus.ListNames
|
|
|
|
# 4. Test Polkit authentication
|
|
pkcheck --action-id org.projectatomic.aptostree1.manage --process $$ --user $USER
|
|
```
|
|
|
|
**Prevention**:
|
|
- Configure appropriate Polkit policies
|
|
- Use proper user authentication mechanisms
|
|
- Implement audit logging for security events
|
|
- Regular security policy reviews
|
|
|
|
### Permission Escalation
|
|
**Problem**: Commands gain unexpected privileges
|
|
|
|
**Causes**:
|
|
- Incorrect file permissions
|
|
- Insecure Polkit policies
|
|
- D-Bus interface vulnerabilities
|
|
- Container escape vulnerabilities
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check file permissions
|
|
ls -la /usr/bin/apt-ostree
|
|
ls -la /var/lib/apt-ostree/
|
|
|
|
# 2. Verify Polkit policy security
|
|
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy
|
|
|
|
# 3. Check container isolation
|
|
bubblewrap --dev-bind / / --proc /proc -- id
|
|
|
|
# 4. Audit privilege usage
|
|
sudo journalctl -u apt-ostreed | grep -i "privilege\|permission\|auth"
|
|
```
|
|
|
|
**Prevention**:
|
|
- Implement principle of least privilege
|
|
- Use secure containerization techniques
|
|
- Regular security audits and penetration testing
|
|
- Proper input validation and sanitization
|
|
|
|
## System-Specific Problems
|
|
|
|
### Debian/Ubuntu Issues
|
|
**Problem**: Commands fail on specific Debian/Ubuntu versions
|
|
|
|
**Causes**:
|
|
- Version-specific dependencies
|
|
- Package compatibility issues
|
|
- System configuration differences
|
|
- Kernel version limitations
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check system version
|
|
cat /etc/os-release
|
|
lsb_release -a
|
|
|
|
# 2. Verify package compatibility
|
|
apt-cache policy apt-ostree
|
|
apt-cache policy libostree-1-1
|
|
|
|
# 3. Check kernel version
|
|
uname -r
|
|
|
|
# 4. Verify system requirements
|
|
dpkg -l | grep -E "(ostree|apt|polkit)"
|
|
```
|
|
|
|
**Prevention**:
|
|
- Test on multiple system versions
|
|
- Document version-specific requirements
|
|
- Implement compatibility checks
|
|
- Use appropriate dependency versions
|
|
|
|
### Architecture-Specific Issues
|
|
**Problem**: Commands fail on specific architectures
|
|
|
|
**Causes**:
|
|
- Architecture-specific bugs
|
|
- Missing architecture support
|
|
- Binary compatibility issues
|
|
- Endianness problems
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check system architecture
|
|
dpkg --print-architecture
|
|
uname -m
|
|
arch
|
|
|
|
# 2. Verify binary compatibility
|
|
file /usr/bin/apt-ostree
|
|
ldd /usr/bin/apt-ostree
|
|
|
|
# 3. Check architecture support
|
|
apt-ostree shlib-backend get-basearch
|
|
|
|
# 4. Test cross-compilation
|
|
cargo build --target aarch64-unknown-linux-gnu
|
|
```
|
|
|
|
**Prevention**:
|
|
- Test on multiple architectures
|
|
- Implement architecture-specific code paths
|
|
- Use portable data formats
|
|
- Regular cross-architecture testing
|
|
|
|
### Container/VM Issues
|
|
**Problem**: Commands fail in containerized or virtualized environments
|
|
|
|
**Causes**:
|
|
- Limited system access
|
|
- Missing hardware support
|
|
- Resource limitations
|
|
- Isolation restrictions
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# 1. Check container environment
|
|
cat /proc/1/cgroup
|
|
systemd-detect-virt
|
|
|
|
# 2. Verify system capabilities
|
|
capsh --print
|
|
|
|
# 3. Check resource limits
|
|
ulimit -a
|
|
cat /proc/self/limits
|
|
|
|
# 4. Test basic functionality
|
|
apt-ostree testutils moo
|
|
```
|
|
|
|
**Prevention**:
|
|
- Test in various container environments
|
|
- Implement graceful degradation
|
|
- Document environment requirements
|
|
- Use appropriate resource limits
|
|
|
|
## Best Practices for Troubleshooting
|
|
|
|
### Systematic Approach
|
|
1. **Identify the problem**: Understand what's failing and why
|
|
2. **Check system state**: Verify system health and configuration
|
|
3. **Test basic functionality**: Ensure core components work
|
|
4. **Isolate the issue**: Narrow down the problem scope
|
|
5. **Apply solutions**: Implement appropriate fixes
|
|
6. **Verify resolution**: Confirm the problem is solved
|
|
7. **Document solution**: Record the problem and solution
|
|
|
|
### Logging and Monitoring
|
|
```bash
|
|
# Enable comprehensive logging
|
|
export APT_OSTREE_LOG_LEVEL=debug
|
|
export RUST_LOG=debug
|
|
|
|
# Monitor system resources
|
|
htop
|
|
iotop
|
|
nethogs
|
|
|
|
# Track command execution
|
|
time sudo apt-ostree internals diagnostics
|
|
```
|
|
|
|
### Testing and Validation
|
|
```bash
|
|
# Test in isolated environment
|
|
sudo apt-ostree testutils script-shell /tmp/test.sh --read-only
|
|
|
|
# Validate system state
|
|
sudo apt-ostree internals validate-state
|
|
|
|
# Run comprehensive diagnostics
|
|
sudo apt-ostree internals diagnostics --verbose
|
|
```
|
|
|
|
### Documentation and Knowledge Base
|
|
- **Record problems**: Document all issues and solutions
|
|
- **Build knowledge base**: Create troubleshooting guides
|
|
- **Share solutions**: Contribute to community knowledge
|
|
- **Regular updates**: Keep documentation current
|
|
|
|
---
|
|
|
|
*This guide covers troubleshooting for apt-ostree development commands. For general troubleshooting, refer to the main User Guide.*
|