Fix YAML linting issues and update system requirements to Debian 13+
- Fix trailing spaces and blank lines in Forgejo workflows - Update system requirements from Ubuntu Jammy/Bookworm to Debian 13+ (Trixie) - Update test treefile to use Debian Trixie instead of Ubuntu Jammy - Update documentation to reflect modern system requirements - Fix yamllint errors for CI/CD functionality - Ensure compatibility with modern OSTree and libapt versions
This commit is contained in:
parent
ec0da91864
commit
3dec23f8f7
85 changed files with 12569 additions and 1088 deletions
645
docs/development-commands-troubleshooting.md
Normal file
645
docs/development-commands-troubleshooting.md
Normal file
|
|
@ -0,0 +1,645 @@
|
|||
# Development Commands Troubleshooting Guide
|
||||
|
||||
## Overview
|
||||
This document provides comprehensive troubleshooting information for apt-ostree development commands. It covers common issues, error messages, and solutions for the testutils, shlib-backend, and internals commands.
|
||||
|
||||
## Table of Contents
|
||||
1. [Common Issues](#common-issues)
|
||||
2. [Error Messages and Solutions](#error-messages-and-solutions)
|
||||
3. [Debugging Techniques](#debugging-techniques)
|
||||
4. [Performance Issues](#performance-issues)
|
||||
5. [Security Issues](#security-issues)
|
||||
6. [System-Specific Problems](#system-specific-problems)
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Command Not Found
|
||||
**Problem**: Development commands are not available or return "command not found"
|
||||
|
||||
**Symptoms**:
|
||||
- `apt-ostree testutils --help` returns "unknown command"
|
||||
- Development commands don't appear in help output
|
||||
|
||||
**Causes**:
|
||||
- Development features not compiled in
|
||||
- Binary built without development features
|
||||
- Feature flags not properly configured
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Verify development features are enabled
|
||||
cargo build --features development
|
||||
|
||||
# 2. Check if binary includes development commands
|
||||
cargo run --features development -- testutils --help
|
||||
|
||||
# 3. Rebuild with all development features
|
||||
cargo build --features development,dev-full
|
||||
|
||||
# 4. Verify feature compilation
|
||||
cargo check --features development
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Always build with `--features development` for development work
|
||||
- Use feature flags consistently across build environments
|
||||
- Document required features in build scripts
|
||||
|
||||
### Permission Denied
|
||||
**Problem**: Commands fail with permission errors
|
||||
|
||||
**Symptoms**:
|
||||
- "Permission denied" errors
|
||||
- "Operation not permitted" messages
|
||||
- Commands fail even when run as root
|
||||
|
||||
**Causes**:
|
||||
- Insufficient user privileges
|
||||
- Polkit policy restrictions
|
||||
- File system permissions
|
||||
- SELinux/AppArmor restrictions
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check user privileges
|
||||
whoami
|
||||
groups $USER
|
||||
|
||||
# 2. Verify Polkit policies
|
||||
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy
|
||||
|
||||
# 3. Check file permissions
|
||||
ls -la /usr/bin/apt-ostree
|
||||
ls -la /var/lib/apt-ostree/
|
||||
|
||||
# 4. Verify daemon is running
|
||||
sudo systemctl status apt-ostreed
|
||||
|
||||
# 5. Check SELinux/AppArmor status
|
||||
getenforce 2>/dev/null || echo "SELinux not available"
|
||||
aa-status 2>/dev/null || echo "AppArmor not available"
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Ensure proper Polkit policies are installed
|
||||
- Configure appropriate user groups and permissions
|
||||
- Test commands in isolated environments first
|
||||
|
||||
### Bubblewrap Issues
|
||||
**Problem**: script-shell command fails with bubblewrap errors
|
||||
|
||||
**Symptoms**:
|
||||
- "bubblewrap: command not found"
|
||||
- "Failed to execute bubblewrap"
|
||||
- Containerization failures
|
||||
|
||||
**Causes**:
|
||||
- Bubblewrap not installed
|
||||
- Insufficient bubblewrap permissions
|
||||
- Kernel security restrictions
|
||||
- User namespace limitations
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check bubblewrap installation
|
||||
which bubblewrap
|
||||
bubblewrap --version
|
||||
|
||||
# 2. Install bubblewrap if missing
|
||||
sudo apt-get install bubblewrap
|
||||
|
||||
# 3. Test bubblewrap functionality
|
||||
bubblewrap --dev-bind / / --proc /proc -- echo "test"
|
||||
|
||||
# 4. Check user namespace support
|
||||
cat /proc/sys/kernel/unprivileged_userns_clone
|
||||
|
||||
# 5. Verify kernel capabilities
|
||||
capsh --print
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Ensure bubblewrap is available in build environment
|
||||
- Test bubblewrap functionality before deployment
|
||||
- Configure appropriate kernel parameters
|
||||
|
||||
### OSTree Repository Issues
|
||||
**Problem**: Commands fail due to OSTree repository problems
|
||||
|
||||
**Symptoms**:
|
||||
- "Repository not found" errors
|
||||
- "Invalid repository" messages
|
||||
- Commit resolution failures
|
||||
|
||||
**Causes**:
|
||||
- OSTree repository not initialized
|
||||
- Repository corruption
|
||||
- Permission issues
|
||||
- Invalid repository path
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check repository status
|
||||
sudo ostree show --repo=/ostree/repo
|
||||
|
||||
# 2. Verify repository integrity
|
||||
sudo ostree fsck --repo=/ostree/repo
|
||||
|
||||
# 3. Check repository permissions
|
||||
ls -la /ostree/repo/
|
||||
|
||||
# 4. Reinitialize repository if needed
|
||||
sudo ostree init --repo=/ostree/repo --mode=bare
|
||||
|
||||
# 5. Check OSTree service status
|
||||
sudo systemctl status ostree-remount
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Initialize OSTree repository during system setup
|
||||
- Regular repository maintenance and integrity checks
|
||||
- Proper backup and recovery procedures
|
||||
|
||||
## Error Messages and Solutions
|
||||
|
||||
### testutils Command Errors
|
||||
|
||||
#### inject-pkglist Errors
|
||||
**Error**: "Failed to open OSTree repository"
|
||||
```bash
|
||||
# Solution: Check repository path and permissions
|
||||
sudo ostree show --repo=/ostree/repo
|
||||
sudo chown -R root:root /ostree/repo
|
||||
```
|
||||
|
||||
**Error**: "Invalid commit reference"
|
||||
```bash
|
||||
# Solution: Verify commit exists and is accessible
|
||||
sudo ostree log --repo=/ostree/repo
|
||||
sudo ostree show --repo=/ostree/repo <commit-hash>
|
||||
```
|
||||
|
||||
**Error**: "Failed to inject package list"
|
||||
```bash
|
||||
# Solution: Check metadata format and permissions
|
||||
sudo ostree show --repo=/ostree/repo --print-metadata-key apt.packages <commit-hash>
|
||||
```
|
||||
|
||||
#### script-shell Errors
|
||||
**Error**: "Failed to create bubblewrap container"
|
||||
```bash
|
||||
# Solution: Check bubblewrap installation and permissions
|
||||
sudo apt-get install bubblewrap
|
||||
sudo chmod +s /usr/bin/bubblewrap
|
||||
```
|
||||
|
||||
**Error**: "Script execution failed"
|
||||
```bash
|
||||
# Solution: Verify script permissions and content
|
||||
chmod +x /tmp/test.sh
|
||||
cat /tmp/test.sh
|
||||
```
|
||||
|
||||
**Error**: "Invalid root path"
|
||||
```bash
|
||||
# Solution: Check path exists and is accessible
|
||||
ls -la /mnt/ostree/deploy/
|
||||
sudo mkdir -p /mnt/ostree/deploy/debian/13/amd64
|
||||
```
|
||||
|
||||
#### generate-synthetic-upgrade Errors
|
||||
**Error**: "Failed to generate upgrade"
|
||||
```bash
|
||||
# Solution: Check system state and dependencies
|
||||
sudo apt-ostree internals diagnostics
|
||||
sudo apt-ostree status
|
||||
```
|
||||
|
||||
**Error**: "Invalid package list"
|
||||
```bash
|
||||
# Solution: Verify package format and availability
|
||||
apt-cache search <package-name>
|
||||
apt-cache show <package-name>
|
||||
```
|
||||
|
||||
### shlib-backend Command Errors
|
||||
|
||||
#### get-basearch Errors
|
||||
**Error**: "Failed to determine architecture"
|
||||
```bash
|
||||
# Solution: Check system architecture detection
|
||||
dpkg --print-architecture
|
||||
uname -m
|
||||
arch
|
||||
```
|
||||
|
||||
**Error**: "Invalid deployment reference"
|
||||
```bash
|
||||
# Solution: Verify deployment exists
|
||||
sudo apt-ostree status
|
||||
sudo ostree log --repo=/ostree/repo
|
||||
```
|
||||
|
||||
#### varsubst-basearch Errors
|
||||
**Error**: "Invalid variable format"
|
||||
```bash
|
||||
# Solution: Check variable syntax
|
||||
echo "arch={{arch}}" | sudo apt-ostree shlib-backend varsubst-basearch
|
||||
```
|
||||
|
||||
**Error**: "Variable substitution failed"
|
||||
```bash
|
||||
# Solution: Verify variable values and format
|
||||
sudo apt-ostree shlib-backend get-basearch
|
||||
```
|
||||
|
||||
#### packagelist-from-commit Errors
|
||||
**Error**: "Commit not found"
|
||||
```bash
|
||||
# Solution: Verify commit exists
|
||||
sudo ostree log --repo=/ostree/repo
|
||||
sudo ostree show --repo=/ostree/repo <commit-hash>
|
||||
```
|
||||
|
||||
**Error**: "Failed to extract package list"
|
||||
```bash
|
||||
# Solution: Check commit metadata and permissions
|
||||
sudo ostree show --repo=/ostree/repo --print-metadata <commit-hash>
|
||||
```
|
||||
|
||||
### internals Command Errors
|
||||
|
||||
#### diagnostics Errors
|
||||
**Error**: "Diagnostics failed"
|
||||
```bash
|
||||
# Solution: Check system state and permissions
|
||||
sudo apt-ostree internals diagnostics --verbose
|
||||
sudo systemctl status apt-ostreed
|
||||
```
|
||||
|
||||
**Error**: "Component check failed"
|
||||
```bash
|
||||
# Solution: Check specific component status
|
||||
sudo apt-ostree internals diagnostics --category ostree
|
||||
sudo apt-ostree internals diagnostics --category apt
|
||||
```
|
||||
|
||||
#### validate-state Errors
|
||||
**Error**: "State validation failed"
|
||||
```bash
|
||||
# Solution: Check system consistency
|
||||
sudo apt-ostree internals validate-state --verbose
|
||||
sudo apt-ostree status
|
||||
```
|
||||
|
||||
**Error**: "Component validation failed"
|
||||
```bash
|
||||
# Solution: Check specific component state
|
||||
sudo apt-ostree internals validate-state --component ostree
|
||||
sudo apt-ostree internals validate-state --component apt
|
||||
```
|
||||
|
||||
#### debug-dump Errors
|
||||
**Error**: "Failed to dump debug information"
|
||||
```bash
|
||||
# Solution: Check permissions and output location
|
||||
sudo apt-ostree internals debug-dump --output /tmp/debug.json
|
||||
ls -la /tmp/debug.json
|
||||
```
|
||||
|
||||
**Error**: "Category dump failed"
|
||||
```bash
|
||||
# Solution: Check specific category availability
|
||||
sudo apt-ostree internals debug-dump --category system-info
|
||||
```
|
||||
|
||||
## Debugging Techniques
|
||||
|
||||
### Verbose Output
|
||||
```bash
|
||||
# Enable verbose output for all commands
|
||||
export APT_OSTREE_LOG_LEVEL=debug
|
||||
export RUST_LOG=debug
|
||||
|
||||
# Run commands with verbose flags
|
||||
sudo apt-ostree testutils script-shell /tmp/test.sh --verbose
|
||||
sudo apt-ostree internals diagnostics --verbose
|
||||
```
|
||||
|
||||
### Log Analysis
|
||||
```bash
|
||||
# Check daemon logs
|
||||
sudo journalctl -u apt-ostreed -f
|
||||
|
||||
# Check system logs
|
||||
sudo journalctl -f
|
||||
|
||||
# Check specific log files
|
||||
sudo tail -f /var/log/apt-ostreed.log
|
||||
```
|
||||
|
||||
### Step-by-Step Debugging
|
||||
```bash
|
||||
# 1. Check basic functionality
|
||||
sudo apt-ostree testutils moo
|
||||
|
||||
# 2. Verify system state
|
||||
sudo apt-ostree internals diagnostics
|
||||
|
||||
# 3. Test specific components
|
||||
sudo apt-ostree shlib-backend get-basearch
|
||||
|
||||
# 4. Check dependencies
|
||||
sudo apt-ostree internals validate-state
|
||||
|
||||
# 5. Generate debug information
|
||||
sudo apt-ostree internals debug-dump --output /tmp/debug.json
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
```bash
|
||||
# Set debugging environment variables
|
||||
export APT_OSTREE_DEBUG=1
|
||||
export APT_OSTREE_LOG_LEVEL=trace
|
||||
export RUST_BACKTRACE=1
|
||||
export RUST_LOG=trace
|
||||
|
||||
# Run commands with debugging
|
||||
sudo -E apt-ostree testutils script-shell /tmp/test.sh
|
||||
```
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### Slow Command Execution
|
||||
**Problem**: Commands take too long to execute
|
||||
|
||||
**Causes**:
|
||||
- Large repository size
|
||||
- Network latency
|
||||
- Insufficient system resources
|
||||
- Inefficient algorithms
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check system resources
|
||||
htop
|
||||
free -h
|
||||
df -h
|
||||
|
||||
# 2. Monitor command performance
|
||||
time sudo apt-ostree internals diagnostics
|
||||
|
||||
# 3. Use profiling tools
|
||||
cargo install flamegraph
|
||||
cargo flamegraph --bin apt-ostree -- internals diagnostics
|
||||
|
||||
# 4. Check repository size
|
||||
du -sh /ostree/repo/
|
||||
sudo ostree summary --repo=/ostree/repo
|
||||
```
|
||||
|
||||
**Optimization Techniques**:
|
||||
- Use appropriate timeouts for long-running operations
|
||||
- Implement caching for frequently accessed data
|
||||
- Optimize database queries and file operations
|
||||
- Use parallel processing where possible
|
||||
|
||||
### Memory Issues
|
||||
**Problem**: Commands consume excessive memory
|
||||
|
||||
**Causes**:
|
||||
- Memory leaks
|
||||
- Large data structures
|
||||
- Inefficient memory usage
|
||||
- Insufficient system memory
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Monitor memory usage
|
||||
ps aux | grep apt-ostree
|
||||
cat /proc/$(pgrep apt-ostree)/status | grep VmRSS
|
||||
|
||||
# 2. Check for memory leaks
|
||||
valgrind --tool=memcheck --leak-check=full apt-ostree internals diagnostics
|
||||
|
||||
# 3. Profile memory usage
|
||||
cargo install cargo-valgrind
|
||||
cargo valgrind test
|
||||
```
|
||||
|
||||
**Optimization Techniques**:
|
||||
- Use streaming for large data processing
|
||||
- Implement proper cleanup and resource management
|
||||
- Use appropriate data structures
|
||||
- Monitor memory usage patterns
|
||||
|
||||
## Security Issues
|
||||
|
||||
### Authentication Failures
|
||||
**Problem**: Commands fail due to authentication issues
|
||||
|
||||
**Causes**:
|
||||
- Invalid user credentials
|
||||
- Expired authentication tokens
|
||||
- Polkit policy restrictions
|
||||
- D-Bus authentication failures
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check user authentication
|
||||
whoami
|
||||
groups $USER
|
||||
|
||||
# 2. Verify Polkit policies
|
||||
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy
|
||||
|
||||
# 3. Check D-Bus authentication
|
||||
dbus-send --system --dest=org.freedesktop.DBus --type=method_call --print-reply /org/freedesktop/DBus org.freedesktop.DBus.ListNames
|
||||
|
||||
# 4. Test Polkit authentication
|
||||
pkcheck --action-id org.projectatomic.aptostree1.manage --process $$ --user $USER
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Configure appropriate Polkit policies
|
||||
- Use proper user authentication mechanisms
|
||||
- Implement audit logging for security events
|
||||
- Regular security policy reviews
|
||||
|
||||
### Permission Escalation
|
||||
**Problem**: Commands gain unexpected privileges
|
||||
|
||||
**Causes**:
|
||||
- Incorrect file permissions
|
||||
- Insecure Polkit policies
|
||||
- D-Bus interface vulnerabilities
|
||||
- Container escape vulnerabilities
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check file permissions
|
||||
ls -la /usr/bin/apt-ostree
|
||||
ls -la /var/lib/apt-ostree/
|
||||
|
||||
# 2. Verify Polkit policy security
|
||||
sudo polkit-policy-file-validate /usr/share/polkit-1/actions/org.projectatomic.aptostree1.policy
|
||||
|
||||
# 3. Check container isolation
|
||||
bubblewrap --dev-bind / / --proc /proc -- id
|
||||
|
||||
# 4. Audit privilege usage
|
||||
sudo journalctl -u apt-ostreed | grep -i "privilege\|permission\|auth"
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Implement principle of least privilege
|
||||
- Use secure containerization techniques
|
||||
- Regular security audits and penetration testing
|
||||
- Proper input validation and sanitization
|
||||
|
||||
## System-Specific Problems
|
||||
|
||||
### Debian/Ubuntu Issues
|
||||
**Problem**: Commands fail on specific Debian/Ubuntu versions
|
||||
|
||||
**Causes**:
|
||||
- Version-specific dependencies
|
||||
- Package compatibility issues
|
||||
- System configuration differences
|
||||
- Kernel version limitations
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check system version
|
||||
cat /etc/os-release
|
||||
lsb_release -a
|
||||
|
||||
# 2. Verify package compatibility
|
||||
apt-cache policy apt-ostree
|
||||
apt-cache policy libostree-1-1
|
||||
|
||||
# 3. Check kernel version
|
||||
uname -r
|
||||
|
||||
# 4. Verify system requirements
|
||||
dpkg -l | grep -E "(ostree|apt|polkit)"
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Test on multiple system versions
|
||||
- Document version-specific requirements
|
||||
- Implement compatibility checks
|
||||
- Use appropriate dependency versions
|
||||
|
||||
### Architecture-Specific Issues
|
||||
**Problem**: Commands fail on specific architectures
|
||||
|
||||
**Causes**:
|
||||
- Architecture-specific bugs
|
||||
- Missing architecture support
|
||||
- Binary compatibility issues
|
||||
- Endianness problems
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check system architecture
|
||||
dpkg --print-architecture
|
||||
uname -m
|
||||
arch
|
||||
|
||||
# 2. Verify binary compatibility
|
||||
file /usr/bin/apt-ostree
|
||||
ldd /usr/bin/apt-ostree
|
||||
|
||||
# 3. Check architecture support
|
||||
apt-ostree shlib-backend get-basearch
|
||||
|
||||
# 4. Test cross-compilation
|
||||
cargo build --target aarch64-unknown-linux-gnu
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Test on multiple architectures
|
||||
- Implement architecture-specific code paths
|
||||
- Use portable data formats
|
||||
- Regular cross-architecture testing
|
||||
|
||||
### Container/VM Issues
|
||||
**Problem**: Commands fail in containerized or virtualized environments
|
||||
|
||||
**Causes**:
|
||||
- Limited system access
|
||||
- Missing hardware support
|
||||
- Resource limitations
|
||||
- Isolation restrictions
|
||||
|
||||
**Solutions**:
|
||||
```bash
|
||||
# 1. Check container environment
|
||||
cat /proc/1/cgroup
|
||||
systemd-detect-virt
|
||||
|
||||
# 2. Verify system capabilities
|
||||
capsh --print
|
||||
|
||||
# 3. Check resource limits
|
||||
ulimit -a
|
||||
cat /proc/self/limits
|
||||
|
||||
# 4. Test basic functionality
|
||||
apt-ostree testutils moo
|
||||
```
|
||||
|
||||
**Prevention**:
|
||||
- Test in various container environments
|
||||
- Implement graceful degradation
|
||||
- Document environment requirements
|
||||
- Use appropriate resource limits
|
||||
|
||||
## Best Practices for Troubleshooting
|
||||
|
||||
### Systematic Approach
|
||||
1. **Identify the problem**: Understand what's failing and why
|
||||
2. **Check system state**: Verify system health and configuration
|
||||
3. **Test basic functionality**: Ensure core components work
|
||||
4. **Isolate the issue**: Narrow down the problem scope
|
||||
5. **Apply solutions**: Implement appropriate fixes
|
||||
6. **Verify resolution**: Confirm the problem is solved
|
||||
7. **Document solution**: Record the problem and solution
|
||||
|
||||
### Logging and Monitoring
|
||||
```bash
|
||||
# Enable comprehensive logging
|
||||
export APT_OSTREE_LOG_LEVEL=debug
|
||||
export RUST_LOG=debug
|
||||
|
||||
# Monitor system resources
|
||||
htop
|
||||
iotop
|
||||
nethogs
|
||||
|
||||
# Track command execution
|
||||
time sudo apt-ostree internals diagnostics
|
||||
```
|
||||
|
||||
### Testing and Validation
|
||||
```bash
|
||||
# Test in isolated environment
|
||||
sudo apt-ostree testutils script-shell /tmp/test.sh --read-only
|
||||
|
||||
# Validate system state
|
||||
sudo apt-ostree internals validate-state
|
||||
|
||||
# Run comprehensive diagnostics
|
||||
sudo apt-ostree internals diagnostics --verbose
|
||||
```
|
||||
|
||||
### Documentation and Knowledge Base
|
||||
- **Record problems**: Document all issues and solutions
|
||||
- **Build knowledge base**: Create troubleshooting guides
|
||||
- **Share solutions**: Contribute to community knowledge
|
||||
- **Regular updates**: Keep documentation current
|
||||
|
||||
---
|
||||
|
||||
*This guide covers troubleshooting for apt-ostree development commands. For general troubleshooting, refer to the main User Guide.*
|
||||
Loading…
Add table
Add a link
Reference in a new issue