robojerk
a23b4e53fd
Some checks failed
Compile apt-layer (v2) / compile (push) Has been cancelled
- Added 20-daemon-integration.sh scriptlet for D-Bus and daemon lifecycle management - Updated 99-main.sh with new daemon subcommands (start, stop, status, install, uninstall, test, layer, deploy, upgrade, rollback) - Enhanced help and usage text for daemon integration - Fixed bash syntax errors in daemon integration scriptlet - Updated compile.sh to include daemon integration in build process - Updated .gitignore to exclude src/rpm-ostree/ reference source - Updated CHANGELOG.md and TODO.md to document daemon integration milestone - Removed src/rpm-ostree/ from git tracking (reference only, not committed)
475 lines
No EOL
8.9 KiB
Markdown
475 lines
No EOL
8.9 KiB
Markdown
# Debugging rpm-ostree
|
|
|
|
## Overview
|
|
|
|
This guide covers debugging techniques and tools for troubleshooting rpm-ostree issues. It includes setting up debug environments, using debugging tools, and solving common problems.
|
|
|
|
## Debug Environment Setup
|
|
|
|
### Enabling Debug Output
|
|
|
|
#### Environment Variables
|
|
```bash
|
|
# Enable all debug messages
|
|
export G_MESSAGES_DEBUG=all
|
|
export RUST_LOG=debug
|
|
export RPMOSTREE_DEBUG=1
|
|
|
|
# Enable specific debug domains
|
|
export G_MESSAGES_DEBUG=rpmostree
|
|
export RUST_LOG=rpmostree=debug
|
|
|
|
# Enable verbose output
|
|
export RPMOSTREE_VERBOSE=1
|
|
```
|
|
|
|
#### Command Line Options
|
|
```bash
|
|
# Enable verbose output
|
|
rpm-ostree --verbose status
|
|
|
|
# Enable debug output
|
|
rpm-ostree --debug status
|
|
|
|
# Enable trace output
|
|
rpm-ostree --trace status
|
|
```
|
|
|
|
### Debug Configuration
|
|
|
|
#### Daemon Debug Configuration
|
|
```ini
|
|
# /etc/rpm-ostree/rpm-ostreed.conf
|
|
[daemon]
|
|
debug=true
|
|
verbose=true
|
|
log-level=debug
|
|
```
|
|
|
|
#### Client Debug Configuration
|
|
```ini
|
|
# ~/.config/rpm-ostree/config
|
|
[client]
|
|
debug=true
|
|
verbose=true
|
|
log-level=debug
|
|
```
|
|
|
|
## Debugging Tools
|
|
|
|
### GDB (GNU Debugger)
|
|
|
|
#### Basic GDB Usage
|
|
```bash
|
|
# Debug rpm-ostree binary
|
|
gdb --args rpm-ostree status
|
|
|
|
# Debug daemon process
|
|
gdb -p $(pidof rpm-ostreed)
|
|
|
|
# Attach to running process
|
|
gdb attach $(pidof rpm-ostreed)
|
|
```
|
|
|
|
#### GDB Commands
|
|
```bash
|
|
# Set breakpoints
|
|
(gdb) break rpmostree_core_new
|
|
(gdb) break rpmostree_daemon_handle_upgrade
|
|
|
|
# Run program
|
|
(gdb) run
|
|
|
|
# Continue execution
|
|
(gdb) continue
|
|
|
|
# Step through code
|
|
(gdb) next
|
|
(gdb) step
|
|
|
|
# Examine variables
|
|
(gdb) print variable_name
|
|
(gdb) print *pointer_variable
|
|
|
|
# Examine call stack
|
|
(gdb) bt
|
|
(gdb) bt full
|
|
|
|
# Examine memory
|
|
(gdb) x/10x pointer_address
|
|
(gdb) x/s string_address
|
|
```
|
|
|
|
#### GDB Scripts
|
|
```bash
|
|
# Create GDB script
|
|
cat > debug_rpm_ostree.gdb << EOF
|
|
set pagination off
|
|
set logging file debug.log
|
|
set logging on
|
|
|
|
break rpmostree_core_new
|
|
commands
|
|
print *core
|
|
continue
|
|
end
|
|
|
|
run status
|
|
EOF
|
|
|
|
# Run with script
|
|
gdb -x debug_rpm_ostree.gdb --args rpm-ostree status
|
|
```
|
|
|
|
### Valgrind
|
|
|
|
#### Memory Leak Detection
|
|
```bash
|
|
# Check for memory leaks
|
|
valgrind --leak-check=full \
|
|
--show-leak-kinds=all \
|
|
--track-origins=yes \
|
|
--verbose \
|
|
--log-file=valgrind.log \
|
|
rpm-ostree status
|
|
```
|
|
|
|
#### Memory Error Detection
|
|
```bash
|
|
# Check for memory errors
|
|
valgrind --tool=memcheck \
|
|
--track-origins=yes \
|
|
--verbose \
|
|
--log-file=valgrind.log \
|
|
rpm-ostree status
|
|
```
|
|
|
|
#### Callgrind Profiling
|
|
```bash
|
|
# Profile function calls
|
|
valgrind --tool=callgrind \
|
|
--callgrind-out-file=callgrind.out \
|
|
rpm-ostree upgrade
|
|
|
|
# Analyze results
|
|
callgrind_annotate callgrind.out
|
|
```
|
|
|
|
### strace
|
|
|
|
#### System Call Tracing
|
|
```bash
|
|
# Trace all system calls
|
|
strace -f -o strace.log rpm-ostree status
|
|
|
|
# Trace specific system calls
|
|
strace -e trace=file,network -f rpm-ostree status
|
|
|
|
# Trace with timestamps
|
|
strace -t -f -o strace.log rpm-ostree status
|
|
|
|
# Trace with relative timestamps
|
|
strace -r -f -o strace.log rpm-ostree status
|
|
```
|
|
|
|
#### Network Call Tracing
|
|
```bash
|
|
# Trace network operations
|
|
strace -e trace=network -f rpm-ostree upgrade
|
|
|
|
# Trace specific network calls
|
|
strace -e trace=connect,accept,sendto,recvfrom -f rpm-ostree upgrade
|
|
```
|
|
|
|
### perf
|
|
|
|
#### Performance Profiling
|
|
```bash
|
|
# Profile CPU usage
|
|
perf record -g rpm-ostree upgrade
|
|
perf report
|
|
|
|
# Profile specific process
|
|
perf record -g -p $(pidof rpm-ostreed)
|
|
perf report
|
|
|
|
# Profile with call graph
|
|
perf record -g --call-graph=dwarf rpm-ostree upgrade
|
|
perf report --call-graph
|
|
```
|
|
|
|
#### Event Analysis
|
|
```bash
|
|
# Analyze system events
|
|
perf record -e sched:* rpm-ostree upgrade
|
|
perf report
|
|
|
|
# Analyze cache misses
|
|
perf record -e cache-misses rpm-ostree upgrade
|
|
perf report
|
|
```
|
|
|
|
## Debugging Specific Components
|
|
|
|
### Daemon Debugging
|
|
|
|
#### Daemon Logs
|
|
```bash
|
|
# View daemon logs
|
|
journalctl -u rpm-ostreed -f
|
|
|
|
# View daemon logs with debug
|
|
journalctl -u rpm-ostreed -f --grep="DEBUG"
|
|
|
|
# View recent daemon logs
|
|
journalctl -u rpm-ostreed --since="10 minutes ago"
|
|
```
|
|
|
|
#### Daemon Process Debugging
|
|
```bash
|
|
# Debug daemon startup
|
|
gdb --args /usr/bin/rpm-ostreed --debug
|
|
|
|
# Attach to running daemon
|
|
gdb -p $(pidof rpm-ostreed)
|
|
|
|
# Check daemon status
|
|
systemctl status rpm-ostreed
|
|
```
|
|
|
|
#### D-Bus Debugging
|
|
```bash
|
|
# Monitor D-Bus messages
|
|
dbus-monitor --system
|
|
|
|
# Monitor specific D-Bus interface
|
|
dbus-monitor --system "interface=org.projectatomic.rpmostree1"
|
|
|
|
# Test D-Bus methods
|
|
gdbus call --system \
|
|
--dest=org.projectatomic.rpmostree1 \
|
|
--object-path=/org/projectatomic/rpmostree1 \
|
|
--method=org.projectatomic.rpmostree1.OS.Status
|
|
```
|
|
|
|
### Package Management Debugging
|
|
|
|
#### Package Resolution
|
|
```bash
|
|
# Debug package resolution
|
|
rpm-ostree --debug db diff $commit1 $commit2
|
|
|
|
# Check package conflicts
|
|
rpm-ostree --debug install --dry-run package-name
|
|
|
|
# Debug dependency resolution
|
|
rpm-ostree --debug override replace package-name
|
|
```
|
|
|
|
#### Package Installation
|
|
```bash
|
|
# Debug package installation
|
|
rpm-ostree --debug install package-name
|
|
|
|
# Check installation logs
|
|
journalctl -u rpm-ostreed --grep="package"
|
|
|
|
# Debug package scripts
|
|
rpm-ostree --debug install --apply-live package-name
|
|
```
|
|
|
|
### OSTree Debugging
|
|
|
|
#### Repository Debugging
|
|
```bash
|
|
# Check repository status
|
|
ostree fsck --repo=/ostree/repo
|
|
|
|
# Debug repository operations
|
|
ostree --verbose log $commit
|
|
|
|
# Check repository objects
|
|
ostree ls $commit
|
|
|
|
# Debug repository pull
|
|
ostree --verbose pull $remote $ref
|
|
```
|
|
|
|
#### Commit Debugging
|
|
```bash
|
|
# Examine commit metadata
|
|
ostree show $commit
|
|
|
|
# Debug commit differences
|
|
ostree diff $commit1 $commit2
|
|
|
|
# Check commit files
|
|
ostree ls -R $commit
|
|
```
|
|
|
|
## Common Debugging Scenarios
|
|
|
|
### Upgrade Failures
|
|
|
|
#### Debug Upgrade Process
|
|
```bash
|
|
# Enable debug output for upgrade
|
|
export G_MESSAGES_DEBUG=all
|
|
export RUST_LOG=debug
|
|
rpm-ostree --debug upgrade
|
|
|
|
# Check upgrade logs
|
|
journalctl -u rpm-ostreed --since="5 minutes ago"
|
|
|
|
# Debug specific upgrade step
|
|
rpm-ostree --debug upgrade --dry-run
|
|
```
|
|
|
|
#### Network Issues
|
|
```bash
|
|
# Check network connectivity
|
|
curl -I https://ostree.example.com/repo/
|
|
|
|
# Debug network operations
|
|
strace -e trace=network -f rpm-ostree upgrade
|
|
|
|
# Check DNS resolution
|
|
nslookup ostree.example.com
|
|
```
|
|
|
|
### Package Installation Issues
|
|
|
|
#### Dependency Conflicts
|
|
```bash
|
|
# Check package conflicts
|
|
rpm-ostree --debug install --dry-run package-name
|
|
|
|
# Debug dependency resolution
|
|
rpm-ostree --debug db diff $current $pending
|
|
|
|
# Check package metadata
|
|
rpm-ostree --debug db list --user
|
|
```
|
|
|
|
#### Script Execution Issues
|
|
```bash
|
|
# Debug package scripts
|
|
rpm-ostree --debug install --apply-live package-name
|
|
|
|
# Check script logs
|
|
journalctl -u rpm-ostreed --grep="script"
|
|
|
|
# Debug script environment
|
|
rpm-ostree --debug install --verbose package-name
|
|
```
|
|
|
|
### Boot Issues
|
|
|
|
#### Bootloader Problems
|
|
```bash
|
|
# Check bootloader configuration
|
|
ls -la /boot/loader/entries/
|
|
|
|
# Debug bootloader update
|
|
rpm-ostree --debug kargs --append="debug"
|
|
|
|
# Check bootloader logs
|
|
journalctl -u systemd-boot
|
|
```
|
|
|
|
#### Kernel Issues
|
|
```bash
|
|
# Debug kernel arguments
|
|
rpm-ostree --debug kargs
|
|
|
|
# Check kernel modules
|
|
rpm-ostree --debug db list --user | grep kernel
|
|
|
|
# Debug initramfs
|
|
rpm-ostree --debug reload
|
|
```
|
|
|
|
## Debug Output Analysis
|
|
|
|
### Log Analysis
|
|
|
|
#### Parsing Debug Logs
|
|
```bash
|
|
# Extract error messages
|
|
grep -i "error" debug.log
|
|
|
|
# Extract warning messages
|
|
grep -i "warning" debug.log
|
|
|
|
# Extract debug messages
|
|
grep -i "debug" debug.log
|
|
|
|
# Extract specific component logs
|
|
grep "rpmostree" debug.log
|
|
```
|
|
|
|
#### Performance Analysis
|
|
```bash
|
|
# Analyze timing information
|
|
grep "time" debug.log | awk '{print $1, $2, $NF}'
|
|
|
|
# Analyze memory usage
|
|
grep "memory" debug.log
|
|
|
|
# Analyze network operations
|
|
grep "network" debug.log
|
|
```
|
|
|
|
### Core Dump Analysis
|
|
|
|
#### Enabling Core Dumps
|
|
```bash
|
|
# Enable core dumps
|
|
ulimit -c unlimited
|
|
|
|
# Set core dump location
|
|
echo "/tmp/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern
|
|
|
|
# Run command that might crash
|
|
rpm-ostree status
|
|
```
|
|
|
|
#### Analyzing Core Dumps
|
|
```bash
|
|
# Analyze core dump
|
|
gdb rpm-ostree core.rpm-ostree.12345
|
|
|
|
# Get backtrace
|
|
(gdb) bt
|
|
|
|
# Get full backtrace
|
|
(gdb) bt full
|
|
|
|
# Examine registers
|
|
(gdb) info registers
|
|
```
|
|
|
|
## Debugging Best Practices
|
|
|
|
### Systematic Approach
|
|
1. **Reproduce the issue**: Ensure consistent reproduction
|
|
2. **Isolate the problem**: Narrow down the scope
|
|
3. **Gather information**: Collect logs and debug output
|
|
4. **Analyze the data**: Look for patterns and clues
|
|
5. **Test solutions**: Verify fixes work
|
|
|
|
### Documentation
|
|
1. **Record steps**: Document reproduction steps
|
|
2. **Save logs**: Preserve debug output and logs
|
|
3. **Note environment**: Document system configuration
|
|
4. **Track changes**: Record what was tried and results
|
|
|
|
### Tools Selection
|
|
1. **Start simple**: Use basic tools first
|
|
2. **Add complexity**: Use advanced tools as needed
|
|
3. **Combine tools**: Use multiple tools together
|
|
4. **Validate results**: Cross-check with different tools
|
|
|
|
---
|
|
|
|
*Effective debugging requires systematic approach, appropriate tools, and thorough analysis. This guide provides the foundation for troubleshooting rpm-ostree issues.* |