particle-os-tools/TROUBLESHOOTING_GUIDE.md

Particle-OS Troubleshooting Guide

This guide documents all issues encountered during Particle-OS development, their solutions, and common troubleshooting steps.

🔧 Issues Fixed & Solutions

1. Script Location Standardization

Issue: Mixed script locations causing confusion and PATH issues

• Some scripts in /usr/local/bin/
• Some scripts in project directory
• Inconsistent naming conventions

Solution: Implemented Option A - Install all scripts to /usr/local/bin/

• Created install-particle-os.sh for production installation
• Created dev-install.sh for development reinstallation
• Standardized script names:
  • apt-layer.sh → apt-layer
  • composefs-alternative.sh → composefs
  • bootc-alternative.sh → bootc
  • bootupd-alternative.sh → bootupd
  • orchestrator.sh → particle-orchestrator
  • oci-integration.sh → particle-oci
  • particle-logrotate.sh → particle-logrotate

Files Updated:

• install-particle-os.sh - Production installation script
• dev-install.sh - Development installation script
• orchestrator.sh - Updated to use standardized paths
• INSTALLATION.md - Complete installation documentation

2. fsverity Integration

Issue: Missing fsverity script causing orchestrator dependency failures

• Orchestrator looking for /usr/local/bin/fsverity
• Custom fsverity-alternative.sh not available

Solution: Use real Ubuntu fsverity package instead of custom script

• Updated orchestrator.sh to use system fsverity command
• Enhanced dependency checking to verify command -v fsverity
• Added fsverity installation instructions to documentation

Files Updated:

• orchestrator.sh - Changed FSVERITY_SCRIPT="fsverity"
• INSTALLATION.md - Added fsverity as system requirement
• install-particle-os.sh - Removed fsverity-alternative.sh from installation

3. Path Reference Issues

Issue: Scripts still referencing old paths after standardization

• apt-layer.sh looking for composefs-alternative.sh instead of composefs
• particle-config.sh using old script names

Solution: Updated all path references to use standardized names

• Updated COMPOSEFS_SCRIPT in apt-layer.sh
• Updated all script paths in particle-config.sh
• Ensured consistency across all configuration files

Files Updated:

• apt-layer.sh - COMPOSEFS_SCRIPT="/usr/local/bin/composefs"
• particle-config.sh - Updated all script paths to standardized names

4. Missing Logging Functions

Issue: composefs and bootupd scripts missing logging functions

• log_info: command not found errors
• Inconsistent logging across scripts

Solution: Need to add missing functions to source files

• Add log_info(), log_warning(), log_error() functions
• Ensure consistent logging format across all scripts
• Update source files and recompile

Status: ⚠️ PENDING - Need to update source files

5. Repetitive Initialization

Issue: apt-layer initializing multiple times during status command

• Recursive self-calls causing performance issues
• Multiple initialization messages

Solution: Fixed recursive calls in rpm-ostree compatibility layer

• Updated rpm_ostree_status() to call internal functions directly
• Updated rpm_ostree_install() to call internal functions directly
• Updated rpm_ostree_cancel() to call internal functions directly

Status: ✅ FIXED - Applied to runtime, needs source update

🚨 Current Issues & Status

High Priority

1. Missing Scripts in Project Directory

   • Issue: Scripts exist in project directory but dev-install.sh can't find them
   • Status: ✅ RESOLVED - Scripts are present: composefs-alternative.sh, bootc-alternative.sh, bootupd-alternative.sh
   • Root Cause: User running dev-install.sh from wrong directory
   • Solution: Run dev-install.sh from the project directory containing the scripts
   • Current State: composefs and bootupd are installed but have missing functions

2. Live Overlay Not Supported

   • Issue: System doesn't support live overlay in VM environment
   • Status: ⚠️ EXPECTED - Normal in VM, not a real issue
   • Impact: Limited testing of live overlay features

Medium Priority

3. Source File Updates Needed

   • Issue: Runtime fixes not applied to source files
   • Status: 📝 PENDING - Need to update source and recompile
   • Impact: Fixes not permanent across recompilations

4. Missing Functions in Compiled Scripts

   • Issue: composefs and bootupd scripts missing init_directories function
   • Status: 🔧 CONFIRMED - init_directories: command not found errors
   • Impact: Scripts fail to initialize properly
   • Solution: Need to add missing functions to source files and recompile

🛠️ Common Troubleshooting Steps

Installation Issues

# Check if scripts are executable
ls -la /usr/local/bin/apt-layer /usr/local/bin/composefs

# Verify PATH includes /usr/local/bin
echo $PATH | grep /usr/local/bin

# Check if configuration is loaded
ls -la /usr/local/etc/particle-config.sh

# Reinstall if needed
sudo ./dev-install.sh

Dependency Issues

# Install system dependencies
sudo apt install -y fsverity

# Check if fsverity is available
command -v fsverity

# Verify all Particle-OS scripts exist
which apt-layer composefs bootc bootupd particle-orchestrator

Permission Issues

# Fix script permissions
sudo chmod +x /usr/local/bin/apt-layer
sudo chmod +x /usr/local/bin/composefs
sudo chmod +x /usr/local/bin/bootc
sudo chmod +x /usr/local/bin/bootupd

# Fix ownership
sudo chown root:root /usr/local/bin/apt-layer

Configuration Issues

# Check configuration loading
sudo apt-layer --init

# Verify workspace directories
ls -la /var/lib/particle-os/
ls -la /var/log/particle-os/
ls -la /var/cache/particle-os/

Missing Function Issues

# Check for missing functions in scripts
grep -n "init_directories" /usr/local/bin/composefs
grep -n "init_directories" /usr/local/bin/bootupd

# Check what functions are defined
grep -n "^[a-zA-Z_][a-zA-Z0-9_]*()" /usr/local/bin/composefs | head -10
grep -n "^[a-zA-Z_][a-zA-Z0-9_]*()" /usr/local/bin/bootupd | head -10

Missing Script Issues

# Check what scripts are available in project directory
ls -la *.sh

# Check what scripts are installed
ls -la /usr/local/bin/apt-layer /usr/local/bin/composefs /usr/local/bin/bootc /usr/local/bin/bootupd

# Find missing source scripts
find . -name "*composefs*" -o -name "*bootc*" -o -name "*bootupd*"

📋 Testing Checklist

Pre-Installation

• All source scripts exist in project directory
• Installation scripts are executable
• System dependencies are installed (fsverity)

Post-Installation

• All scripts installed to /usr/local/bin/
• Scripts are executable and in PATH
• Configuration file is installed
• Workspace directories are created

Functionality Testing

• apt-layer --help works
• composefs --help works
• bootc --help works
• bootupd --help works
• particle-orchestrator help works
• apt-layer status works
• apt-layer --init works

🔄 Development Workflow

Making Changes

1. Edit source files in src/ directory
2. Recompile scripts using compile.sh
3. Test changes with sudo ./dev-install.sh
4. Verify functionality

Testing Changes

# Quick reinstall
sudo ./dev-install.sh

# Test specific component
sudo apt-layer --help
sudo composefs --help

# Test integration
sudo particle-orchestrator help

Production Deployment

# Full installation with backup
sudo ./install-particle-os.sh

# Verify installation
which apt-layer composefs bootc bootupd particle-orchestrator

📚 Reference Information

Standardized Script Names

Source	Installed As	Purpose
apt-layer.sh	apt-layer	Package layer management
composefs-alternative.sh	composefs	ComposeFS image management
bootc-alternative.sh	bootc	Bootable container management
bootupd-alternative.sh	bootupd	Bootloader management
orchestrator.sh	particle-orchestrator	System orchestration
oci-integration.sh	particle-oci	OCI integration
particle-logrotate.sh	particle-logrotate	Log rotation management

Key Directories

• /usr/local/bin/ - Installed scripts
• /usr/local/etc/particle-config.sh - Configuration file
• /var/lib/particle-os/ - Workspace directory
• /var/log/particle-os/ - Log directory
• /var/cache/particle-os/ - Cache directory

System Dependencies

• fsverity - File integrity verification
• podman or docker - Container runtime
• squashfs-tools - ComposeFS support
• jq - JSON processing
• coreutils - Basic utilities

🎯 Next Steps

1. Locate missing scripts - Find composefs, bootc, bootupd in project
2. Update source files - Apply runtime fixes to source
3. Test full integration - Verify all components work together
4. Document working process - Update README with working examples

---

Last Updated: 2025-01-27 Version: 1.0 Status: Active Development