particle-os-tools/src/composefs/README.md

Ubuntu uBlue ComposeFS Alternative - Modular System

A modular, self-contained alternative to ComposeFS for Ubuntu uBlue systems, providing content-addressable layered filesystem functionality using overlayfs and squashfs.

Overview

This modular system breaks down the monolithic composefs-alternative.sh into logical, maintainable scriptlets that are compiled into a single self-contained executable. The system provides:

• Content-addressable layers with automatic deduplication
• Immutable layers using SquashFS compression
• Multi-layer image support with overlayfs mounting
• Parallel hash generation for optimal performance
• Comprehensive cleanup and maintenance tools
• Ubuntu uBlue integration with unified configuration

Architecture

Modular Design

The system is organized into focused scriptlets that handle specific functionality:

src/composefs/
├── scriptlets/           # Individual functional modules
│   ├── 00-header.sh     # Header, shared functions, and utilities
│   ├── 01-dependencies.sh # Dependency checking and validation
│   ├── 02-hash.sh       # Content hash generation (parallel processing)
│   ├── 03-layers.sh     # Layer management and creation
│   ├── 04-images.sh     # Image management and mounting
│   ├── 05-listing.sh    # Listing, reporting, and status functions
│   ├── 06-cleanup.sh    # Cleanup and maintenance operations
│   └── 99-main.sh       # Main dispatch and help system
├── config/              # Configuration files (JSON)
├── compile.sh           # Build system for merging scriptlets
└── README.md           # This documentation

Scriptlet Functions

00-header.sh - Header and Shared Functions

• Global cleanup variables and trap handlers
• Security validation functions (validate_path, validate_image_name)
• System introspection utilities (get_system_info, calculate_disk_usage)
• Root privilege checking and directory initialization

01-dependencies.sh - Dependency Checking

• Comprehensive dependency validation for all required tools
• Kernel module availability checking (squashfs, overlay)
• Detailed error reporting for missing components

02-hash.sh - Content Hash Generation

• Parallel hash generation using xargs for optimal performance
• Content-addressable layer ID creation
• Fallback to sequential processing if parallel fails
• Progress indication for large datasets

03-layers.sh - Layer Management

• Layer creation with SquashFS compression
• Content-addressable layer ID generation
• Layer deduplication and existence checking
• Layer mounting and cleanup

04-images.sh - Image Management

• Multi-layer image creation from source directories
• OverlayFS mounting with proper layer stacking
• Mount point validation and cleanup
• Image metadata management

05-listing.sh - Listing and Reporting

• Comprehensive image, layer, and mount listing
• Optimized layer reference counting with caching
• System status reporting with health checks
• Disk usage calculation (accounting for deduplication)

06-cleanup.sh - Cleanup and Maintenance

• Unreferenced layer cleanup
• Orphaned mount information cleanup
• Image removal with dependency checking
• Full system cleanup operations

99-main.sh - Main Dispatch

• Command-line argument parsing
• Comprehensive help system
• Main function orchestration
• Error handling and usage display

Compilation System

Build Process

The compile.sh script provides a sophisticated build system that:

1. Validates dependencies (jq, bash)
2. Checks scriptlet integrity and syntax
3. Embeds configuration files (JSON) with size warnings
4. Merges all scriptlets in the correct order
5. Generates a self-contained executable with proper headers
6. Validates the final script syntax
7. Provides detailed progress reporting

Usage

# Compile with default output path
cd src/composefs
bash compile.sh

# Compile with custom output path
bash compile.sh -o /path/to/custom/composefs-alternative.sh

# Show help
bash compile.sh -h

Output

The compilation produces composefs-alternative.sh with:

• Self-contained functionality - no external dependencies beyond system tools
• Ubuntu uBlue integration - sources ublue-config.sh if available
• Embedded configurations - JSON configs embedded as associative arrays
• Comprehensive error handling - robust validation and cleanup
• Performance optimizations - parallel processing and caching

Features

Core Functionality

• Content-addressable layers: SHA256-based layer identification
• Automatic deduplication: Identical content creates single layer
• Multi-layer images: Stack multiple layers for complex filesystems
• Immutable layers: SquashFS compression ensures layer integrity
• OverlayFS mounting: Read-write overlays on immutable base layers

Performance Features

• Parallel hash generation: Multi-core processing for large datasets
• Cached reference counting: Optimized layer usage tracking
• Compression optimization: XZ compression with progress indication
• Memory-efficient processing: Streaming operations for large files

Security Features

• Path traversal protection: Validates all input paths
• Input sanitization: Character set restrictions and validation
• Privilege escalation prevention: Root requirement enforcement
• Secure temporary file handling: Automatic cleanup with traps

Management Features

• Comprehensive status reporting: System health and usage information
• Automatic cleanup: Unreferenced layer and orphaned mount cleanup
• Health monitoring: Detection of orphaned mounts and unreferenced layers
• Detailed logging: Integration with Ubuntu uBlue logging system

Usage Examples

Basic Operations

# Create a multi-layer image
sudo ./composefs-alternative.sh create my-app /path/to/base /path/to/apps

# Mount the image
sudo ./composefs-alternative.sh mount my-app /mnt/my-app

# List all images
sudo ./composefs-alternative.sh list-images

# Show system status
sudo ./composefs-alternative.sh status

# Clean up unreferenced layers
sudo ./composefs-alternative.sh cleanup

# Unmount and remove
sudo ./composefs-alternative.sh unmount /mnt/my-app
sudo ./composefs-alternative.sh remove my-app

Advanced Usage

# Create image with multiple layers
sudo ./composefs-alternative.sh create complex-app \
    /path/to/base \
    /path/to/runtime \
    /path/to/applications \
    /path/to/configs

# List layers with reference counts
sudo ./composefs-alternative.sh list-layers

# Check system health
sudo ./composefs-alternative.sh status

# Full system cleanup
sudo ./composefs-alternative.sh cleanup

System Requirements

Dependencies

• Linux kernel: squashfs and overlay modules
• squashfs-tools: For layer compression and mounting
• jq: JSON processing and validation
• coreutils: System utilities (du, stat, etc.)
• util-linux: Mount utilities (mount, umount, etc.)

Installation

# Ubuntu/Debian
sudo apt update
sudo apt install squashfs-tools jq coreutils util-linux

# Ensure kernel modules are loaded
sudo modprobe squashfs overlay

Development

Adding New Features

1. Create a new scriptlet in scriptlets/ with appropriate numbering
2. Add the scriptlet to compile.sh in the correct order
3. Update documentation and examples
4. Test thoroughly with various scenarios

Scriptlet Guidelines

• Single responsibility: Each scriptlet should handle one functional area
• Error handling: Use log_error and log_warning from ublue-config.sh
• Security: Validate all inputs and sanitize paths
• Performance: Consider parallel processing for expensive operations
• Documentation: Include clear comments and usage examples

Testing

# Test compilation
cd src/composefs
bash compile.sh

# Test syntax validation
bash -n ../composefs-alternative.sh

# Test basic functionality
sudo ./composefs-alternative.sh help
sudo ./composefs-alternative.sh status

Integration with Ubuntu uBlue

The ComposeFS alternative integrates seamlessly with Ubuntu uBlue systems:

• Configuration sourcing: Automatically sources ublue-config.sh
• Unified logging: Uses uBlue logging functions and conventions
• Path consistency: Follows uBlue directory structure conventions
• Error handling: Consistent with uBlue error reporting patterns

Performance Considerations

Large Datasets

• Parallel processing: Hash generation uses multiple CPU cores
• Compression: XZ compression reduces storage requirements
• Deduplication: Identical content creates single layers
• Caching: Layer reference counting is cached for performance

Memory Usage

• Streaming operations: Large files are processed in streams
• Temporary file management: Automatic cleanup prevents disk bloat
• Progress indication: Long operations show progress to prevent timeouts

Troubleshooting

Common Issues

1. Missing dependencies: Install required packages (squashfs-tools, jq)
2. Kernel modules: Ensure squashfs and overlay modules are loaded
3. Permissions: Script requires root privileges for filesystem operations
4. Disk space: Ensure sufficient space for layer creation and mounting

Debug Information

# Check system status
sudo ./composefs-alternative.sh status

# Verify dependencies
sudo ./composefs-alternative.sh help

# Check kernel modules
lsmod | grep -E "(squashfs|overlay)"

Future Enhancements

Planned Features

• External configuration loading: Support for large external config files
• Compression options: Configurable compression algorithms
• Layer encryption: Optional layer encryption for sensitive data
• Network layer support: Remote layer fetching and caching
• API integration: REST API for remote management

Scalability Improvements

• Distributed processing: Multi-node hash generation
• Layer streaming: Streaming layer creation for very large datasets
• Compression optimization: Adaptive compression based on content type
• Cache management: Intelligent layer caching and eviction

Contributing

Development Workflow

1. Fork the repository and create a feature branch
2. Add new scriptlets or modify existing ones
3. Update the compile script if adding new modules
4. Test thoroughly with various scenarios
5. Update documentation and examples
6. Submit a pull request with detailed description

Code Standards

• Bash best practices: Follow shell scripting best practices
• Error handling: Comprehensive error checking and reporting
• Security: Input validation and sanitization
• Documentation: Clear comments and usage examples
• Testing: Include test cases for new functionality

License

This project follows the same license as the main Ubuntu uBlue System Tools project.

Support

For issues, questions, or contributions:

• Documentation: Check this README and inline comments
• Examples: Review usage examples in the help system
• Testing: Use the status command for system diagnostics
• Development: Follow the modular development guidelines

---

Note: This modular structure provides the best of both worlds - organized development with unified deployment. The compile script ensures that users always get a single, self-contained script while developers can work on individual components efficiently. The compilation system is not just a simple concatenation tool, but a sophisticated build system that handles complex requirements while maintaining simplicity and reliability.