Initial commit: Particle-OS tools repository
- Complete Particle-OS rebranding from uBlue-OS - Professional installation system with standardized paths - Self-initialization system with --init and --reset commands - Enhanced error messages and dependency checking - Comprehensive testing infrastructure - All source scriptlets updated with runtime improvements - Clean codebase with redundant files moved to archive - Complete documentation suite
This commit is contained in:
commit
74c7bede5f
125 changed files with 66318 additions and 0 deletions
337
src/composefs/README.md
Normal file
337
src/composefs/README.md
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
# Ubuntu uBlue ComposeFS Alternative - Modular System
|
||||
|
||||
A modular, self-contained alternative to ComposeFS for Ubuntu uBlue systems, providing content-addressable layered filesystem functionality using overlayfs and squashfs.
|
||||
|
||||
## Overview
|
||||
|
||||
This modular system breaks down the monolithic `composefs-alternative.sh` into logical, maintainable scriptlets that are compiled into a single self-contained executable. The system provides:
|
||||
|
||||
- **Content-addressable layers** with automatic deduplication
|
||||
- **Immutable layers** using SquashFS compression
|
||||
- **Multi-layer image support** with overlayfs mounting
|
||||
- **Parallel hash generation** for optimal performance
|
||||
- **Comprehensive cleanup** and maintenance tools
|
||||
- **Ubuntu uBlue integration** with unified configuration
|
||||
|
||||
## Architecture
|
||||
|
||||
### Modular Design
|
||||
|
||||
The system is organized into focused scriptlets that handle specific functionality:
|
||||
|
||||
```
|
||||
src/composefs/
|
||||
├── scriptlets/ # Individual functional modules
|
||||
│ ├── 00-header.sh # Header, shared functions, and utilities
|
||||
│ ├── 01-dependencies.sh # Dependency checking and validation
|
||||
│ ├── 02-hash.sh # Content hash generation (parallel processing)
|
||||
│ ├── 03-layers.sh # Layer management and creation
|
||||
│ ├── 04-images.sh # Image management and mounting
|
||||
│ ├── 05-listing.sh # Listing, reporting, and status functions
|
||||
│ ├── 06-cleanup.sh # Cleanup and maintenance operations
|
||||
│ └── 99-main.sh # Main dispatch and help system
|
||||
├── config/ # Configuration files (JSON)
|
||||
├── compile.sh # Build system for merging scriptlets
|
||||
└── README.md # This documentation
|
||||
```
|
||||
|
||||
### Scriptlet Functions
|
||||
|
||||
#### **00-header.sh** - Header and Shared Functions
|
||||
- Global cleanup variables and trap handlers
|
||||
- Security validation functions (`validate_path`, `validate_image_name`)
|
||||
- System introspection utilities (`get_system_info`, `calculate_disk_usage`)
|
||||
- Root privilege checking and directory initialization
|
||||
|
||||
#### **01-dependencies.sh** - Dependency Checking
|
||||
- Comprehensive dependency validation for all required tools
|
||||
- Kernel module availability checking (squashfs, overlay)
|
||||
- Detailed error reporting for missing components
|
||||
|
||||
#### **02-hash.sh** - Content Hash Generation
|
||||
- **Parallel hash generation** using xargs for optimal performance
|
||||
- Content-addressable layer ID creation
|
||||
- Fallback to sequential processing if parallel fails
|
||||
- Progress indication for large datasets
|
||||
|
||||
#### **03-layers.sh** - Layer Management
|
||||
- Layer creation with SquashFS compression
|
||||
- Content-addressable layer ID generation
|
||||
- Layer deduplication and existence checking
|
||||
- Layer mounting and cleanup
|
||||
|
||||
#### **04-images.sh** - Image Management
|
||||
- Multi-layer image creation from source directories
|
||||
- OverlayFS mounting with proper layer stacking
|
||||
- Mount point validation and cleanup
|
||||
- Image metadata management
|
||||
|
||||
#### **05-listing.sh** - Listing and Reporting
|
||||
- Comprehensive image, layer, and mount listing
|
||||
- Optimized layer reference counting with caching
|
||||
- System status reporting with health checks
|
||||
- Disk usage calculation (accounting for deduplication)
|
||||
|
||||
#### **06-cleanup.sh** - Cleanup and Maintenance
|
||||
- Unreferenced layer cleanup
|
||||
- Orphaned mount information cleanup
|
||||
- Image removal with dependency checking
|
||||
- Full system cleanup operations
|
||||
|
||||
#### **99-main.sh** - Main Dispatch
|
||||
- Command-line argument parsing
|
||||
- Comprehensive help system
|
||||
- Main function orchestration
|
||||
- Error handling and usage display
|
||||
|
||||
## Compilation System
|
||||
|
||||
### Build Process
|
||||
|
||||
The `compile.sh` script provides a sophisticated build system that:
|
||||
|
||||
1. **Validates dependencies** (jq, bash)
|
||||
2. **Checks scriptlet integrity** and syntax
|
||||
3. **Embeds configuration files** (JSON) with size warnings
|
||||
4. **Merges all scriptlets** in the correct order
|
||||
5. **Generates a self-contained executable** with proper headers
|
||||
6. **Validates the final script** syntax
|
||||
7. **Provides detailed progress reporting**
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Compile with default output path
|
||||
cd src/composefs
|
||||
bash compile.sh
|
||||
|
||||
# Compile with custom output path
|
||||
bash compile.sh -o /path/to/custom/composefs-alternative.sh
|
||||
|
||||
# Show help
|
||||
bash compile.sh -h
|
||||
```
|
||||
|
||||
### Output
|
||||
|
||||
The compilation produces `composefs-alternative.sh` with:
|
||||
- **Self-contained functionality** - no external dependencies beyond system tools
|
||||
- **Ubuntu uBlue integration** - sources `ublue-config.sh` if available
|
||||
- **Embedded configurations** - JSON configs embedded as associative arrays
|
||||
- **Comprehensive error handling** - robust validation and cleanup
|
||||
- **Performance optimizations** - parallel processing and caching
|
||||
|
||||
## Features
|
||||
|
||||
### Core Functionality
|
||||
- **Content-addressable layers**: SHA256-based layer identification
|
||||
- **Automatic deduplication**: Identical content creates single layer
|
||||
- **Multi-layer images**: Stack multiple layers for complex filesystems
|
||||
- **Immutable layers**: SquashFS compression ensures layer integrity
|
||||
- **OverlayFS mounting**: Read-write overlays on immutable base layers
|
||||
|
||||
### Performance Features
|
||||
- **Parallel hash generation**: Multi-core processing for large datasets
|
||||
- **Cached reference counting**: Optimized layer usage tracking
|
||||
- **Compression optimization**: XZ compression with progress indication
|
||||
- **Memory-efficient processing**: Streaming operations for large files
|
||||
|
||||
### Security Features
|
||||
- **Path traversal protection**: Validates all input paths
|
||||
- **Input sanitization**: Character set restrictions and validation
|
||||
- **Privilege escalation prevention**: Root requirement enforcement
|
||||
- **Secure temporary file handling**: Automatic cleanup with traps
|
||||
|
||||
### Management Features
|
||||
- **Comprehensive status reporting**: System health and usage information
|
||||
- **Automatic cleanup**: Unreferenced layer and orphaned mount cleanup
|
||||
- **Health monitoring**: Detection of orphaned mounts and unreferenced layers
|
||||
- **Detailed logging**: Integration with Ubuntu uBlue logging system
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Operations
|
||||
|
||||
```bash
|
||||
# Create a multi-layer image
|
||||
sudo ./composefs-alternative.sh create my-app /path/to/base /path/to/apps
|
||||
|
||||
# Mount the image
|
||||
sudo ./composefs-alternative.sh mount my-app /mnt/my-app
|
||||
|
||||
# List all images
|
||||
sudo ./composefs-alternative.sh list-images
|
||||
|
||||
# Show system status
|
||||
sudo ./composefs-alternative.sh status
|
||||
|
||||
# Clean up unreferenced layers
|
||||
sudo ./composefs-alternative.sh cleanup
|
||||
|
||||
# Unmount and remove
|
||||
sudo ./composefs-alternative.sh unmount /mnt/my-app
|
||||
sudo ./composefs-alternative.sh remove my-app
|
||||
```
|
||||
|
||||
### Advanced Usage
|
||||
|
||||
```bash
|
||||
# Create image with multiple layers
|
||||
sudo ./composefs-alternative.sh create complex-app \
|
||||
/path/to/base \
|
||||
/path/to/runtime \
|
||||
/path/to/applications \
|
||||
/path/to/configs
|
||||
|
||||
# List layers with reference counts
|
||||
sudo ./composefs-alternative.sh list-layers
|
||||
|
||||
# Check system health
|
||||
sudo ./composefs-alternative.sh status
|
||||
|
||||
# Full system cleanup
|
||||
sudo ./composefs-alternative.sh cleanup
|
||||
```
|
||||
|
||||
## System Requirements
|
||||
|
||||
### Dependencies
|
||||
- **Linux kernel**: squashfs and overlay modules
|
||||
- **squashfs-tools**: For layer compression and mounting
|
||||
- **jq**: JSON processing and validation
|
||||
- **coreutils**: System utilities (du, stat, etc.)
|
||||
- **util-linux**: Mount utilities (mount, umount, etc.)
|
||||
|
||||
### Installation
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt update
|
||||
sudo apt install squashfs-tools jq coreutils util-linux
|
||||
|
||||
# Ensure kernel modules are loaded
|
||||
sudo modprobe squashfs overlay
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Adding New Features
|
||||
|
||||
1. **Create a new scriptlet** in `scriptlets/` with appropriate numbering
|
||||
2. **Add the scriptlet** to `compile.sh` in the correct order
|
||||
3. **Update documentation** and examples
|
||||
4. **Test thoroughly** with various scenarios
|
||||
|
||||
### Scriptlet Guidelines
|
||||
|
||||
- **Single responsibility**: Each scriptlet should handle one functional area
|
||||
- **Error handling**: Use `log_error` and `log_warning` from ublue-config.sh
|
||||
- **Security**: Validate all inputs and sanitize paths
|
||||
- **Performance**: Consider parallel processing for expensive operations
|
||||
- **Documentation**: Include clear comments and usage examples
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Test compilation
|
||||
cd src/composefs
|
||||
bash compile.sh
|
||||
|
||||
# Test syntax validation
|
||||
bash -n ../composefs-alternative.sh
|
||||
|
||||
# Test basic functionality
|
||||
sudo ./composefs-alternative.sh help
|
||||
sudo ./composefs-alternative.sh status
|
||||
```
|
||||
|
||||
## Integration with Ubuntu uBlue
|
||||
|
||||
The ComposeFS alternative integrates seamlessly with Ubuntu uBlue systems:
|
||||
|
||||
- **Configuration sourcing**: Automatically sources `ublue-config.sh`
|
||||
- **Unified logging**: Uses uBlue logging functions and conventions
|
||||
- **Path consistency**: Follows uBlue directory structure conventions
|
||||
- **Error handling**: Consistent with uBlue error reporting patterns
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Large Datasets
|
||||
- **Parallel processing**: Hash generation uses multiple CPU cores
|
||||
- **Compression**: XZ compression reduces storage requirements
|
||||
- **Deduplication**: Identical content creates single layers
|
||||
- **Caching**: Layer reference counting is cached for performance
|
||||
|
||||
### Memory Usage
|
||||
- **Streaming operations**: Large files are processed in streams
|
||||
- **Temporary file management**: Automatic cleanup prevents disk bloat
|
||||
- **Progress indication**: Long operations show progress to prevent timeouts
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Missing dependencies**: Install required packages (squashfs-tools, jq)
|
||||
2. **Kernel modules**: Ensure squashfs and overlay modules are loaded
|
||||
3. **Permissions**: Script requires root privileges for filesystem operations
|
||||
4. **Disk space**: Ensure sufficient space for layer creation and mounting
|
||||
|
||||
### Debug Information
|
||||
|
||||
```bash
|
||||
# Check system status
|
||||
sudo ./composefs-alternative.sh status
|
||||
|
||||
# Verify dependencies
|
||||
sudo ./composefs-alternative.sh help
|
||||
|
||||
# Check kernel modules
|
||||
lsmod | grep -E "(squashfs|overlay)"
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
- **External configuration loading**: Support for large external config files
|
||||
- **Compression options**: Configurable compression algorithms
|
||||
- **Layer encryption**: Optional layer encryption for sensitive data
|
||||
- **Network layer support**: Remote layer fetching and caching
|
||||
- **API integration**: REST API for remote management
|
||||
|
||||
### Scalability Improvements
|
||||
- **Distributed processing**: Multi-node hash generation
|
||||
- **Layer streaming**: Streaming layer creation for very large datasets
|
||||
- **Compression optimization**: Adaptive compression based on content type
|
||||
- **Cache management**: Intelligent layer caching and eviction
|
||||
|
||||
## Contributing
|
||||
|
||||
### Development Workflow
|
||||
1. **Fork the repository** and create a feature branch
|
||||
2. **Add new scriptlets** or modify existing ones
|
||||
3. **Update the compile script** if adding new modules
|
||||
4. **Test thoroughly** with various scenarios
|
||||
5. **Update documentation** and examples
|
||||
6. **Submit a pull request** with detailed description
|
||||
|
||||
### Code Standards
|
||||
- **Bash best practices**: Follow shell scripting best practices
|
||||
- **Error handling**: Comprehensive error checking and reporting
|
||||
- **Security**: Input validation and sanitization
|
||||
- **Documentation**: Clear comments and usage examples
|
||||
- **Testing**: Include test cases for new functionality
|
||||
|
||||
## License
|
||||
|
||||
This project follows the same license as the main Ubuntu uBlue System Tools project.
|
||||
|
||||
## Support
|
||||
|
||||
For issues, questions, or contributions:
|
||||
- **Documentation**: Check this README and inline comments
|
||||
- **Examples**: Review usage examples in the help system
|
||||
- **Testing**: Use the status command for system diagnostics
|
||||
- **Development**: Follow the modular development guidelines
|
||||
|
||||
---
|
||||
|
||||
**Note**: This modular structure provides the best of both worlds - organized development with unified deployment. The compile script ensures that users always get a single, self-contained script while developers can work on individual components efficiently. The compilation system is not just a simple concatenation tool, but a sophisticated build system that handles complex requirements while maintaining simplicity and reliability.
|
||||
Loading…
Add table
Add a link
Reference in a new issue