2025-07-14 08:07:01 -07:00

18 KiB

Raw Blame History

Particle-OS apt-layer Tool - Modular Structure

This directory contains the modular source code for the Particle-OS apt-layer Tool, organized into logical scriptlets that are compiled into a single unified script.

Particle-OS is aiming to be a near 1:1 implementation of the Fedora ublue-os, but using Ubuntu/Debian as a base. ublue-os uses rpm-ostree which obviously won't work on Ubuntu/Debian systems. apt-layer.sh is a proof of concept to make a 1:1 feature complete imitation of rpm-ostree (knowing we're using apt and not dnf, keeping those limitations in mind).

📁 Directory Structure

src/apt-layer/
├── compile.sh              # Compilation script (merges all scriptlets)
├── config/                 # Configuration files (JSON)
│   ├── apt-layer-settings.json     # Main configuration
│   └── package-validation.json     # Package validation rules
├── scriptlets/             # Individual scriptlet files
│   ├── 00-header.sh        # Shared utility functions, global cleanup, system detection
│   ├── 01-dependencies.sh  # Dependency checking and validation
│   ├── 02-transactions.sh  # Transaction management and rollback
│   ├── 03-traditional.sh   # Traditional chroot-based layer creation
│   ├── 04-container.sh     # Container-based layer creation (Apx-style)
│   ├── 05-live-overlay.sh  # Live system layering (rpm-ostree style)
│   ├── 06-oci-integration.sh # OCI export/import functionality
│   ├── 07-bootloader.sh    # Bootloader integration
│   ├── 09-atomic-deployment.sh # Atomic deployment system
│   ├── 10-rpm-ostree-compat.sh # rpm-ostree compatibility layer
│   ├── 15-ostree-atomic.sh # OSTree atomic package management
│   ├── 24-dpkg-direct-install.sh # Direct dpkg Installation (Performance Optimization)
│   └── 99-main.sh          # Main dispatch and help
├── README.md               # This file
└── CHANGELOG.md            # Version history and changes

🚀 Usage

Compiling the Unified Script

# Navigate to the apt-layer directory
cd src/apt-layer

# Run the compilation script
bash compile.sh

This will generate apt-layer.sh in the project root directory.

Development Workflow

Edit Individual Scriptlets: Modify the specific scriptlet files in scriptlets/
Test Changes: Make your changes and test individual components
Compile: Run bash compile.sh to merge all scriptlets
Deploy: The unified apt-layer.sh is ready for distribution

📋 Scriptlet Descriptions

Core Scriptlets (All Implemented)

00-header.sh: Shared utility functions, global cleanup, and system detection helpers
01-dependencies.sh: Package dependency validation and kernel module checking
02-transactions.sh: Transaction management with automatic rollback
03-traditional.sh: Traditional chroot-based layer creation
04-container.sh: Container-based layer creation (Apx-style) ✅ IMPLEMENTED
05-live-overlay.sh: Live system layering (rpm-ostree style) ✅ IMPLEMENTED
06-oci-integration.sh: OCI export/import functionality ✅ IMPLEMENTED
07-bootloader.sh: Bootloader integration (UEFI/GRUB/systemd-boot) ✅ IMPLEMENTED
09-atomic-deployment.sh: Atomic deployment system ✅ IMPLEMENTED
10-rpm-ostree-compat.sh: rpm-ostree compatibility layer ✅ IMPLEMENTED
15-ostree-atomic.sh: OSTree atomic package management ✅ IMPLEMENTED
24-dpkg-direct-install.sh: Direct dpkg Installation (Performance Optimization) ✅ IMPLEMENTED
99-main.sh: Main command dispatch and help system

🔧 Benefits of This Structure

✅ Modular Development

Each component can be developed and tested independently
Easy to locate and modify specific functionality
Clear separation of concerns

✅ Unified Deployment

Single apt-layer.sh file for end users
No complex dependency management
Professional distribution format

✅ Maintainable Code

Logical organization by functionality
Easy to add new features
Clear documentation per component

✅ Version Control Friendly

Small, focused files are easier to review
Clear commit history per feature
Reduced merge conflicts

🏗️ Architecture Overview

Core Components

Transaction Management: Atomic operations with automatic rollback
ComposeFS Integration: Uses the modular ComposeFS backend
Dependency Validation: Comprehensive dependency checking
Error Handling: Robust error handling and recovery

Layer Creation Methods

Traditional: Chroot-based package installation ✅ IMPLEMENTED
- Uses chroot for isolation
- Direct package installation in isolated environment
- Suitable for most use cases
Container (Apx-style): Isolated container-based installation ✅ IMPLEMENTED
- Uses container technology (Podman/Docker/systemd-nspawn)
- Complete isolation from host system
- Reproducible and secure package installation
- Named after the Apx package manager's isolation approach
Live Overlay: Runtime package installation ✅ IMPLEMENTED
- Installs packages on running system using overlayfs
- Provides immediate package availability
- Supports commit/rollback operations

Integration Points

ComposeFS Backend: Uses the modular composefs-alternative.sh
Particle-OS Config: Integrates with unified configuration system
Bootloader: Automatic boot entry management ✅ IMPLEMENTED
OCI: Container image export/import ✅ IMPLEMENTED

🚀 Quick Start

Basic Layer Creation

# Create a gaming layer
sudo ./apt-layer.sh particle-os/base/24.04 particle-os/gaming/24.04 steam wine

# List available images
sudo ./apt-layer.sh --list

# Show image information
sudo ./apt-layer.sh --info particle-os/gaming/24.04

Container-based Layer Creation (Apx-style Isolation)

Apx-style isolation follows the same architectural pattern as the Apx package manager, which uses OSTree + Container Isolation. This approach provides a secure, isolated environment for package installation while maintaining the immutable, layered system architecture.

Architecture Overview:

Base System: Immutable base layer (OSTree in Apx, ComposeFS in apt-layer)
Isolation Layer: Container-based package installation
Result: New immutable layer with container-derived changes

Key Benefits:

Isolation: Package installations run in isolated containers, preventing conflicts with the host system
Reproducibility: Same packages installed in the same base image always produce identical results
Security: Container isolation prevents malicious packages from affecting the host
Clean Environment: Each installation starts with a fresh, clean base image
Multi-runtime Support: Works with Podman, Docker, or systemd-nspawn
Immutable Layers: Results in immutable, atomic layers just like OSTree

How it Works (Same Pattern as Apx):

Starts with immutable base layer (ComposeFS instead of OSTree)
Creates a temporary container from the base image
Installs packages inside the isolated container environment
Extracts the changes and creates a new immutable layer
Cleans up the temporary container
Results in an immutable layer that can be deployed atomically

Comparison with rpm-ostree and Apx:

Aspect	rpm-ostree	Apx	apt-layer (Apx-style)
Base System	OSTree commits	OSTree commits	ComposeFS layers
Isolation Method	Direct RPM installation	Container isolation	Container isolation
Package Format	RPM packages	RPM packages	DEB packages
Installation Process	Direct in OSTree tree	Container → OSTree commit	Container → ComposeFS layer
Dependency Resolution	DNF/RPM	DNF/RPM	APT
Transaction Safety	OSTree atomic	OSTree atomic	ComposeFS atomic
Reproducibility	OSTree commit hashes	OSTree commit hashes	ComposeFS layer hashes
Cross-Platform	Red Hat/Fedora	Red Hat/Fedora	Ubuntu/Debian

Key Architectural Similarities:

Apx: OSTree base + Container isolation + OSTree commits = Immutable layers
apt-layer: ComposeFS base + Container isolation + ComposeFS layers = Immutable layers
rpm-ostree: OSTree base + Direct installation + OSTree commits = Immutable layers

The Pattern is the Same: All three approaches create immutable, layered systems. The difference is in the underlying technologies:

rpm-ostree: Uses OSTree + RPM directly
Apx: Uses OSTree + Container isolation + RPM
apt-layer: Uses ComposeFS + Container isolation + APT

What is ComposeFS? ComposeFS is a modern filesystem technology that provides immutable, layered filesystem capabilities similar to OSTree. It creates read-only, deduplicated layers that can be composed together to form a complete filesystem. Each layer contains the changes from the previous layer, creating an immutable, versioned filesystem structure. ComposeFS layers are content-addressable (identified by cryptographic hashes) and can be efficiently shared and distributed, making it ideal for container images and immutable system deployments.

How ComposeFS Works:

Base Layer: Starts with an immutable base filesystem layer
Layer Creation: New layers contain only the changes (deltas) from the previous layer
Composition: Multiple layers are composed together to create the final filesystem
Immutability: Each layer is read-only and cannot be modified once created
Efficiency: Deduplication and compression reduce storage requirements
Distribution: Layers can be independently distributed and cached

We're not departing from how rpm-ostree works - we're following the same immutable layering pattern but with Ubuntu/Debian technologies instead of Red Hat technologies.

# Container-based layer creation (Apx-style)
sudo ./apt-layer.sh --container particle-os/base/24.04 particle-os/gaming/24.04 steam wine

# Container-based layer with multiple packages
sudo ./apt-layer.sh --container particle-os/base/24.04 particle-os/dev/24.04 vscode git nodejs npm

# Container status and information
sudo ./apt-layer.sh --container-status

Live System Installation

# Live system installation (rpm-ostree style)
sudo ./apt-layer.sh --live-install firefox

# Live overlay management
sudo ./apt-layer.sh --live-overlay start
sudo ./apt-layer.sh --live-overlay status
sudo ./apt-layer.sh --live-overlay commit
sudo ./apt-layer.sh --live-overlay rollback

Direct dpkg Installation (Performance Optimization)

# Direct dpkg installation (faster, more controlled)
sudo ./apt-layer.sh --dpkg-install curl wget

# Container-based dpkg installation
sudo ./apt-layer.sh --container-dpkg particle-os/base/24.04 particle-os/dev/24.04 vscode git

# Live system dpkg installation
sudo ./apt-layer.sh --live-dpkg firefox

# Download-only mode (for offline installation)
DPKG_DOWNLOAD_ONLY=true sudo ./apt-layer.sh --dpkg-install curl

# Force installation with dependency issues
DPKG_FORCE_DEPENDS=true sudo ./apt-layer.sh --dpkg-install package-name

# Use specific chroot directory
DPKG_CHROOT_DIR=/path/to/chroot sudo ./apt-layer.sh --dpkg-install package-name

OCI Integration

# OCI export
sudo ./apt-layer.sh --oci-export ubuntu-ublue/gaming/24.04 my-registry/gaming:latest

# OCI import
sudo ./apt-layer.sh --oci-import my-registry/gaming:latest ubuntu-ublue/gaming/24.04

# OCI status
sudo ./apt-layer.sh --oci-status

Bootloader Management

# Bootloader management
sudo ./apt-layer.sh bootloader status
sudo ./apt-layer.sh bootloader list-entries
sudo ./apt-layer.sh bootloader set-default particle-os/gaming/24.04

Kernel arguments (rpm-ostree compatibility)

sudo ./apt-layer.sh kargs add rd.break=pre-mount
sudo ./apt-layer.sh kargs list
sudo ./apt-layer.sh kargs remove rd.break=pre-mount

OSTree Atomic Package Management

# Atomic OSTree package management
sudo ./apt-layer.sh ostree compose install firefox vlc
sudo ./apt-layer.sh ostree compose remove package-name
sudo ./apt-layer.sh ostree compose update

# View atomic history
sudo ./apt-layer.sh ostree log
sudo ./apt-layer.sh ostree diff commit1 commit2
sudo ./apt-layer.sh ostree status
sudo ./apt-layer.sh ostree rollback commit-id
sudo ./apt-layer.sh ostree cleanup

rpm-ostree Compatibility

# Full rpm-ostree command compatibility
sudo ./apt-layer.sh install firefox
sudo ./apt-layer.sh upgrade
sudo ./apt-layer.sh rebase particle-os/gaming/24.04
sudo ./apt-layer.sh rollback
sudo ./apt-layer.sh status
sudo ./apt-layer.sh diff
sudo ./apt-layer.sh db list
sudo ./apt-layer.sh cleanup

🔧 Configuration

The apt-layer tool integrates with the Particle-OS configuration system and includes a comprehensive JSON-based configuration system:

Particle-OS Integration

# Configuration is automatically loaded from:
# /usr/local/etc/particle-config.sh

# Key configuration variables:
WORKSPACE="/var/lib/particle-os"
COMPOSEFS_SCRIPT="/usr/local/bin/composefs-alternative.sh"
CONTAINER_RUNTIME="podman"

JSON Configuration System

The tool includes embedded JSON configuration files for enterprise-grade configurability:

apt-layer-settings.json: Global settings, feature toggles, and defaults
security-policy.json: Security policies, signature requirements, and blocked packages
users.json: RBAC user definitions and access control
audit-settings.json: Audit logging policies and compliance frameworks
backup-policy.json: Backup frequency, retention, and encryption settings
signing-policy.json: Layer signing methods and trusted keys
oci-settings.json: OCI registry configuration and authentication
package-management.json: Repository policies and dependency resolution
maintenance.json: Automated maintenance and cleanup policies

All configuration files are automatically embedded in the compiled script and can be overridden via command-line arguments for enterprise deployment flexibility.

🛠️ Development Guidelines

Adding New Scriptlets

Create the scriptlet file in scriptlets/ with appropriate naming
Add to compile.sh in the correct order
Update this README with the new scriptlet description
Test thoroughly before committing

Scriptlet Naming Convention

00-header.sh: Shared utility functions, global cleanup, and system detection
01-XX.sh: Dependencies and validation
02-XX.sh: Core functionality
03-XX.sh: Layer creation methods
04-XX.sh: Advanced features
05-XX.sh: Live system features
06-XX.sh: OCI integration
07-XX.sh: Bootloader integration
09-XX.sh: Atomic deployment
10-XX.sh: Compatibility layers
15-XX.sh: OSTree atomic features
24-XX.sh: Performance optimizations
99-main.sh: Main dispatch (always last)

Error Handling

All scriptlets should:

Use the unified logging system (log_info, log_error, etc.)
Include proper error handling and cleanup
Integrate with transaction management when appropriate

ComposeFS Modular System: Backend filesystem layer
BootC Modular System: Container-native boot system
Particle-OS Configuration: Unified configuration system

🎯 Development Phases

✅ Phase 1: Core Stability (COMPLETED)

Modular architecture implementation
Transaction management system
Traditional layer creation
ComposeFS backend integration

✅ Phase 2: Enhanced Features (COMPLETED)

Container-based layer creation
OCI integration
Live system layering

✅ Phase 3: Bootloader Integration (COMPLETED)

Multi-bootloader support (UEFI/GRUB/systemd-boot)
Kernel arguments management
Boot entry management
Atomic deployment integration

✅ Phase 4: OSTree Atomic Package Management (COMPLETED)

OSTree atomic commits for package operations
Atomic deployment with rollback capabilities
Versioned package history
Direct dpkg installation optimization
Live overlay system with DNS fixes

✅ Phase 5: rpm-ostree Compatibility (COMPLETED)

Full rpm-ostree command compatibility
Atomic deployment system
Live overlay system
Bootloader integration
OCI integration

🎯 Current Status

✅ COMPLETED MAJOR MILESTONES:

OSTree/Atomic Workflow Implemented:
- All apt-layer ostree compose commands (install, remove, update) create atomic, versioned commits
- apt-layer ostree log, diff, status, rollback, cleanup fully implemented and tested
- Overlay and dpkg install workflow robust, with DNS fixes for WSL and offline .deb install support
- Log function bug fixed (commit history now displays correctly)
Testing & Validation:
- All atomic/OSTree commands tested and confirmed functional
- Overlay and atomic install workflows validated, including rollback readiness

🔄 NEXT PRIORITIES:

Further test rollback and deployment activation
Document overlay/atomic best practices and known caveats
Continue integration and optimization of atomic/OSTree workflow
Add more automated tests for edge cases (optional)

🛠️ COMPILATION SYSTEM ENHANCEMENTS:

Add source file dependency validation - Validate that all required functions exist in source
- Add validation that all required functions exist in source scriptlets
- Add dependency validation during compilation
- Add error checking for missing source files
- Add function dependency graph validation
- Add cross-scriptlet function reference checking

🎯 Scope Reduction Summary

As of July 2025, Particle-OS apt-layer has been successfully reduced to core rpm-ostree-like features only. All advanced, enterprise, cloud, multi-tenant, admin, compliance, and security features have been archived to archive/apt-layer/scriptlets/.

Current Focus:

Atomic deployment, rollback, status, diff, cleanup - Core rpm-ostree functionality
Live overlay and container-based layering - Immutable system management
Bootloader and kargs management - System boot configuration
OCI/ComposeFS integration - Container and filesystem integration
Direct dpkg install - Performance optimization for apt/deb systems
OSTree atomic package management - True atomic package operations

18 KiB Raw Blame History