# Particle-OS apt-layer Tool - Modular Structure

This directory contains the modular source code for the Particle-OS apt-layer Tool, organized into logical scriptlets that are compiled into a single unified script.

Particle-OS is aiming to be a near 1:1 implementation of the Fedora ublue-os, but using Ubuntu/Debian as a base. ublue-os uses rpm-ostree which obviously won't work on Ubuntu/Debian systems. apt-layer.sh is a proof of concept to make a 1:1 feature complete imitation of rpm-ostree (knowing we're using apt and not dnf, keeping those limitations in mind).

## 📁 Directory Structure

```
src/apt-layer/
├── compile.sh              # Compilation script (merges all scriptlets)
├── config/                 # Configuration files (JSON)
│   ├── apt-layer-settings.json     # Main configuration
│   └── package-validation.json     # Package validation rules
├── scriptlets/             # Individual scriptlet files
│   ├── 00-header.sh        # Shared utility functions, global cleanup, system detection
│   ├── 01-dependencies.sh  # Dependency checking and validation
│   ├── 02-transactions.sh  # Transaction management and rollback
│   ├── 03-traditional.sh   # Traditional chroot-based layer creation
│   ├── 04-container.sh     # Container-based layer creation (Apx-style)
│   ├── 05-live-overlay.sh  # Live system layering (rpm-ostree style)
│   ├── 06-oci-integration.sh # OCI export/import functionality
│   ├── 07-bootloader.sh    # Bootloader integration
│   ├── 09-atomic-deployment.sh # Atomic deployment system
│   ├── 10-rpm-ostree-compat.sh # rpm-ostree compatibility layer
│   ├── 15-ostree-atomic.sh # OSTree atomic package management
│   ├── 24-dpkg-direct-install.sh # Direct dpkg Installation (Performance Optimization)
│   └── 99-main.sh          # Main dispatch and help
├── README.md               # This file
└── CHANGELOG.md            # Version history and changes
```

## 🚀 Usage

### Compiling the Unified Script

```bash
# Navigate to the apt-layer directory
cd src/apt-layer

# Run the compilation script
bash compile.sh
```

This will generate `apt-layer.sh` in the project root directory.

### Development Workflow

1. **Edit Individual Scriptlets**: Modify the specific scriptlet files in `scriptlets/`
2. **Test Changes**: Make your changes and test individual components
3. **Compile**: Run `bash compile.sh` to merge all scriptlets
4. **Deploy**: The unified `apt-layer.sh` is ready for distribution

## 📋 Scriptlet Descriptions

### Core Scriptlets (All Implemented)

- **00-header.sh**: Shared utility functions, global cleanup, and system detection helpers
- **01-dependencies.sh**: Package dependency validation and kernel module checking
- **02-transactions.sh**: Transaction management with automatic rollback
- **03-traditional.sh**: Traditional chroot-based layer creation
- **04-container.sh**: Container-based layer creation (Apx-style) ✅ **IMPLEMENTED**
- **05-live-overlay.sh**: Live system layering (rpm-ostree style) ✅ **IMPLEMENTED**
- **06-oci-integration.sh**: OCI export/import functionality ✅ **IMPLEMENTED**
- **07-bootloader.sh**: Bootloader integration (UEFI/GRUB/systemd-boot) ✅ **IMPLEMENTED**
- **09-atomic-deployment.sh**: Atomic deployment system ✅ **IMPLEMENTED**
- **10-rpm-ostree-compat.sh**: rpm-ostree compatibility layer ✅ **IMPLEMENTED**
- **15-ostree-atomic.sh**: OSTree atomic package management ✅ **IMPLEMENTED**
- **24-dpkg-direct-install.sh**: Direct dpkg Installation (Performance Optimization) ✅ **IMPLEMENTED**
- **99-main.sh**: Main command dispatch and help system

## 🔧 Benefits of This Structure

### ✅ **Modular Development**
- Each component can be developed and tested independently
- Easy to locate and modify specific functionality
- Clear separation of concerns

### ✅ **Unified Deployment**
- Single `apt-layer.sh` file for end users
- No complex dependency management
- Professional distribution format

### ✅ **Maintainable Code**
- Logical organization by functionality
- Easy to add new features
- Clear documentation per component

### ✅ **Version Control Friendly**
- Small, focused files are easier to review
- Clear commit history per feature
- Reduced merge conflicts

## 🏗️ Architecture Overview

### **Core Components**

1. **Transaction Management**: Atomic operations with automatic rollback
2. **ComposeFS Integration**: Uses the modular ComposeFS backend
3. **Dependency Validation**: Comprehensive dependency checking
4. **Error Handling**: Robust error handling and recovery

### **Layer Creation Methods**

1. **Traditional**: Chroot-based package installation ✅ **IMPLEMENTED**
   - Uses chroot for isolation
   - Direct package installation in isolated environment
   - Suitable for most use cases

2. **Container (Apx-style)**: Isolated container-based installation ✅ **IMPLEMENTED**
   - Uses container technology (Podman/Docker/systemd-nspawn)
   - Complete isolation from host system
   - Reproducible and secure package installation
   - Named after the Apx package manager's isolation approach

3. **Live Overlay**: Runtime package installation ✅ **IMPLEMENTED**
   - Installs packages on running system using overlayfs
   - Provides immediate package availability
   - Supports commit/rollback operations

### **Integration Points**

- **ComposeFS Backend**: Uses the modular `composefs-alternative.sh`
- **Particle-OS Config**: Integrates with unified configuration system
- **Bootloader**: Automatic boot entry management ✅ **IMPLEMENTED**
- **OCI**: Container image export/import ✅ **IMPLEMENTED**

## 🚀 Quick Start

### Basic Layer Creation

```bash
# Create a gaming layer
sudo ./apt-layer.sh particle-os/base/24.04 particle-os/gaming/24.04 steam wine

# List available images
sudo ./apt-layer.sh --list

# Show image information
sudo ./apt-layer.sh --info particle-os/gaming/24.04
```

### Container-based Layer Creation (Apx-style Isolation)

Apx-style isolation follows the same architectural pattern as the Apx package manager, which uses **OSTree + Container Isolation**. This approach provides a secure, isolated environment for package installation while maintaining the immutable, layered system architecture.

**Architecture Overview:**
- **Base System**: Immutable base layer (OSTree in Apx, ComposeFS in apt-layer)
- **Isolation Layer**: Container-based package installation
- **Result**: New immutable layer with container-derived changes

**Key Benefits:**
- **Isolation**: Package installations run in isolated containers, preventing conflicts with the host system
- **Reproducibility**: Same packages installed in the same base image always produce identical results
- **Security**: Container isolation prevents malicious packages from affecting the host
- **Clean Environment**: Each installation starts with a fresh, clean base image
- **Multi-runtime Support**: Works with Podman, Docker, or systemd-nspawn
- **Immutable Layers**: Results in immutable, atomic layers just like OSTree

**How it Works (Same Pattern as Apx):**
1. Starts with immutable base layer (ComposeFS instead of OSTree)
2. Creates a temporary container from the base image
3. Installs packages inside the isolated container environment
4. Extracts the changes and creates a new immutable layer
5. Cleans up the temporary container
6. Results in an immutable layer that can be deployed atomically

**Comparison with rpm-ostree and Apx:**

| Aspect | rpm-ostree | Apx | apt-layer (Apx-style) |
|--------|------------|-----|----------------------|
| **Base System** | OSTree commits | OSTree commits | ComposeFS layers |
| **Isolation Method** | Direct RPM installation | Container isolation | Container isolation |
| **Package Format** | RPM packages | RPM packages | DEB packages |
| **Installation Process** | Direct in OSTree tree | Container → OSTree commit | Container → ComposeFS layer |
| **Dependency Resolution** | DNF/RPM | DNF/RPM | APT |
| **Transaction Safety** | OSTree atomic | OSTree atomic | ComposeFS atomic |
| **Reproducibility** | OSTree commit hashes | OSTree commit hashes | ComposeFS layer hashes |
| **Cross-Platform** | Red Hat/Fedora | Red Hat/Fedora | Ubuntu/Debian |

**Key Architectural Similarities:**
- **Apx**: OSTree base + Container isolation + OSTree commits = Immutable layers
- **apt-layer**: ComposeFS base + Container isolation + ComposeFS layers = Immutable layers
- **rpm-ostree**: OSTree base + Direct installation + OSTree commits = Immutable layers

**The Pattern is the Same:**
All three approaches create immutable, layered systems. The difference is in the underlying technologies:
- **rpm-ostree**: Uses OSTree + RPM directly
- **Apx**: Uses OSTree + Container isolation + RPM
- **apt-layer**: Uses ComposeFS + Container isolation + APT

**What is ComposeFS?**
ComposeFS is a modern filesystem technology that provides immutable, layered filesystem capabilities similar to OSTree. It creates read-only, deduplicated layers that can be composed together to form a complete filesystem. Each layer contains the changes from the previous layer, creating an immutable, versioned filesystem structure. ComposeFS layers are content-addressable (identified by cryptographic hashes) and can be efficiently shared and distributed, making it ideal for container images and immutable system deployments.

**How ComposeFS Works:**
1. **Base Layer**: Starts with an immutable base filesystem layer
2. **Layer Creation**: New layers contain only the changes (deltas) from the previous layer
3. **Composition**: Multiple layers are composed together to create the final filesystem
4. **Immutability**: Each layer is read-only and cannot be modified once created
5. **Efficiency**: Deduplication and compression reduce storage requirements
6. **Distribution**: Layers can be independently distributed and cached

We're not departing from how rpm-ostree works - we're following the same immutable layering pattern but with Ubuntu/Debian technologies instead of Red Hat technologies.

```bash
# Container-based layer creation (Apx-style)
sudo ./apt-layer.sh --container particle-os/base/24.04 particle-os/gaming/24.04 steam wine

# Container-based layer with multiple packages
sudo ./apt-layer.sh --container particle-os/base/24.04 particle-os/dev/24.04 vscode git nodejs npm

# Container status and information
sudo ./apt-layer.sh --container-status
```

### Live System Installation

```bash
# Live system installation (rpm-ostree style)
sudo ./apt-layer.sh --live-install firefox

# Live overlay management
sudo ./apt-layer.sh --live-overlay start
sudo ./apt-layer.sh --live-overlay status
sudo ./apt-layer.sh --live-overlay commit
sudo ./apt-layer.sh --live-overlay rollback
```

### Direct dpkg Installation (Performance Optimization)

```bash
# Direct dpkg installation (faster, more controlled)
sudo ./apt-layer.sh --dpkg-install curl wget

# Container-based dpkg installation
sudo ./apt-layer.sh --container-dpkg particle-os/base/24.04 particle-os/dev/24.04 vscode git

# Live system dpkg installation
sudo ./apt-layer.sh --live-dpkg firefox

# Download-only mode (for offline installation)
DPKG_DOWNLOAD_ONLY=true sudo ./apt-layer.sh --dpkg-install curl

# Force installation with dependency issues
DPKG_FORCE_DEPENDS=true sudo ./apt-layer.sh --dpkg-install package-name

# Use specific chroot directory
DPKG_CHROOT_DIR=/path/to/chroot sudo ./apt-layer.sh --dpkg-install package-name
```

### OCI Integration

```bash
# OCI export
sudo ./apt-layer.sh --oci-export ubuntu-ublue/gaming/24.04 my-registry/gaming:latest

# OCI import
sudo ./apt-layer.sh --oci-import my-registry/gaming:latest ubuntu-ublue/gaming/24.04

# OCI status
sudo ./apt-layer.sh --oci-status
```

### Bootloader Management

```bash
# Bootloader management
sudo ./apt-layer.sh bootloader status
sudo ./apt-layer.sh bootloader list-entries
sudo ./apt-layer.sh bootloader set-default particle-os/gaming/24.04
```

### Kernel arguments (rpm-ostree compatibility)
```bash
sudo ./apt-layer.sh kargs add rd.break=pre-mount
sudo ./apt-layer.sh kargs list
sudo ./apt-layer.sh kargs remove rd.break=pre-mount
```

### OSTree Atomic Package Management

```bash
# Atomic OSTree package management
sudo ./apt-layer.sh ostree compose install firefox vlc
sudo ./apt-layer.sh ostree compose remove package-name
sudo ./apt-layer.sh ostree compose update

# View atomic history
sudo ./apt-layer.sh ostree log
sudo ./apt-layer.sh ostree diff commit1 commit2
sudo ./apt-layer.sh ostree status
sudo ./apt-layer.sh ostree rollback commit-id
sudo ./apt-layer.sh ostree cleanup
```

### rpm-ostree Compatibility

```bash
# Full rpm-ostree command compatibility
sudo ./apt-layer.sh install firefox
sudo ./apt-layer.sh upgrade
sudo ./apt-layer.sh rebase particle-os/gaming/24.04
sudo ./apt-layer.sh rollback
sudo ./apt-layer.sh status
sudo ./apt-layer.sh diff
sudo ./apt-layer.sh db list
sudo ./apt-layer.sh cleanup
```

## 🔧 Configuration

The apt-layer tool integrates with the Particle-OS configuration system and includes a comprehensive JSON-based configuration system:

### Particle-OS Integration
```bash
# Configuration is automatically loaded from:
# /usr/local/etc/particle-config.sh

# Key configuration variables:
WORKSPACE="/var/lib/particle-os"
COMPOSEFS_SCRIPT="/usr/local/bin/composefs-alternative.sh"
CONTAINER_RUNTIME="podman"
```

### JSON Configuration System
The tool includes embedded JSON configuration files for enterprise-grade configurability:

- **apt-layer-settings.json**: Global settings, feature toggles, and defaults
- **security-policy.json**: Security policies, signature requirements, and blocked packages
- **users.json**: RBAC user definitions and access control
- **audit-settings.json**: Audit logging policies and compliance frameworks
- **backup-policy.json**: Backup frequency, retention, and encryption settings
- **signing-policy.json**: Layer signing methods and trusted keys
- **oci-settings.json**: OCI registry configuration and authentication
- **package-management.json**: Repository policies and dependency resolution
- **maintenance.json**: Automated maintenance and cleanup policies

All configuration files are automatically embedded in the compiled script and can be overridden via command-line arguments for enterprise deployment flexibility.

## 🛠️ Development Guidelines

### Adding New Scriptlets

1. **Create the scriptlet file** in `scriptlets/` with appropriate naming
2. **Add to compile.sh** in the correct order
3. **Update this README** with the new scriptlet description
4. **Test thoroughly** before committing

### Scriptlet Naming Convention

- **00-header.sh**: Shared utility functions, global cleanup, and system detection
- **01-XX.sh**: Dependencies and validation
- **02-XX.sh**: Core functionality
- **03-XX.sh**: Layer creation methods
- **04-XX.sh**: Advanced features
- **05-XX.sh**: Live system features
- **06-XX.sh**: OCI integration
- **07-XX.sh**: Bootloader integration
- **09-XX.sh**: Atomic deployment
- **10-XX.sh**: Compatibility layers
- **15-XX.sh**: OSTree atomic features
- **24-XX.sh**: Performance optimizations
- **99-main.sh**: Main dispatch (always last)

### Error Handling

All scriptlets should:
- Use the unified logging system (`log_info`, `log_error`, etc.)
- Include proper error handling and cleanup
- Integrate with transaction management when appropriate

## 📚 Related Documentation

- **[ComposeFS Modular System](../composefs/README.md)**: Backend filesystem layer
- **[BootC Modular System](../bootc/README.md)**: Container-native boot system
- **[Particle-OS Configuration](../../particle-config.sh)**: Unified configuration system

## 🎯 Development Phases

### ✅ Phase 1: Core Stability (COMPLETED)
- [x] Modular architecture implementation
- [x] Transaction management system
- [x] Traditional layer creation
- [x] ComposeFS backend integration

### ✅ Phase 2: Enhanced Features (COMPLETED)
- [x] Container-based layer creation
- [x] OCI integration
- [x] Live system layering

### ✅ Phase 3: Bootloader Integration (COMPLETED)
- [x] Multi-bootloader support (UEFI/GRUB/systemd-boot)
- [x] Kernel arguments management
- [x] Boot entry management
- [x] Atomic deployment integration

### ✅ Phase 4: OSTree Atomic Package Management (COMPLETED)
- [x] OSTree atomic commits for package operations
- [x] Atomic deployment with rollback capabilities
- [x] Versioned package history
- [x] Direct dpkg installation optimization
- [x] Live overlay system with DNS fixes

### ✅ Phase 5: rpm-ostree Compatibility (COMPLETED)
- [x] Full rpm-ostree command compatibility
- [x] Atomic deployment system
- [x] Live overlay system
- [x] Bootloader integration
- [x] OCI integration

## 🎯 Current Status

### ✅ **COMPLETED MAJOR MILESTONES:**
- **OSTree/Atomic Workflow Implemented:**
  - All `apt-layer ostree compose` commands (install, remove, update) create atomic, versioned commits
  - `apt-layer ostree log`, `diff`, `status`, `rollback`, `cleanup` fully implemented and tested
  - Overlay and dpkg install workflow robust, with DNS fixes for WSL and offline `.deb` install support
  - Log function bug fixed (commit history now displays correctly)
- **Testing & Validation:**
  - All atomic/OSTree commands tested and confirmed functional
  - Overlay and atomic install workflows validated, including rollback readiness

### 🔄 **NEXT PRIORITIES:**
- [ ] Further test rollback and deployment activation
- [ ] Document overlay/atomic best practices and known caveats
- [ ] Continue integration and optimization of atomic/OSTree workflow
- [ ] Add more automated tests for edge cases (optional)

### 🛠️ **COMPILATION SYSTEM ENHANCEMENTS:**
- [ ] **Add source file dependency validation** - Validate that all required functions exist in source
  - [ ] Add validation that all required functions exist in source scriptlets
  - [ ] Add dependency validation during compilation
  - [ ] Add error checking for missing source files
  - [ ] Add function dependency graph validation
  - [ ] Add cross-scriptlet function reference checking

## 🎯 Scope Reduction Summary

As of July 2025, Particle-OS apt-layer has been **successfully reduced to core rpm-ostree-like features only**. All advanced, enterprise, cloud, multi-tenant, admin, compliance, and security features have been archived to `archive/apt-layer/scriptlets/`.

**Current Focus:**
- **Atomic deployment, rollback, status, diff, cleanup** - Core rpm-ostree functionality
- **Live overlay and container-based layering** - Immutable system management
- **Bootloader and kargs management** - System boot configuration
- **OCI/ComposeFS integration** - Container and filesystem integration
- **Direct dpkg install** - Performance optimization for apt/deb systems
- **OSTree atomic package management** - True atomic package operations