deb-mock/dev_notes/plugins.md
2025-08-03 22:16:04 +00:00

594 lines
No EOL
22 KiB
Markdown

# Deb-Mock Plugin System Roadmap
## Overview
This document outlines the roadmap for implementing a comprehensive plugin system in **Deb-Mock**, inspired by Fedora's Mock plugin architecture but adapted specifically for Debian-based systems. The plugin system will provide extensible functionality for build environment management, performance optimization, and Debian-specific features.
## Plugin Hook System
Based on [Mock's Plugin Hooks](https://rpm-software-management.github.io/mock/Plugin-Hooks), **Deb-Mock** implements a comprehensive hook system that allows plugins to integrate at specific points in the build lifecycle.
### Hook Points
| Hook Name | Description | When Called | Use Cases |
|-----------|-------------|-------------|-----------|
| `clean` | Clean up plugin resources | After chroot cleanup | Resource cleanup, cache management |
| `earlyprebuild` | Very early build stage | Before SRPM rebuild, before dependencies | Environment setup, pre-validation |
| `initfailed` | Chroot initialization failed | When chroot creation fails | Error reporting, cleanup |
| `list_snapshots` | List available snapshots | When `--list-snapshots` is used | Snapshot management |
| `make_snapshot` | Create a snapshot | When snapshot creation is requested | State preservation |
| `mount_root` | Mount chroot directory | Before preinit, chroot exists | Filesystem mounting |
| `postbuild` | After build completion | After RPM/SRPM build (success/failure) | Result processing, cleanup |
| `postchroot` | After chroot command | After `mock chroot` command | Cache updates, cleanup |
| `postclean` | After chroot cleanup | After chroot content deletion | Resource cleanup |
| `postdeps` | After dependency installation | Dependencies installed, before build | Environment verification |
| `postinit` | After chroot initialization | Chroot ready for dependencies | Cache creation, setup |
| `postshell` | After shell exit | After `mock shell` command | Cache updates, cleanup |
| `postupdate` | After package updates | After successful package updates | Cache updates |
| `postumount` | After unmounting | All inner mounts unmounted | Filesystem cleanup |
| `postapt` | After APT operations | After any package manager action | Package state tracking |
| `prebuild` | Before build starts | After BuildRequires, before RPM build | Build preparation |
| `prechroot` | Before chroot command | Before `mock chroot` command | Cache restoration |
| `preinit` | Before chroot initialization | Only chroot/result dirs exist | Cache restoration, setup |
| `preshell` | Before shell prompt | Before `mock shell` prompt | Cache restoration, setup |
| `preapt` | Before APT operations | Before any package manager action | Cache preparation |
| `process_logs` | Process build logs | After build log completion | Log compression, analysis |
| `remove_snapshot` | Remove snapshot | When snapshot removal requested | Snapshot cleanup |
| `rollback_to` | Rollback to snapshot | When rollback requested | State restoration |
| `scrub` | Scrub chroot | When chroot scrubbing requested | Deep cleanup |
### Hook Registration
Plugins register hooks using the same pattern as Mock:
```python
# In plugin __init__ method
plugins.add_hook("postbuild", self.post_build_handler)
plugins.add_hook("preinit", self.pre_init_handler)
plugins.add_hook("clean", self.cleanup_handler)
```
## Mock Plugin Analysis & Deb-Mock Implementation Status
### ✅ Already Implemented (Core Features)
| Mock Plugin | Deb-Mock Status | Implementation Notes |
|-------------|----------------|---------------------|
| **CCache** | ✅ Implemented | Integrated into cache management system |
| **RootCache** | ✅ Implemented | Root cache with validation and cleanup |
| **YumCache** | ✅ Implemented | Package cache for downloaded .deb files |
### 🔄 High Priority Plugins (Easy Implementation)
#### 1. BindMount Plugin
- **Purpose**: Mount host directories/files into chroot environments
- **Fedora-specific**: No - generic Linux feature
- **Implementation Complexity**: Low - uses `mount --bind`
- **Priority**: 🔴 **High** - Essential for development workflows
- **Use Cases**: Source code mounting, shared libraries, development tools
- **Hooks**: `mount_root`, `postumount`
#### 2. CompressLogs Plugin
- **Purpose**: Compress build logs to save disk space
- **Fedora-specific**: No - generic compression functionality
- **Implementation Complexity**: Low - uses `gzip/xz`
- **Priority**: 🟡 **Medium** - Nice to have feature
- **Use Cases**: CI/CD environments, long-term log storage
- **Hooks**: `process_logs`
#### 3. ChrootScan Plugin
- **Purpose**: Copy specific files from chroot after build (core dumps, logs, artifacts)
- **Fedora-specific**: No - generic file scanning and copying
- **Implementation Complexity**: Low - file pattern matching and copying
- **Priority**: 🟡 **Medium** - Useful for debugging and analysis
- **Use Cases**: Debugging failed builds, collecting build artifacts
- **Hooks**: `postbuild`
#### 4. Tmpfs Plugin
- **Purpose**: Use tmpfs for faster I/O operations in chroot
- **Fedora-specific**: No - Linux kernel feature
- **Implementation Complexity**: Low - mount tmpfs filesystem
- **Priority**: 🟡 **Medium** - Performance optimization
- **Use Cases**: High-performance builds, memory-rich systems
- **Hooks**: `mount_root`, `postumount`
### 🔄 Medium Priority Plugins (Moderate Complexity)
#### 5. BuildrootLock Plugin
- **Purpose**: Generate reproducible build environment lockfiles
- **Fedora-specific**: No - generic reproducibility concept
- **Implementation Complexity**: Moderate - JSON lockfile generation and validation
- **Priority**: 🔴 **High** - Critical for reproducible builds
- **Use Cases**: CI/CD reproducibility, build environment consistency
- **Hooks**: `postdeps`, `postbuild`
#### 6. Export-Buildroot-Image Plugin
- **Purpose**: Export chroot as OCI container image
- **Fedora-specific**: No - OCI standard
- **Implementation Complexity**: Moderate - debootstrap + OCI tools integration
- **Priority**: 🟡 **Medium** - Container integration
- **Use Cases**: Container-based builds, deployment pipelines
- **Hooks**: `postbuild`
#### 7. Mount Plugin
- **Purpose**: Mount additional filesystems in chroot
- **Fedora-specific**: No - generic filesystem mounting
- **Implementation Complexity**: Moderate - filesystem mounting and management
- **Priority**: 🟡 **Medium** - Advanced use cases
- **Use Cases**: Network filesystems, specialized storage
- **Hooks**: `mount_root`, `postumount`
### 🔄 Low Priority Plugins (Complex/Advanced)
#### 8. Overlayfs Plugin
- **Purpose**: Use overlayfs for faster chroot operations
- **Fedora-specific**: No - Linux kernel feature
- **Implementation Complexity**: High - overlayfs management and optimization
- **Priority**: 🟢 **Low** - Advanced optimization
- **Use Cases**: High-frequency builds, resource optimization
- **Hooks**: `mount_root`, `postumount`
#### 9. LvmRoot Plugin
- **Purpose**: Use LVM for chroot storage management
- **Fedora-specific**: No - Linux LVM
- **Implementation Complexity**: High - LVM volume management
- **Priority**: 🟢 **Low** - Advanced storage management
- **Use Cases**: Large-scale deployments, storage optimization
- **Hooks**: `mount_root`, `postclean`, `postumount`
### ❌ Fedora-Specific Plugins (Not Directly Applicable)
| Mock Plugin | Why Not Applicable | Deb-Mock Alternative |
|-------------|-------------------|---------------------|
| **PackageState** | RPM-specific package state tracking | Debian package state tracking |
| **PMRequest** | RPM package manager requests | APT package manager requests |
| **rpkg-preprocessor** | RPM-specific preprocessing | Debian source preprocessing |
| **SELinux** | Fedora SELinux policies | AppArmor integration |
| **Sign** | RPM signing mechanisms | Debian package signing |
| **Scm** | Git/SVN integration | Git integration (generic) |
## Debian-Specific Plugin Opportunities
### 1. AppArmor Plugin (Debian Alternative to SELinux)
- **Purpose**: AppArmor profile management for chroot environments
- **Implementation**: AppArmor profile generation and enforcement
- **Priority**: 🟡 **Medium** - Security enhancement
- **Use Cases**: Secure build environments, policy enforcement
- **Hooks**: `preinit`, `postinit`, `clean`
### 2. DebianSource Plugin
- **Purpose**: Handle Debian source package formats (.dsc, .orig.tar.gz, .debian.tar.gz)
- **Implementation**: Source package validation and preprocessing
- **Priority**: 🔴 **High** - Core Debian functionality
- **Use Cases**: Source package handling, format validation
- **Hooks**: `earlyprebuild`, `prebuild`
### 3. APTRequest Plugin
- **Purpose**: Handle APT package manager requests and dependency resolution
- **Implementation**: APT dependency resolution and package installation
- **Priority**: 🔴 **High** - Core package management
- **Use Cases**: Dependency management, package installation
- **Hooks**: `preapt`, `postapt`
### 4. DebianSign Plugin
- **Purpose**: Debian package signing (debsign, dpkg-sig)
- **Implementation**: GPG signing of .deb packages and .changes files
- **Priority**: 🟡 **Medium** - Package security
- **Use Cases**: Package signing, security compliance
- **Hooks**: `postbuild`
## Implementation Roadmap
### Phase 1: Hook System Infrastructure
#### Plugin Hook Manager
```python
# deb_mock/plugins/hook_manager.py
class HookManager:
"""Manages plugin hooks and their execution"""
def __init__(self):
self.hooks = {}
def add_hook(self, hook_name: str, callback):
"""Register a hook callback"""
if hook_name not in self.hooks:
self.hooks[hook_name] = []
self.hooks[hook_name].append(callback)
def call_hook(self, hook_name: str, context: dict = None):
"""Execute all registered hooks for a given hook name"""
if hook_name not in self.hooks:
return
context = context or {}
for callback in self.hooks[hook_name]:
try:
callback(context)
except Exception as e:
# Log hook execution errors but don't fail the build
print(f"Warning: Hook {hook_name} failed: {e}")
def get_hook_names(self) -> list:
"""Get list of available hook names"""
return list(self.hooks.keys())
```
#### Enhanced Base Plugin Class
```python
# deb_mock/plugins/base.py
class BasePlugin:
"""Base class for all Deb-Mock plugins"""
def __init__(self, config, hook_manager):
self.config = config
self.hook_manager = hook_manager
self.enabled = self._is_enabled()
self._register_hooks()
def _is_enabled(self) -> bool:
"""Check if plugin is enabled in configuration"""
return self.config.plugins.get(self.name, {}).get('enabled', False)
def _register_hooks(self):
"""Register plugin hooks with the hook manager"""
# Override in subclasses to register specific hooks
pass
# Hook method stubs - override in subclasses as needed
def clean(self, context): pass
def earlyprebuild(self, context): pass
def initfailed(self, context): pass
def list_snapshots(self, context): pass
def make_snapshot(self, context): pass
def mount_root(self, context): pass
def postbuild(self, context): pass
def postchroot(self, context): pass
def postclean(self, context): pass
def postdeps(self, context): pass
def postinit(self, context): pass
def postshell(self, context): pass
def postupdate(self, context): pass
def postumount(self, context): pass
def postapt(self, context): pass
def prebuild(self, context): pass
def prechroot(self, context): pass
def preinit(self, context): pass
def preshell(self, context): pass
def preapt(self, context): pass
def process_logs(self, context): pass
def remove_snapshot(self, context): pass
def rollback_to(self, context): pass
def scrub(self, context): pass
```
### Phase 2: Core Plugins with Hook Integration
#### BindMount Plugin with Hooks
```python
# deb_mock/plugins/bind_mount.py
class BindMountPlugin(BasePlugin):
"""Mount host directories into chroot environments"""
def _register_hooks(self):
"""Register bind mount hooks"""
self.hook_manager.add_hook("mount_root", self.mount_root)
self.hook_manager.add_hook("postumount", self.postumount)
def mount_root(self, context):
"""Mount bind mounts when chroot is mounted"""
if not self.enabled:
return
chroot_path = context.get('chroot_path')
if not chroot_path:
return
for host_path, chroot_mount_path in self.config.plugins.bind_mount.mounts:
full_chroot_path = os.path.join(chroot_path, chroot_mount_path.lstrip('/'))
subprocess.run(['mount', '--bind', host_path, full_chroot_path])
def postumount(self, context):
"""Unmount bind mounts when chroot is unmounted"""
if not self.enabled:
return
chroot_path = context.get('chroot_path')
if not chroot_path:
return
for host_path, chroot_mount_path in self.config.plugins.bind_mount.mounts:
full_chroot_path = os.path.join(chroot_path, chroot_mount_path.lstrip('/'))
subprocess.run(['umount', full_chroot_path])
```
#### RootCache Plugin with Hooks
```python
# deb_mock/plugins/root_cache.py
class RootCachePlugin(BasePlugin):
"""Root cache management with hook integration"""
def _register_hooks(self):
"""Register root cache hooks"""
self.hook_manager.add_hook("preinit", self.preinit)
self.hook_manager.add_hook("postinit", self.postinit)
self.hook_manager.add_hook("postchroot", self.postchroot)
self.hook_manager.add_hook("postshell", self.postshell)
self.hook_manager.add_hook("clean", self.clean)
def preinit(self, context):
"""Restore chroot from cache before initialization"""
if not self.enabled:
return
chroot_name = context.get('chroot_name')
if self._cache_exists(chroot_name):
self._restore_from_cache(chroot_name, context)
def postinit(self, context):
"""Create cache after successful initialization"""
if not self.enabled:
return
chroot_name = context.get('chroot_name')
self._create_cache(chroot_name, context)
def postchroot(self, context):
"""Update cache after chroot operations"""
if not self.enabled:
return
chroot_name = context.get('chroot_name')
self._update_cache(chroot_name, context)
def postshell(self, context):
"""Update cache after shell operations"""
if not self.enabled:
return
chroot_name = context.get('chroot_name')
self._update_cache(chroot_name, context)
def clean(self, context):
"""Clean up cache resources"""
if not self.enabled:
return
self._cleanup_old_caches()
```
#### CompressLogs Plugin with Hooks
```python
# deb_mock/plugins/compress_logs.py
class CompressLogsPlugin(BasePlugin):
"""Compress build logs with hook integration"""
def _register_hooks(self):
"""Register log compression hooks"""
self.hook_manager.add_hook("process_logs", self.process_logs)
def process_logs(self, context):
"""Compress build logs after build completion"""
if not self.enabled:
return
log_dir = context.get('log_dir')
if not log_dir:
return
compression = self.config.plugins.compress_logs.compression
level = self.config.plugins.compress_logs.level
for log_file in Path(log_dir).glob('*.log'):
if compression == 'gzip':
subprocess.run(['gzip', f'-{level}', str(log_file)])
elif compression == 'xz':
subprocess.run(['xz', f'-{level}', str(log_file)])
```
### Phase 3: Debian-Specific Plugins with Hooks
#### APTRequest Plugin with Hooks
```python
# deb_mock/plugins/apt_request.py
class APTRequestPlugin(BasePlugin):
"""APT package manager integration with hooks"""
def _register_hooks(self):
"""Register APT request hooks"""
self.hook_manager.add_hook("preapt", self.prequest)
self.hook_manager.add_hook("postapt", self.postrequest)
def prequest(self, context):
"""Handle pre-APT operations"""
if not self.enabled:
return
operation = context.get('operation')
packages = context.get('packages', [])
# Log APT operations
self._log_apt_operation(operation, packages)
# Update package cache if needed
if self.config.plugins.apt_request.update_cache:
self._update_package_cache(context)
def postrequest(self, context):
"""Handle post-APT operations"""
if not self.enabled:
return
operation = context.get('operation')
result = context.get('result')
# Track package state changes
self._update_package_state(operation, result)
# Update caches if packages were modified
if result and result.get('packages_modified'):
self._invalidate_caches(context)
```
#### DebianSource Plugin with Hooks
```python
# deb_mock/plugins/debian_source.py
class DebianSourcePlugin(BasePlugin):
"""Debian source package handling with hooks"""
def _register_hooks(self):
"""Register source package hooks"""
self.hook_manager.add_hook("earlyprebuild", self.earlyprebuild)
self.hook_manager.add_hook("prebuild", self.prebuild)
def earlyprebuild(self, context):
"""Validate and prepare source package"""
if not self.enabled:
return
source_package = context.get('source_package')
if not source_package:
return
# Validate source package format
self._validate_source_package(source_package)
# Extract source package if needed
if self.config.plugins.debian_source.extract_patches:
self._extract_source_package(source_package, context)
def prebuild(self, context):
"""Prepare source package for building"""
if not self.enabled:
return
source_package = context.get('source_package')
build_dir = context.get('build_dir')
# Apply patches if needed
self._apply_patches(source_package, build_dir)
# Verify checksums if enabled
if self.config.plugins.debian_source.validate_checksums:
self._verify_checksums(source_package)
```
## Configuration Integration
### Plugin Configuration in YAML
```yaml
# deb-mock.yaml
plugins:
bind_mount:
enabled: true
mounts:
- host_path: "/home/user/project"
chroot_path: "/builddir/project"
root_cache:
enabled: true
cache_dir: "/var/cache/deb-mock/root-cache"
max_age_days: 7
compress_logs:
enabled: true
compression: "gzip"
level: 9
tmpfs:
enabled: true
size: "2G"
mount_point: "/tmp"
debian_source:
enabled: true
validate_checksums: true
extract_patches: true
apt_request:
enabled: true
update_cache: true
install_recommends: false
track_package_state: true
```
### CLI Integration
```bash
# Enable specific plugins
deb-mock --enable-plugin bind_mount,compress_logs build package.dsc
# Plugin-specific options
deb-mock --plugin-option bind_mount:mounts=[("/host/path", "/chroot/path")] build package.dsc
# List available plugins
deb-mock list-plugins
# Show plugin help
deb-mock plugin-help bind_mock
# List available hooks
deb-mock list-hooks
# Show hook usage
deb-mock hook-info postbuild
```
## Hook Context Information
Each hook receives a context dictionary with relevant information:
### Common Context Keys
- `chroot_name`: Name of the chroot
- `chroot_path`: Path to the chroot directory
- `source_package`: Source package being built
- `build_dir`: Build directory
- `log_dir`: Log directory
- `result_dir`: Result directory
- `operation`: Current operation being performed
- `config`: Current configuration
- `result`: Operation result (for post-hooks)
### Hook-Specific Context
- `mount_root`: `chroot_path`, `mount_points`
- `postbuild`: `build_result`, `artifacts`, `build_log`
- `preapt/postapt`: `operation`, `packages`, `result`
- `process_logs`: `log_files`, `log_dir`
- `clean`: `chroot_name`, `cleanup_type`
## Priority Recommendations
### Immediate (Next Sprint)
1. **Hook System Infrastructure** - Core hook management
2. **BindMount Plugin** - Essential for development workflows
3. **RootCache Plugin** - Critical for performance
4. **CompressLogs Plugin** - Simple but useful
### Short Term (Next Month)
1. **Tmpfs Plugin** - Performance optimization
2. **ChrootScan Plugin** - Debugging support
3. **DebianSource Plugin** - Core Debian functionality
4. **APTRequest Plugin** - Enhanced package management
### Medium Term (Next Quarter)
1. **Export-Buildroot-Image Plugin** - Container integration
2. **AppArmor Plugin** - Security enhancement
3. **BuildrootLock Plugin** - Reproducible builds
4. **Mount Plugin** - Advanced filesystem support
### Long Term (Future)
1. **Overlayfs Plugin** - Advanced optimization
2. **LvmRoot Plugin** - Advanced storage
3. **DebianSign Plugin** - Package security
4. **Snapshot Plugin** - State management
## Conclusion
This enhanced plugin system with comprehensive hooks provides **Deb-Mock** with the same level of extensibility as Mock while being specifically tailored for Debian-based systems. The hook system allows plugins to integrate seamlessly at the right points in the build lifecycle, providing maximum flexibility and power.
The implementation strategy focuses on:
- **Comprehensive hook coverage** matching Mock's capabilities
- **Debian-specific adaptations** for package management and workflows
- **Performance optimization** through intelligent hook usage
- **Extensibility** through a well-designed plugin architecture
This approach ensures that **Deb-Mock** can provide the same level of functionality as Mock while being specifically tailored to Debian-based systems and workflows.