From 6768634f28e4b88fa7ba89950005eafcf00f15ef Mon Sep 17 00:00:00 2001 From: robojerk Date: Wed, 27 Aug 2025 19:42:18 -0700 Subject: [PATCH] added todo.txt --- DEBIAN_README.md | 318 ++++++++++++++++++ debian_module_adapter.py | 295 ++++++++++++++++ debian_package_converter.py | 425 ++++++++++++++++++++++++ examples/debian-basic/debian-server.yml | 56 ++++ examples/debian-gnome/debian-gnome.yml | 86 +++++ examples/debian-kde/debian-kde.yml | 55 +++ modules.json | 3 + modules/apt-ostree/Containerfile | 21 ++ modules/apt-ostree/entrypoint.sh | 36 ++ modules/apt-ostree/module.yml | 25 ++ modules/apt-ostree/ostree-wrapper.sh | 214 ++++++++++++ modules/apt/Containerfile | 22 ++ modules/apt/apt-wrapper.sh | 276 +++++++++++++++ modules/apt/entrypoint.sh | 36 ++ modules/apt/module.yml | 47 +++ modules/deb-mock/Containerfile | 22 ++ modules/deb-mock/entrypoint.sh | 36 ++ modules/deb-mock/mock-wrapper.sh | 186 +++++++++++ modules/deb-mock/module.yml | 20 ++ todo.txt | 99 ++++++ 20 files changed, 2278 insertions(+) create mode 100644 DEBIAN_README.md create mode 100644 debian_module_adapter.py create mode 100644 debian_package_converter.py create mode 100644 examples/debian-basic/debian-server.yml create mode 100644 examples/debian-gnome/debian-gnome.yml create mode 100644 examples/debian-kde/debian-kde.yml create mode 100644 modules/apt-ostree/Containerfile create mode 100644 modules/apt-ostree/entrypoint.sh create mode 100644 modules/apt-ostree/module.yml create mode 100644 modules/apt-ostree/ostree-wrapper.sh create mode 100644 modules/apt/Containerfile create mode 100644 modules/apt/apt-wrapper.sh create mode 100644 modules/apt/entrypoint.sh create mode 100644 modules/apt/module.yml create mode 100644 modules/deb-mock/Containerfile create mode 100644 modules/deb-mock/entrypoint.sh create mode 100644 modules/deb-mock/mock-wrapper.sh create mode 100644 modules/deb-mock/module.yml create mode 100644 todo.txt diff --git a/DEBIAN_README.md b/DEBIAN_README.md new file mode 100644 index 0000000..1ca0a1b --- /dev/null +++ b/DEBIAN_README.md @@ -0,0 +1,318 @@ +# Debian Modules for Blue-Build + +This document provides comprehensive information about the Debian modules available in blue-build, which offer 1:1 compatibility with the original Fedora modules while providing full Debian support. + +## Overview + +The Debian modules are designed to maintain the exact same file structure and user experience as the original blue-build system, ensuring that users familiar with the Fedora version will intuitively understand how to use the Debian implementation. + +## Supported Debian Versions + +- **Debian 13 (Trixie)** - Current stable release +- **Debian 14 (Forky)** - Testing release +- **Debian Sid** - Unstable release + +## Available Modules + +### 1. apt Module + +The `apt` module replaces the `dnf` module and provides Debian package management capabilities. + +#### Features +- Package installation and removal +- Repository management (PPA, backports, custom) +- GPG key management +- Task-based package groups (replaces RPM groups) +- .deb package file support +- Repository cleanup options + +#### Example Usage +```yaml +- type: apt + repos: + cleanup: true + backports: true + ppa: + - ppa:ondrej/php + keys: + - https://example.com/gpg.key + install: + skip-unavailable: true + packages: + - nginx + - postgresql + - php8.2 + task-install: + with-optional: true + packages: + - gnome-desktop-environment + remove: + packages: + - nano + - less +``` + +#### Key Differences from dnf +- **COPR → PPA**: Use `ppa:` instead of `copr:` +- **RPM Groups → Debian Tasks**: Use `task-install` instead of `group-install` +- **RPM Fusion → Backports**: Enable `backports: true` for additional packages +- **Package Files**: Supports `.deb` files instead of `.rpm` + +### 2. apt-ostree Module + +The `apt-ostree` module replaces the `rpm-ostree` module and provides Debian package layering with ostree. + +#### Features +- Package layering using ostree +- Repository and key management +- Package replacement and removal +- Automatic ostree commits +- Fallback to regular apt when ostree operations fail + +#### Example Usage +```yaml +- type: apt-ostree + repos: + - https://example.com/repo.list + keys: + - https://example.com/gpg.key + install: + - starship + - brave-browser + remove: + - firefox + replace: + - from-repo: https://example.com/repo + packages: + - php8.2 + - php8.2-common +``` + +#### Key Differences from rpm-ostree +- Uses Debian package database format +- Integrates with `apt-ostree` tool +- Debian-specific package layering + +### 3. deb-mock Module + +The `deb-mock` module replaces the `mock` module and provides Debian build environment management. + +#### Features +- Build environment setup (pbuilder, sbuild, schroot) +- Build dependency management +- Custom repository configuration +- Build script execution +- Artifact collection + +#### Example Usage +```yaml +- type: deb-mock + environment: bookworm-amd64 + packages: + - build-essential + - devscripts + - debhelper + repositories: + - deb http://deb.debian.org/debian bookworm main + - deb http://deb.debian.org/debian bookworm-backports main + build-script: | + #!/bin/bash + set -e + dpkg-buildpackage -b -us -uc + artifacts: + - ../*.deb + - ../*.dsc + - ../*.tar.gz +``` + +#### Key Differences from mock +- **Chroot → Environment**: Use `environment` instead of `chroot` +- **Builddeps → Packages**: Use `packages` for build dependencies +- **Resultdir → Artifacts**: Use `artifacts` for output files + +## Migration Guide + +### Converting from Fedora to Debian + +#### 1. Update Module Types +```yaml +# Before (Fedora) +- type: dnf +- type: rpm-ostree +- type: mock + +# After (Debian) +- type: apt +- type: apt-ostree +- type: deb-mock +``` + +#### 2. Convert Repositories +```yaml +# Before (Fedora) +repos: + copr: + - atim/starship + nonfree: rpmfusion + +# After (Debian) +repos: + ppa: + - ppa:atim/starship + backports: true +``` + +#### 3. Convert Package Groups +```yaml +# Before (Fedora) +group-install: + packages: + - development-tools + +# After (Debian) +task-install: + packages: + - development-tools +``` + +#### 4. Convert Package Specifications +```yaml +# Before (Fedora) +install: + packages: + - https://example.com/package.rpm + +# After (Debian) +install: + packages: + - https://example.com/package.deb +``` + +## Compatibility Layer + +The system includes a compatibility layer that automatically converts Fedora module configurations to Debian equivalents: + +### Automatic Conversions +- **Module Types**: `dnf` → `apt`, `rpm-ostree` → `apt-ostree`, `mock` → `deb-mock` +- **Repositories**: `copr:` → `ppa:`, `rpmfusion` → `backports` +- **Package Groups**: `group-install` → `task-install` +- **Package Names**: Common package name mappings (e.g., `gcc-c++` → `g++`) + +### Usage +```python +from debian_module_adapter import DebianModuleAdapter + +adapter = DebianModuleAdapter() +adapted_config = adapter.adapt_module_config(original_config) +result = adapter.execute_module(adapted_config) +``` + +## Example Recipes + +### Basic Debian Server +See `examples/debian-basic/debian-server.yml` for a complete server setup example. + +### GNOME Desktop +See `examples/debian-gnome/debian-gnome.yml` for a GNOME desktop environment. + +### KDE Desktop +See `examples/debian-kde/debian-kde.yml` for a KDE Plasma desktop environment. + +## Testing + +### Module Testing +```bash +# Test apt module +cd blue-build-modules/modules/apt +docker build -t apt-module . +docker run --rm -e BLUEBUILD_MODULE_CONFIG='{"type":"apt","install":{"packages":["curl"]}}' apt-module + +# Test apt-ostree module +cd blue-build-modules/modules/apt-ostree +docker build -t apt-ostree-module . +docker run --rm -e BLUEBUILD_MODULE_CONFIG='{"type":"apt-ostree","install":["curl"]}' apt-ostree-module + +# Test deb-mock module +cd blue-build-modules/modules/deb-mock +docker build -t deb-mock-module . +docker run --rm -e BLUEBUILD_MODULE_CONFIG='{"type":"deb-mock","environment":"bookworm-amd64"}' deb-mock-module +``` + +### Integration Testing +```bash +# Test with blue-build-cli +blue-build build examples/debian-basic/debian-server.yml + +# Test with existing Debian tools +python3 debian_module_adapter.py +python3 debian_package_converter.py +``` + +## Troubleshooting + +### Common Issues + +#### 1. Package Not Found +- Ensure the package name is correct for Debian +- Check if the package is available in the specified repositories +- Use `apt-cache search ` to find package names + +#### 2. Repository Issues +- Verify GPG keys are correctly imported +- Check repository URLs are accessible +- Ensure repository format is correct for Debian + +#### 3. Build Environment Issues +- Verify build dependencies are installed +- Check if the target architecture is supported +- Ensure sufficient disk space for build artifacts + +### Debug Mode +Enable debug logging by setting the `DEBUG` environment variable: +```bash +export DEBUG=1 +blue-build build recipe.yml +``` + +## Contributing + +### Adding New Package Mappings +Edit `debian_package_converter.py` to add new package name mappings: +```python +self.package_mappings.update({ + "new-rpm-package": "new-debian-package", + "another-rpm-pkg": "another-debian-pkg" +}) +``` + +### Adding New Repository Mappings +Edit `debian_package_converter.py` to add new repository mappings: +```python +self.repo_mappings.update({ + "new-rpm-repo": "new-debian-repo" +}) +``` + +### Testing Changes +1. Make your changes +2. Run the test suite +3. Test with real Debian images +4. Submit a pull request + +## Support + +### Documentation +- [Blue-Build Documentation](https://blue-build.org/) +- [Debian Package Management](https://wiki.debian.org/PackageManagement) +- [Debian Tasks](https://wiki.debian.org/tasksel) + +### Community +- [Blue-Build Community](https://github.com/blue-build) +- [Debian Community](https://www.debian.org/support) + +### Issues +Report issues and feature requests through the project's issue tracker. + +## License + +This project is licensed under the same terms as the original blue-build project. diff --git a/debian_module_adapter.py b/debian_module_adapter.py new file mode 100644 index 0000000..def9a44 --- /dev/null +++ b/debian_module_adapter.py @@ -0,0 +1,295 @@ +#!/usr/bin/env python3 +""" +Debian Module Adapter + +This module provides a compatibility layer between the new Debian modules +and existing Debian tools, ensuring seamless integration. +""" + +import json +import os +import subprocess +import tempfile +from typing import Dict, List, Optional, Any, Union +from dataclasses import dataclass, asdict +from pathlib import Path +import logging + +@dataclass +class ModuleConfig: + """Represents a module configuration""" + type: str + config: Dict[str, Any] + module_path: str + working_directory: str + +@dataclass +class ModuleResult: + """Represents the result of module execution""" + success: bool + output: str + error: Optional[str] + artifacts: List[str] + metadata: Dict[str, Any] + +class DebianModuleAdapter: + """Adapter for Debian modules to integrate with existing tools""" + + def __init__(self, config_dir: str = "./config/modules"): + self.config_dir = Path(config_dir) + self.config_dir.mkdir(parents=True, exist_ok=True) + self.logger = self._setup_logging() + + # Module type mappings + self.module_mappings = { + "apt": "apt", + "apt-ostree": "apt-ostree", + "deb-mock": "deb-mock", + "dnf": "apt", # Map dnf to apt for compatibility + "rpm-ostree": "apt-ostree", # Map rpm-ostree to apt-ostree + "mock": "deb-mock" # Map mock to deb-mock + } + + def _setup_logging(self) -> logging.Logger: + """Setup logging for the adapter""" + logger = logging.getLogger('debian-module-adapter') + logger.setLevel(logging.INFO) + + if not logger.handlers: + handler = logging.FileHandler(self.config_dir / "adapter.log") + formatter = logging.Formatter( + '%(asctime)s - %(name)s - %(levelname)s - %(message)s' + ) + handler.setFormatter(formatter) + logger.addHandler(handler) + + return logger + + def adapt_module_config(self, original_config: Dict[str, Any]) -> ModuleConfig: + """Adapt a module configuration for Debian compatibility""" + module_type = original_config.get("type", "") + + # Map module types if needed + if module_type in self.module_mappings: + adapted_type = self.module_mappings[module_type] + self.logger.info(f"Adapting module type from {module_type} to {adapted_type}") + else: + adapted_type = module_type + + # Create adapted configuration + adapted_config = self._create_adapted_config(original_config, adapted_type) + + return ModuleConfig( + type=adapted_type, + config=adapted_config, + module_path=f"modules/{adapted_type}", + working_directory=os.getcwd() + ) + + def _create_adapted_config(self, original_config: Dict[str, Any], adapted_type: str) -> Dict[str, Any]: + """Create an adapted configuration for the target module type""" + adapted_config = original_config.copy() + adapted_config["type"] = adapted_type + + # Handle specific adaptations based on module type + if adapted_type == "apt": + adapted_config = self._adapt_to_apt(original_config) + elif adapted_type == "apt-ostree": + adapted_config = self._adapt_to_apt_ostree(original_config) + elif adapted_type == "deb-mock": + adapted_config = self._adapt_to_deb_mock(original_config) + + return adapted_config + + def _adapt_to_apt(self, config: Dict[str, Any]) -> Dict[str, Any]: + """Adapt configuration for apt module""" + adapted = config.copy() + + # Convert RPM-specific fields to Debian equivalents + if "group-install" in adapted: + adapted["task-install"] = { + "packages": adapted.pop("group-install")["packages"], + "with-optional": adapted.pop("group-install").get("with-optional", False) + } + + if "group-remove" in adapted: + adapted["task-remove"] = { + "packages": adapted.pop("group-remove")["packages"] + } + + # Convert COPR repos to PPA + if "repos" in adapted and "copr" in adapted["repos"]: + if "ppa" not in adapted["repos"]: + adapted["repos"]["ppa"] = [] + + copr_repos = adapted["repos"]["copr"] + if isinstance(copr_repos, list): + for copr in copr_repos: + # Convert COPR format to PPA format + ppa_name = f"ppa:{copr.replace('/', '/')}" + adapted["repos"]["ppa"].append(ppa_name) + + # Remove COPR section + del adapted["repos"]["copr"] + + # Convert RPM Fusion to Debian backports + if "repos" in adapted and "nonfree" in adapted["repos"]: + if adapted["repos"]["nonfree"] == "rpmfusion": + adapted["repos"]["backports"] = True + del adapted["repos"]["nonfree"] + + return adapted + + def _adapt_to_apt_ostree(self, config: Dict[str, Any]) -> Dict[str, Any]: + """Adapt configuration for apt-ostree module""" + adapted = config.copy() + + # Convert RPM-specific fields to Debian equivalents + if "group-install" in adapted: + adapted["task-install"] = { + "packages": adapted.pop("group-install")["packages"], + "with-optional": adapted.pop("group-install").get("with-optional", False) + } + + if "group-remove" in adapted: + adapted["task-remove"] = { + "packages": adapted.pop("group-remove")["packages"] + } + + return adapted + + def _adapt_to_deb_mock(self, config: Dict[str, Any]) -> Dict[str, Any]: + """Adapt configuration for deb-mock module""" + adapted = config.copy() + + # Convert RPM-specific fields to Debian equivalents + if "chroot" in adapted: + adapted["environment"] = adapted.pop("chroot") + + if "builddeps" in adapted: + adapted["packages"] = adapted.pop("builddeps") + + if "resultdir" in adapted: + adapted["artifacts"] = adapted.pop("resultdir") + + return adapted + + def execute_module(self, module_config: ModuleConfig) -> ModuleResult: + """Execute a Debian module""" + try: + self.logger.info(f"Executing module: {module_config.type}") + + # Set up environment variables + env = os.environ.copy() + env["BLUEBUILD_MODULE_CONFIG"] = json.dumps(module_config.config) + + # Execute the module + result = subprocess.run( + ["docker", "run", "--rm", "-v", f"{module_config.working_directory}:/workspace", + "-w", "/workspace", "-e", f"BLUEBUILD_MODULE_CONFIG={json.dumps(module_config.config)}", + f"ghcr.io/blue-build/modules/{module_config.type}:latest"], + capture_output=True, + text=True, + env=env, + cwd=module_config.working_directory + ) + + if result.returncode == 0: + return ModuleResult( + success=True, + output=result.stdout, + error=None, + artifacts=self._extract_artifacts(result.stdout), + metadata={"return_code": result.returncode} + ) + else: + return ModuleResult( + success=False, + output=result.stdout, + error=result.stderr, + artifacts=[], + metadata={"return_code": result.returncode} + ) + + except Exception as e: + self.logger.error(f"Error executing module {module_config.type}: {e}") + return ModuleResult( + success=False, + output="", + error=str(e), + artifacts=[], + metadata={"error_type": type(e).__name__} + ) + + def _extract_artifacts(self, output: str) -> List[str]: + """Extract artifact information from module output""" + artifacts = [] + + # Look for artifact patterns in output + lines = output.split('\n') + for line in lines: + if "artifact:" in line.lower() or "created:" in line.lower(): + # Extract file paths from the line + parts = line.split() + for part in parts: + if part.startswith('/') or part.endswith(('.deb', '.dsc', '.tar.gz')): + artifacts.append(part) + + return artifacts + + def validate_module_config(self, module_config: ModuleConfig) -> bool: + """Validate a module configuration""" + try: + # Check if module type is supported + if module_config.type not in ["apt", "apt-ostree", "deb-mock"]: + self.logger.error(f"Unsupported module type: {module_config.type}") + return False + + # Check if required fields are present + required_fields = self._get_required_fields(module_config.type) + for field in required_fields: + if field not in module_config.config: + self.logger.error(f"Missing required field: {field}") + return False + + return True + + except Exception as e: + self.logger.error(f"Error validating module config: {e}") + return False + + def _get_required_fields(self, module_type: str) -> List[str]: + """Get required fields for a module type""" + if module_type == "apt": + return ["type"] + elif module_type == "apt-ostree": + return ["type"] + elif module_type == "deb-mock": + return ["type"] + else: + return [] + + def get_module_info(self, module_type: str) -> Dict[str, Any]: + """Get information about a module type""" + module_info = { + "apt": { + "description": "Debian package management with apt", + "capabilities": ["package_installation", "repository_management", "task_management"], + "supported_formats": [".deb", "package_names"], + "repository_types": ["ppa", "backports", "custom"] + }, + "apt-ostree": { + "description": "Debian package layering with ostree", + "capabilities": ["package_layering", "ostree_integration", "atomic_updates"], + "supported_formats": [".deb", "package_names"], + "repository_types": ["ppa", "backports", "custom"] + }, + "deb-mock": { + "description": "Debian build environment management", + "capabilities": ["build_environment", "package_building", "artifact_collection"], + "supported_formats": ["source_packages", ".deb"], + "build_tools": ["pbuilder", "sbuild", "schroot"] + } + } + + return module_info.get(module_type, {}) diff --git a/debian_package_converter.py b/debian_package_converter.py new file mode 100644 index 0000000..88aa640 --- /dev/null +++ b/debian_package_converter.py @@ -0,0 +1,425 @@ +#!/usr/bin/env python3 +""" +Debian Package Converter + +This module handles package format conversions between RPM and Debian formats, +ensuring compatibility when migrating from Fedora to Debian systems. +""" + +import json +import os +import subprocess +import tempfile +from typing import Dict, List, Optional, Any, Tuple +from dataclasses import dataclass, asdict +from pathlib import Path +import logging +import re + +@dataclass +class PackageInfo: + """Represents package information""" + name: str + version: str + architecture: str + format: str # "rpm" or "deb" + dependencies: List[str] + provides: List[str] + conflicts: List[str] + description: str + +@dataclass +class ConversionResult: + """Represents the result of a package conversion""" + success: bool + original_package: str + converted_package: Optional[str] + error: Optional[str] + warnings: List[str] + metadata: Dict[str, Any] + +class DebianPackageConverter: + """Converts packages between RPM and Debian formats""" + + def __init__(self, config_dir: str = "./config/converter"): + self.config_dir = Path(config_dir) + self.config_dir.mkdir(parents=True, exist_ok=True) + self.logger = self._setup_logging() + + # Package name mappings (RPM -> Debian) + self.package_mappings = self._load_package_mappings() + + # Repository mappings + self.repo_mappings = { + "rpmfusion": "debian-backports", + "copr": "ppa", + "epel": "debian-backports" + } + + def _setup_logging(self) -> logging.Logger: + """Setup logging for the converter""" + logger = logging.getLogger('debian-package-converter') + logger.setLevel(logging.INFO) + + if not logger.handlers: + handler = logging.FileHandler(self.config_dir / "converter.log") + formatter = logging.Formatter( + '%(asctime)s - %(name)s - %(levelname)s - %(message)s' + ) + handler.setFormatter(formatter) + logger.addHandler(handler) + + return logger + + def _load_package_mappings(self) -> Dict[str, str]: + """Load package name mappings from configuration""" + mappings_file = self.config_dir / "package_mappings.json" + + if mappings_file.exists(): + with open(mappings_file, 'r') as f: + return json.load(f) + else: + # Default mappings for common packages + default_mappings = { + # Development tools + "gcc": "gcc", + "gcc-c++": "g++", + "make": "make", + "cmake": "cmake", + "autoconf": "autoconf", + "automake": "automake", + "libtool": "libtool", + "pkg-config": "pkg-config", + + # System tools + "systemd": "systemd", + "systemd-sysv": "systemd-sysv", + "udev": "udev", + "dbus": "dbus", + + # Desktop environments + "gnome-shell": "gnome-shell", + "gnome-session": "gnome-session", + "gnome-control-center": "gnome-control-center", + "kde-workspace": "kde-plasma-workspace", + "kde-runtime": "kde-runtime", + + # Common libraries + "glib2": "libglib2.0-0", + "glib2-devel": "libglib2.0-dev", + "gtk3": "libgtk-3-0", + "gtk3-devel": "libgtk-3-dev", + "qt5-qtbase": "qtbase5-dev", + "qt5-qtbase-devel": "qtbase5-dev", + + # Network tools + "curl": "curl", + "wget": "wget", + "openssh": "openssh-client", + "openssh-server": "openssh-server", + + # Text editors + "vim": "vim", + "emacs": "emacs", + "nano": "nano", + + # Shells + "bash": "bash", + "zsh": "zsh", + "fish": "fish", + + # Package managers + "dnf": "apt", + "yum": "apt", + "rpm": "dpkg" + } + + # Save default mappings + with open(mappings_file, 'w') as f: + json.dump(default_mappings, f, indent=2) + + return default_mappings + + def convert_package_name(self, rpm_name: str) -> str: + """Convert an RPM package name to its Debian equivalent""" + # Remove RPM-specific suffixes + clean_name = rpm_name.replace("-devel", "").replace("-debuginfo", "").replace("-doc", "") + + # Check if we have a direct mapping + if clean_name in self.package_mappings: + return self.package_mappings[clean_name] + + # Handle common patterns + if clean_name.endswith("-devel"): + base_name = clean_name[:-6] + if base_name in self.package_mappings: + return f"{self.package_mappings[base_name]}-dev" + + # Handle library packages + if clean_name.startswith("lib") and clean_name.endswith("-devel"): + lib_name = clean_name[3:-6] + return f"lib{lib_name}-dev" + + # Handle Python packages + if clean_name.startswith("python3-"): + return clean_name # Python packages often have the same names + + # Handle Perl packages + if clean_name.startswith("perl-"): + perl_name = clean_name[5:] + return f"lib{perl_name}-perl" + + # If no mapping found, return the original name + self.logger.warning(f"No mapping found for package: {rpm_name}") + return clean_name + + def convert_repository(self, rpm_repo: str) -> str: + """Convert an RPM repository to its Debian equivalent""" + # Handle COPR repositories + if rpm_repo.startswith("copr:"): + copr_parts = rpm_repo.split(":") + if len(copr_parts) >= 3: + user = copr_parts[1] + project = copr_parts[2] + return f"ppa:{user}/{project}" + + # Handle RPM Fusion + if "rpmfusion" in rpm_repo: + return "debian-backports" + + # Handle EPEL + if "epel" in rpm_repo: + return "debian-backports" + + # If no mapping found, return the original + return rpm_repo + + def convert_package_spec(self, rpm_spec: str) -> str: + """Convert an RPM package specification to Debian format""" + # Handle URL specifications + if rpm_spec.startswith("http") and rpm_spec.endswith(".rpm"): + return rpm_spec.replace(".rpm", ".deb") + + # Handle package names + if not rpm_spec.startswith("http"): + return self.convert_package_name(rpm_spec) + + return rpm_spec + + def convert_dependencies(self, rpm_deps: List[str]) -> List[str]: + """Convert RPM dependencies to Debian equivalents""" + converted_deps = [] + + for dep in rpm_deps: + # Handle version constraints + if ">=" in dep: + parts = dep.split(">=") + pkg_name = self.convert_package_name(parts[0]) + version = parts[1] + converted_deps.append(f"{pkg_name} (>= {version})") + elif "<=" in dep: + parts = dep.split("<=") + pkg_name = self.convert_package_name(parts[0]) + version = parts[1] + converted_deps.append(f"{pkg_name} (<= {version})") + elif "=" in dep and not dep.startswith("="): + parts = dep.split("=") + pkg_name = self.convert_package_name(parts[0]) + version = parts[1] + converted_deps.append(f"{pkg_name} (= {version})") + else: + converted_deps.append(self.convert_package_name(dep)) + + return converted_deps + + def convert_group_to_task(self, rpm_group: str) -> str: + """Convert an RPM group to a Debian task""" + group_mappings = { + "development-tools": "development-tools", + "system-tools": "system-tools", + "network-tools": "network-tools", + "graphical-internet": "graphical-internet", + "text-internet": "text-internet", + "office": "office", + "graphics": "graphics", + "sound-and-video": "sound-and-video", + "games": "games", + "education": "education", + "scientific": "scientific", + "documentation": "documentation" + } + + return group_mappings.get(rpm_group, rpm_group) + + def convert_module_config(self, rpm_config: Dict[str, Any]) -> Dict[str, Any]: + """Convert an entire RPM module configuration to Debian format""" + deb_config = rpm_config.copy() + + # Convert module type + if deb_config.get("type") == "dnf": + deb_config["type"] = "apt" + elif deb_config.get("type") == "rpm-ostree": + deb_config["type"] = "apt-ostree" + elif deb_config.get("type") == "mock": + deb_config["type"] = "deb-mock" + + # Convert repositories + if "repos" in deb_config: + if "copr" in deb_config["repos"]: + deb_config["repos"]["ppa"] = [] + for copr in deb_config["repos"]["copr"]: + ppa = self.convert_repository(f"copr:{copr}") + if ppa.startswith("ppa:"): + deb_config["repos"]["ppa"].append(ppa) + del deb_config["repos"]["copr"] + + if "nonfree" in deb_config["repos"]: + if deb_config["repos"]["nonfree"] == "rpmfusion": + deb_config["repos"]["backports"] = True + del deb_config["repos"]["nonfree"] + + # Convert package installations + if "install" in deb_config and "packages" in deb_config["install"]: + packages = deb_config["install"]["packages"] + converted_packages = [] + + for pkg in packages: + if isinstance(pkg, str): + converted_packages.append(self.convert_package_spec(pkg)) + elif isinstance(pkg, dict) and "packages" in pkg: + converted_pkg = pkg.copy() + converted_pkg["packages"] = [self.convert_package_name(p) for p in pkg["packages"]] + converted_packages.append(converted_pkg) + else: + converted_packages.append(pkg) + + deb_config["install"]["packages"] = converted_packages + + # Convert package removals + if "remove" in deb_config and "packages" in deb_config["remove"]: + deb_config["remove"]["packages"] = [ + self.convert_package_name(pkg) for pkg in deb_config["remove"]["packages"] + ] + + # Convert groups to tasks + if "group-install" in deb_config: + deb_config["task-install"] = { + "packages": [self.convert_group_to_task(group) for group in deb_config["group-install"]["packages"]], + "with-optional": deb_config["group-install"].get("with-optional", False) + } + del deb_config["group-install"] + + if "group-remove" in deb_config: + deb_config["task-remove"] = { + "packages": [self.convert_group_to_task(group) for group in deb_config["group-remove"]["packages"]] + } + del deb_config["group-remove"] + + return deb_config + + def validate_debian_package(self, package_name: str) -> bool: + """Validate if a Debian package name is valid""" + # Basic validation - Debian package names should be lowercase, no spaces + if not package_name or " " in package_name: + return False + + # Check if it's a valid package name pattern + valid_pattern = re.compile(r'^[a-z0-9][a-z0-9+\-\.]+$') + return bool(valid_pattern.match(package_name)) + + def get_package_info(self, package_name: str, format_type: str = "deb") -> Optional[PackageInfo]: + """Get information about a package""" + try: + if format_type == "deb": + # Use apt-cache for Debian packages + result = subprocess.run( + ["apt-cache", "show", package_name], + capture_output=True, + text=True, + check=True + ) + + # Parse apt-cache output + info = self._parse_apt_cache_output(result.stdout) + return info + + elif format_type == "rpm": + # Use rpm for RPM packages + result = subprocess.run( + ["rpm", "-qi", package_name], + capture_output=True, + text=True, + check=True + ) + + # Parse rpm output + info = self._parse_rpm_output(result.stdout) + return info + + except subprocess.CalledProcessError: + self.logger.warning(f"Package not found: {package_name}") + return None + except Exception as e: + self.logger.error(f"Error getting package info: {e}") + return None + + def _parse_apt_cache_output(self, output: str) -> PackageInfo: + """Parse apt-cache output to extract package information""" + lines = output.split('\n') + info = { + "name": "", + "version": "", + "architecture": "", + "format": "deb", + "dependencies": [], + "provides": [], + "conflicts": [], + "description": "" + } + + for line in lines: + if line.startswith("Package:"): + info["name"] = line.split(":", 1)[1].strip() + elif line.startswith("Version:"): + info["version"] = line.split(":", 1)[1].strip() + elif line.startswith("Architecture:"): + info["architecture"] = line.split(":", 1)[1].strip() + elif line.startswith("Depends:"): + deps = line.split(":", 1)[1].strip() + info["dependencies"] = [d.strip() for d in deps.split(",")] + elif line.startswith("Provides:"): + provides = line.split(":", 1)[1].strip() + info["provides"] = [p.strip() for p in provides.split(",")] + elif line.startswith("Conflicts:"): + conflicts = line.split(":", 1)[1].strip() + info["conflicts"] = [c.strip() for c in conflicts.split(",")] + elif line.startswith("Description:"): + info["description"] = line.split(":", 1)[1].strip() + + return PackageInfo(**info) + + def _parse_rpm_output(self, output: str) -> PackageInfo: + """Parse rpm output to extract package information""" + lines = output.split('\n') + info = { + "name": "", + "version": "", + "architecture": "", + "format": "rpm", + "dependencies": [], + "provides": [], + "conflicts": [], + "description": "" + } + + for line in lines: + if line.startswith("Name"): + info["name"] = line.split(":", 1)[1].strip() + elif line.startswith("Version"): + info["version"] = line.split(":", 1)[1].strip() + elif line.startswith("Architecture"): + info["architecture"] = line.split(":", 1)[1].strip() + elif line.startswith("Summary"): + info["description"] = line.split(":", 1)[1].strip() + + return PackageInfo(**info) diff --git a/examples/debian-basic/debian-server.yml b/examples/debian-basic/debian-server.yml new file mode 100644 index 0000000..bedba21 --- /dev/null +++ b/examples/debian-basic/debian-server.yml @@ -0,0 +1,56 @@ +--- +name: debian-server +description: Basic Debian server image with essential packages +base-image: debian:bookworm-slim +image-version: latest + +modules: + - type: apt + repos: + cleanup: true + backports: true + keys: + - https://deb.debian.org/debian/dists/bookworm/Release.gpg + install: + skip-unavailable: true + packages: + - curl + - wget + - vim + - htop + - nginx + - postgresql + - python3 + - python3-pip + - git + - build-essential + remove: + packages: + - nano + - less + task-install: + with-optional: false + packages: + - development-tools + - system-tools + + - type: script + snippets: + - echo "Debian server setup completed" + - echo "OS Version: {{ os_version }}" + - systemctl enable nginx + - systemctl enable postgresql + + - type: files + files: + - source: ./nginx.conf + dest: /etc/nginx/nginx.conf + - source: ./postgresql.conf + dest: /etc/postgresql/postgresql.conf + + - type: systemd + units: + - name: nginx.service + enabled: true + - name: postgresql.service + enabled: true diff --git a/examples/debian-gnome/debian-gnome.yml b/examples/debian-gnome/debian-gnome.yml new file mode 100644 index 0000000..541553c --- /dev/null +++ b/examples/debian-gnome/debian-gnome.yml @@ -0,0 +1,86 @@ +--- +name: debian-gnome +description: Debian GNOME desktop environment with essential applications +base-image: debian:bookworm-slim +image-version: latest + +modules: + - type: apt + repos: + cleanup: true + backports: true + ppa: + - ppa:gnome3-team/gnome3 + - ppa:gnome3-team/gnome3-staging + keys: + - https://deb.debian.org/debian/dists/bookworm/Release.gpg + install: + skip-unavailable: true + packages: + - gnome-shell + - gnome-session + - gnome-control-center + - gnome-terminal + - gnome-software + - nautilus + - gedit + - firefox-esr + - libreoffice + - gimp + - inkscape + - vlc + - rhythmbox + - totem + - evolution + - gnome-calendar + - gnome-contacts + - gnome-maps + - gnome-weather + - gnome-tweaks + - dconf-editor + - gnome-shell-extensions + task-install: + with-optional: true + packages: + - gnome-desktop-environment + - gnome-desktop-environment-apps + remove: + packages: + - nano + - less + - vim-tiny + + - type: script + snippets: + - echo "GNOME desktop setup completed" + - echo "OS Version: {{ os_version }}" + - systemctl set-default graphical.target + - systemctl enable gdm + + - type: gschema-overrides + overrides: + - schema: org.gnome.desktop.interface + key: enable-hot-corners + value: true + - schema: org.gnome.desktop.interface + key: show-battery-percentage + value: true + - schema: org.gnome.desktop.background + key: picture-uri + value: "file:///usr/share/backgrounds/debian/default.jpg" + + - type: gnome-extensions + extensions: + - name: "user-theme@gnome-shell-extensions.gcampax.github.com" + enabled: true + - name: "dash-to-dock@micxgx.gmail.com" + enabled: true + - name: "workspace-indicator@gnome-shell-extensions.gcampax.github.com" + enabled: true + + - type: systemd + units: + - name: gdm.service + enabled: true + - name: NetworkManager.service + enabled: true diff --git a/examples/debian-kde/debian-kde.yml b/examples/debian-kde/debian-kde.yml new file mode 100644 index 0000000..3852848 --- /dev/null +++ b/examples/debian-kde/debian-kde.yml @@ -0,0 +1,55 @@ +--- +name: debian-kde +description: Debian KDE Plasma desktop environment +base-image: debian:bookworm-slim +image-version: latest + +modules: + - type: apt + repos: + cleanup: true + backports: true + ppa: + - ppa:kubuntu-ppa/backports + keys: + - https://deb.debian.org/debian/dists/bookworm/Release.gpg + install: + skip-unavailable: true + packages: + - kde-plasma-desktop + - kde-standard + - dolphin + - konsole + - kate + - firefox-esr + - libreoffice + - gimp + - vlc + - amarok + - k3b + - kdenlive + - digikam + task-install: + with-optional: true + packages: + - kde-plasma-desktop + - kde-plasma-desktop-apps + remove: + packages: + - nano + - less + - vim-tiny + + - type: script + snippets: + - echo "KDE desktop setup completed" + - echo "OS Version: {{ os_version }}" + - systemctl set-default graphical.target + - systemctl enable sddm + + - type: systemd + units: + - name: sddm.service + enabled: true + - name: NetworkManager.service + enabled: true diff --git a/modules.json b/modules.json index be662e0..55e9ccf 100644 --- a/modules.json +++ b/modules.json @@ -12,6 +12,9 @@ "https://raw.githubusercontent.com/blue-build/modules/main/modules/justfiles/module.yml", "https://raw.githubusercontent.com/blue-build/modules/main/modules/rpm-ostree/module.yml", "https://raw.githubusercontent.com/blue-build/modules/main/modules/dnf/module.yml", + "https://raw.githubusercontent.com/blue-build/modules/main/modules/apt/module.yml", + "https://raw.githubusercontent.com/blue-build/modules/main/modules/apt-ostree/module.yml", + "https://raw.githubusercontent.com/blue-build/modules/main/modules/deb-mock/module.yml", "https://raw.githubusercontent.com/blue-build/modules/main/modules/os-release/module.yml", "https://raw.githubusercontent.com/blue-build/modules/main/modules/kargs/module.yml", "https://raw.githubusercontent.com/blue-build/modules/main/modules/initramfs/module.yml", diff --git a/modules/apt-ostree/Containerfile b/modules/apt-ostree/Containerfile new file mode 100644 index 0000000..d46e1f0 --- /dev/null +++ b/modules/apt-ostree/Containerfile @@ -0,0 +1,21 @@ +FROM debian:bookworm-slim + +# Install required packages +RUN apt-get update && apt-get install -y \ + apt \ + apt-transport-https \ + ca-certificates \ + curl \ + gnupg \ + ostree \ + && rm -rf /var/lib/apt/lists/* + +# Copy module files +COPY entrypoint.sh /usr/local/bin/ +COPY ostree-wrapper.sh /usr/local/bin/ + +# Make scripts executable +RUN chmod +x /usr/local/bin/entrypoint.sh /usr/local/bin/ostree-wrapper.sh + +# Set entrypoint +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] diff --git a/modules/apt-ostree/entrypoint.sh b/modules/apt-ostree/entrypoint.sh new file mode 100644 index 0000000..da94356 --- /dev/null +++ b/modules/apt-ostree/entrypoint.sh @@ -0,0 +1,36 @@ +#!/bin/bash +set -euo pipefail + +# Log function +log() { + echo "[APT-OSTREE-MODULE] $1" +} + +# Check if we have the required environment variables +if [ -z "${BLUEBUILD_MODULE_CONFIG:-}" ]; then + log "ERROR: BLUEBUILD_MODULE_CONFIG environment variable is not set" + exit 1 +fi + +# Parse the module configuration +log "Parsing module configuration..." +CONFIG_FILE=$(echo "$BLUEBUILD_MODULE_CONFIG" | jq -r '.config_file // empty') +if [ -n "$CONFIG_FILE" ]; then + log "Using config file: $CONFIG_FILE" + MODULE_CONFIG=$(cat "$CONFIG_FILE") +else + log "Using inline module configuration" + MODULE_CONFIG="$BLUEBUILD_MODULE_CONFIG" +fi + +# Extract module type and validate +MODULE_TYPE=$(echo "$MODULE_CONFIG" | jq -r '.type // empty') +if [ "$MODULE_TYPE" != "apt-ostree" ]; then + log "ERROR: Invalid module type: $MODULE_TYPE (expected: apt-ostree)" + exit 1 +fi + +log "Starting apt-ostree module execution..." + +# Execute the ostree wrapper with the configuration +exec /usr/local/bin/ostree-wrapper.sh "$MODULE_CONFIG" diff --git a/modules/apt-ostree/module.yml b/modules/apt-ostree/module.yml new file mode 100644 index 0000000..346621d --- /dev/null +++ b/modules/apt-ostree/module.yml @@ -0,0 +1,25 @@ +name: apt-ostree +shortdesc: The apt-ostree module offers pseudo-declarative package and repository management using apt-ostree. +example: | + type: apt-ostree + repos: + - https://ppa.launchpadcontent.net/ondrej/php/ubuntu/dists/jammy/Release.gpg + - https://brave-browser-apt-release.s3.brave.com/brave-browser.list + keys: + - https://brave-browser-apt-release.s3.brave.com/brave-core.asc + optfix: + - Tabby # needed because tabby installs into `/opt/Tabby` + - brave.com + install: + - starship + - brave-browser + - https://github.com/Eugeny/tabby/releases/download/v1.0.209/tabby-1.0.209-linux-x64.deb + remove: + - firefox + - firefox-l10n-all + replace: + - from-repo: https://ppa.launchpadcontent.net/ondrej/php/ubuntu/dists/jammy/Release.gpg + packages: + - php8.2 + - php8.2-common + - php8.2-cli diff --git a/modules/apt-ostree/ostree-wrapper.sh b/modules/apt-ostree/ostree-wrapper.sh new file mode 100644 index 0000000..91875be --- /dev/null +++ b/modules/apt-ostree/ostree-wrapper.sh @@ -0,0 +1,214 @@ +#!/bin/bash +set -euo pipefail + +# Log function +log() { + echo "[OSTREE-WRAPPER] $1" +} + +# Function to add repository files +add_repos() { + local repos_config="$1" + + if [ -n "$repos_config" ]; then + local repos=$(echo "$repos_config" | jq -r '.[]? // empty') + for repo in $repos; do + if [[ $repo == http* ]]; then + log "Downloading repository file: $repo" + curl -fsSL "$repo" -o "/etc/apt/sources.list.d/$(basename "$repo")" + else + log "Copying repository file: $repo" + cp "$repo" "/etc/apt/sources.list.d/" + fi + done + fi +} + +# Function to add GPG keys +add_keys() { + local keys_config="$1" + + if [ -n "$keys_config" ]; then + local keys=$(echo "$keys_config" | jq -r '.[]? // empty') + for key in $keys; do + if [[ $key == http* ]]; then + log "Downloading and adding GPG key: $key" + curl -fsSL "$key" | apt-key add - + else + log "Adding GPG key: $key" + apt-key add "$key" + fi + done + fi +} + +# Function to handle /opt/ symlinking +handle_optfix() { + local optfix_config="$1" + if [ -n "$optfix_config" ]; then + local optfixes=$(echo "$optfix_config" | jq -r '.[]? // empty') + for optfix in $optfixes; do + log "Setting up /opt/ symlink for: $optfix" + mkdir -p "/opt/$optfix" + ln -sf "/opt/$optfix" "/usr/local/$optfix" + done + fi +} + +# Function to install packages using ostree +install_packages() { + local install_config="$1" + + if [ -n "$install_config" ]; then + local packages=$(echo "$install_config" | jq -r '.[]? // empty') + for package in $packages; do + if [[ $package == http* && $package == *.deb ]]; then + log "Downloading and installing .deb package: $package" + curl -fsSL "$package" -o /tmp/package.deb + # Use ostree to install the package + ostree admin install-pkg /tmp/package.deb || { + log "WARNING: Failed to install package via ostree: $package" + # Fallback to regular dpkg + dpkg -i /tmp/package.deb || apt-get install -f -y + } + rm -f /tmp/package.deb + else + log "Installing package via ostree: $package" + # Use ostree to install the package + ostree admin install-pkg "$package" || { + log "WARNING: Failed to install package via ostree: $package" + # Fallback to regular apt + apt-get install -y "$package" || { + log "ERROR: Failed to install package $package" + exit 1 + } + } + fi + done + fi +} + +# Function to remove packages using ostree +remove_packages() { + local remove_config="$1" + + if [ -n "$remove_config" ]; then + local packages=$(echo "$remove_config" | jq -r '.[]? // empty') + for package in $packages; do + log "Removing package via ostree: $package" + # Use ostree to remove the package + ostree admin uninstall-pkg "$package" || { + log "WARNING: Failed to remove package via ostree: $package" + # Fallback to regular apt + apt-get remove -y "$package" || { + log "WARNING: Failed to remove package $package" + } + } + done + fi +} + +# Function to replace packages using ostree +replace_packages() { + local replace_config="$1" + + if [ -n "$replace_config" ]; then + local replacements=$(echo "$replace_config" | jq -c '.[]') + echo "$replacements" | while read -r replacement; do + local from_repo=$(echo "$replacement" | jq -r '.from-repo // empty') + local packages=$(echo "$replacement" | jq -r '.packages[]? // empty') + + log "Replacing packages from repository: $from_repo" + for package in $packages; do + log "Replacing package via ostree: $package" + # Use ostree to replace the package + ostree admin replace-pkg "$package" || { + log "WARNING: Failed to replace package via ostree: $package" + # Fallback to regular apt + apt-get install -y "$package" || { + log "ERROR: Failed to replace package $package" + exit 1 + } + } + done + done + fi +} + +# Function to commit ostree changes +commit_changes() { + log "Committing ostree changes" + + # Get current ostree ref + local current_ref=$(ostree refs --list | head -n1) + if [ -n "$current_ref" ]; then + log "Current ostree ref: $current_ref" + + # Create new commit with changes + local new_ref="${current_ref}-$(date +%Y%m%d-%H%M%S)" + log "Creating new ostree ref: $new_ref" + + # Commit the changes + ostree commit --branch="$new_ref" --tree=dir=/ || { + log "WARNING: Failed to commit ostree changes" + } + else + log "WARNING: No ostree refs found" + fi +} + +# Main execution +main() { + local module_config="$1" + + log "Starting apt-ostree module execution" + + # Handle repositories + if echo "$module_config" | jq -e '.repos' >/dev/null 2>&1; then + log "Processing repositories configuration" + add_repos "$(echo "$module_config" | jq -c '.repos')" + fi + + # Handle GPG keys + if echo "$module_config" | jq -e '.keys' >/dev/null 2>&1; then + log "Processing GPG keys configuration" + add_keys "$(echo "$module_config" | jq -c '.keys')" + fi + + # Handle /opt/ symlinking + if echo "$module_config" | jq -e '.optfix' >/dev/null 2>&1; then + log "Processing optfix configuration" + handle_optfix "$(echo "$module_config" | jq -c '.optfix')" + fi + + # Handle package installation + if echo "$module_config" | jq -e '.install' >/dev/null 2>&1; then + log "Processing package installation" + install_packages "$(echo "$module_config" | jq -c '.install')" + fi + + # Handle package removal + if echo "$module_config" | jq -e '.remove' >/dev/null 2>&1; then + log "Processing package removal" + remove_packages "$(echo "$module_config" | jq -c '.remove')" + fi + + # Handle package replacement + if echo "$module_config" | jq -e '.replace' >/dev/null 2>&1; then + log "Processing package replacement" + replace_packages "$(echo "$module_config" | jq -c '.replace')" + fi + + # Commit ostree changes + commit_changes + + # Clean up + log "Cleaning up package cache" + apt-get clean + apt-get autoremove -y + + log "apt-ostree module execution completed successfully" +} + +# Execute main function with the module configuration +main "$1" diff --git a/modules/apt/Containerfile b/modules/apt/Containerfile new file mode 100644 index 0000000..9d195e8 --- /dev/null +++ b/modules/apt/Containerfile @@ -0,0 +1,22 @@ +FROM debian:bookworm-slim + +# Install required packages +RUN apt-get update && apt-get install -y \ + apt \ + apt-transport-https \ + ca-certificates \ + curl \ + gnupg \ + software-properties-common \ + tasksel \ + && rm -rf /var/lib/apt/lists/* + +# Copy module files +COPY entrypoint.sh /usr/local/bin/ +COPY apt-wrapper.sh /usr/local/bin/ + +# Make scripts executable +RUN chmod +x /usr/local/bin/entrypoint.sh /usr/local/bin/apt-wrapper.sh + +# Set entrypoint +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] diff --git a/modules/apt/apt-wrapper.sh b/modules/apt/apt-wrapper.sh new file mode 100644 index 0000000..f3eccb5 --- /dev/null +++ b/modules/apt/apt-wrapper.sh @@ -0,0 +1,276 @@ +#!/bin/bash +set -euo pipefail + +# Log function +log() { + echo "[APT-WRAPPER] $1" +} + +# Function to add repository files +add_repos() { + local repos_config="$1" + local cleanup=false + + # Check if cleanup is enabled + if echo "$repos_config" | jq -e '.cleanup' >/dev/null 2>&1; then + cleanup=$(echo "$repos_config" | jq -r '.cleanup') + fi + + # Add repository files + if echo "$repos_config" | jq -e '.files' >/dev/null 2>&1; then + local files=$(echo "$repos_config" | jq -r '.files[]? // empty') + for file in $files; do + if [[ $file == http* ]]; then + log "Downloading repository file: $file" + curl -fsSL "$file" -o "/etc/apt/sources.list.d/$(basename "$file")" + else + log "Copying repository file: $file" + cp "$file" "/etc/apt/sources.list.d/" + fi + done + fi + + # Add PPA repositories + if echo "$repos_config" | jq -e '.ppa' >/dev/null 2>&1; then + local ppas=$(echo "$repos_config" | jq -r '.ppa[]? // empty') + for ppa in $ppas; do + log "Adding PPA: $ppa" + add-apt-repository -y "$ppa" + done + fi + + # Add GPG keys + if echo "$repos_config" | jq -e '.keys' >/dev/null 2>&1; then + local keys=$(echo "$repos_config" | jq -r '.keys[]? // empty') + for key in $keys; do + if [[ $key == http* ]]; then + log "Downloading and adding GPG key: $key" + curl -fsSL "$key" | apt-key add - + else + log "Adding GPG key: $key" + apt-key add "$key" + fi + done + fi + + # Enable backports if requested + if echo "$repos_config" | jq -e '.backports' >/dev/null 2>&1; then + local backports=$(echo "$repos_config" | jq -r '.backports') + if [ "$backports" = "true" ]; then + log "Enabling backports repository" + echo "deb http://deb.debian.org/debian $(lsb_release -cs)-backports main" > /etc/apt/sources.list.d/backports.list + fi + fi + + # Update package lists + log "Updating package lists" + apt-get update + + # Clean up repositories if requested + if [ "$cleanup" = "true" ]; then + log "Cleaning up added repositories" + rm -f /etc/apt/sources.list.d/*.list + apt-get update + fi +} + +# Function to handle /opt/ symlinking +handle_optfix() { + local optfix_config="$1" + if [ -n "$optfix_config" ]; then + local optfixes=$(echo "$optfix_config" | jq -r '.[]? // empty') + for optfix in $optfixes; do + log "Setting up /opt/ symlink for: $optfix" + mkdir -p "/opt/$optfix" + ln -sf "/opt/$optfix" "/usr/local/$optfix" + done + fi +} + +# Function to install packages +install_packages() { + local install_config="$1" + local skip_unavailable=false + + # Check if skip-unavailable is enabled + if echo "$install_config" | jq -e '.skip-unavailable' >/dev/null 2>&1; then + skip_unavailable=$(echo "$install_config" | jq -r '.skip-unavailable') + fi + + # Install packages + if echo "$install_config" | jq -e '.packages' >/dev/null 2>&1; then + local packages=$(echo "$install_config" | jq -r '.packages[]? // empty') + for package in $packages; do + if echo "$package" | jq -e '.repo' >/dev/null 2>&1; then + # Package from specific repository + local repo=$(echo "$package" | jq -r '.repo') + local repo_packages=$(echo "$package" | jq -r '.packages[]? // empty') + for repo_package in $repo_packages; do + log "Installing $repo_package from repository $repo" + apt-get install -y "$repo_package" || { + if [ "$skip_unavailable" = "true" ]; then + log "WARNING: Package $repo_package not available, skipping" + else + log "ERROR: Failed to install package $repo_package" + exit 1 + fi + } + done + else + # Direct package specification + if [[ $package == http* && $package == *.deb ]]; then + log "Downloading and installing .deb package: $package" + curl -fsSL "$package" -o /tmp/package.deb + dpkg -i /tmp/package.deb || apt-get install -f -y + rm -f /tmp/package.deb + else + log "Installing package: $package" + apt-get install -y "$package" || { + if [ "$skip_unavailable" = "true" ]; then + log "WARNING: Package $package not available, skipping" + else + log "ERROR: Failed to install package $package" + exit 1 + fi + } + fi + fi + done + fi +} + +# Function to remove packages +remove_packages() { + local remove_config="$1" + if echo "$remove_config" | jq -e '.packages' >/dev/null 2>&1; then + local packages=$(echo "$remove_config" | jq -r '.packages[]? // empty') + for package in $packages; do + log "Removing package: $package" + apt-get remove -y "$package" || { + log "WARNING: Failed to remove package $package" + } + done + fi +} + +# Function to replace packages +replace_packages() { + local replace_config="$1" + if echo "$replace_config" | jq -e '.[]' >/dev/null 2>&1; then + local replacements=$(echo "$replace_config" | jq -c '.[]') + echo "$replacements" | while read -r replacement; do + local from_repo=$(echo "$replacement" | jq -r '.from-repo // empty') + local packages=$(echo "$replacement" | jq -r '.packages[]? // empty') + local skip_unavailable=$(echo "$replacement" | jq -r '.skip-unavailable // false') + + log "Replacing packages from repository: $from_repo" + for package in $packages; do + log "Replacing package: $package" + apt-get install -y "$package" || { + if [ "$skip_unavailable" = "true" ]; then + log "WARNING: Package $package not available, skipping" + else + log "ERROR: Failed to replace package $package" + exit 1 + fi + } + done + done + fi +} + +# Function to install task packages +install_tasks() { + local task_config="$1" + if echo "$task_config" | jq -e '.packages' >/dev/null 2>&1; then + local with_optional=$(echo "$task_config" | jq -r '.with-optional // false') + local packages=$(echo "$task_config" | jq -r '.packages[]? // empty') + + for task in $packages; do + log "Installing task: $task" + if [ "$with_optional" = "true" ]; then + tasksel install "$task" || { + log "WARNING: Failed to install task $task" + } + else + tasksel install "$task" || { + log "WARNING: Failed to install task $task" + } + fi + done + fi +} + +# Function to remove task packages +remove_tasks() { + local task_config="$1" + if echo "$task_config" | jq -e '.packages' >/dev/null 2>&1; then + local packages=$(echo "$task_config" | jq -r '.packages[]? // empty') + + for task in $packages; do + log "Removing task: $task" + tasksel remove "$task" || { + log "WARNING: Failed to remove task $task" + } + done + fi +} + +# Main execution +main() { + local module_config="$1" + + log "Starting apt module execution" + + # Handle repositories + if echo "$module_config" | jq -e '.repos' >/dev/null 2>&1; then + log "Processing repositories configuration" + add_repos "$(echo "$module_config" | jq -c '.repos')" + fi + + # Handle /opt/ symlinking + if echo "$module_config" | jq -e '.optfix' >/dev/null 2>&1; then + log "Processing optfix configuration" + handle_optfix "$(echo "$module_config" | jq -c '.optfix')" + fi + + # Handle package installation + if echo "$module_config" | jq -e '.install' >/dev/null 2>&1; then + log "Processing package installation" + install_packages "$(echo "$module_config" | jq -c '.install')" + fi + + # Handle package removal + if echo "$module_config" | jq -e '.remove' >/dev/null 2>&1; then + log "Processing package removal" + remove_packages "$(echo "$module_config" | jq -c '.remove')" + fi + + # Handle package replacement + if echo "$module_config" | jq -e '.replace' >/dev/null 2>&1; then + log "Processing package replacement" + replace_packages "$(echo "$module_config" | jq -c '.replace')" + fi + + # Handle task installation + if echo "$module_config" | jq -e '.task-install' >/dev/null 2>&1; then + log "Processing task installation" + install_tasks "$(echo "$module_config" | jq -c '.task-install')" + fi + + # Handle task removal + if echo "$module_config" | jq -e '.task-remove' >/dev/null 2>&1; then + log "Processing task removal" + remove_tasks "$(echo "$module_config" | jq -c '.task-remove')" + fi + + # Clean up + log "Cleaning up package cache" + apt-get clean + apt-get autoremove -y + + log "apt module execution completed successfully" +} + +# Execute main function with the module configuration +main "$1" diff --git a/modules/apt/entrypoint.sh b/modules/apt/entrypoint.sh new file mode 100644 index 0000000..dd6c0e6 --- /dev/null +++ b/modules/apt/entrypoint.sh @@ -0,0 +1,36 @@ +#!/bin/bash +set -euo pipefail + +# Log function +log() { + echo "[APT-MODULE] $1" +} + +# Check if we have the required environment variables +if [ -z "${BLUEBUILD_MODULE_CONFIG:-}" ]; then + log "ERROR: BLUEBUILD_MODULE_CONFIG environment variable is not set" + exit 1 +fi + +# Parse the module configuration +log "Parsing module configuration..." +CONFIG_FILE=$(echo "$BLUEBUILD_MODULE_CONFIG" | jq -r '.config_file // empty') +if [ -n "$CONFIG_FILE" ]; then + log "Using config file: $CONFIG_FILE" + MODULE_CONFIG=$(cat "$CONFIG_FILE") +else + log "Using inline module configuration" + MODULE_CONFIG="$BLUEBUILD_MODULE_CONFIG" +fi + +# Extract module type and validate +MODULE_TYPE=$(echo "$MODULE_CONFIG" | jq -r '.type // empty') +if [ "$MODULE_TYPE" != "apt" ]; then + log "ERROR: Invalid module type: $MODULE_TYPE (expected: apt)" + exit 1 +fi + +log "Starting apt module execution..." + +# Execute the apt wrapper with the configuration +exec /usr/local/bin/apt-wrapper.sh "$MODULE_CONFIG" diff --git a/modules/apt/module.yml b/modules/apt/module.yml new file mode 100644 index 0000000..2d9ce73 --- /dev/null +++ b/modules/apt/module.yml @@ -0,0 +1,47 @@ +name: apt +shortdesc: The apt module offers pseudo-declarative package and repository management using apt. +example: | + type: apt + repos: + cleanup: true # clean up added repos after module is done + files: + - https://brave-browser-apt-release.s3.brave.com/brave-browser.list + - custom.list + ppa: + - ppa:ondrej/php + - ppa:git-core/ppa + keys: + - https://brave-browser-apt-release.s3.brave.com/brave-core.asc + backports: true # enable backports repository + optfix: # performs symlinking for `/opt/` to allow certain packages to install + - Tabby # needed because tabby installs into `/opt/Tabby/` + - brave.com + install: + skip-unavailable: true # skip unavailable packages + packages: + - repo: brave-browser + packages: + - brave-browser + - starship + - https://github.com/Eugeny/tabby/releases/download/v1.0.209/tabby-1.0.209-linux-x64.deb + - kubectl.deb + remove: + packages: + - firefox + - firefox-l10n-all + replace: + - from-repo: ppa:ondrej/php + skip-unavailable: true # skip unavailable packages + packages: + - php8.2 + - php8.2-common + - php8.2-cli + task-install: + with-optional: true # install optional packages from task + packages: + - gnome-desktop-environment + - kde-plasma-desktop + - xfce4 + task-remove: + packages: + - development-tools diff --git a/modules/deb-mock/Containerfile b/modules/deb-mock/Containerfile new file mode 100644 index 0000000..7b1c015 --- /dev/null +++ b/modules/deb-mock/Containerfile @@ -0,0 +1,22 @@ +FROM debian:bookworm-slim + +# Install required packages +RUN apt-get update && apt-get install -y \ + build-essential \ + devscripts \ + debhelper \ + deb-mock \ + pbuilder \ + sbuild \ + schroot \ + && rm -rf /var/lib/apt/lists/* + +# Copy module files +COPY entrypoint.sh /usr/local/bin/ +COPY mock-wrapper.sh /usr/local/bin/ + +# Make scripts executable +RUN chmod +x /usr/local/bin/entrypoint.sh /usr/local/bin/mock-wrapper.sh + +# Set entrypoint +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] diff --git a/modules/deb-mock/entrypoint.sh b/modules/deb-mock/entrypoint.sh new file mode 100644 index 0000000..de8d9a9 --- /dev/null +++ b/modules/deb-mock/entrypoint.sh @@ -0,0 +1,36 @@ +#!/bin/bash +set -euo pipefail + +# Log function +log() { + echo "[DEB-MOCK-MODULE] $1" +} + +# Check if we have the required environment variables +if [ -z "${BLUEBUILD_MODULE_CONFIG:-}" ]; then + log "ERROR: BLUEBUILD_MODULE_CONFIG environment variable is not set" + exit 1 +fi + +# Parse the module configuration +log "Parsing module configuration..." +CONFIG_FILE=$(echo "$BLUEBUILD_MODULE_CONFIG" | jq -r '.config_file // empty') +if [ -n "$CONFIG_FILE" ]; then + log "Using config file: $CONFIG_FILE" + MODULE_CONFIG=$(cat "$CONFIG_FILE") +else + log "Using inline module configuration" + MODULE_CONFIG="$BLUEBUILD_MODULE_CONFIG" +fi + +# Extract module type and validate +MODULE_TYPE=$(echo "$MODULE_CONFIG" | jq -r '.type // empty') +if [ "$MODULE_TYPE" != "deb-mock" ]; then + log "ERROR: Invalid module type: $MODULE_TYPE (expected: deb-mock)" + exit 1 +fi + +log "Starting deb-mock module execution..." + +# Execute the mock wrapper with the configuration +exec /usr/local/bin/mock-wrapper.sh "$MODULE_CONFIG" diff --git a/modules/deb-mock/mock-wrapper.sh b/modules/deb-mock/mock-wrapper.sh new file mode 100644 index 0000000..e6ad381 --- /dev/null +++ b/modules/deb-mock/mock-wrapper.sh @@ -0,0 +1,186 @@ +#!/bin/bash +set -euo pipefail + +# Log function +log() { + echo "[MOCK-WRAPPER] $1" +} + +# Function to setup build environment +setup_environment() { + local env_config="$1" + local environment=$(echo "$env_config" | jq -r '.environment // "bookworm-amd64"') + + log "Setting up build environment: $environment" + + # Parse environment (e.g., "bookworm-amd64" -> suite="bookworm", arch="amd64") + local suite=$(echo "$environment" | cut -d'-' -f1) + local arch=$(echo "$environment" | cut -d'-' -f2) + + log "Suite: $suite, Architecture: $arch" + + # Create pbuilder environment + log "Creating pbuilder environment" + pbuilder create --distribution "$suite" --architecture "$arch" --basetgz "/var/cache/pbuilder/$suite-$arch-base.tgz" || { + log "WARNING: Failed to create pbuilder environment, trying sbuild" + sbuild-createchroot --dist="$suite" --arch="$arch" "/var/cache/sbuild/$suite-$arch" http://deb.debian.org/debian/ || { + log "ERROR: Failed to create build environment" + exit 1 + } + } +} + +# Function to install build dependencies +install_dependencies() { + local packages_config="$1" + + if [ -n "$packages_config" ]; then + local packages=$(echo "$packages_config" | jq -r '.[]? // empty') + for package in $packages; do + log "Installing build dependency: $package" + apt-get install -y "$package" || { + log "WARNING: Failed to install package $package" + } + done + fi +} + +# Function to configure repositories +configure_repositories() { + local repos_config="$1" + + if [ -n "$repos_config" ]; then + local repos=$(echo "$repos_config" | jq -r '.[]? // empty') + for repo in $repos; do + log "Adding repository: $repo" + echo "$repo" >> /etc/apt/sources.list.d/deb-mock.list + done + + # Update package lists + apt-get update + fi +} + +# Function to execute build script +execute_build() { + local build_script="$1" + + if [ -n "$build_script" ]; then + log "Executing build script" + + # Create temporary build script + local script_file="/tmp/build-script.sh" + echo "$build_script" > "$script_file" + chmod +x "$script_file" + + # Execute the build script + "$script_file" || { + log "ERROR: Build script failed" + exit 1 + } + + # Clean up + rm -f "$script_file" + else + log "No build script provided, skipping build execution" + fi +} + +# Function to collect artifacts +collect_artifacts() { + local artifacts_config="$1" + + if [ -n "$artifacts_config" ]; then + local artifacts=$(echo "$artifacts_config" | jq -r '.[]? // empty') + local artifacts_dir="/tmp/artifacts" + + mkdir -p "$artifacts_dir" + + for artifact in $artifacts; do + log "Collecting artifact: $artifact" + + # Handle glob patterns + if [[ $artifact == *"*"* ]]; then + # Expand glob pattern + for file in $artifact; do + if [ -e "$file" ]; then + log "Copying artifact: $file" + cp "$file" "$artifacts_dir/" + fi + done + else + # Single file + if [ -e "$artifact" ]; then + log "Copying artifact: $artifact" + cp "$artifact" "$artifacts_dir/" + fi + fi + done + + log "Artifacts collected in: $artifacts_dir" + ls -la "$artifacts_dir" + fi +} + +# Function to cleanup build environment +cleanup_environment() { + log "Cleaning up build environment" + + # Clean package cache + apt-get clean + apt-get autoremove -y + + # Clean build artifacts + rm -rf /tmp/build-* + + # Clean source packages + rm -f ../*.deb ../*.dsc ../*.tar.gz ../*.buildinfo ../*.changes +} + +# Main execution +main() { + local module_config="$1" + + log "Starting deb-mock module execution" + + # Setup build environment + if echo "$module_config" | jq -e '.environment' >/dev/null 2>&1; then + log "Processing environment configuration" + setup_environment "$(echo "$module_config" | jq -c '.environment')" + else + log "Using default environment: bookworm-amd64" + setup_environment '{"environment": "bookworm-amd64"}' + fi + + # Install build dependencies + if echo "$module_config" | jq -e '.packages' >/dev/null 2>&1; then + log "Processing packages configuration" + install_dependencies "$(echo "$module_config" | jq -c '.packages')" + fi + + # Configure repositories + if echo "$module_config" | jq -e '.repositories' >/dev/null 2>&1; then + log "Processing repositories configuration" + configure_repositories "$(echo "$module_config" | jq -c '.repositories')" + fi + + # Execute build script + if echo "$module_config" | jq -e '.build-script' >/dev/null 2>&1; then + log "Processing build script configuration" + execute_build "$(echo "$module_config" | jq -r '.build-script')" + fi + + # Collect artifacts + if echo "$module_config" | jq -e '.artifacts' >/dev/null 2>&1; then + log "Processing artifacts configuration" + collect_artifacts "$(echo "$module_config" | jq -c '.artifacts')" + fi + + # Cleanup + cleanup_environment + + log "deb-mock module execution completed successfully" +} + +# Execute main function with the module configuration +main "$1" diff --git a/modules/deb-mock/module.yml b/modules/deb-mock/module.yml new file mode 100644 index 0000000..8200579 --- /dev/null +++ b/modules/deb-mock/module.yml @@ -0,0 +1,20 @@ +name: deb-mock +shortdesc: The deb-mock module offers Debian build environment management using deb-mock. +example: | + type: deb-mock + environment: bookworm-amd64 + packages: + - build-essential + - devscripts + - debhelper + repositories: + - deb http://deb.debian.org/debian bookworm main + - deb http://deb.debian.org/debian bookworm-backports main + build-script: | + #!/bin/bash + set -e + dpkg-buildpackage -b -us -uc + artifacts: + - ../*.deb + - ../*.dsc + - ../*.tar.gz diff --git a/todo.txt b/todo.txt new file mode 100644 index 0000000..f187722 --- /dev/null +++ b/todo.txt @@ -0,0 +1,99 @@ +# Blue-Build Modules Debian Support Tasks + +## Overview +This file contains tasks specific to the blue-build-modules project for adding Debian support. + +## Current Status +✅ **COMPLETED**: Created apt module (replaces dnf) +✅ **COMPLETED**: Created apt-ostree module (replaces rpm-ostree) +✅ **COMPLETED**: Created deb-mock module (replaces mock) +✅ **COMPLETED**: Updated modules.json with new modules +✅ **COMPLETED**: Created module compatibility layer +✅ **COMPLETED**: Created package converter +✅ **COMPLETED**: Created example recipes +✅ **COMPLETED**: Created comprehensive documentation + +## Remaining Tasks + +### Module Testing (HIGH PRIORITY) ( Untested ) +- [ ] Test apt module with basic package installation +- [ ] Test apt module with repository management +- [ ] Test apt module with package removal +- [ ] Test apt module with task installation +- [ ] Test apt module with package replacement +- [ ] Test apt module with PPA repositories +- [ ] Test apt module with backports +- [ ] Test apt module with .deb package files + +- [ ] Test apt-ostree module with package layering +- [ ] Test apt-ostree module with package replacement +- [ ] Test apt-ostree module with repository management +- [ ] Test apt-ostree module fallback to regular apt + +- [ ] Test deb-mock module with build environment setup +- [ ] Test deb-mock module with package building +- [ ] Test deb-mock module with artifact collection +- [ ] Test deb-mock module with different architectures + +### Integration Testing ( Untested ) +- [ ] Test module loading from modules.json +- [ ] Test module execution in build pipeline +- [ ] Test compatibility layer with existing tools +- [ ] Test package format conversions +- [ ] Test repository mappings (COPR → PPA, RPM Fusion → Backports) + +### Build and Deployment ( Untested ) +- [ ] Build Docker images for all Debian modules +- [ ] Test module containers individually +- [ ] Push module images to registry +- [ ] Sign module images with cosign +- [ ] Update build scripts if needed + +### Documentation and Examples +- [ ] Test all example recipes +- [ ] Create additional specialized examples +- [ ] Add troubleshooting guides +- [ ] Create migration tutorials +- [ ] Add performance benchmarks + +### Quality Assurance +- [ ] Code review for all new modules +- [ ] Security review of module scripts +- [ ] Performance testing +- [ ] Error handling validation +- [ ] Logging and debugging improvements + +## Module-Specific Tasks + +### apt Module +- [ ] Validate PPA repository handling +- [ ] Test backports repository integration +- [ ] Verify task-based package groups +- [ ] Test repository cleanup functionality +- [ ] Validate GPG key management + +### apt-ostree Module +- [ ] Test ostree integration +- [ ] Validate package layering +- [ ] Test fallback mechanisms +- [ ] Verify ostree commit creation +- [ ] Test with different ostree versions + +### deb-mock Module +- [ ] Test pbuilder integration +- [ ] Test sbuild integration +- [ ] Validate build environment creation +- [ ] Test artifact collection +- [ ] Verify build script execution + +## Dependencies +- apt-ostree tool must be fully functional +- deb-mock tool must be compatible +- Debian base images must be available +- Docker build environment must be ready + +## Notes +- All core modules are implemented and ready for testing +- Focus should be on testing and validation +- Modules maintain 1:1 compatibility with Fedora equivalents +- File structure preserved for user familiarity