apt-ostree/docs/.old/apt-ostree-daemon-plan/architecture/tree-composition.md

🏗️ apt-ostree Tree Composition Architecture

📋 Overview

This document outlines the tree composition architecture for apt-ostree, based on analysis of how rpm-ostree implements tree building from packages. Tree composition is the process of creating custom OSTree trees by installing packages and committing the result to an OSTree repository.

🏗️ Architecture Overview

Component Separation

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   CLI Client    │    │   Rust Core     │    │   Rust Daemon   │
│   (apt-ostree)  │◄──►│   (DBus)        │◄──►│   (aptostreed)  │
│                 │    │                 │    │                 │
│ • compose       │    │ • Client Logic  │    │ • Tree Building │
│ • tree          │    │ • DBus Client   │    │ • Package       │
│ • image         │    │ • Progress      │    │ • OSTree Ops    │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Responsibility Distribution

CLI Client (apt-ostree)

• Command parsing for compose subcommands
• User interface and progress display
• DBus communication with daemon
• File handling for treefiles and inputs

Daemon (apt-ostreed)

• Tree building from packages
• Package installation and dependency resolution
• OSTree commit creation
• Build environment management

🔍 rpm-ostree Implementation Analysis

CLI Commands Structure

Based on rpmostree-builtin-compose.cxx, rpm-ostree provides these compose subcommands:

static RpmOstreeCommand compose_subcommands[] = {
  { "tree", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
    "Process a \"treefile\"; install packages and commit the result to an OSTree repository",
    rpmostree_compose_builtin_tree },
  { "install",
    (RpmOstreeBuiltinFlags)(RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD
                            | RPM_OSTREE_BUILTIN_FLAG_REQUIRES_ROOT),
    "Install packages into a target path", rpmostree_compose_builtin_install },
  { "postprocess",
    (RpmOstreeBuiltinFlags)(RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD
                            | RPM_OSTREE_BUILTIN_FLAG_REQUIRES_ROOT),
    "Perform final postprocessing on an installation root", rpmostree_compose_builtin_postprocess },
  { "commit",
    (RpmOstreeBuiltinFlags)(RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD
                            | RPM_OSTREE_BUILTIN_FLAG_REQUIRES_ROOT),
    "Commit a target path to an OSTree repository", rpmostree_compose_builtin_commit },
  { "extensions", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
    "Download RPM packages guaranteed to depsolve with a base OSTree",
    rpmostree_compose_builtin_extensions },
  { "container-encapsulate", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
    "Generate a reproducible \"chunked\" container image (using RPM data) from an OSTree commit",
    rpmostree_compose_builtin_container_encapsulate },
  { "image", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
    "Generate a reproducible \"chunked\" container image (using RPM data) from a treefile",
    rpmostree_compose_builtin_image },
  { "rootfs", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD, "Generate a root filesystem tree from a treefile",
    rpmostree_compose_builtin_rootfs },
  { "build-chunked-oci", RPM_OSTREE_BUILTIN_FLAG_LOCAL_CMD,
    "Generate a \"chunked\" OCI archive from an input rootfs",
    rpmostree_compose_builtin_build_chunked_oci },
  { NULL, (RpmOstreeBuiltinFlags)0, NULL, NULL }
};

Key Insights from rpm-ostree

1. Local Commands: Most compose commands are LOCAL_CMD (don't require daemon)
2. Root Requirements: Package installation requires root privileges
3. Rust Integration: Many commands delegate to Rust implementation
4. Treefile Processing: Uses declarative treefile format for composition

🚀 apt-ostree Implementation Strategy

1. CLI Command Structure

// src/main.rs - Compose command handling
async fn compose_commands(args: &[String]) -> AptOstreeResult<()> {
    if args.is_empty() {
        show_compose_help();
        return Ok(());
    }
    
    let subcommand = &args[0];
    match subcommand.as_str() {
        "tree" => compose_tree(&args[1..]).await?,
        "install" => compose_install(&args[1..]).await?,
        "postprocess" => compose_postprocess(&args[1..]).await?,
        "commit" => compose_commit(&args[1..]).await?,
        "extensions" => compose_extensions(&args[1..]).await?,
        "container-encapsulate" => compose_container_encapsulate(&args[1..]).await?,
        "image" => compose_image(&args[1..]).await?,
        "rootfs" => compose_rootfs(&args[1..]).await?,
        "build-chunked-oci" => compose_build_chunked_oci(&args[1..]).await?,
        _ => {
            println!("❌ Unknown compose subcommand: {}", subcommand);
            show_compose_help();
        }
    }
    Ok(())
}

2. Tree Composition Workflow

Tree Command Implementation

// src/compose/tree.rs
pub struct TreeComposer {
    ostree_repo: Arc<RwLock<Repo>>,
    apt_manager: Arc<AptManager>,
    build_root: PathBuf,
}

impl TreeComposer {
    pub async fn compose_tree(&self, treefile_path: &Path) -> Result<String, Error> {
        // 1. Parse treefile (YAML/JSON configuration)
        let treefile = self.parse_treefile(treefile_path).await?;
        
        // 2. Set up build environment
        self.setup_build_environment(&treefile).await?;
        
        // 3. Install base packages
        self.install_base_packages(&treefile.base_packages).await?;
        
        // 4. Install additional packages
        self.install_additional_packages(&treefile.packages).await?;
        
        // 5. Apply customizations
        self.apply_customizations(&treefile.customizations).await?;
        
        // 6. Post-process installation
        self.post_process_installation().await?;
        
        // 7. Commit to OSTree repository
        let commit_hash = self.commit_tree(&treefile.commit_message).await?;
        
        Ok(commit_hash)
    }
    
    async fn parse_treefile(&self, path: &Path) -> Result<Treefile, Error> {
        let content = tokio::fs::read_to_string(path).await?;
        let treefile: Treefile = serde_yaml::from_str(&content)?;
        Ok(treefile)
    }
    
    async fn setup_build_environment(&self, treefile: &Treefile) -> Result<(), Error> {
        // Create build root directory
        tokio::fs::create_dir_all(&self.build_root).await?;
        
        // Set up package sources
        self.setup_package_sources(&treefile.repositories).await?;
        
        // Initialize APT cache
        self.apt_manager.update_cache().await?;
        
        Ok(())
    }
    
    async fn install_base_packages(&self, packages: &[String]) -> Result<(), Error> {
        for package in packages {
            self.apt_manager.install_package(package).await?;
        }
        Ok(())
    }
    
    async fn install_additional_packages(&self, packages: &[String]) -> Result<(), Error> {
        // Resolve dependencies
        let all_packages = self.apt_manager.resolve_dependencies(packages).await?;
        
        // Install packages
        for package in all_packages {
            self.apt_manager.install_package(&package).await?;
        }
        
        Ok(())
    }
    
    async fn apply_customizations(&self, customizations: &Customizations) -> Result<(), Error> {
        // Apply file modifications
        for file_mod in &customizations.files {
            self.apply_file_modification(file_mod).await?;
        }
        
        // Apply package overrides
        for override_pkg in &customizations.package_overrides {
            self.apply_package_override(override_pkg).await?;
        }
        
        // Apply system modifications
        for sys_mod in &customizations.system_modifications {
            self.apply_system_modification(sys_mod).await?;
        }
        
        Ok(())
    }
    
    async fn post_process_installation(&self) -> Result<(), Error> {
        // Run package post-installation scripts
        self.run_post_install_scripts().await?;
        
        // Update package database
        self.update_package_database().await?;
        
        // Clean up temporary files
        self.cleanup_build_artifacts().await?;
        
        Ok(())
    }
    
    async fn commit_tree(&self, message: &str) -> Result<String, Error> {
        // Create OSTree commit from build root
        let commit_hash = self.ostree_repo
            .write()
            .await
            .commit_tree(
                &self.build_root,
                message,
                None, // No parent commit for new tree
            )
            .await?;
        
        Ok(commit_hash)
    }
}

3. Treefile Format

# Example treefile for apt-ostree
apiVersion: "apt-ostree/v1"
kind: "Treefile"

metadata:
  name: "debian-silverblue"
  version: "13.0"
  description: "Custom Debian Silverblue tree"

base:
  ostree_ref: "debian/13/x86_64/silverblue"
  packages:
    - "systemd"
    - "bash"
    - "coreutils"

packages:
  - "vim"
  - "git"
  - "curl"
  - "wget"

repositories:
  - name: "debian"
    url: "http://deb.debian.org/debian"
    distribution: "trixie"
    components: ["main", "contrib", "non-free"]
  - name: "debian-security"
    url: "http://security.debian.org/debian-security"
    distribution: "trixie-security"
    components: ["main", "contrib", "non-free"]

customizations:
  files:
    - path: "/etc/hostname"
      content: "debian-silverblue"
      mode: "0644"
    - path: "/etc/motd"
      content: "Welcome to Debian Silverblue!"
      mode: "0644"
  
  package_overrides:
    - name: "vim"
      version: "2:9.0.1378-1"
      action: "replace"
  
  system_modifications:
    - type: "kernel_args"
      action: "append"
      value: "console=ttyS0,115200"
    - type: "initramfs"
      action: "regenerate"
      args: ["--add-drivers", "virtio_console"]

commit:
  message: "Custom Debian Silverblue tree with development tools"
  ref: "debian/13/x86_64/silverblue-custom"

4. Package Installation in Tree Composition

APT Integration for Tree Building

// src/compose/apt_integration.rs
pub struct AptTreeIntegration {
    apt_manager: Arc<AptManager>,
    build_root: PathBuf,
}

impl AptTreeIntegration {
    pub async fn install_packages_for_tree(
        &self,
        packages: &[String],
        build_root: &Path,
    ) -> Result<(), Error> {
        // 1. Set up APT configuration for build root
        self.setup_apt_config(build_root).await?;
        
        // 2. Resolve package dependencies
        let all_packages = self.apt_manager.resolve_dependencies(packages).await?;
        
        // 3. Download packages
        let package_paths = self.apt_manager.download_packages(&all_packages).await?;
        
        // 4. Extract packages to build root
        for (package, path) in all_packages.iter().zip(package_paths.iter()) {
            self.extract_package_to_build_root(package, path, build_root).await?;
        }
        
        // 5. Execute package scripts
        self.execute_package_scripts(&all_packages, build_root).await?;
        
        // 6. Update package database
        self.update_package_database(build_root).await?;
        
        Ok(())
    }
    
    async fn setup_apt_config(&self, build_root: &Path) -> Result<(), Error> {
        // Create APT configuration directory
        let apt_dir = build_root.join("etc/apt");
        tokio::fs::create_dir_all(&apt_dir).await?;
        
        // Copy APT sources
        let sources_path = apt_dir.join("sources.list");
        let sources_content = self.generate_sources_list().await?;
        tokio::fs::write(sources_path, sources_content).await?;
        
        // Set up APT preferences
        let preferences_path = apt_dir.join("preferences");
        let preferences_content = self.generate_preferences().await?;
        tokio::fs::write(preferences_path, preferences_content).await?;
        
        Ok(())
    }
    
    async fn extract_package_to_build_root(
        &self,
        package: &str,
        package_path: &Path,
        build_root: &Path,
    ) -> Result<(), Error> {
        // Extract DEB package contents
        let package_contents = self.extract_deb_package(package_path).await?;
        
        // Apply files to build root
        for (file_path, file_content) in package_contents.files {
            let full_path = build_root.join(&file_path);
            
            // Create parent directories
            if let Some(parent) = full_path.parent() {
                tokio::fs::create_dir_all(parent).await?;
            }
            
            // Write file content
            tokio::fs::write(&full_path, file_content).await?;
        }
        
        // Store package scripts
        if let Some(scripts) = package_contents.scripts {
            self.store_package_scripts(package, scripts, build_root).await?;
        }
        
        Ok(())
    }
}

5. Container Image Generation

OCI Image Creation from Trees

// src/compose/container.rs
pub struct ContainerGenerator {
    ostree_repo: Arc<RwLock<Repo>>,
    build_root: PathBuf,
}

impl ContainerGenerator {
    pub async fn generate_container_image(
        &self,
        tree_ref: &str,
        image_ref: &str,
        options: &ContainerOptions,
    ) -> Result<String, Error> {
        // 1. Extract tree to temporary directory
        let tree_path = self.extract_tree(tree_ref).await?;
        
        // 2. Generate container metadata
        let metadata = self.generate_container_metadata(tree_ref, options).await?;
        
        // 3. Create container layers
        let layers = self.create_container_layers(&tree_path, options).await?;
        
        // 4. Build OCI image
        let image_path = self.build_oci_image(metadata, layers).await?;
        
        // 5. Push to registry (if specified)
        if let Some(registry) = &options.registry {
            self.push_to_registry(&image_path, registry).await?;
        }
        
        Ok(image_path)
    }
    
    async fn extract_tree(&self, tree_ref: &str) -> Result<PathBuf, Error> {
        // Extract OSTree commit to temporary directory
        let temp_dir = tempfile::tempdir()?;
        let tree_path = temp_dir.path().to_path_buf();
        
        self.ostree_repo
            .write()
            .await
            .checkout(tree_ref, &tree_path)
            .await?;
        
        Ok(tree_path)
    }
    
    async fn generate_container_metadata(
        &self,
        tree_ref: &str,
        options: &ContainerOptions,
    ) -> Result<ContainerMetadata, Error> {
        // Generate container configuration
        let config = ContainerConfig {
            architecture: options.architecture.clone(),
            os: "linux".to_string(),
            created: chrono::Utc::now(),
            author: options.author.clone(),
            labels: options.labels.clone(),
            entrypoint: options.entrypoint.clone(),
            cmd: options.cmd.clone(),
            working_dir: options.working_dir.clone(),
            env: options.env.clone(),
            volumes: options.volumes.clone(),
        };
        
        Ok(ContainerMetadata {
            config,
            layers: Vec::new(),
            history: Vec::new(),
        })
    }
}

🔐 Security and Privileges

1. Privilege Requirements

// Security checks for tree composition
impl TreeComposer {
    pub async fn check_compose_privileges(&self, treefile: &Treefile) -> Result<(), SecurityError> {
        // Check if user has permission to compose trees
        if !self.security_manager.can_compose_trees().await? {
            return Err(SecurityError::InsufficientPrivileges(
                "Tree composition requires elevated privileges".to_string(),
            ));
        }
        
        // Check repository access permissions
        if !self.security_manager.can_access_repository(&treefile.base.ostree_ref).await? {
            return Err(SecurityError::RepositoryAccessDenied(
                treefile.base.ostree_ref.clone(),
            ));
        }
        
        // Check package source permissions
        for repo in &treefile.repositories {
            if !self.security_manager.can_access_package_source(repo).await? {
                return Err(SecurityError::PackageSourceAccessDenied(repo.url.clone()));
            }
        }
        
        Ok(())
    }
}

2. Sandboxed Build Environment

// Sandboxed package installation
impl AptTreeIntegration {
    pub async fn install_packages_sandboxed(
        &self,
        packages: &[String],
        build_root: &Path,
    ) -> Result<(), Error> {
        // Create sandboxed environment
        let mut sandbox = self.create_sandbox().await?;
        
        // Mount build root
        sandbox.bind_mount(build_root, "/build")?;
        
        // Mount package cache
        sandbox.bind_mount("/var/cache/apt", "/var/cache/apt")?;
        
        // Execute package installation in sandbox
        let output = sandbox.exec(
            &["apt-get", "install", "-y"],
            &packages,
        ).await?;
        
        if !output.status.success() {
            return Err(Error::PackageInstallationFailed(output.stderr));
        }
        
        Ok(())
    }
}

📊 Performance Optimization

1. Parallel Package Processing

// Parallel package installation
impl AptTreeIntegration {
    pub async fn install_packages_parallel(
        &self,
        packages: &[String],
        build_root: &Path,
    ) -> Result<(), Error> {
        let mut tasks = JoinSet::new();
        
        // Spawn parallel download tasks
        for package in packages {
            let package = package.clone();
            let apt_manager = self.apt_manager.clone();
            
            tasks.spawn(async move {
                apt_manager.download_package(&package).await
            });
        }
        
        // Collect downloaded packages
        let mut downloaded_packages = Vec::new();
        while let Some(result) = tasks.join_next().await {
            downloaded_packages.push(result??);
        }
        
        // Install packages in dependency order
        let sorted_packages = self.sort_packages_by_dependencies(&downloaded_packages).await?;
        for package in sorted_packages {
            self.install_package(&package, build_root).await?;
        }
        
        Ok(())
    }
}

2. Caching Strategy

// Package and metadata caching
impl TreeComposer {
    pub async fn setup_caching(&self) -> Result<(), Error> {
        // Set up package cache
        let package_cache = self.build_root.join("var/cache/apt");
        tokio::fs::create_dir_all(&package_cache).await?;
        
        // Set up metadata cache
        let metadata_cache = self.build_root.join("var/lib/apt");
        tokio::fs::create_dir_all(&metadata_cache).await?;
        
        // Copy existing caches if available
        if let Ok(existing_cache) = tokio::fs::read_dir("/var/cache/apt").await {
            for entry in existing_cache {
                let entry = entry?;
                let dest = package_cache.join(entry.file_name());
                if entry.file_type().await?.is_file() {
                    tokio::fs::copy(entry.path(), dest).await?;
                }
            }
        }
        
        Ok(())
    }
}

🧪 Testing Strategy

1. Unit Tests

#[cfg(test)]
mod tests {
    use super::*;
    
    #[tokio::test]
    async fn test_treefile_parsing() {
        let treefile_content = r#"
apiVersion: "apt-ostree/v1"
kind: "Treefile"
metadata:
  name: "test-tree"
packages:
  - "vim"
  - "git"
        "#;
        
        let treefile: Treefile = serde_yaml::from_str(treefile_content).unwrap();
        assert_eq!(treefile.metadata.name, "test-tree");
        assert_eq!(treefile.packages.len(), 2);
    }
    
    #[tokio::test]
    async fn test_package_installation() {
        let composer = TreeComposer::new().await.unwrap();
        let result = composer.install_base_packages(&["test-package"]).await;
        assert!(result.is_ok());
    }
}

2. Integration Tests

#[tokio::test]
async fn test_full_tree_composition() {
    // Create test treefile
    let treefile = create_test_treefile().await?;
    
    // Set up test environment
    let composer = TreeComposer::new().await?;
    
    // Compose tree
    let commit_hash = composer.compose_tree(&treefile).await?;
    
    // Verify result
    assert!(!commit_hash.is_empty());
    
    // Verify packages are installed
    let installed_packages = composer.list_installed_packages().await?;
    assert!(installed_packages.contains(&"vim".to_string()));
}

🚀 Future Enhancements

1. Advanced Treefile Features

• Conditional packages based on architecture or features
• Package variants and alternatives
• Custom package sources and repositories
• Build-time hooks and scripts

2. Performance Improvements

• Incremental builds using layer caching
• Parallel package processing with dependency analysis
• Distributed builds across multiple machines
• Build artifact caching and reuse

3. Integration Features

• CI/CD integration for automated tree building
• Version control integration with Git
• Build monitoring and progress tracking
• Artifact signing and verification

This architecture provides a solid foundation for implementing production-ready tree composition in apt-ostree, maintaining compatibility with the rpm-ostree ecosystem while leveraging the strengths of the Debian/Ubuntu package management system.