bootc-docs/internals/bootc-internals-technical-guide.md

# bootc internals - Technical Guide

## Overview

`bootc internals` is a hidden command that provides internal-only operations for system maintenance, debugging, and integration with other tools. These commands are not intended for regular users but are essential for system administrators, developers, and automated tooling.

## Purpose

The internals commands serve several critical functions:

1. **System Maintenance**: Low-level system operations and cleanup
2. **Integration**: Proxy frontends for other tools and services
3. **Debugging**: Testing and diagnostic capabilities
4. **Development**: Internal tooling and schema generation
5. **Systemd Integration**: Generator and service management

## Command Structure

```rust
pub(crate) enum InternalsOpts {
    SystemdGenerator { normal_dir, early_dir, late_dir },
    FixupEtcFstab,
    PrintJsonSchema { of: SchemaType },
    Fsverity(FsverityOpts),
    Fsck,
    Cleanup,
    Relabel { as_path, path },
    OstreeExt { args },
    Cfs { args },
    OstreeContainer { args },
    TestComposefs,
    LoopbackCleanupHelper { device },
    AllocateCleanupLoopback { file_path },
    BootcInstallCompletion { sysroot, stateroot },
    Reboot,
    PublishRhsmFacts,  // Feature-gated
    DumpCliJson,       // Feature-gated
    DirDiff { pristine_etc, current_etc, new_etc, perform_merge },
}
```

## Core Commands

### 1. Systemd Generator

**Purpose**: Generate systemd units for bootc integration
**Usage**: `bootc internals systemd-generator <normal_dir> [early_dir] [late_dir]`

```rust
InternalsOpts::SystemdGenerator {
    normal_dir: Utf8PathBuf,
    early_dir: Option<Utf8PathBuf>,
    late_dir: Option<Utf8PathBuf>,
} => {
    let unit_dir = &Dir::open_ambient_dir(normal_dir, cap_std::ambient_authority())?;
    crate::generator::generator(root, unit_dir)
}
```

**Functionality**:
- Generates systemd units for bootc services
- Handles fstab editing for composefs systems
- Creates bootc-fstab-edit.service when needed
- Integrates with systemd's generator system

**Implementation**:
```rust
pub(crate) fn fstab_generator_impl(root: &Dir, unit_dir: &Dir) -> Result<bool> {
    // Check if system is ostree-booted
    if !is_ostree_booted_in(root)? {
        return Ok(false);
    }

    // Check /etc/fstab for anaconda stamp
    if let Some(fd) = root.open_optional("etc/fstab")? {
        let mut from_anaconda = false;
        for line in fd.lines() {
            let line = line?;
            if line.contains(BOOTC_EDITED_STAMP) {
                return Ok(false); // Already processed
            }
            if line.contains(FSTAB_ANACONDA_STAMP) {
                from_anaconda = true;
            }
        }
        
        if from_anaconda {
            generate_fstab_editor(unit_dir)?;
            return Ok(true);
        }
    }
    Ok(false)
}
```

### 2. Filesystem Consistency Check (Fsck)

**Purpose**: Perform consistency checking on bootc systems
**Usage**: `bootc internals fsck`

```rust
InternalsOpts::Fsck => {
    let sysroot = &get_storage().await?;
    crate::fsck::fsck(&sysroot, std::io::stdout().lock()).await?;
    Ok(())
}
```

**Functionality**:
- Validates OSTree repository integrity
- Checks composefs repository consistency
- Verifies deployment states
- Validates system configuration

**Implementation**:
```rust
pub async fn fsck(sysroot: &Storage, mut out: impl Write) -> Result<()> {
    let ostree = sysroot.get_ostree()?;
    let repo = ostree.repo();
    
    // Run all registered fsck checks
    for check in FSCK_CHECKS {
        match check(sysroot).await {
            Ok(Ok(())) => writeln!(out, "✓ {}", check.name())?,
            Ok(Err(e)) => writeln!(out, "✗ {}: {}", check.name(), e)?,
            Err(e) => writeln!(out, "✗ {}: {}", check.name(), e)?,
        }
    }
    Ok(())
}
```

### 3. Cleanup Operations

**Purpose**: Perform cleanup actions on bootc systems
**Usage**: `bootc internals cleanup`

```rust
InternalsOpts::Cleanup => {
    let sysroot = get_storage().await?;
    crate::deploy::cleanup(&sysroot).await
}
```

**Functionality**:
- Remove old deployments
- Clean up unused container images
- Garbage collect OSTree repository
- Remove temporary files

### 4. Filesystem Verity (Fsverity)

**Purpose**: Manage fsverity integrity verification
**Usage**: `bootc internals fsverity <subcommand>`

```rust
pub(crate) enum FsverityOpts {
    Measure { path: Utf8PathBuf },
    Enable { path: Utf8PathBuf },
}

// Implementation
InternalsOpts::Fsverity(args) => match args {
    FsverityOpts::Measure { path } => {
        let fd = std::fs::File::open(&path)?;
        let digest: fsverity::Sha256HashValue = fsverity::measure_verity(&fd)?;
        println!("{}", digest.to_hex());
        Ok(())
    }
    FsverityOpts::Enable { path } => {
        let fd = std::fs::File::open(&path)?;
        fsverity::enable_verity_raw::<fsverity::Sha256HashValue>(&fd)?;
        Ok(())
    }
}
```

**Subcommands**:
- **`measure`**: Calculate fsverity digest of a file
- **`enable`**: Enable fsverity verification on a file

### 5. Proxy Commands

#### OstreeExt Proxy
**Purpose**: Proxy frontend for `ostree-ext` CLI
**Usage**: `bootc internals ostree-ext [args...]`

```rust
InternalsOpts::OstreeExt { args } => {
    ostree_ext::cli::run_from_iter(["ostree-ext".into()].into_iter().chain(args)).await
}
```

#### Cfs Proxy
**Purpose**: Proxy frontend for `cfsctl` CLI
**Usage**: `bootc internals cfs [args...]`

```rust
InternalsOpts::Cfs { args } => {
    crate::cfsctl::run_from_iter(args.iter()).await
}
```

#### OstreeContainer Proxy
**Purpose**: Proxy frontend for legacy `ostree container` CLI
**Usage**: `bootc internals ostree-container [args...]`

```rust
InternalsOpts::OstreeContainer { args } => {
    ostree_ext::cli::run_from_iter(
        ["ostree-ext".into(), "container".into()]
            .into_iter()
            .chain(args),
    ).await
}
```

### 6. System Maintenance

#### Fixup /etc/fstab
**Purpose**: Fix /etc/fstab for composefs systems
**Usage**: `bootc internals fixup-etc-fstab`

```rust
InternalsOpts::FixupEtcFstab => {
    crate::deploy::fixup_etc_fstab(&root)
}
```

#### Relabel
**Purpose**: SELinux relabeling
**Usage**: `bootc internals relabel --as-path <path> <target_path>`

```rust
InternalsOpts::Relabel { as_path, path } => {
    let root = &Dir::open_ambient_dir("/", cap_std::ambient_authority())?;
    let path = path.strip_prefix("/")?;
    let sepolicy = &ostree::SePolicy::new(&gio::File::for_path("/"), gio::Cancellable::NONE)?;
    crate::lsm::relabel_recurse(root, path, as_path.as_deref(), sepolicy)?;
    Ok(())
}
```

### 7. Testing and Development

#### Test Composefs
**Purpose**: Ensure composefs repository is initialized
**Usage**: `bootc internals test-composefs`

```rust
InternalsOpts::TestComposefs => {
    let storage = get_storage().await?;
    let cfs = storage.get_ensure_composefs()?;
    let testdata = b"some test data";
    let testdata_digest = openssl::sha::sha256(testdata);
    let mut w = SplitStreamWriter::new(&cfs, None, Some(testdata_digest));
    w.write_inline(testdata);
    let object = cfs.write_stream(w, Some("testobject"))?.to_hex();
    // Verify expected digest
    assert_eq!(object, "expected_digest");
    Ok(())
}
```

#### Loopback Device Management
**Purpose**: Manage loopback devices for testing
**Usage**: 
- `bootc internals loopback-cleanup-helper --device <device>`
- `bootc internals allocate-cleanup-loopback --file-path <path>`

```rust
InternalsOpts::LoopbackCleanupHelper { device } => {
    crate::blockdev::run_loopback_cleanup_helper(&device).await
}

InternalsOpts::AllocateCleanupLoopback { file_path: _ } => {
    let temp_file = tempfile::NamedTempFile::new()?;
    let temp_path = temp_file.path();
    
    let loopback = crate::blockdev::LoopbackDevice::new(temp_path)?;
    println!("Created loopback device: {}", loopback.path());
    
    // Cleanup
    drop(loopback);
    println!("Successfully closed loopback device");
    Ok(())
}
```

### 8. Installation Completion

**Purpose**: Complete installation from ostree-ext
**Usage**: `bootc internals bootc-install-completion --sysroot <path> --stateroot <name>`

```rust
InternalsOpts::BootcInstallCompletion { sysroot, stateroot } => {
    let rootfs = &Dir::open_ambient_dir("/", cap_std::ambient_authority())?;
    crate::install::completion::run_from_ostree(rootfs, &sysroot, &stateroot).await
}
```

### 9. System Operations

#### Reboot
**Purpose**: Initiate reboot (for testing)
**Usage**: `bootc internals reboot`

```rust
InternalsOpts::Reboot => {
    crate::reboot::reboot()
}
```

#### Print JSON Schema
**Purpose**: Generate JSON schemas (for development)
**Usage**: `bootc internals print-json-schema --of <type>`

```rust
InternalsOpts::PrintJsonSchema { of } => {
    let schema = match of {
        SchemaType::Host => schema_for!(crate::spec::Host),
        SchemaType::Progress => schema_for!(crate::progress_jsonl::Event),
    };
    let mut stdout = std::io::stdout().lock();
    serde_json::to_writer_pretty(&mut stdout, &schema)?;
    Ok(())
}
```

### 10. Feature-Gated Commands

#### RHSM Facts Publishing
**Purpose**: Publish subscription-manager facts
**Usage**: `bootc internals publish-rhsm-facts` (requires `rhsm` feature)

```rust
#[cfg(feature = "rhsm")]
InternalsOpts::PublishRhsmFacts => {
    crate::rhsm::publish_facts(&root).await
}
```

#### CLI JSON Dump
**Purpose**: Dump CLI structure as JSON
**Usage**: `bootc internals dump-cli-json` (requires `docgen` feature)

```rust
#[cfg(feature = "docgen")]
InternalsOpts::DumpCliJson => {
    use clap::CommandFactory;
    let cmd = Opt::command();
    let json = crate::cli_json::dump_cli_json(&cmd)?;
    println!("{}", json);
    Ok(())
}
```

### 11. Directory Diff Testing

**Purpose**: Test etc-diff/etc-merge functionality
**Usage**: `bootc internals dir-diff <pristine_etc> <current_etc> <new_etc> [--perform-merge]`

```rust
InternalsOpts::DirDiff {
    pristine_etc,
    current_etc,
    new_etc,
    perform_merge,
} => {
    // Test 3-way merge functionality
    let pristine_dir = Dir::open_ambient_dir(&pristine_etc, cap_std::ambient_authority())?;
    let current_dir = Dir::open_ambient_dir(&current_etc, cap_std::ambient_authority())?;
    let new_dir = Dir::open_ambient_dir(&new_etc, cap_std::ambient_authority())?;
    
    let (pristine_files, current_files, new_files) =
        traverse_etc(&pristine_dir, &current_dir, &new_dir)?;
    
    if perform_merge {
        let diff = compute_diff(&pristine_files, &current_files)?;
        merge(&current_dir, &current_files, &new_dir, &new_files, diff)?;
    }
    Ok(())
}
```

## Architecture

### 1. Command Routing

The internals commands are routed through the main CLI dispatcher:

```rust
match opt {
    Opt::Internals(opts) => match opts {
        InternalsOpts::SystemdGenerator { ... } => { ... }
        InternalsOpts::Fsck => { ... }
        InternalsOpts::Cleanup => { ... }
        // ... other commands
    }
}
```

### 2. Error Handling

All internals commands use consistent error handling:

```rust
#[context("Operation name")]
pub(crate) fn operation() -> Result<()> {
    // Implementation with proper error context
}
```

### 3. Integration Points

- **Systemd**: Generator integration for service management
- **OSTree**: Direct OSTree operations and repository management
- **Composefs**: Composefs repository operations
- **SELinux**: Security labeling and policy enforcement
- **Filesystem**: Low-level filesystem operations

## Security Considerations

### 1. Privilege Requirements

Most internals commands require root privileges:

```rust
pub(crate) fn require_root(is_container: bool) -> Result<()> {
    ensure!(
        rustix::process::getuid().is_root(),
        if is_container {
            "The user inside the container from which you are running this command must be root"
        } else {
            "This command must be executed as the root user"
        }
    );
    Ok(())
}
```

### 2. Hidden Commands

All internals commands are hidden from help output:

```rust
#[clap(subcommand)]
#[clap(hide = true)]
Internals(InternalsOpts),
```

### 3. Internal Use Only

Commands are marked as internal-only and not intended for regular users.

## Performance Characteristics

### 1. Systemd Generator

- **Fast**: Minimal overhead during boot
- **Conditional**: Only runs when needed
- **Efficient**: Direct filesystem operations

### 2. Fsck Operations

- **Comprehensive**: Checks all system components
- **Parallel**: Can run multiple checks concurrently
- **Detailed**: Provides specific error information

### 3. Cleanup Operations

- **Safe**: Only removes confirmed unused data
- **Efficient**: Batch operations where possible
- **Atomic**: Operations are atomic where possible

## Troubleshooting

### 1. Common Issues

#### Generator Not Running
```bash
# Check systemd generator status
systemctl status bootc-systemd-generator

# Check generator logs
journalctl -u bootc-systemd-generator
```

#### Fsck Failures
```bash
# Run fsck manually
bootc internals fsck

# Check specific components
bootc internals fsck --component ostree
```

#### Cleanup Issues
```bash
# Run cleanup manually
bootc internals cleanup

# Check storage usage
df -h /sysroot
```

### 2. Debug Commands

```bash
# Enable debug logging
RUST_LOG=debug bootc internals <command>

# Check system state
bootc status

# Verify system integrity
bootc internals fsck
```

## Best Practices

### 1. Usage Guidelines

- **Internal Only**: Don't use in production scripts
- **Root Required**: Always run as root
- **Testing**: Use in test environments first
- **Documentation**: Document any custom usage

### 2. Integration Patterns

- **Systemd**: Use generators for service integration
- **Monitoring**: Use fsck for health checks
- **Maintenance**: Use cleanup for storage management
- **Development**: Use test commands for validation

### 3. Error Handling

- **Logging**: Always check logs for errors
- **Recovery**: Have recovery procedures ready
- **Testing**: Test commands before production use
- **Monitoring**: Monitor system health after operations

This technical guide provides comprehensive understanding of the bootc internals system's architecture, implementation, and usage patterns.