apt-ostree/.notes/research/skopeo.md

# Skopeo Integration in apt-layer

## TLDR - Quick Reference

### Basic Commands

**Inspect a container image:**
```sh
skopeo inspect docker://ubuntu:24.04
```

**Copy image between registries:**
```sh
skopeo copy docker://ubuntu:24.04 docker://myregistry/ubuntu:24.04
```

**Copy image to local directory:**
```sh
skopeo copy docker://ubuntu:24.04 dir:/path/to/local/directory
```

**Copy local directory to registry:**
```sh
skopeo copy dir:/path/to/local/directory docker://myregistry/myimage:latest
```

**List available tags:**
```sh
skopeo list-tags docker://ubuntu
```

**Login to registry:**
```sh
skopeo login --username username myregistry.com
```

**Delete image from registry:**
```sh
skopeo delete docker://myregistry/image:tag
```

### Quick Example
```sh
# Import OCI image to apt-layer
apt-layer --oci-import ubuntu:24.04 my-base/24.04

# Export apt-layer image to OCI
apt-layer --oci-export my-gaming/24.04 myregistry/gaming:latest

# Inspect image before import
skopeo inspect docker://ubuntu:24.04
```

---

## Skopeo Commands Reference

Based on the [official skopeo documentation](https://www.mankier.com/1/skopeo), skopeo provides the following commands:

### Core Commands

| Command | Purpose | Usage Example |
|---------|---------|---------------|
| `skopeo copy` | Copy images between locations | `skopeo copy docker://src docker://dest` |
| `skopeo inspect` | Inspect image metadata | `skopeo inspect docker://ubuntu:24.04` |
| `skopeo list-tags` | List available tags | `skopeo list-tags docker://ubuntu` |
| `skopeo delete` | Delete image from registry | `skopeo delete docker://registry/image:tag` |

### Authentication Commands

| Command | Purpose | Usage Example |
|---------|---------|---------------|
| `skopeo login` | Login to registry | `skopeo login --username user registry.com` |
| `skopeo logout` | Logout from registry | `skopeo logout registry.com` |

### Signature Commands

| Command | Purpose | Usage Example |
|---------|---------|---------------|
| `skopeo standalone-sign` | Sign image without daemon | `skopeo standalone-sign --key key.pem image` |
| `skopeo standalone-verify` | Verify image signature | `skopeo standalone-verify --key key.pem image` |
| `skopeo generate-sigstore-key` | Generate Sigstore key | `skopeo generate-sigstore-key --output key.pem` |

### Utility Commands

| Command | Purpose | Usage Example |
|---------|---------|---------------|
| `skopeo manifest-digest` | Get manifest digest | `skopeo manifest-digest manifest.json` |
| `skopeo sync` | Sync images between registries | `skopeo sync --src docker --dest dir registry` |

### Transport Types

Skopeo supports various transport types:
- `docker://` - Docker registry
- `dir://` - Local directory
- `oci://` - OCI directory
- `containers-storage://` - Podman storage
- `docker-archive://` - Docker tar archive
- `oci-archive://` - OCI tar archive

---

## Overview

apt-layer uses [skopeo](https://github.com/containers/skopeo) for OCI (Open Container Initiative) container image operations, mirroring the approach used by rpm-ostree. Both rpm-ostree and apt-layer use **podman as their primary container runtime** and **skopeo specifically for OCI operations**.

**Key Role:** Skopeo serves as the specialized OCI tool in apt-layer for:
- Container image inspection and validation
- Image copying between registries and local storage
- Image format conversion (OCI ↔ ComposeFS)
- Registry authentication and signature verification

**Container Runtime:** Podman serves as the primary container runtime for:
- Running containers for package installation
- Building and managing container images
- Container lifecycle management
- Interactive development and testing

---

## Package Structure

### Debian/Ubuntu Package

**Package Name:** `skopeo`  
**Purpose:** OCI container image operations  
**Contains:**
- `/usr/bin/skopeo` - Main skopeo executable
- `/usr/share/man/man1/skopeo.1.gz` - Manual page
- `/usr/share/doc/skopeo/` - Documentation

**Dependencies:**
- `libgpgme11` - GPG Made Easy library
- `libostree-1-1` - OSTree library
- `containers-common` - Container utilities

### Installation

**Debian/Ubuntu:**
```sh
sudo apt install -y skopeo
```

**Fedora/RHEL:**
```sh
sudo dnf install -y skopeo
```

---

## Skopeo Usage in apt-layer

### 1. Tool Usage Strategy

apt-layer and rpm-ostree use a specialized approach for different types of operations:

**OCI Operations (skopeo):**
- Image inspection and validation
- Image copying between registries
- Image format conversion
- Signature verification
- Registry operations without running containers

**Container Runtime Operations (podman):**
- Running containers for package installation
- Building and managing container images
- Container lifecycle management
- Interactive development and testing

```bash
# apt-layer automatically detects and uses the appropriate tool
if command -v skopeo &> /dev/null; then
    OCI_TOOL="skopeo"
    log_info "Using skopeo for OCI operations" "apt-layer"
elif command -v podman &> /dev/null; then
    OCI_TOOL="podman"
    log_info "Using podman for OCI operations (fallback)" "apt-layer"
else
    OCI_TOOL="docker"
    log_info "Using docker for OCI operations (fallback)" "apt-layer"
fi

# Container runtime is always podman when available
if command -v podman &> /dev/null; then
    CONTAINER_RUNTIME="podman"
    log_info "Using podman as container runtime" "apt-layer"
else
    CONTAINER_RUNTIME="docker"
    log_info "Using docker as container runtime" "apt-layer"
fi
```

### 2. Image Import Operations

**Import OCI image as ComposeFS:**
```bash
# apt-layer command
apt-layer --oci-import ubuntu:24.04 my-base/24.04

# Underlying skopeo operation
skopeo copy docker://ubuntu:24.04 dir:/tmp/oci-import-12345
```

**Process:**
1. `skopeo copy` downloads the OCI image to a local directory
2. apt-layer extracts the filesystem from the OCI structure
3. apt-layer creates a ComposeFS image from the extracted filesystem
4. The ComposeFS image is stored in apt-layer's image workspace

### 3. Image Export Operations

**Export ComposeFS image to OCI:**
```bash
# apt-layer command
apt-layer --oci-export my-gaming/24.04 myregistry/gaming:latest

# Underlying skopeo operation
skopeo copy dir:/tmp/oci-export-12345 docker://myregistry/gaming:latest
```

**Process:**
1. apt-layer mounts the ComposeFS image
2. apt-layer creates an OCI directory structure from the mounted filesystem
3. `skopeo copy` uploads the OCI directory to the registry
4. The image is available in the container registry

### 4. Image Inspection and Validation

**Inspect container images:**
```bash
# Direct skopeo usage
skopeo inspect docker://ubuntu:24.04

# apt-layer integration
apt-layer --oci-info ubuntu:24.04
```

**List available tags:**
```bash
# Direct skopeo usage
skopeo list-tags docker://ubuntu

# apt-layer integration
apt-layer --oci-list-tags ubuntu
```

**Validate image before import:**
```bash
# Check if image exists and is accessible
if ! skopeo inspect "docker://$image_name" >/dev/null 2>&1; then
    log_error "Invalid OCI image: $image_name" "apt-layer"
    return 1
fi

# Check available tags
skopeo list-tags "docker://$registry/$image" | grep -q "$tag"
```

**Returns:**
- Image metadata (layers, architecture, OS)
- Labels and annotations
- Creation date and size information
- Digest and signature information
- Available tags for the image

### 5. Registry Authentication

**Authentication with registries:**
```bash
# Login to registry (handled by podman)
podman login myregistry.com

# skopeo uses the same authentication
skopeo copy docker://myregistry.com/image:tag dir:/local/path

# Both podman and skopeo share authentication configuration
# from ~/.docker/config.json or ~/.config/containers/auth.json
```

### 6. Image Signing and Verification

**Generate Sigstore key:**
```bash
# Generate signing key
skopeo generate-sigstore-key --output signing-key.pem
```

**Sign image:**
```bash
# Sign image with standalone signing
skopeo standalone-sign --key signing-key.pem docker://myregistry/image:tag
```

**Verify image signature:**
```bash
# Verify image signature
skopeo standalone-verify --key signing-key.pem docker://myregistry/image:tag
```

**apt-layer integration:**
```bash
# Sign apt-layer image before export
apt-layer --oci-sign my-gaming/24.04 signing-key.pem

# Verify imported image
apt-layer --oci-verify ubuntu:24.04 signing-key.pem
```

### 7. Advanced Operations

**Get manifest digest:**
```bash
# Get digest for verification
skopeo manifest-digest manifest.json
```

**Sync images between registries:**
```bash
# Sync all tags from one registry to another
skopeo sync --src docker --dest docker registry1.com registry2.com
```

**Delete images from registry:**
```bash
# Remove image from registry
skopeo delete docker://myregistry/image:tag
```

**apt-layer integration:**
```bash
# Sync apt-layer images to backup registry
apt-layer --oci-sync myregistry.com backup-registry.com

# Clean up old images
apt-layer --oci-cleanup myregistry.com --older-than 30d
```

---

## Skopeo vs Container Runtimes

### Skopeo (OCI Operations Only)

**Use Cases:**
- Image inspection and validation
- Image copying between registries
- Image format conversion
- Signature verification
- Registry operations without running containers

**Limitations:**
- Cannot run containers
- Cannot build images
- Limited to OCI operations

### Podman (Primary Container Runtime)

**Use Cases:**
- Running containers for package installation
- Building and managing container images
- Container lifecycle management
- Interactive development and testing
- OCI operations (when skopeo unavailable)

**Integration:**
- apt-layer uses podman as the primary container runtime (like rpm-ostree)
- skopeo handles specialized OCI operations
- Both work together in the apt-layer ecosystem

### Docker (Fallback Container Runtime)

**Use Cases:**
- Running containers when podman unavailable
- Building images when podman unavailable
- Container operations in environments without podman

**Note:** apt-layer and rpm-ostree prefer podman over docker for container operations

---

## OCI Integration Workflow

### 1. Import Workflow

```bash
# Step 1: Inspect the image
skopeo inspect docker://ubuntu:24.04

# Step 2: Copy image to local directory
skopeo copy docker://ubuntu:24.04 dir:/tmp/oci-import

# Step 3: Extract filesystem
# apt-layer extracts the root filesystem from the OCI structure

# Step 4: Create ComposeFS image
mkcomposefs /tmp/extracted-rootfs my-base/24.04 --digest-store=/path/to/objects

# Step 5: Cleanup
rm -rf /tmp/oci-import
```

### 2. Export Workflow

```bash
# Step 1: Mount ComposeFS image
mount -t composefs -o basedir=/path/to/objects my-gaming/24.04 /mnt/composefs

# Step 2: Create OCI directory structure
# apt-layer creates manifest.json, config.json, and layer files

# Step 3: Copy to registry
skopeo copy dir:/tmp/oci-export docker://myregistry/gaming:latest

# Step 4: Unmount and cleanup
umount /mnt/composefs
rm -rf /tmp/oci-export
```

---

## Integration with apt-layer Features

### 1. OSTree Atomic Operations

```bash
# Import OCI image and create OSTree commit
apt-layer ostree compose install --from-oci ubuntu:24.04

# Export OSTree deployment as OCI image
apt-layer ostree compose export my-deployment myregistry/deployment:latest
```

### 2. Container-based Package Installation

```bash
# Use OCI image as base for package installation (uses podman)
apt-layer --container ubuntu:24.04 my-dev/24.04 vscode git

# Export result back to OCI (uses skopeo)
apt-layer --oci-export my-dev/24.04 myregistry/dev:latest
```

### 3. Live System Integration

```bash
# Import OCI image for live system base
apt-layer --live-import ubuntu:24.04

# Export live system changes as OCI image
apt-layer --live-export myregistry/live-changes:latest
```

---

## Error Handling and Validation

### 1. Image Validation

```bash
# Validate image before import
if ! skopeo inspect "docker://$image_name" >/dev/null 2>&1; then
    log_error "Invalid OCI image: $image_name" "apt-layer"
    return 1
fi
```

### 2. Registry Connectivity

```bash
# Test registry connectivity
if ! skopeo inspect "docker://$registry/$image" >/dev/null 2>&1; then
    log_error "Cannot access registry: $registry" "apt-layer"
    return 1
fi
```

### 3. Authentication Errors

```bash
# Handle authentication failures
if ! skopeo copy "docker://$source" "docker://$destination"; then
    log_error "Authentication failed or insufficient permissions" "apt-layer"
    log_info "Try: podman login $registry" "apt-layer"
    log_info "Note: podman and skopeo share authentication configuration" "apt-layer"
    return 1
fi
```

---

## Configuration

### 1. Registry Configuration

**File:** `/etc/containers/registries.conf`  
**Purpose:** Configure registry mirrors, authentication, and security

```ini
[[registry]]
prefix = "docker.io"
location = "docker.io"
insecure = false
blocked = false

[[registry]]
prefix = "myregistry.com"
location = "myregistry.com"
insecure = true
```

### 2. Authentication

**File:** `~/.docker/config.json` (shared with podman/docker)  
**Purpose:** Store registry credentials

```json
{
  "auths": {
    "myregistry.com": {
      "auth": "base64-encoded-credentials"
    }
  }
}
```

### 3. Policy Configuration

**File:** `/etc/containers/policy.json`  
**Purpose:** Define signature verification policies

```json
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ]
}
```

---

## Troubleshooting

### Common Issues

**1. Authentication Errors:**
```bash
# Error: authentication required
# Solution: Login to registry (podman and skopeo share auth)
podman login myregistry.com
# or use skopeo directly
skopeo login --username username myregistry.com
```

**2. Network Connectivity:**
```bash
# Error: connection refused
# Solution: Check network and firewall
curl -I https://registry-1.docker.io/v2/
```

**3. Image Not Found:**
```bash
# Error: manifest unknown
# Solution: Verify image name and tag
skopeo list-tags docker://ubuntu
```

**4. Insufficient Permissions:**
```bash
# Error: permission denied
# Solution: Check registry permissions
skopeo inspect docker://myregistry/private-image
```

**5. Signature Verification Errors:**
```bash
# Error: signature verification failed
# Solution: Check signing key and policy
skopeo standalone-verify --key key.pem docker://image:tag
```

**6. Transport Type Errors:**
```bash
# Error: unsupported transport type
# Solution: Use correct transport prefix
skopeo copy docker://image:tag dir:/local/path
skopeo copy oci://image:tag docker://registry/image:tag
```

### Debug Mode

```bash
# Enable debug output
export SKOPEO_DEBUG=1
apt-layer --oci-import ubuntu:24.04 my-base/24.04
```

---

## Integration Notes

- **Podman-First Approach:** apt-layer uses podman as the primary container runtime (like rpm-ostree)
- **Skopeo for OCI:** skopeo handles specialized OCI operations (inspection, copying, conversion)
- **ComposeFS Integration:** Seamless conversion between OCI and ComposeFS formats
- **Registry Support:** Full support for Docker Hub, private registries, and local storage
- **Signature Verification:** Built-in support for image signatures and verification
- **Authentication:** Shared authentication between podman and skopeo for consistent experience
- **Error Handling:** Comprehensive error handling with helpful diagnostic messages

---

## References

- [Skopeo GitHub Repository](https://github.com/containers/skopeo)
- [Skopeo Documentation](https://github.com/containers/skopeo/blob/main/README.md)
- [Skopeo Man Page](https://www.mankier.com/1/skopeo)
- [Skopeo Copy Man Page](https://www.mankier.com/1/skopeo-copy)
- [Skopeo Inspect Man Page](https://www.mankier.com/1/skopeo-inspect)
- [Skopeo List-Tags Man Page](https://www.mankier.com/1/skopeo-list-tags)
- [Skopeo Login Man Page](https://www.mankier.com/1/skopeo-login)
- [Skopeo Delete Man Page](https://www.mankier.com/1/skopeo-delete)
- [Skopeo Standalone-Sign Man Page](https://www.mankier.com/1/skopeo-standalone-sign)
- [Skopeo Standalone-Verify Man Page](https://www.mankier.com/1/skopeo-standalone-verify)
- [Skopeo Generate-Sigstore-Key Man Page](https://www.mankier.com/1/skopeo-generate-sigstore-key)
- [Skopeo Sync Man Page](https://www.mankier.com/1/skopeo-sync)
- [OCI Specification](https://github.com/opencontainers/image-spec)
- [Container Tools Documentation](https://github.com/containers/toolbox)
- [rpm-ostree Skopeo Integration](https://github.com/coreos/rpm-ostree)