apt-ostree/.notes/research/atomic-filesystems.md

# Understanding Atomic Filesystems: rpm-ostree, apt-ostree, and Ubuntu Integration

## Overview

Fedora Atomic Desktops, including spins like Silverblue, Kinoite (KDE Plasma), Bazzite, and Bluefin, leverage rpm-ostree to provide a unique approach to operating system management built around an immutable core filesystem. This differs significantly from traditional Linux distributions and introduces some nuances in how the filesystem is structured and interact with applications.

**NEW: apt-ostree Integration**
The apt-ostree project brings similar atomic filesystem concepts to Ubuntu/Debian systems, adapting OSTree's immutable filesystem model for APT package management.

## 1. The Immutable Root Filesystem

### **Read-Only Core**
- The core operating system (located at `/` and everything under `/usr`) is mounted as read-only
- This enhances stability and security by preventing accidental or malicious modifications to the base system
- **apt-ostree adaptation**: Same principle applies, but with DEB packages instead of RPM packages

### **Version Control**
- rpm-ostree functions like "Git for operating system binaries", allowing for atomic updates and rollbacks
- **apt-ostree equivalent**: Uses OSTree for atomic commits of DEB package layers
- Updates are applied as a whole, transactional unit, rather than piecemeal package installations

### **Transactional Updates**
- When you perform an OS update on a Fedora Atomic Desktop, rpm-ostree downloads and prepares the new version in the background
- **apt-ostree process**: Similar workflow but with APT package resolution and DEB extraction
- New combined image is created, with previous version available for rollback

## 2. Writable Directories and User Data

### **Separate Writable Areas**
- While the core OS is immutable, directories like `/etc` and `/var` remain writable
- **Ubuntu considerations**: Same structure, but Ubuntu may have additional writable locations

### **User Data Preservation**
- User data is stored separately (typically in `/var/home`, symlinked to `/home`)
- **apt-ostree mapping**:
  ```
  /home → /var/home (symlink)
  /opt → /var/opt (symlink)
  /usr/local → /var/usrlocal (symlink)
  /srv → /var/srv (symlink)
  /root → /var/roothome (symlink)
  /mnt → /var/mnt (symlink)
  /tmp → /var/tmp (symlink)
  ```

### **Symlinks for Compatibility**
- Fedora Atomic Desktops utilize symlinks to redirect expected writable locations into `/var`
- **apt-ostree implementation**: Same symlink structure for Ubuntu compatibility

## 3. Application Management and Layering

### **Containerized Applications**
- **Flatpaks**: Core philosophy for most software installations
- **Ubuntu alternatives**: Snap packages and Flatpaks both work with apt-ostree
- Applications run in isolated environments and are not part of the base filesystem

### **Package Layering**
- **rpm-ostree**: Allows "layering" additional RPM packages on top of the base OS image
- **apt-ostree**: Layers DEB packages on top of the base Ubuntu system
- **mmdebstrap integration**: Used for creating clean base images and package layers

### **Development Environments**
- **Toolbox/Devcontainers**: Containerized development environments
- **Ubuntu equivalents**: Docker containers, LXD containers, or chroot environments
- Keeps development tools isolated from the host system

## 4. Filesystem Structure for apt-ostree

### **Base Filesystem Layout**
```
/
├── ostree/                    # OSTree repository and deployments
│   ├── repo/                 # OSTree repository
│   ├── deploy/               # Deployed systems
│   └── boot/                 # Boot configurations
├── var/                      # Writable data (shared across deployments)
│   ├── home/                 # User home directories
│   ├── opt/                  # Optional application software
│   ├── usrlocal/             # Locally installed software
│   ├── etc/                  # System configuration (merged on upgrade)
│   └── tmp/                  # Temporary files
├── etc/                      # System configuration (writable)
└── usr/                      # Read-only system software
```

### **mmdebstrap Integration**
```bash
# Create base system with mmdebstrap
sudo mmdebstrap --arch=amd64 --variant=minbase \
    --include=systemd,systemd-sysv,ostree \
    noble /tmp/ubuntu-base

# Create package layer
sudo mmdebstrap --arch=amd64 \
    --include=ubuntu-desktop-minimal,gnome-shell \
    noble /tmp/ubuntu-desktop /tmp/ubuntu-base

# Create OSTree commit
sudo ostree commit --repo=/ostree/repo \
    --branch=ubuntu/24.04/x86_64/desktop \
    --parent=ubuntu/24.04/x86_64/base \
    /tmp/ubuntu-desktop
```

## 5. Ubuntu-Specific Considerations

### **Package Management Differences**
- **APT vs DNF**: Different package managers with different dependency resolution
- **DEB vs RPM**: Different package formats and metadata
- **Repository structure**: Ubuntu uses different repository organization

### **Filesystem Choices**
- **Btrfs**: Recommended for root and `/var/home` (same as Fedora)
- **Ext4**: Alternative for simpler setups
- **LVM**: Supported for advanced partitioning

### **Bootloader Configuration**
- **GRUB**: Primary bootloader for Ubuntu systems
- **Secure Boot**: Ubuntu-specific key management
- **UEFI**: Modern boot method support

## 6. OSTree Atomic Filesystem Best Practices (Debian/Ubuntu Focus)

### **Core Principles**
- Root and `/usr` are always read-only; only `/etc` and `/var` are writable
- Use symlinks/bind mounts for: `/home`, `/opt`, `/srv`, `/root`, `/usr/local`, `/mnt`, `/tmp`
- `/var` is shared across deployments; initial content is copied on first boot, not overwritten on upgrade
- `/etc` is merged on upgrade; defaults should be in `/usr/etc`
- Package layering creates new deployments; all changes are atomic and require reboot

### **User and Group Management**
- Static users/groups: use `nss-altfiles` or `systemd-sysusers`
- Dynamic user creation: handled through systemd services
- **apt-ostree consideration**: DEB package scripts may create users/groups

### **Package Script Execution**
- **Pre-installation scripts**: Run during package extraction
- **Post-installation scripts**: Run after filesystem assembly
- **Sandboxing**: Scripts run in controlled environment with bubblewrap
- **Rollback**: Failed scripts trigger automatic rollback

## 7. Testing and Validation

### **Filesystem Validation Tasks**
- [ ] Validate all symlinks/bind mounts at boot and after upgrade
- [ ] Test package install/remove/upgrade for packages writing to `/var`, `/opt`, `/usr/local`
- [ ] Test `/etc` merge behavior with complex configurations
- [ ] Test user/group management and persistence
- [ ] Test container support and isolation
- [ ] Document any Debian/Ubuntu-specific quirks

### **mmdebstrap Testing**
```bash
# Test base system creation
time sudo mmdebstrap --arch=amd64 --variant=minbase noble /tmp/test-base

# Test package layering
time sudo mmdebstrap --arch=amd64 --include=systemd,ostree noble /tmp/test-layered /tmp/test-base

# Test OSTree integration
sudo ostree commit --repo=/ostree/repo --branch=test/base /tmp/test-layered
```

### **Performance Benchmarks**
```bash
# mmdebstrap vs debootstrap timing
time sudo debootstrap --arch=amd64 --variant=minbase noble /tmp/debootstrap-test
# Typical: 3-5 minutes

time sudo mmdebstrap --arch=amd64 --variant=minbase noble /tmp/mmdebstrap-test
# Typical: 1-2 minutes
```

## 8. Development Workflow with apt-ostree

### **Local Development Environment**
```bash
# Set up isolated OSTree repository
mkdir -p ~/ostree-dev/repo
ostree init --repo=~/ostree-dev/repo --mode=archive-z2

# Create test packages
mkdir -p ~/ostree-dev/test-packages
apt download hello cowsay fortune-mod

# Extract and commit packages
for pkg in *.deb; do
    dpkg-deb -R "$pkg" "extracted-${pkg%.deb}"
done

ostree commit --repo=~/ostree-dev/repo \
    --branch=test/packages \
    --subject="Test Packages" \
    extracted-*
```

### **Integration Testing**
```bash
# Test apt-ostree with local repository
cd ~/ostree-dev/apt-ostree
cargo build --release
./target/release/apt-ostree --repo=~/ostree-dev/repo status
./target/release/apt-ostree --repo=~/ostree-dev/repo install hello
```

## 9. Security Considerations

### **Filesystem Security**
- Read-only root filesystem prevents tampering
- Symlink structure prevents privilege escalation
- OSTree commits are cryptographically verified

### **Package Security**
- APT package verification (GPG signatures)
- OSTree commit verification
- Sandboxed script execution

### **Runtime Security**
- Container isolation for applications
- Systemd security features
- AppArmor/SELinux integration

## 10. Migration and Deployment

### **From Traditional Ubuntu**
- User data migration from `/home` to `/var/home`
- Configuration migration from `/etc` to `/usr/etc`
- Package state migration to OSTree layers

### **Deployment Strategies**
- **Incremental**: Layer packages on existing system
- **Full**: Complete system replacement
- **Hybrid**: Combine with snap/Flatpak applications

### **Rollback Procedures**
- Automatic rollback on failed updates
- Manual rollback to previous deployments
- Emergency recovery procedures

## Conclusion

Fedora Atomic Desktops and their Ubuntu equivalents through apt-ostree offer a robust and reliable computing experience built around an immutable core. The filesystem structure and application handling are distinct from traditional Linux distributions, with a strong emphasis on containerization and clear separation between the base operating system and user data.

**apt-ostree Integration Benefits:**
- **Atomic Updates**: Transactional system updates with rollback capability
- **Immutable Core**: Enhanced security and stability
- **Package Layering**: Flexible software installation
- **Ubuntu Compatibility**: Works with existing Ubuntu ecosystem
- **mmdebstrap Integration**: Fast and reliable system creation

While this approach may require some adjustment for users accustomed to traditional package management, the benefits in terms of stability, security, and reproducibility are substantial. The integration of mmdebstrap provides significant performance improvements for development and testing workflows.

---

*Based on upstream OSTree documentation, Fedora Atomic Desktop research, and apt-ostree development experience. Adapted for Ubuntu/Debian ecosystem integration.*