No description
Find a file
Gerald Pinder 8069006c03
feat: Stages (#173)
## Stages

A new property (`stages`) is being added to the recipe file schema. This
property will allow users to define a list of Containerfile stages each
with their own modules. Stages can be used to compile programs, perform
parallel operations, and copy the results into the final image without
contaminating the final image.

### Module Support

Currently the only modules that work out-of-the-box are `copy`,
`script`, `files`, and `containerfile`. Other modules are dependent on
the programs installed on the image. In order to better support some of
our essential modules, a setup script is ran at the start of each stage
that is not `scratch`. This script will install `curl`, `wget`, `bash`,
and `grep` and use the package manager for the detected distributions.

At this time, the following distributions are supported:

- Debian
- Ubuntu
- Fedora
- Alpine

Contributions to increase the size of this list is
[welcome](https://github.com/blue-build/cli)!

### Syntax

- **Required**
- `from` - The full image ref (image name + tag). This will be set in
the `FROM` statement of the stage.
- `name` - The name of the stage. This is used when referencing the
stage when using the `from:` property in the `copy` module.
- `modules` - The list of modules to execute. The exact same syntax used
by the main recipe `modules:` property.
- **Optional**
- `shell` - Allows a user to pass in an array of strings that are passed
directly into the [`SHELL`
instruction](https://docs.docker.com/reference/dockerfile/#shell).

#### Example

```yaml
stages:
- name: ubuntu-test
  from: ubuntu
  modules:
  - type: files
    files:
    - usr: /usr
  - type: script
    scripts:
    - example.sh
    snippets:
    - echo "test" > /test.txt
  - type: test-module
  - type: containerfile
    containerfiles:
    - labels
    snippets:
    - RUN echo "This is a snippet"
```

### Tasks
- [x] `from-file:` - Allows the user to store their stages in a separate
file so it can be included in multiple recipes
- [x] `no-cache:` - This will be useful for stages that want to pull the
latest changes from a git repo and not have to rely on the base image
getting an update for the build to be triggered again.
- [x] Add setup script to be able to install necessary programs to run
`bluebuild` modules in stages
- [x] Check for circular dependencies and error out

## `copy` module

This is a 1-1 for the [`COPY`
instruction](https://docs.docker.com/reference/dockerfile/#copy). It has
the ability to copy files between stages, making this a very important
addition to complete functionality for the stages feature. Each use of
this "module" will become its own layer.

### Decision to use `--link`

We use the `--link`
[option](https://docs.docker.com/reference/dockerfile/#benefits-of-using---link)
which allows that layer to have the same hash if the files haven't
changed regardless of if the previous instructions have changed. This
allows these layers to not have to be re-downloaded on the user's
computer if the copied files haven't changed.

### Syntax

- **Required**
- `src` - The source directory/file from the repo OR when `from:` is set
the image/stage that is specified.
  - `dest` - The destination directory/file inside the working image.
- **Optional**
  - `from` - The stage/image to copy from.

#### Example

```yaml
modules:
- type: copy
  from: ubuntu-test
  src: /test.txt
  dest: /
```

### Tasks
- [x] make `from:` optional
- [x] Add README.md and module.yml

## Feature gating

Gating this feature until we release for `v0.9.0`. The plan will be to
build all features (including this one) for main branch builds. This
means that these features will be available when using the `main` image
and consequently the `use_unstable_cli:` option on the GitHub Action.
All future `v0.9.0` features will be gated as well to allow for patches
to `v0.8`.

### Tasks
- [x] Build `--all-features` on non-tagged builds
- [x] Add stages and copy features
2024-05-18 13:23:50 +00:00
.cargo chore: Remove nightly flags 2024-02-09 15:22:27 -05:00
.github chore: Fix checkout for podman-build 2024-05-15 11:35:20 -04:00
.helix chore: Remove builtin-podman code 2024-03-23 15:54:15 -04:00
integration-tests feat: Stages (#173) 2024-05-18 13:23:50 +00:00
recipe feat: Stages (#173) 2024-05-18 13:23:50 +00:00
scripts feat: Stages (#173) 2024-05-18 13:23:50 +00:00
src feat: Stages (#173) 2024-05-18 13:23:50 +00:00
template feat: Stages (#173) 2024-05-18 13:23:50 +00:00
utils feat: Stages (#173) 2024-05-18 13:23:50 +00:00
.earthlyignore Add ability to use incremental caching for rust builds in Earthfile 2023-12-27 16:36:16 -05:00
.envrc feat: add flakehub entry + nix flake (#109) 2024-03-05 21:34:29 +00:00
.gitignore feat: add flakehub entry + nix flake (#109) 2024-03-05 21:34:29 +00:00
.rusty-hook.toml feat: Stages (#173) 2024-05-18 13:23:50 +00:00
build.rs fix: Git sha not present during cargo install (#176) 2024-05-05 00:44:22 -04:00
Cargo.lock feat: Stages (#173) 2024-05-18 13:23:50 +00:00
Cargo.toml feat: Stages (#173) 2024-05-18 13:23:50 +00:00
CHANGELOG.md chore: Release 2024-05-16 20:39:06 -04:00
cliff.toml chore: Add cargo release files 2024-02-13 16:33:43 -05:00
cosign.pub fix: Remove check for specific branches for signing (#114) 2024-03-05 09:18:43 -05:00
Earthfile feat: Stages (#173) 2024-05-18 13:23:50 +00:00
flake.lock feat: add flakehub entry + nix flake (#109) 2024-03-05 21:34:29 +00:00
flake.nix feat: add flakehub entry + nix flake (#109) 2024-03-05 21:34:29 +00:00
install.sh chore: Release 2024-05-16 20:39:06 -04:00
justfile fix: Use shebang in release recipe 2024-04-22 17:07:40 -04:00
LICENSE Add LICENSE 2023-10-01 20:57:24 +00:00
modules.json feat: Stages (#173) 2024-05-18 13:23:50 +00:00
README.md docs: Update docker/podman install instructions 2024-05-14 10:58:12 -04:00
rust-toolchain.toml feat: Bugreport command (#28) 2024-01-31 09:51:13 -05:00
rustfmt.toml feat: upgrades (#26) 2024-01-27 19:07:03 -05:00

BlueBuild. A minimal logo with a blue-billed duck holding a golden wrench in its beak.

BlueBuild

BlueBuild's command line program that builds Containerfiles and custom images based on your recipe.yml.

Requirements

The bluebuild tool takes advantage of newer build features. Specifically bind, cache, and tmpfs mounts on the RUN instructions. We support using the following tools and their versions:

  • Docker - v23 and above
  • Podman - v4 and above
  • Buildah - v1.24 and above

Installation

Distrobox

We package a fedora-toolbox and alpine image with all the tools needed to run bluebuild. You can use distrobox to run the application without needing to install it on your machine.

# fedora-toolbox
distrobox create blue-build --image ghcr.io/blue-build/cli
# alpine
distrobox create blue-build --image ghcr.io/blue-build/cli:latest-alpine

By default, the bluebuild commands will not be visible outside of the distrobox itself. You will need to enter the distrobox, and either run the commands from inside the distrobox, or export the distrobox commands for use outside the distrobox.

Refer to the distrobox documentation for more information.

Running commands from within distrobox

[user@host]$ bluebuild help
ERROR
[user@host]$ distrobox enter blue-build
[user@blue-build]$ bluebuild help
A CLI tool built for creating Containerfile templates based on the Ublue Community Project
...

Exporting commands to run outside distrobox

[user@blue-build]$ distrobox-export --bin $(which bluebuild)
[user@blue-build]$ exit
[user@host]$ bluebuild help
A CLI tool built for creating Containerfile templates based on the Ublue Community Project
...

Cargo

This is the best way to install as it gives you the opportunity to build for your specific environment.

cargo install --locked blue-build

Podman/Docker

This will install the binary on your system in /usr/local/bin. This is only a linux-gnu version.

podman run --pull always --rm ghcr.io/blue-build/cli:latest-installer | bash
docker run --pull always --rm ghcr.io/blue-build/cli:latest-installer | bash

Nix Flake

You can install this CLI through the Nix flake on Flakehub

Non-nixos

You can install BlueBuild to your global package environment on non-nixos systems by running

# you can replace "*" with a specific tag
nix profile install https://flakehub.com/f/bluebuild/cli/*.tar.gz#bluebuild

NixOS

If you are using a dedicated flake to manage your dependencies, you can add BlueBuild as a flake input throught the fh cli (that can be installed through nixpkgs) and add bluebuild to it.

{pkgs,inputs,...}: {
    ...
    environment.SystemPackages = [
        inputs.bluebuild.packages.${pkgs.system}.bluebuild # change bluebuild with the fh added input name
    ];
    ...
}

If you are not using a dedicated nix flake, you can add the BlueBuild flake as a variable inside your /etc/nixos/*.nix configuration, though this requires you to run nixos-rebuild with the --impure variable, it is not advisable to do so.

{pkgs,...}:
let
    bluebuild = builtins.fetchTarball "https://flakehub.com/f/bluebuild/cli/*.tar.gz";
in {
    ...
    environment.SystemPackages = [
        bluebuild.packages.${pkgs.system}.bluebuild
    ];
    ...
}

You can also use nix develop .# in this repos directory to run a nix shell with development dependencies and some helful utilities for building BlueBuild!

Github Install Script

bash <(curl -s https://raw.githubusercontent.com/blue-build/cli/main/install.sh)

How to use

Templating

Once you have the CLI tool installed, you can run the following to pull in your recipe file to generate a Containerfile.

bluebuild template -o <CONTAINERFILE> <RECIPE_FILE>

You can then use this with podman or buildah to build and publish your image. Further options can be viewed by running bluebuild template --help

Building

If you don't care about the details of the template, you can run the build command.

bluebuild build ./recipes/recipe.yaml

This will template out the file and build with buildah or podman.

Completions

The bluebuild completions command generates shell completions, printed to stdout. These completions can be stored for integration in your shell environment. For example, on a system with bash-completion installed:

# user completions
$ bluebuild completions bash > ~/.local/share/bash-completion/completions/bluebuild
# system-wide completions
$ bluebuild completions bash | sudo tee /usr/share/bash-completion/completions/bluebuild

Subsequent invocations of bluebuild will respond to <Tab> autocompletions:

$ bluebuild # press <Tab>
-v           -V           --help       template     bug-report
-q           --verbose    --version    upgrade      completions
-h           --quiet      build        rebase       help

Currently, bluebuild completions are available for bash, zsh, fish, powershell, and elvish shell environments.

Local Builds

Rebase

If you want to test your changes, you can do so by using the rebase command. This will create an image as a .tar.gz file, store it in /etc/bluebuild, an run rpm-ostree rebase on that newly built file.

sudo bluebuild rebase recipes/recipe.yml

You can initiate an immediate restart by adding the --reboot/-r option.

Upgrade

When you've rebased onto a local image archive, you can update your image for your recipe by running:

sudo bluebuild upgrade recipes/recipe.yml

The --reboot argument can be used with this command as well.

CI Builds

GitHub

You can use our GitHub Action by using the following .github/workflows/build.yaml:

name: bluebuild
on:
  schedule:
    - cron: "00 17 * * *" # build at 17:00 UTC every day
                          # (20 minutes after last ublue images start building)
  push:
    paths-ignore: # don't rebuild if only documentation has changed
      - "**.md"
  pull_request:
  workflow_dispatch: # allow manually triggering builds
jobs:
  bluebuild:
    name: Build Custom Image
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    strategy:
      fail-fast: false # stop GH from cancelling all matrix builds if one fails
      matrix:
        recipe:
          # !! Add your recipes here
          - recipe.yml
    steps:
       # the build is fully handled by the reusable github action
      - name: Build Custom Image
        uses: blue-build/github-action@v1.0.0
        with:
          recipe: ${{ matrix.recipe }}
          cosign_private_key: ${{ secrets.SIGNING_SECRET }}
          registry_token: ${{ github.token }}
          pr_event_number: ${{ github.event.number }}
Gitlab

We also support GitLab CI! Fun fact, this project started out as a way to build these images in GitLab. You will want to make use of GitLab's Secure Files feature for using your cosign private key for signing. Here's an example of a .gitlab-ci.yml:

workflow:
  rules:
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push"
      when: never
    - if: "$CI_COMMIT_TAG"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: "$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS"
      when: never
    - if: "$CI_COMMIT_BRANCH"

stages:
  - build

build-image:
  stage: build
  image:
    name: ghcr.io/blue-build/cli:main
    entrypoint: [""]
  services:
    - docker:dind
  parallel:
    matrix:
      - RECIPE:
          # Add your recipe files here
          - recipe.yml
  variables:
    # Setup a secure connection with docker-in-docker service
    # https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: /certs
    DOCKER_TLS_VERIFY: 1
    DOCKER_CERT_PATH: $DOCKER_TLS_CERTDIR/client
  before_script:
    # Pulls secure files into the build
    - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
    - export COSIGN_PRIVATE_KEY=$(cat .secure_files/cosign.key)
  script:
    - sleep 5 # Wait a bit for the docker-in-docker service to start
    - bluebuild build --push ./recipes/$RECIPE

Future Features

  • Stages for parallel building (useful for compiling programs for your image)
  • Automatic download and management of image keys for seamless signed image rebasing
  • Module command for easy 3rd party plugin management
  • Create an init command to create a repo for you to start out
  • Setup the project to allow installing with cargo-binstall