feat: Stages (#173)

## Stages

A new property (`stages`) is being added to the recipe file schema. This
property will allow users to define a list of Containerfile stages each
with their own modules. Stages can be used to compile programs, perform
parallel operations, and copy the results into the final image without
contaminating the final image.

### Module Support

Currently the only modules that work out-of-the-box are `copy`,
`script`, `files`, and `containerfile`. Other modules are dependent on
the programs installed on the image. In order to better support some of
our essential modules, a setup script is ran at the start of each stage
that is not `scratch`. This script will install `curl`, `wget`, `bash`,
and `grep` and use the package manager for the detected distributions.

At this time, the following distributions are supported:

- Debian
- Ubuntu
- Fedora
- Alpine

Contributions to increase the size of this list is
[welcome](https://github.com/blue-build/cli)!

### Syntax

- **Required**
- `from` - The full image ref (image name + tag). This will be set in
the `FROM` statement of the stage.
- `name` - The name of the stage. This is used when referencing the
stage when using the `from:` property in the `copy` module.
- `modules` - The list of modules to execute. The exact same syntax used
by the main recipe `modules:` property.
- **Optional**
- `shell` - Allows a user to pass in an array of strings that are passed
directly into the [`SHELL`
instruction](https://docs.docker.com/reference/dockerfile/#shell).

#### Example

```yaml
stages:
- name: ubuntu-test
  from: ubuntu
  modules:
  - type: files
    files:
    - usr: /usr
  - type: script
    scripts:
    - example.sh
    snippets:
    - echo "test" > /test.txt
  - type: test-module
  - type: containerfile
    containerfiles:
    - labels
    snippets:
    - RUN echo "This is a snippet"
```

### Tasks
- [x] `from-file:` - Allows the user to store their stages in a separate
file so it can be included in multiple recipes
- [x] `no-cache:` - This will be useful for stages that want to pull the
latest changes from a git repo and not have to rely on the base image
getting an update for the build to be triggered again.
- [x] Add setup script to be able to install necessary programs to run
`bluebuild` modules in stages
- [x] Check for circular dependencies and error out

## `copy` module

This is a 1-1 for the [`COPY`
instruction](https://docs.docker.com/reference/dockerfile/#copy). It has
the ability to copy files between stages, making this a very important
addition to complete functionality for the stages feature. Each use of
this "module" will become its own layer.

### Decision to use `--link`

We use the `--link`
[option](https://docs.docker.com/reference/dockerfile/#benefits-of-using---link)
which allows that layer to have the same hash if the files haven't
changed regardless of if the previous instructions have changed. This
allows these layers to not have to be re-downloaded on the user's
computer if the copied files haven't changed.

### Syntax

- **Required**
- `src` - The source directory/file from the repo OR when `from:` is set
the image/stage that is specified.
  - `dest` - The destination directory/file inside the working image.
- **Optional**
  - `from` - The stage/image to copy from.

#### Example

```yaml
modules:
- type: copy
  from: ubuntu-test
  src: /test.txt
  dest: /
```

### Tasks
- [x] make `from:` optional
- [x] Add README.md and module.yml

## Feature gating

Gating this feature until we release for `v0.9.0`. The plan will be to
build all features (including this one) for main branch builds. This
means that these features will be available when using the `main` image
and consequently the `use_unstable_cli:` option on the GitHub Action.
All future `v0.9.0` features will be gated as well to allow for patches
to `v0.8`.

### Tasks
- [x] Build `--all-features` on non-tagged builds
- [x] Add stages and copy features

template/templates/modules/akmods/akmods.j2

@@ -1,4 +1,5 @@
 {%- for info in recipe.modules_ext.get_akmods_info_list(os_version) %}
+# Stage for AKmod {{ info.stage_name }}
 FROM scratch as stage-akmods-{{ info.stage_name }}
 COPY --from=ghcr.io/ublue-os/{{ info.images.0 }} /rpms /rpms
 COPY --from=ghcr.io/ublue-os/{{ info.images.1 }} /rpms /rpms

template/templates/modules/containerfile/README.md

@@ -4,7 +4,7 @@
 Only compiler-based builds can use this module as it is built-in to the BlueBuild CLI tool.
 :::
 
-The `containerfile` module is a tool for adding custom [`Containerfile`](https://github.com/containers/common/blob/main/docs/Containerfile.5.md) instructions for custom image builds. This is useful when you wish to use some feature directly available in a `Containerfile`, but not in a bash module, such as copying from other OCI images with `COPY --from`.
+The `containerfile` module is a tool for adding custom [`Containerfile`](https://github.com/containers/common/blob/main/docs/Containerfile.5.md) instructions for custom image builds. This is useful when you wish to use some feature directly available in a `Containerfile`, but not in a bash module, such as using a `RUN` instruction with custom mounts.
 
 Since standard compiler-based BlueBuild image builds generate a `Containerfile` from your recipe, there is no need to manage it yourself. However, we know that we also have technical users that would like to have the ability to customize their `Containerfile`. This is where the `containerfile` module comes into play. 
 
@@ -18,10 +18,10 @@ The `snippets` property is the easiest to use when you just need to insert a few
 modules:
   - type: containerfile
     snippets:
-      - COPY --from=docker.io/mikefarah/yq /usr/bin/yq /usr/bin/yq
+      - RUN --mount=type=tmpfs,target=/tmp /some/script.sh
 ```
 
-This makes it really easy to copy a file or program from another image.
+This makes it really easy to add individual, custom instructions.
 
 :::note
 **NOTE:** Each entry of a snippet will be its own layer in the final `Containerfile`.
@@ -66,7 +66,7 @@ If you wanted to have some `snippets` run before any `containerfiles` have, you
 modules:
   - type: containerfile
     snippets:
-      - COPY --from=docker.io/mikefarah/yq /usr/bin/yq /usr/bin/yq
+      - RUN --mount=type=tmpfs,target=/tmp /some/script.sh
   - type: containerfile
     containerfiles:
       - example

template/templates/modules/containerfile/module.yml

@@ -4,7 +4,7 @@ readme: https://raw.githubusercontent.com/blue-build/cli/main/template/templates
 example: |
   type: containerfile
   snippets:
-    - COPY ./config/example-dir/example-file.txt /usr/etc/example/
+    - RUN --mount=type=tmpfs,target=/tmp /some/script.sh
   containerfiles:
     - example
     - subroutine

template/templates/modules/copy/README.md

@@ -0,0 +1,44 @@
+# `copy`
+
+:::caution
+Only compiler-based builds can use this module as it is built-in to the BlueBuild CLI tool.
+:::
+
+:::note
+**NOTE:** This module is currently only available with the `use_unstable_cli` option on the GHA or using the `main` image.
+:::
+
+The `copy` module is a short-hand method of adding a [`COPY`](https://docs.docker.com/reference/dockerfile/#copy) instruction into the image. This can be used to copy files from images, other stages, or even from the build context. 
+
+## Usage
+
+The `copy` module's properties are a 1-1 match with the `COPY` instruction containing `src`, `dest`, and `from` (optional). The example below will `COPY` the file `/usr/bin/yq` from `docker.io/mikefarah/yq` into `/usr/bin/`.
+
+```yaml
+mdoules:
+- type: copy
+  from: docker.io/mikefarah/yq
+  src: /usr/bin/yq
+  dest: /usr/bin/
+```
+
+Creating an instruction like:
+
+```dockerfile
+COPY --linked --from=docker.io/mikefarah/yq /usr/bin/yq /usr/bin/
+```
+
+Omitting `from:` will allow copying from the build context:
+
+```yaml
+mdoules:
+- type: copy
+  src: file/to/copy.conf
+  dest: /usr/etc/app/
+```
+
+Creating an instruction like:
+
+```dockerfile
+COPY --linked file/to/copy.conf /usr/etc/app/
+```

template/templates/modules/copy/copy.j2

@@ -0,0 +1,4 @@
+{%- if let Some((from_img, src, dest)) = module.get_copy_args() %}
+COPY{% if let Some(from_img) = from_img %} --from={{ from_img }}{% endif %} {{ src }} {{ dest }}
+{%- endif %}
+

template/templates/modules/copy/module.yml

@@ -0,0 +1,8 @@
+name: copy
+shortdesc: The copy module is a direct translation of the `COPY` instruction in a Containerfile.
+readme: https://raw.githubusercontent.com/blue-build/cli/main/template/templates/modules/copy/README.md
+example: |
+  type: copy
+  from: docker.io/mikefarah/yq
+  src: /usr/bin/yq
+  dest: /usr/bin/

template/templates/modules/modules.j2

@@ -1,41 +1,66 @@
-# Key RUN
-RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys \
-  mkdir -p /usr/etc/pki/containers/ \
-  && cp /tmp/keys/* /usr/etc/pki/containers/ \
-  && ostree container commit
-
-# Bin RUN
-RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins \
-  mkdir -p /usr/bin/ \
-  && cp /tmp/bins/* /usr/bin/ \
-  && ostree container commit
-
+{% macro main_modules_run(modules_ext, os_version) %}
 # Module RUNs
-{%- for module in recipe.modules_ext.modules %}
-  {%- if let Some(type) = module.module_type %}
-    {%- if type == "containerfile" %}
-      {%- include "modules/containerfile/containerfile.j2" %}
-    {%- else %}
+  {%- for module in modules_ext.modules %}
+    {%- if let Some(module) = module.required_fields %}
+      {%- if module.no_cache %}
+ARG CACHEBUST="{{ build_id }}"
+      {%- endif %}
+
+      {%- if module.module_type == "containerfile" %}
+        {%- include "modules/containerfile/containerfile.j2" %}
+      {%- else if module.module_type == "copy" %}
+        {%- include "modules/copy/copy.j2" %}
+      {%- else %}
 RUN \
-      {%- if files_dir_exists %}
+        {%- if files_dir_exists %}
   --mount=type=bind,from=stage-files,src=/files,dst=/tmp/files,rw \
-      {%- else %}
+        {%- else %}
   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw \
-      {%- endif %}
-      {%- if let Some(source) = module.source %}
+        {%- endif %}
+        {%- if let Some(source) = module.source %}
   --mount=type=bind,from={{ source }},src=/modules,dst=/tmp/modules,rw \
-      {%- else %}
+        {%- else %}
   --mount=type=bind,from=stage-modules,src=/modules,dst=/tmp/modules,rw \
-      {%- endif %}
-      {%- if type == "akmods" %}
+        {%- endif %}
+        {%- if module.module_type == "akmods" %}
   --mount=type=bind,from=stage-akmods-{{ module.generate_akmods_info(os_version).stage_name }},src=/rpms,dst=/tmp/rpms,rw \
-      {%- endif %}
+        {%- endif %}
   --mount=type=bind,from=ghcr.io/blue-build/cli:{{ exports_tag }}-build-scripts,src=/scripts/,dst=/tmp/scripts/ \
   --mount=type=cache,dst=/var/cache/rpm-ostree,id=rpm-ostree-cache-{{ recipe.name }}-{{ recipe.image_version }},sharing=locked \
-  source /tmp/scripts/exports.sh \
-  && /tmp/scripts/run_module.sh {{ type }} '{{ module.print_module_context() }}'
+  /tmp/scripts/run_module.sh '{{ module.module_type }}' '{{ module.print_module_context() }}' \
+  && ostree container commit
+      {%- endif %}
     {%- endif %}
-  {%- endif %}
-{%- endfor %}
+  {%- endfor %}
+{% endmacro %}
+{% macro stage_modules_run(modules_ext, os_version) %}
+# Module RUNs
+  {%- for module in modules_ext.modules %}
+    {%- if let Some(module) = module.required_fields %}
 
+      {%- if module.no_cache %}
+ARG CACHEBUST="{{ build_id }}"
+      {%- endif %}
 
+      {%- if module.module_type == "containerfile" %}
+        {%- include "modules/containerfile/containerfile.j2" %}
+      {%- else if module.module_type == "copy" %}
+        {%- include "modules/copy/copy.j2" %}
+      {%- else %}
+RUN \
+        {%- if files_dir_exists %}
+  --mount=type=bind,from=stage-files,src=/files,dst=/tmp/files,rw \
+        {%- else %}
+  --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw \
+        {%- endif %}
+        {%- if let Some(source) = module.source %}
+  --mount=type=bind,from={{ source }},src=/modules,dst=/tmp/modules,rw \
+        {%- else %}
+  --mount=type=bind,from=stage-modules,src=/modules,dst=/tmp/modules,rw \
+        {%- endif %}
+  --mount=type=bind,from=ghcr.io/blue-build/cli:{{ exports_tag }}-build-scripts,src=/scripts/,dst=/tmp/scripts/ \
+  /tmp/scripts/run_module.sh '{{ module.module_type }}' '{{ module.print_module_context() }}'
+      {%- endif %}
+    {%- endif %}
+  {%- endfor %}
+{% endmacro %}