This adds the ability for the user to add alternate tags to their
images. These are designed to replace the `latest` and timestamp (e.g.
`20240429`) tags.
This will help make the Containerfile just a little bit easier to read
(ignoring all the mounts lol). This would also allow us to add logic
later to support modules that run executables other than `*.sh`.
As I was re-arranging my recipe files, I needed a way to ensure that the
order of my recipe is correct without having to read through the
generated `Containerfile`. So I added a `-d`/`--display-full-recipe` arg
to `template` that will print out all of your modules in the order
defined by following the `from-file` property.
```
$> bluebuild template --help
Generate a Containerfile from a recipe
Usage: bluebuild template [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to create a template from
Options:
-o, --output <OUTPUT>
File to output to instead of STDOUT
--registry <REGISTRY>
The registry domain the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
--registry-namespace <REGISTRY_NAMESPACE>
The registry namespace the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
-d, --display-full-recipe
Instead of creating a Containerfile, display the full recipe after traversing all `from-file` properties.
This can be used to help debug the order you defined your recipe.
-t, --syntax-theme <SYNTAX_THEME>
Choose a theme for the syntax highlighting for the Containerfile or Yaml.
The default is `mocha-dark`.
[possible values: mocha-dark, ocean-dark, ocean-light, eighties-dark, inspired-github, solarized-dark, solarized-light]
-s, --squash
Puts the build in a `squash-stage` and COPY's the results to the final stage as one layer.
WARN: This doesn't work with the docker driver as it has been deprecated.
NOTE: Squash has a performance benefit for the newer versions of podman and buildah.
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-v, --verbose...
Increase logging verbosity
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-q, --quiet...
Decrease logging verbosity
-h, --help
Print help (see a summary with '-h')
```
Preview of Containerfile/Dockerfile syntax highlighting:
Preview of Yaml highlighting:
This change updates the akmods module to pull the new nvidia images. The
new property will be a boolean with property name of `nvidia`. If a user
continues to use the old `nvidia-version` property, a warning will be
printed telling them to switch to the new property. The old images will
still be allowed to be used to support backwards compatibility.
Previous implementation wasn't taking into account the modules failing
to be read properly and were just being ignored. This changes that to
error out if any `from-file:` usage doesn't deserialize properly.
Recipe files can now be put into their own directory `./recipes/`. This
directory is NEVER copied into the build so changes to a recipe will no
longer cause cache misses for builds. Here is an example of my build
changing the second to last module and only requiring the last 2 `RUN`
layers to be run again.
```
=> CACHED [stage-config 1/1] COPY ./config /config 0.0s
=> CACHED [stage-modules 1/2] COPY --from=ghcr.io/blue-build/modules:latest /modules /modules 0.0s
=> CACHED [stage-modules 2/2] COPY ./modules /modules 0.0s
=> CACHED [stage-keys 1/1] COPY cosign.pub /keys/jp-desktop-gaming.pub 0.0s
=> CACHED [stage-4 2/16] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys mkdir -p /usr/etc/pki/containers/ && cp /tmp/keys/* /usr/et 0.0s
=> CACHED [stage-bins 1/3] COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /bins/cosign 0.0s
=> CACHED [stage-bins 2/3] COPY --from=docker.io/mikefarah/yq /usr/bin/yq /bins/yq 0.0s
=> CACHED [stage-bins 3/3] COPY --from=ghcr.io/blue-build/cli:main-installer /out/bluebuild /bins/bluebuild 0.0s
=> CACHED [stage-4 3/16] RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins mkdir -p /usr/bin/ && cp /tmp/bins/* /usr/bin/ && ostree 0.0s
=> CACHED [stage-4 4/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 5/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 6/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 7/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 8/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 9/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 10/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 11/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 12/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 13/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 14/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> [stage-4 15/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from= 33.4s
=> [stage-4 16/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from=s 0.7s
```
Support was also added to put all build files into `./files/` instead of
`./config/`. This is an all or nothing operation, meaning if there
exists a directory of `files` then the `config` directory will be
completely ignored. Work will have to be done in
https://github.com/blue-build/modules to allow users to put their files
directly in `./files/` and not `./files/files` for the `files` module or
`./files/scripts` for the scripts module.
Support was also added to move the `./config/containerfiles/` directory
to the root of the project. Now the directories you can find in the root
of projects are:
```
files/
containerfiles/
recipes/
```
I noticed that making changes to the exports script before a release
could cause modules to not build properly if breaking changes got pushed
out. To prevent this, I'm making it so that the hash of the commit is
put in the tag for the exports script image and that the CLI tool will
use that hash when building the `Containerfile`.
### Buildah/Podman support
Buildah and podman can make heavy use of the squash feature. Something
that I've noticed when trying to build from inside of a container,
requiring intermediate layers with mounts causes build times to
skyrocket. Build times are much faster when using the `--squash`
functionality (seen as `--layers=false`).
Here are the following results from my personal build using both squash
and non-squash functionality.
#### Squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop (digest: sha256:60f743ba322041918d302e7e7f10438c59502e19343c294064bacb676c8eb7b7)
ostree chunk layers already present: 65
custom layers already present: 3
custom layers needed: 1 (814.0 MB)
```
All changes appear to show as a single custom layer. Any small change
even at the end of the build appears to require completely downloading
the new layer (squash only squashes additional layers on top of the base
layer). This makes sense as layers cannot currently be downloaded by
diff.
#### Non-squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest (digest: sha256:0658b51febfcbaa1722961b7a6d2b197d3823a6228e330f45dd1e1aaefd145c5)
ostree chunk layers already present: 65
custom layers already present: 4
custom layers needed: 15 (942.4 MB)
```
As expected, there are more layers when not squashing and the size is
slightly bigger. Most likely due to there being extra information stored
in the layers that is subsequently removed.
### Docker support
Docker is apparently [no longer
supporting](https://github.com/docker/buildx/issues/1287) the use of the
`--squash` arg. The use of squash will not be available for the docker
driver in this case.
There are 2 new args available that allow the user to specify which
program to use for building and inspecting images. If the user doesn't
provide an argument, the tool will determine which program to use like
it has been.
Help text:
```
Build an image from a recipe
Usage: bluebuild build [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to build an image
Options:
-p, --push
Push the image with all the tags.
Requires `--registry`, `--username`, and `--password` if not building in CI.
-c, --compression-format <COMPRESSION_FORMAT>
The compression format the images will be pushed in
[default: gzip]
[possible values: gzip, zstd]
-n, --no-retry-push
Block `bluebuild` from retrying to push the image
--retry-count <RETRY_COUNT>
The number of times to retry pushing the image
[default: 1]
-f, --force
Allow `bluebuild` to overwrite an existing Containerfile without confirmation.
This is not needed if the Containerfile is in .gitignore or has already been built by `bluebuild`.
-a, --archive <ARCHIVE>
Archives the built image into a tarfile in the specified directory
--registry <REGISTRY>
The registry's domain name
-v, --verbose...
Increase logging verbosity
-q, --quiet...
Decrease logging verbosity
--registry-namespace <REGISTRY_NAMESPACE>
The url path to your base project images
[aliases: registry-path]
-U, --username <USERNAME>
The username to login to the container registry
-P, --password <PASSWORD>
The password to login to the container registry
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-h, --help
Print help (see a summary with '-h')
```
I had initially added distrobox installation instructions to the website
(see [here](https://github.com/blue-build/website/pull/45), but it has
been suggested that the CLI readme is a more natural place for them.
I have tested the distrobox installation method and procedures on my
home machine and aside from an odd bug (see
https://github.com/blue-build/cli/issues/145) everything is working.
This exists in the legacy template:
489ebea77b/build.sh (L13C1-L13C39)
And is also used in many modules, though with
`MODULE_DIRECTORY="${MODULE_DIRECTORY:-"/tmp/modules"}"` to not break
things.
And is documented on the website, because I assumed it existed:
https://blue-build.org/reference/module/#module_directory
I know this probably won't change much, but it's still useful for the
scripts.
