This change updates the akmods module to pull the new nvidia images. The
new property will be a boolean with property name of `nvidia`. If a user
continues to use the old `nvidia-version` property, a warning will be
printed telling them to switch to the new property. The old images will
still be allowed to be used to support backwards compatibility.
Previous implementation wasn't taking into account the modules failing
to be read properly and were just being ignored. This changes that to
error out if any `from-file:` usage doesn't deserialize properly.
Recipe files can now be put into their own directory `./recipes/`. This
directory is NEVER copied into the build so changes to a recipe will no
longer cause cache misses for builds. Here is an example of my build
changing the second to last module and only requiring the last 2 `RUN`
layers to be run again.
```
=> CACHED [stage-config 1/1] COPY ./config /config 0.0s
=> CACHED [stage-modules 1/2] COPY --from=ghcr.io/blue-build/modules:latest /modules /modules 0.0s
=> CACHED [stage-modules 2/2] COPY ./modules /modules 0.0s
=> CACHED [stage-keys 1/1] COPY cosign.pub /keys/jp-desktop-gaming.pub 0.0s
=> CACHED [stage-4 2/16] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys mkdir -p /usr/etc/pki/containers/ && cp /tmp/keys/* /usr/et 0.0s
=> CACHED [stage-bins 1/3] COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /bins/cosign 0.0s
=> CACHED [stage-bins 2/3] COPY --from=docker.io/mikefarah/yq /usr/bin/yq /bins/yq 0.0s
=> CACHED [stage-bins 3/3] COPY --from=ghcr.io/blue-build/cli:main-installer /out/bluebuild /bins/bluebuild 0.0s
=> CACHED [stage-4 3/16] RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins mkdir -p /usr/bin/ && cp /tmp/bins/* /usr/bin/ && ostree 0.0s
=> CACHED [stage-4 4/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 5/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 6/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 7/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 8/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 9/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 10/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 11/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 12/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 13/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 14/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> [stage-4 15/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from= 33.4s
=> [stage-4 16/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from=s 0.7s
```
Support was also added to put all build files into `./files/` instead of
`./config/`. This is an all or nothing operation, meaning if there
exists a directory of `files` then the `config` directory will be
completely ignored. Work will have to be done in
https://github.com/blue-build/modules to allow users to put their files
directly in `./files/` and not `./files/files` for the `files` module or
`./files/scripts` for the scripts module.
Support was also added to move the `./config/containerfiles/` directory
to the root of the project. Now the directories you can find in the root
of projects are:
```
files/
containerfiles/
recipes/
```
I noticed that making changes to the exports script before a release
could cause modules to not build properly if breaking changes got pushed
out. To prevent this, I'm making it so that the hash of the commit is
put in the tag for the exports script image and that the CLI tool will
use that hash when building the `Containerfile`.
### Buildah/Podman support
Buildah and podman can make heavy use of the squash feature. Something
that I've noticed when trying to build from inside of a container,
requiring intermediate layers with mounts causes build times to
skyrocket. Build times are much faster when using the `--squash`
functionality (seen as `--layers=false`).
Here are the following results from my personal build using both squash
and non-squash functionality.
#### Squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop (digest: sha256:60f743ba322041918d302e7e7f10438c59502e19343c294064bacb676c8eb7b7)
ostree chunk layers already present: 65
custom layers already present: 3
custom layers needed: 1 (814.0 MB)
```
All changes appear to show as a single custom layer. Any small change
even at the end of the build appears to require completely downloading
the new layer (squash only squashes additional layers on top of the base
layer). This makes sense as layers cannot currently be downloaded by
diff.
#### Non-squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest (digest: sha256:0658b51febfcbaa1722961b7a6d2b197d3823a6228e330f45dd1e1aaefd145c5)
ostree chunk layers already present: 65
custom layers already present: 4
custom layers needed: 15 (942.4 MB)
```
As expected, there are more layers when not squashing and the size is
slightly bigger. Most likely due to there being extra information stored
in the layers that is subsequently removed.
### Docker support
Docker is apparently [no longer
supporting](https://github.com/docker/buildx/issues/1287) the use of the
`--squash` arg. The use of squash will not be available for the docker
driver in this case.
There are 2 new args available that allow the user to specify which
program to use for building and inspecting images. If the user doesn't
provide an argument, the tool will determine which program to use like
it has been.
Help text:
```
Build an image from a recipe
Usage: bluebuild build [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to build an image
Options:
-p, --push
Push the image with all the tags.
Requires `--registry`, `--username`, and `--password` if not building in CI.
-c, --compression-format <COMPRESSION_FORMAT>
The compression format the images will be pushed in
[default: gzip]
[possible values: gzip, zstd]
-n, --no-retry-push
Block `bluebuild` from retrying to push the image
--retry-count <RETRY_COUNT>
The number of times to retry pushing the image
[default: 1]
-f, --force
Allow `bluebuild` to overwrite an existing Containerfile without confirmation.
This is not needed if the Containerfile is in .gitignore or has already been built by `bluebuild`.
-a, --archive <ARCHIVE>
Archives the built image into a tarfile in the specified directory
--registry <REGISTRY>
The registry's domain name
-v, --verbose...
Increase logging verbosity
-q, --quiet...
Decrease logging verbosity
--registry-namespace <REGISTRY_NAMESPACE>
The url path to your base project images
[aliases: registry-path]
-U, --username <USERNAME>
The username to login to the container registry
-P, --password <PASSWORD>
The password to login to the container registry
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-h, --help
Print help (see a summary with '-h')
```
I had initially added distrobox installation instructions to the website
(see [here](https://github.com/blue-build/website/pull/45), but it has
been suggested that the CLI readme is a more natural place for them.
I have tested the distrobox installation method and procedures on my
home machine and aside from an odd bug (see
https://github.com/blue-build/cli/issues/145) everything is working.
This exists in the legacy template:
489ebea77b/build.sh (L13C1-L13C39)
And is also used in many modules, though with
`MODULE_DIRECTORY="${MODULE_DIRECTORY:-"/tmp/modules"}"` to not break
things.
And is documented on the website, because I assumed it existed:
https://blue-build.org/reference/module/#module_directory
I know this probably won't change much, but it's still useful for the
scripts.
I've been investigating more into how ostree works and how it relates to
running `ostree container commit` for each layer. I've decided to move
our pre-installed bins and public keys into their own stages and then
bind mount them into a `RUN` instruction so that we can just use `cp` to
get the files into the image and then call `ostree container commit`.
Now all of our layers in the image (after the base image) will be in the
ostree commit tree.
All supported versions of podman, buildah, and docker support the zstd
compression format. This format should allow users to pull less data
when updating their computers.
This fix involves creating a new function for the `BuildDriver` trait
called `build_tag_push`. In order to get the proper logic in place to
make use of `docker buildx`, I had to create a separate function that
would construct the build command to include all of the tags necessary
for pushing. A default implementation of `build_tag_push` will be used
for `podman` and `buildah` which was originally from the build command's
functions. Now that we have custom logic for docker builds, we can take
advantage of using the GitHub cache features without having the `--load`
arg which had a big negative effect on build times. We can now also use
docker for creating local `oci-archive` tarballs for local rebasing.
Making use of the `oci-archive` will require the user to create a
`docker-container` builder as it is not supported on the standard
`docker` builder.
https://docs.docker.com/build/exporters/oci-docker/
https://github.com/blue-build/cli/issues/118#issuecomment-1992536977
as mentioned here reverting to the bash files module simplifies the
build process (and allows for better usage of `ostree container
commit`). It also means there are less overall layers and (ideally) a
smaller image as a result. Tested locally and template generation seems
to work just fine
