As I was re-arranging my recipe files, I needed a way to ensure that the
order of my recipe is correct without having to read through the
generated `Containerfile`. So I added a `-d`/`--display-full-recipe` arg
to `template` that will print out all of your modules in the order
defined by following the `from-file` property.
```
$> bluebuild template --help
Generate a Containerfile from a recipe
Usage: bluebuild template [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to create a template from
Options:
-o, --output <OUTPUT>
File to output to instead of STDOUT
--registry <REGISTRY>
The registry domain the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
--registry-namespace <REGISTRY_NAMESPACE>
The registry namespace the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
-d, --display-full-recipe
Instead of creating a Containerfile, display the full recipe after traversing all `from-file` properties.
This can be used to help debug the order you defined your recipe.
-t, --syntax-theme <SYNTAX_THEME>
Choose a theme for the syntax highlighting for the Containerfile or Yaml.
The default is `mocha-dark`.
[possible values: mocha-dark, ocean-dark, ocean-light, eighties-dark, inspired-github, solarized-dark, solarized-light]
-s, --squash
Puts the build in a `squash-stage` and COPY's the results to the final stage as one layer.
WARN: This doesn't work with the docker driver as it has been deprecated.
NOTE: Squash has a performance benefit for the newer versions of podman and buildah.
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-v, --verbose...
Increase logging verbosity
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-q, --quiet...
Decrease logging verbosity
-h, --help
Print help (see a summary with '-h')
```
Preview of Containerfile/Dockerfile syntax highlighting:
Preview of Yaml highlighting:
Recipe files can now be put into their own directory `./recipes/`. This
directory is NEVER copied into the build so changes to a recipe will no
longer cause cache misses for builds. Here is an example of my build
changing the second to last module and only requiring the last 2 `RUN`
layers to be run again.
```
=> CACHED [stage-config 1/1] COPY ./config /config 0.0s
=> CACHED [stage-modules 1/2] COPY --from=ghcr.io/blue-build/modules:latest /modules /modules 0.0s
=> CACHED [stage-modules 2/2] COPY ./modules /modules 0.0s
=> CACHED [stage-keys 1/1] COPY cosign.pub /keys/jp-desktop-gaming.pub 0.0s
=> CACHED [stage-4 2/16] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys mkdir -p /usr/etc/pki/containers/ && cp /tmp/keys/* /usr/et 0.0s
=> CACHED [stage-bins 1/3] COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /bins/cosign 0.0s
=> CACHED [stage-bins 2/3] COPY --from=docker.io/mikefarah/yq /usr/bin/yq /bins/yq 0.0s
=> CACHED [stage-bins 3/3] COPY --from=ghcr.io/blue-build/cli:main-installer /out/bluebuild /bins/bluebuild 0.0s
=> CACHED [stage-4 3/16] RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins mkdir -p /usr/bin/ && cp /tmp/bins/* /usr/bin/ && ostree 0.0s
=> CACHED [stage-4 4/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 5/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 6/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 7/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 8/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 9/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 10/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 11/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 12/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 13/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 14/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> [stage-4 15/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from= 33.4s
=> [stage-4 16/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from=s 0.7s
```
Support was also added to put all build files into `./files/` instead of
`./config/`. This is an all or nothing operation, meaning if there
exists a directory of `files` then the `config` directory will be
completely ignored. Work will have to be done in
https://github.com/blue-build/modules to allow users to put their files
directly in `./files/` and not `./files/files` for the `files` module or
`./files/scripts` for the scripts module.
Support was also added to move the `./config/containerfiles/` directory
to the root of the project. Now the directories you can find in the root
of projects are:
```
files/
containerfiles/
recipes/
```
I noticed that making changes to the exports script before a release
could cause modules to not build properly if breaking changes got pushed
out. To prevent this, I'm making it so that the hash of the commit is
put in the tag for the exports script image and that the CLI tool will
use that hash when building the `Containerfile`.
### Buildah/Podman support
Buildah and podman can make heavy use of the squash feature. Something
that I've noticed when trying to build from inside of a container,
requiring intermediate layers with mounts causes build times to
skyrocket. Build times are much faster when using the `--squash`
functionality (seen as `--layers=false`).
Here are the following results from my personal build using both squash
and non-squash functionality.
#### Squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop (digest: sha256:60f743ba322041918d302e7e7f10438c59502e19343c294064bacb676c8eb7b7)
ostree chunk layers already present: 65
custom layers already present: 3
custom layers needed: 1 (814.0 MB)
```
All changes appear to show as a single custom layer. Any small change
even at the end of the build appears to require completely downloading
the new layer (squash only squashes additional layers on top of the base
layer). This makes sense as layers cannot currently be downloaded by
diff.
#### Non-squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest (digest: sha256:0658b51febfcbaa1722961b7a6d2b197d3823a6228e330f45dd1e1aaefd145c5)
ostree chunk layers already present: 65
custom layers already present: 4
custom layers needed: 15 (942.4 MB)
```
As expected, there are more layers when not squashing and the size is
slightly bigger. Most likely due to there being extra information stored
in the layers that is subsequently removed.
### Docker support
Docker is apparently [no longer
supporting](https://github.com/docker/buildx/issues/1287) the use of the
`--squash` arg. The use of squash will not be available for the docker
driver in this case.
There are 2 new args available that allow the user to specify which
program to use for building and inspecting images. If the user doesn't
provide an argument, the tool will determine which program to use like
it has been.
Help text:
```
Build an image from a recipe
Usage: bluebuild build [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to build an image
Options:
-p, --push
Push the image with all the tags.
Requires `--registry`, `--username`, and `--password` if not building in CI.
-c, --compression-format <COMPRESSION_FORMAT>
The compression format the images will be pushed in
[default: gzip]
[possible values: gzip, zstd]
-n, --no-retry-push
Block `bluebuild` from retrying to push the image
--retry-count <RETRY_COUNT>
The number of times to retry pushing the image
[default: 1]
-f, --force
Allow `bluebuild` to overwrite an existing Containerfile without confirmation.
This is not needed if the Containerfile is in .gitignore or has already been built by `bluebuild`.
-a, --archive <ARCHIVE>
Archives the built image into a tarfile in the specified directory
--registry <REGISTRY>
The registry's domain name
-v, --verbose...
Increase logging verbosity
-q, --quiet...
Decrease logging verbosity
--registry-namespace <REGISTRY_NAMESPACE>
The url path to your base project images
[aliases: registry-path]
-U, --username <USERNAME>
The username to login to the container registry
-P, --password <PASSWORD>
The password to login to the container registry
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-h, --help
Print help (see a summary with '-h')
```
All supported versions of podman, buildah, and docker support the zstd
compression format. This format should allow users to pull less data
when updating their computers.
This fix involves creating a new function for the `BuildDriver` trait
called `build_tag_push`. In order to get the proper logic in place to
make use of `docker buildx`, I had to create a separate function that
would construct the build command to include all of the tags necessary
for pushing. A default implementation of `build_tag_push` will be used
for `podman` and `buildah` which was originally from the build command's
functions. Now that we have custom logic for docker builds, we can take
advantage of using the GitHub cache features without having the `--load`
arg which had a big negative effect on build times. We can now also use
docker for creating local `oci-archive` tarballs for local rebasing.
Making use of the `oci-archive` will require the user to create a
`docker-container` builder as it is not supported on the standard
`docker` builder.
https://docs.docker.com/build/exporters/oci-docker/
the `os_version` is defaulting to the `image_tag` inside containers and
causing our template to use latest tag
---------
Co-authored-by: Gerald Pinder <gmpinder@gmail.com>
This separates out the module template logic so that it is easier to
undertstand what each section does. This will also allow creating in
repo documentation for any special modules that don't use a bash base.
An issue was filed for adding retry logic to our push_image command in
the CLI. https://github.com/blue-build/cli/issues/79.
This PR adds:
- **retry flag**
- `-r`
- defaults to true
- **retry_count flag**
- `--retry_count`
- defaults to 1
This functionality will be extended to our other services in build
(podman api as well once that is hooked up), but this is the initial
ground work
---------
Co-authored-by: Gerald Pinder <gmpinder@gmail.com>
The akmods module require having the /rpms directory put into /tmp/rpms.
By default we will mount the akmods image with the `main-{{ os_version
}}` tag.
If a user supplies `base` for the akmods module in their recipe, it will
pull that image tag instead and mount the resulting /rpms.
```yaml
modules:
- type: akmods
base: surface
install:
- openrazer
```
This would pull the image `ghcr.io/ublue-os/akmods:surface-39`.
A user can also supply `nvidia-version` with the numerical version of
the driver you would like to use. Doing so will mount the appropriate
`akmods-nvidia` image with the version of the driver you want in the
tag.
```yaml
modules:
- type: akmods
nvidia-version: 545
install:
- nvidia
```
This would pull the image `ghcr.io/ublue-os/akmods-nvidia:main-39-545`
and `ghcr.io/ublue-os/akmods:main-39`.
This uses bind mount like all the other modules so these files will not
persist into the final image.
To keep things as consistent as possible, we're switching to using the
GITHUB_TOKEN env var for login instead. Env vars were also all pulled
out into their own constants to make things more consistent.
This change also includes prioritizing public/private key signing over
OIDC keyless for GitHub for an easier transition. It would require the
user to delete their `cosign.pub` file from their repo in order to start
using the keyless method.
the `SIGSTORE_ID_TOKEN` env var is not declared inside of github
actions, however the block that does OIDC signing requires it, this
changes it so that it matches for `GITHUB_TOKEN` in the second block
instead of `SIGSTORE_ID_TOKEN`, (hopefully) using OIDC instead of pubkey
auth
This change will follow the patter that was established with the GitLab
pipeline. Users will be able to have their images signed by GitHub's
OIDC. This will allow users to rebase directly onto a signed image and
forego using the siging module.
