This also includes a new `login` command. The signing and CI logic is now using the Driver trait system along with a new experimental sigstore signing driver. New static macros have also been created to make implementation management easier for `Command` usage and `Driver` trait implementation calls.
---------
Co-authored-by: xyny <60004820+xynydev@users.noreply.github.com>
This will be used for running containers for various tasks. There will
be a way to take all output from the process and a way to display output
from a running container like our builds have.
This allows spawned children processes to be killed when a user performs
a Ctrl-C operation. Before this change, children processes would end up
continuing in the background with the user unaware.
The `build` subcommand can now take in any number of recipe files and
will build them all in parallel. Along with this new ability, I've added
a way to easily distinguish which part of the build log belongs to which
recipe. Check out the `docker_build` action of this PR for an example.
## Tasks
- [x] Make build log follow same pattern as normal logs to keep things
consistent
- [x] Update color ranges based on @xynydev 's feedback
- [x] Deal with ANSI control characters in log output
- [x] Add [`indicatif`](https://crates.io/crates/indicatif) to make logs
look nicer
- [x] Add ability to print logs to a file
This updates the `template` subcommand to be `generate`. The `template`
usage will continue to work as an alias to `generate`. A new `switch`
command is added that will manage both `rpm-ostree rebase` and
`rpm-ostree upgrade` and is fully replacing the respective subcommands
as a breaking change.
The new `switch` command is under the feature flag `switch` and will
currently only build for the `main` branch builds until it is moved as a
default feature (`v0.9.0`).
Closes#159
As I was re-arranging my recipe files, I needed a way to ensure that the
order of my recipe is correct without having to read through the
generated `Containerfile`. So I added a `-d`/`--display-full-recipe` arg
to `template` that will print out all of your modules in the order
defined by following the `from-file` property.
```
$> bluebuild template --help
Generate a Containerfile from a recipe
Usage: bluebuild template [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to create a template from
Options:
-o, --output <OUTPUT>
File to output to instead of STDOUT
--registry <REGISTRY>
The registry domain the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
--registry-namespace <REGISTRY_NAMESPACE>
The registry namespace the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
-d, --display-full-recipe
Instead of creating a Containerfile, display the full recipe after traversing all `from-file` properties.
This can be used to help debug the order you defined your recipe.
-t, --syntax-theme <SYNTAX_THEME>
Choose a theme for the syntax highlighting for the Containerfile or Yaml.
The default is `mocha-dark`.
[possible values: mocha-dark, ocean-dark, ocean-light, eighties-dark, inspired-github, solarized-dark, solarized-light]
-s, --squash
Puts the build in a `squash-stage` and COPY's the results to the final stage as one layer.
WARN: This doesn't work with the docker driver as it has been deprecated.
NOTE: Squash has a performance benefit for the newer versions of podman and buildah.
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-v, --verbose...
Increase logging verbosity
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-q, --quiet...
Decrease logging verbosity
-h, --help
Print help (see a summary with '-h')
```
Preview of Containerfile/Dockerfile syntax highlighting:
Preview of Yaml highlighting:
Recipe files can now be put into their own directory `./recipes/`. This
directory is NEVER copied into the build so changes to a recipe will no
longer cause cache misses for builds. Here is an example of my build
changing the second to last module and only requiring the last 2 `RUN`
layers to be run again.
```
=> CACHED [stage-config 1/1] COPY ./config /config 0.0s
=> CACHED [stage-modules 1/2] COPY --from=ghcr.io/blue-build/modules:latest /modules /modules 0.0s
=> CACHED [stage-modules 2/2] COPY ./modules /modules 0.0s
=> CACHED [stage-keys 1/1] COPY cosign.pub /keys/jp-desktop-gaming.pub 0.0s
=> CACHED [stage-4 2/16] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys mkdir -p /usr/etc/pki/containers/ && cp /tmp/keys/* /usr/et 0.0s
=> CACHED [stage-bins 1/3] COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /bins/cosign 0.0s
=> CACHED [stage-bins 2/3] COPY --from=docker.io/mikefarah/yq /usr/bin/yq /bins/yq 0.0s
=> CACHED [stage-bins 3/3] COPY --from=ghcr.io/blue-build/cli:main-installer /out/bluebuild /bins/bluebuild 0.0s
=> CACHED [stage-4 3/16] RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins mkdir -p /usr/bin/ && cp /tmp/bins/* /usr/bin/ && ostree 0.0s
=> CACHED [stage-4 4/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 5/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 6/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 7/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 8/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 9/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 10/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 11/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 12/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 13/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 14/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> [stage-4 15/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from= 33.4s
=> [stage-4 16/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from=s 0.7s
```
Support was also added to put all build files into `./files/` instead of
`./config/`. This is an all or nothing operation, meaning if there
exists a directory of `files` then the `config` directory will be
completely ignored. Work will have to be done in
https://github.com/blue-build/modules to allow users to put their files
directly in `./files/` and not `./files/files` for the `files` module or
`./files/scripts` for the scripts module.
Support was also added to move the `./config/containerfiles/` directory
to the root of the project. Now the directories you can find in the root
of projects are:
```
files/
containerfiles/
recipes/
```
I noticed that making changes to the exports script before a release
could cause modules to not build properly if breaking changes got pushed
out. To prevent this, I'm making it so that the hash of the commit is
put in the tag for the exports script image and that the CLI tool will
use that hash when building the `Containerfile`.
### Buildah/Podman support
Buildah and podman can make heavy use of the squash feature. Something
that I've noticed when trying to build from inside of a container,
requiring intermediate layers with mounts causes build times to
skyrocket. Build times are much faster when using the `--squash`
functionality (seen as `--layers=false`).
Here are the following results from my personal build using both squash
and non-squash functionality.
#### Squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop (digest: sha256:60f743ba322041918d302e7e7f10438c59502e19343c294064bacb676c8eb7b7)
ostree chunk layers already present: 65
custom layers already present: 3
custom layers needed: 1 (814.0 MB)
```
All changes appear to show as a single custom layer. Any small change
even at the end of the build appears to require completely downloading
the new layer (squash only squashes additional layers on top of the base
layer). This makes sense as layers cannot currently be downloaded by
diff.
#### Non-squash upgrade:
```
$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-desktop-gaming:latest (digest: sha256:0658b51febfcbaa1722961b7a6d2b197d3823a6228e330f45dd1e1aaefd145c5)
ostree chunk layers already present: 65
custom layers already present: 4
custom layers needed: 15 (942.4 MB)
```
As expected, there are more layers when not squashing and the size is
slightly bigger. Most likely due to there being extra information stored
in the layers that is subsequently removed.
### Docker support
Docker is apparently [no longer
supporting](https://github.com/docker/buildx/issues/1287) the use of the
`--squash` arg. The use of squash will not be available for the docker
driver in this case.
There are 2 new args available that allow the user to specify which
program to use for building and inspecting images. If the user doesn't
provide an argument, the tool will determine which program to use like
it has been.
Help text:
```
Build an image from a recipe
Usage: bluebuild build [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to build an image
Options:
-p, --push
Push the image with all the tags.
Requires `--registry`, `--username`, and `--password` if not building in CI.
-c, --compression-format <COMPRESSION_FORMAT>
The compression format the images will be pushed in
[default: gzip]
[possible values: gzip, zstd]
-n, --no-retry-push
Block `bluebuild` from retrying to push the image
--retry-count <RETRY_COUNT>
The number of times to retry pushing the image
[default: 1]
-f, --force
Allow `bluebuild` to overwrite an existing Containerfile without confirmation.
This is not needed if the Containerfile is in .gitignore or has already been built by `bluebuild`.
-a, --archive <ARCHIVE>
Archives the built image into a tarfile in the specified directory
--registry <REGISTRY>
The registry's domain name
-v, --verbose...
Increase logging verbosity
-q, --quiet...
Decrease logging verbosity
--registry-namespace <REGISTRY_NAMESPACE>
The url path to your base project images
[aliases: registry-path]
-U, --username <USERNAME>
The username to login to the container registry
-P, --password <PASSWORD>
The password to login to the container registry
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-h, --help
Print help (see a summary with '-h')
```
All supported versions of podman, buildah, and docker support the zstd
compression format. This format should allow users to pull less data
when updating their computers.
This fix involves creating a new function for the `BuildDriver` trait
called `build_tag_push`. In order to get the proper logic in place to
make use of `docker buildx`, I had to create a separate function that
would construct the build command to include all of the tags necessary
for pushing. A default implementation of `build_tag_push` will be used
for `podman` and `buildah` which was originally from the build command's
functions. Now that we have custom logic for docker builds, we can take
advantage of using the GitHub cache features without having the `--load`
arg which had a big negative effect on build times. We can now also use
docker for creating local `oci-archive` tarballs for local rebasing.
Making use of the `oci-archive` will require the user to create a
`docker-container` builder as it is not supported on the standard
`docker` builder.
https://docs.docker.com/build/exporters/oci-docker/
the `os_version` is defaulting to the `image_tag` inside containers and
causing our template to use latest tag
---------
Co-authored-by: Gerald Pinder <gmpinder@gmail.com>
