This changes the Dockerfile to pull the cosign container image from GHCR
instead of Google Cloud. This helps the Sigstore team manage their cloud
spend (as GHCR is provided for free and Google Cloud Artifact Registry
is not).
Note the container hash does not change and images are posted to both
locations upon cosign's release process.
This also includes a new `login` command. The signing and CI logic is now using the Driver trait system along with a new experimental sigstore signing driver. New static macros have also been created to make implementation management easier for `Command` usage and `Driver` trait implementation calls.
---------
Co-authored-by: xyny <60004820+xynydev@users.noreply.github.com>
This will be used for running containers for various tasks. There will
be a way to take all output from the process and a way to display output
from a running container like our builds have.
This allows spawned children processes to be killed when a user performs
a Ctrl-C operation. Before this change, children processes would end up
continuing in the background with the user unaware.
The `build` subcommand can now take in any number of recipe files and
will build them all in parallel. Along with this new ability, I've added
a way to easily distinguish which part of the build log belongs to which
recipe. Check out the `docker_build` action of this PR for an example.
## Tasks
- [x] Make build log follow same pattern as normal logs to keep things
consistent
- [x] Update color ranges based on @xynydev 's feedback
- [x] Deal with ANSI control characters in log output
- [x] Add [`indicatif`](https://crates.io/crates/indicatif) to make logs
look nicer
- [x] Add ability to print logs to a file
This updates the `template` subcommand to be `generate`. The `template`
usage will continue to work as an alias to `generate`. A new `switch`
command is added that will manage both `rpm-ostree rebase` and
`rpm-ostree upgrade` and is fully replacing the respective subcommands
as a breaking change.
The new `switch` command is under the feature flag `switch` and will
currently only build for the `main` branch builds until it is moved as a
default feature (`v0.9.0`).
Closes#159
## Stages
A new property (`stages`) is being added to the recipe file schema. This
property will allow users to define a list of Containerfile stages each
with their own modules. Stages can be used to compile programs, perform
parallel operations, and copy the results into the final image without
contaminating the final image.
### Module Support
Currently the only modules that work out-of-the-box are `copy`,
`script`, `files`, and `containerfile`. Other modules are dependent on
the programs installed on the image. In order to better support some of
our essential modules, a setup script is ran at the start of each stage
that is not `scratch`. This script will install `curl`, `wget`, `bash`,
and `grep` and use the package manager for the detected distributions.
At this time, the following distributions are supported:
- Debian
- Ubuntu
- Fedora
- Alpine
Contributions to increase the size of this list is
[welcome](https://github.com/blue-build/cli)!
### Syntax
- **Required**
- `from` - The full image ref (image name + tag). This will be set in
the `FROM` statement of the stage.
- `name` - The name of the stage. This is used when referencing the
stage when using the `from:` property in the `copy` module.
- `modules` - The list of modules to execute. The exact same syntax used
by the main recipe `modules:` property.
- **Optional**
- `shell` - Allows a user to pass in an array of strings that are passed
directly into the [`SHELL`
instruction](https://docs.docker.com/reference/dockerfile/#shell).
#### Example
```yaml
stages:
- name: ubuntu-test
from: ubuntu
modules:
- type: files
files:
- usr: /usr
- type: script
scripts:
- example.sh
snippets:
- echo "test" > /test.txt
- type: test-module
- type: containerfile
containerfiles:
- labels
snippets:
- RUN echo "This is a snippet"
```
### Tasks
- [x] `from-file:` - Allows the user to store their stages in a separate
file so it can be included in multiple recipes
- [x] `no-cache:` - This will be useful for stages that want to pull the
latest changes from a git repo and not have to rely on the base image
getting an update for the build to be triggered again.
- [x] Add setup script to be able to install necessary programs to run
`bluebuild` modules in stages
- [x] Check for circular dependencies and error out
## `copy` module
This is a 1-1 for the [`COPY`
instruction](https://docs.docker.com/reference/dockerfile/#copy). It has
the ability to copy files between stages, making this a very important
addition to complete functionality for the stages feature. Each use of
this "module" will become its own layer.
### Decision to use `--link`
We use the `--link`
[option](https://docs.docker.com/reference/dockerfile/#benefits-of-using---link)
which allows that layer to have the same hash if the files haven't
changed regardless of if the previous instructions have changed. This
allows these layers to not have to be re-downloaded on the user's
computer if the copied files haven't changed.
### Syntax
- **Required**
- `src` - The source directory/file from the repo OR when `from:` is set
the image/stage that is specified.
- `dest` - The destination directory/file inside the working image.
- **Optional**
- `from` - The stage/image to copy from.
#### Example
```yaml
modules:
- type: copy
from: ubuntu-test
src: /test.txt
dest: /
```
### Tasks
- [x] make `from:` optional
- [x] Add README.md and module.yml
## Feature gating
Gating this feature until we release for `v0.9.0`. The plan will be to
build all features (including this one) for main branch builds. This
means that these features will be available when using the `main` image
and consequently the `use_unstable_cli:` option on the GitHub Action.
All future `v0.9.0` features will be gated as well to allow for patches
to `v0.8`.
### Tasks
- [x] Build `--all-features` on non-tagged builds
- [x] Add stages and copy features
As I was re-arranging my recipe files, I needed a way to ensure that the
order of my recipe is correct without having to read through the
generated `Containerfile`. So I added a `-d`/`--display-full-recipe` arg
to `template` that will print out all of your modules in the order
defined by following the `from-file` property.
```
$> bluebuild template --help
Generate a Containerfile from a recipe
Usage: bluebuild template [OPTIONS] [RECIPE]
Arguments:
[RECIPE]
The recipe file to create a template from
Options:
-o, --output <OUTPUT>
File to output to instead of STDOUT
--registry <REGISTRY>
The registry domain the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
--registry-namespace <REGISTRY_NAMESPACE>
The registry namespace the image will be published to.
This is used for modules that need to know where the image is being published (i.e. the signing module).
-d, --display-full-recipe
Instead of creating a Containerfile, display the full recipe after traversing all `from-file` properties.
This can be used to help debug the order you defined your recipe.
-t, --syntax-theme <SYNTAX_THEME>
Choose a theme for the syntax highlighting for the Containerfile or Yaml.
The default is `mocha-dark`.
[possible values: mocha-dark, ocean-dark, ocean-light, eighties-dark, inspired-github, solarized-dark, solarized-light]
-s, --squash
Puts the build in a `squash-stage` and COPY's the results to the final stage as one layer.
WARN: This doesn't work with the docker driver as it has been deprecated.
NOTE: Squash has a performance benefit for the newer versions of podman and buildah.
-B, --build-driver <BUILD_DRIVER>
Select which driver to use to build your image
[possible values: buildah, podman, docker]
-v, --verbose...
Increase logging verbosity
-I, --inspect-driver <INSPECT_DRIVER>
Select which driver to use to inspect images
[possible values: skopeo, podman, docker]
-q, --quiet...
Decrease logging verbosity
-h, --help
Print help (see a summary with '-h')
```
Preview of Containerfile/Dockerfile syntax highlighting:
Preview of Yaml highlighting:
Recipe files can now be put into their own directory `./recipes/`. This
directory is NEVER copied into the build so changes to a recipe will no
longer cause cache misses for builds. Here is an example of my build
changing the second to last module and only requiring the last 2 `RUN`
layers to be run again.
```
=> CACHED [stage-config 1/1] COPY ./config /config 0.0s
=> CACHED [stage-modules 1/2] COPY --from=ghcr.io/blue-build/modules:latest /modules /modules 0.0s
=> CACHED [stage-modules 2/2] COPY ./modules /modules 0.0s
=> CACHED [stage-keys 1/1] COPY cosign.pub /keys/jp-desktop-gaming.pub 0.0s
=> CACHED [stage-4 2/16] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys mkdir -p /usr/etc/pki/containers/ && cp /tmp/keys/* /usr/et 0.0s
=> CACHED [stage-bins 1/3] COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /bins/cosign 0.0s
=> CACHED [stage-bins 2/3] COPY --from=docker.io/mikefarah/yq /usr/bin/yq /bins/yq 0.0s
=> CACHED [stage-bins 3/3] COPY --from=ghcr.io/blue-build/cli:main-installer /out/bluebuild /bins/bluebuild 0.0s
=> CACHED [stage-4 3/16] RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins mkdir -p /usr/bin/ && cp /tmp/bins/* /usr/bin/ && ostree 0.0s
=> CACHED [stage-4 4/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 5/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 6/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 7/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 8/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 9/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 10/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 11/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 12/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 13/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> CACHED [stage-4 14/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind 0.0s
=> [stage-4 15/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from= 33.4s
=> [stage-4 16/16] RUN --mount=type=tmpfs,target=/var --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw --mount=type=bind,from=s 0.7s
```
Support was also added to put all build files into `./files/` instead of
`./config/`. This is an all or nothing operation, meaning if there
exists a directory of `files` then the `config` directory will be
completely ignored. Work will have to be done in
https://github.com/blue-build/modules to allow users to put their files
directly in `./files/` and not `./files/files` for the `files` module or
`./files/scripts` for the scripts module.
Support was also added to move the `./config/containerfiles/` directory
to the root of the project. Now the directories you can find in the root
of projects are:
```
files/
containerfiles/
recipes/
```
the `os_version` is defaulting to the `image_tag` inside containers and
causing our template to use latest tag
---------
Co-authored-by: Gerald Pinder <gmpinder@gmail.com>
