particle-os/particle-os-cli
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: Stages (#173)

Browse source 
## Stages

A new property (`stages`) is being added to the recipe file schema. This
property will allow users to define a list of Containerfile stages each
with their own modules. Stages can be used to compile programs, perform
parallel operations, and copy the results into the final image without
contaminating the final image.

### Module Support

Currently the only modules that work out-of-the-box are `copy`,
`script`, `files`, and `containerfile`. Other modules are dependent on
the programs installed on the image. In order to better support some of
our essential modules, a setup script is ran at the start of each stage
that is not `scratch`. This script will install `curl`, `wget`, `bash`,
and `grep` and use the package manager for the detected distributions.

At this time, the following distributions are supported:

- Debian
- Ubuntu
- Fedora
- Alpine

Contributions to increase the size of this list is
[welcome](https://github.com/blue-build/cli)!

### Syntax

- **Required**
- `from` - The full image ref (image name + tag). This will be set in
the `FROM` statement of the stage.
- `name` - The name of the stage. This is used when referencing the
stage when using the `from:` property in the `copy` module.
- `modules` - The list of modules to execute. The exact same syntax used
by the main recipe `modules:` property.
- **Optional**
- `shell` - Allows a user to pass in an array of strings that are passed
directly into the [`SHELL`
instruction](https://docs.docker.com/reference/dockerfile/#shell).

#### Example

```yaml
stages:
- name: ubuntu-test
  from: ubuntu
  modules:
  - type: files
    files:
    - usr: /usr
  - type: script
    scripts:
    - example.sh
    snippets:
    - echo "test" > /test.txt
  - type: test-module
  - type: containerfile
    containerfiles:
    - labels
    snippets:
    - RUN echo "This is a snippet"
```

### Tasks
- [x] `from-file:` - Allows the user to store their stages in a separate
file so it can be included in multiple recipes
- [x] `no-cache:` - This will be useful for stages that want to pull the
latest changes from a git repo and not have to rely on the base image
getting an update for the build to be triggered again.
- [x] Add setup script to be able to install necessary programs to run
`bluebuild` modules in stages
- [x] Check for circular dependencies and error out

## `copy` module

This is a 1-1 for the [`COPY`
instruction](https://docs.docker.com/reference/dockerfile/#copy). It has
the ability to copy files between stages, making this a very important
addition to complete functionality for the stages feature. Each use of
this "module" will become its own layer.

### Decision to use `--link`

We use the `--link`
[option](https://docs.docker.com/reference/dockerfile/#benefits-of-using---link)
which allows that layer to have the same hash if the files haven't
changed regardless of if the previous instructions have changed. This
allows these layers to not have to be re-downloaded on the user's
computer if the copied files haven't changed.

### Syntax

- **Required**
- `src` - The source directory/file from the repo OR when `from:` is set
the image/stage that is specified.
  - `dest` - The destination directory/file inside the working image.
- **Optional**
  - `from` - The stage/image to copy from.

#### Example

```yaml
modules:
- type: copy
  from: ubuntu-test
  src: /test.txt
  dest: /
```

### Tasks
- [x] make `from:` optional
- [x] Add README.md and module.yml

## Feature gating

Gating this feature until we release for `v0.9.0`. The plan will be to
build all features (including this one) for main branch builds. This
means that these features will be available when using the `main` image
and consequently the `use_unstable_cli:` option on the GitHub Action.
All future `v0.9.0` features will be gated as well to allow for patches
to `v0.8`.

### Tasks
- [x] Build `--all-features` on non-tagged builds
- [x] Add stages and copy features
This commit is contained in:
Gerald Pinder 2024-05-18 09:23:50 -04:00 • committed by GitHub
parent 8308e5b285
commit 8069006c03
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
31 changed files with 742 additions and 119 deletions
Download patch file Download diff file Expand all files Collapse all files

6
recipe/Cargo.toml
View file

@ -14,11 +14,17 @@ chrono = "0.4"
indexmap = { version = "2", features = ["serde"] }
anyhow.workspace = true
colored.workspace = true
log.workspace = true
serde.workspace = true
serde_yaml.workspace = true
serde_json.workspace = true
typed-builder.workspace = true
[features]
default = []
stages = []
copy = []
[lints]
workspace = true

4
recipe/src/lib.rs
View file

@ -2,8 +2,12 @@ pub mod akmods_info;
pub mod module;
pub mod module_ext;
pub mod recipe;
pub mod stage;
pub mod stages_ext;
pub use akmods_info::*;
pub use module::*;
pub use module_ext::*;
pub use recipe::*;
pub use stage::*;
pub use stages_ext::*;

159
recipe/src/module.rs
View file

@ -1,6 +1,8 @@
use std::{borrow::Cow, process};
use std::{borrow::Cow, path::PathBuf, process};
use anyhow::{bail, Result};
use blue_build_utils::syntax_highlighting::highlight_ser;
use colored::Colorize;
use indexmap::IndexMap;
use log::{error, trace, warn};
use serde::{Deserialize, Serialize};
@ -9,54 +11,34 @@ use typed_builder::TypedBuilder;
use crate::{AkmodsInfo, ModuleExt};
#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
pub struct Module<'a> {
#[builder(default, setter(into, strip_option))]
#[serde(rename = "type", skip_serializing_if = "Option::is_none")]
pub module_type: Option<Cow<'a, str>>,
#[builder(default, setter(into, strip_option))]
#[serde(rename = "from-file", skip_serializing_if = "Option::is_none")]
pub from_file: Option<Cow<'a, str>>,
#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder, Default)]
pub struct ModuleRequiredFields<'a> {
#[builder(default, setter(into))]
#[serde(rename = "type")]
pub module_type: Cow<'a, str>,
#[builder(default, setter(into, strip_option))]
#[serde(skip_serializing_if = "Option::is_none")]
pub source: Option<Cow<'a, str>>,
#[builder(default)]
#[serde(rename = "no-cache", default, skip_serializing_if = "is_false")]
pub no_cache: bool,
#[serde(flatten)]
#[builder(default, setter(into))]
pub config: IndexMap<String, Value>,
}
impl<'a> Module<'a> {
/// Get's any child modules.
///
/// # Errors
/// Will error if the module cannot be
/// deserialized or the user uses another
/// property alongside `from-file:`.
pub fn get_modules(modules: &[Self]) -> Result<Vec<Self>> {
let mut found_modules = vec![];
for module in modules {
found_modules.extend(
match module.from_file.as_ref() {
None => vec![module.clone()],
Some(file_name) => {
if module.module_type.is_some() || module.source.is_some() {
bail!("You cannot use the `type:` or `source:` property with `from-file:`");
}
Self::get_modules(&ModuleExt::parse_module_from_file(file_name)?.modules)?
}
}
.into_iter(),
);
}
Ok(found_modules)
}
#[allow(clippy::trivially_copy_pass_by_ref)]
const fn is_false(b: &bool) -> bool {
!*b
}
impl<'a> ModuleRequiredFields<'a> {
#[must_use]
pub fn get_module_type_list(&'a self, typ: &str, list_key: &str) -> Option<Vec<String>> {
if self.module_type.as_ref()? == typ {
if self.module_type == typ {
Some(
self.config
.get(list_key)?
@ -88,6 +70,24 @@ impl<'a> Module<'a> {
})
}
#[must_use]
#[allow(clippy::missing_const_for_fn)]
pub fn get_copy_args(&'a self) -> Option<(Option<&'a str>, &'a str, &'a str)> {
#[cfg(feature = "copy")]
{
Some((
self.config.get("from").and_then(|from| from.as_str()),
self.config.get("src")?.as_str()?,
self.config.get("dest")?.as_str()?,
))
}
#[cfg(not(feature = "copy"))]
{
None
}
}
#[must_use]
pub fn generate_akmods_info(&'a self, os_version: &u64) -> AkmodsInfo {
#[derive(Debug, Copy, Clone)]
@ -161,3 +161,90 @@ impl<'a> Module<'a> {
.build()
}
}
#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder, Default)]
pub struct Module<'a> {
#[builder(default, setter(strip_option))]
#[serde(flatten, skip_serializing_if = "Option::is_none")]
pub required_fields: Option<ModuleRequiredFields<'a>>,
#[builder(default, setter(into, strip_option))]
#[serde(rename = "from-file", skip_serializing_if = "Option::is_none")]
pub from_file: Option<Cow<'a, str>>,
}
impl<'a> Module<'a> {
/// Get's any child modules.
///
/// # Errors
/// Will error if the module cannot be
/// deserialized or the user uses another
/// property alongside `from-file:`.
pub fn get_modules(
modules: &[Self],
traversed_files: Option<Vec<PathBuf>>,
) -> Result<Vec<Self>> {
let mut found_modules = vec![];
let traversed_files = traversed_files.unwrap_or_default();
for module in modules {
found_modules.extend(
match &module {
Module {
required_fields: Some(_),
from_file: None,
} => vec![module.clone()],
Module {
required_fields: None,
from_file: Some(file_name),
} => {
let file_name = PathBuf::from(file_name.as_ref());
if traversed_files.contains(&file_name) {
bail!(
"{} File {} has already been parsed:\n{traversed_files:?}",
"Circular dependency detected!".bright_red(),
file_name.display().to_string().bold(),
);
}
let mut traversed_files = traversed_files.clone();
traversed_files.push(file_name.clone());
Self::get_modules(
&ModuleExt::parse(&file_name)?.modules,
Some(traversed_files),
)?
}
_ => {
let from_example = Self::builder().from_file("test.yml").build();
let module_example = Self::example();
bail!(
"Improper format for module. Must be in the format like:\n{}\n{}\n\n{}",
highlight_ser(&module_example, "yaml", None)?,
"or".bold(),
highlight_ser(&from_example, "yaml", None)?
);
}
}
.into_iter(),
);
}
Ok(found_modules)
}
#[must_use]
pub fn example() -> Self {
Self::builder()
.required_fields(
ModuleRequiredFields::builder()
.module_type("module-name")
.config(IndexMap::from_iter([
("module".to_string(), Value::String("config".to_string())),
("goes".to_string(), Value::String("here".to_string())),
]))
.build(),
)
.build()
}
}

20
recipe/src/module_ext.rs
View file

@ -15,12 +15,12 @@ pub struct ModuleExt<'a> {
}
impl ModuleExt<'_> {
/// # Parse a module file returning a [`ModuleExt`]
/// Parse a module file returning a [`ModuleExt`]
///
/// # Errors
/// Can return an `anyhow` Error if the file cannot be read or deserialized
/// into a [`ModuleExt`]
pub fn parse_module_from_file(file_name: &str) -> Result<Self> {
pub fn parse(file_name: &Path) -> Result<Self> {
let legacy_path = Path::new(CONFIG_PATH);
let recipe_path = Path::new(RECIPE_PATH);
@ -52,8 +52,20 @@ impl ModuleExt<'_> {
self.modules
.iter()
.filter(|module| module.module_type.as_ref().is_some_and(|t| t == "akmods"))
.map(|module| module.generate_akmods_info(os_version))
.filter(|module| {
module
.required_fields
.as_ref()
.is_some_and(|rf| rf.module_type == "akmods")
})
.filter_map(|module| {
Some(
module
.required_fields
.as_ref()?
.generate_akmods_info(os_version),
)
})
.filter(|image| seen.insert(image.clone()))
.collect()
}

22
recipe/src/recipe.rs
View file

@ -12,7 +12,7 @@ use serde::{Deserialize, Serialize};
use serde_yaml::Value;
use typed_builder::TypedBuilder;
use crate::{Module, ModuleExt};
use crate::{Module, ModuleExt, StagesExt};
/// The build recipe.
///
@ -60,6 +60,14 @@ pub struct Recipe<'a> {
#[builder(default, setter(into, strip_option))]
pub alt_tags: Option<Vec<Cow<'a, str>>>,
/// The stages extension of the recipe.
///
/// This hold the list of stages that can
/// be used to build software outside of
/// the final build image.
#[serde(flatten, skip_serializing_if = "Option::is_none")]
pub stages_ext: Option<StagesExt<'a>>,
/// The modules extension of the recipe.
///
/// This holds the list of modules to be run on the image.
@ -207,7 +215,17 @@ impl<'a> Recipe<'a> {
let mut recipe = serde_yaml::from_str::<Recipe>(&file)
.map_err(blue_build_utils::serde_yaml_err(&file))?;
recipe.modules_ext.modules = Module::get_modules(&recipe.modules_ext.modules)?.into();
recipe.modules_ext.modules = Module::get_modules(&recipe.modules_ext.modules, None)?.into();
#[cfg(feature = "stages")]
if let Some(ref mut stages_ext) = recipe.stages_ext {
stages_ext.stages = crate::Stage::get_stages(&stages_ext.stages, None)?.into();
}
#[cfg(not(feature = "stages"))]
{
recipe.stages_ext = None;
}
Ok(recipe)
}

159
recipe/src/stage.rs Normal file
View file

@ -0,0 +1,159 @@
use std::{borrow::Cow, path::PathBuf};
use anyhow::{bail, Result};
use blue_build_utils::syntax_highlighting::highlight_ser;
use colored::Colorize;
use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
use crate::{Module, ModuleExt, StagesExt};
/// Contains the required fields for a stage.
#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
pub struct StageRequiredFields<'a> {
/// The name of the stage.
///
/// This can then be referenced in the `copy`
/// module using the `from:` property.
#[builder(setter(into))]
pub name: Cow<'a, str>,
/// The base image of the stage.
///
/// This is set directly in a `FROM` instruction.
#[builder(setter(into))]
pub from: Cow<'a, str>,
/// The shell to use in the stage.
#[builder(default, setter(into, strip_option))]
#[serde(skip_serializing_if = "Option::is_none")]
pub shell: Option<Vec<Cow<'a, str>>>,
/// The modules extension for the stage
#[serde(flatten)]
pub modules_ext: ModuleExt<'a>,
}
/// Corresponds to a stage in a Containerfile
///
/// A stage has its own list of modules to run which
/// allows the user to reuse the modules thats provided to the main build.
#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
pub struct Stage<'a> {
/// The requied fields for a stage.
#[builder(default, setter(strip_option))]
#[serde(flatten, skip_serializing_if = "Option::is_none")]
pub required_fields: Option<StageRequiredFields<'a>>,
/// A reference to another recipe file containing
/// one or more stages.
///
/// An imported recipe file can contain just a single stage like:
///
/// ```yaml
/// name: blue-build
/// image: rust
/// modules:
/// - type: containerfile
/// snippets:
/// - |
/// RUN cargo install blue-build --debug --all-features --target x86_64-unknown-linux-gnu \
/// && mkdir -p /out/ \
/// && mv $CARGO_HOME/bin/bluebuild /out/bluebuild
/// ```
///
/// Or it can contain multiple stages:
///
/// ```yaml
/// stages:
/// - name: blue-build
/// image: rust
/// modules:
/// - type: containerfile
/// snippets:
/// - |
/// RUN cargo install blue-build --debug --all-features --target x86_64-unknown-linux-gnu \
/// && mkdir -p /out/ \
/// && mv $CARGO_HOME/bin/bluebuild /out/bluebuild
/// - name: hello-world
/// image: alpine
/// modules:
/// - type: script
/// snippets:
/// - echo "Hello World!"
/// ```
#[builder(default, setter(into, strip_option))]
#[serde(rename = "from-file", skip_serializing_if = "Option::is_none")]
pub from_file: Option<Cow<'a, str>>,
}
impl<'a> Stage<'a> {
/// Get's any child stages.
///
/// # Errors
/// Will error if the stage cannot be
/// deserialized or the user uses another
/// property alongside `from-file:`.
pub fn get_stages(stages: &[Self], traversed_files: Option<Vec<PathBuf>>) -> Result<Vec<Self>> {
let mut found_stages = vec![];
let traversed_files = traversed_files.unwrap_or_default();
for stage in stages {
found_stages.extend(
match stage {
Stage {
required_fields: Some(_),
from_file: None,
} => vec![stage.clone()],
Stage {
required_fields: None,
from_file: Some(file_name),
} => {
let file_name = PathBuf::from(file_name.as_ref());
if traversed_files.contains(&file_name) {
bail!(
"{} File {} has already been parsed:\n{traversed_files:?}",
"Circular dependency detected!".bright_red(),
file_name.display().to_string().bold(),
);
}
let mut tf = traversed_files.clone();
tf.push(file_name.clone());
Self::get_stages(&StagesExt::parse(&file_name)?.stages, Some(tf))?
}
_ => {
let from_example = Stage::builder().from_file("path/to/stage.yml").build();
let stage_example = Self::example();
bail!(
"Improper format for stage. Must be in the format like:\n{}\n{}\n\n{}",
highlight_ser(&stage_example, "yaml", None)?,
"or".bold(),
highlight_ser(&from_example, "yaml", None)?
);
}
}
.into_iter(),
);
}
Ok(found_stages)
}
#[must_use]
pub fn example() -> Self {
Stage::builder()
.required_fields(
StageRequiredFields::builder()
.name("stage-name")
.from("build/image:here")
.modules_ext(
ModuleExt::builder()
.modules(vec![Module::example()])
.build(),
)
.build(),
)
.build()
}
}

61
recipe/src/stages_ext.rs Normal file
View file

@ -0,0 +1,61 @@
use std::{borrow::Cow, fs, path::Path};
use anyhow::{Context, Result};
use blue_build_utils::constants::{CONFIG_PATH, RECIPE_PATH};
use log::warn;
use serde::{Deserialize, Serialize};
use typed_builder::TypedBuilder;
use crate::{Module, Stage};
#[derive(Default, Serialize, Clone, Deserialize, Debug, TypedBuilder)]
pub struct StagesExt<'a> {
#[builder(default, setter(into))]
pub stages: Cow<'a, [Stage<'a>]>,
}
impl<'a> StagesExt<'a> {
/// Parse a module file returning a [`StagesExt`]
///
/// # Errors
/// Can return an `anyhow` Error if the file cannot be read or deserialized
/// into a [`StagesExt`]
pub fn parse(file_name: &Path) -> Result<Self> {
let legacy_path = Path::new(CONFIG_PATH);
let recipe_path = Path::new(RECIPE_PATH);
let file_path = if recipe_path.exists() && recipe_path.is_dir() {
recipe_path.join(file_name)
} else {
warn!("Use of {CONFIG_PATH} for recipes is deprecated, please move your recipe files into {RECIPE_PATH}");
legacy_path.join(file_name)
};
let file = fs::read_to_string(&file_path)
.context(format!("Failed to open {}", file_path.display()))?;
serde_yaml::from_str::<Self>(&file).map_or_else(
|_| -> Result<Self> {
let mut stage = serde_yaml::from_str::<Stage>(&file)
.map_err(blue_build_utils::serde_yaml_err(&file))?;
if let Some(ref mut rf) = stage.required_fields {
rf.modules_ext.modules =
Module::get_modules(&rf.modules_ext.modules, None)?.into();
}
Ok(Self::builder().stages(vec![stage]).build())
},
|mut stages_ext| -> Result<Self> {
let mut stages: Vec<Stage> =
stages_ext.stages.iter().map(ToOwned::to_owned).collect();
for stage in &mut stages {
if let Some(ref mut rf) = stage.required_fields {
rf.modules_ext.modules =
Module::get_modules(&rf.modules_ext.modules, None)?.into();
}
}
stages_ext.stages = stages.into();
Ok(stages_ext)
},
)
}
}