Similar to rd.break for dracut this allows a user to specify:
- --break or --break=*
- to get a shell before each stage is run
- --break=stage.name
- to get a shell each time the stage with that name is run
- example: --break=org.osbuild.copy
- --break=stage.id
- to get a shell each time the stage with that ID is run
- get the ID for the stages for your manifest by running
osbuild on the manifest with --inspect
- example: --break=dc6e3a66fef3ebe7c815eb24d348215b9e5e2ed0cd808c15ebbe85fc73181a86
and get a bash shell where they can inspect the environment to debug
and develop OSBuild stages.
To workaround the issue that inside macOS containers the ownership
cannot be preserved we introduce a new environment that can be
used to forcefully relax the use of `cp -a`.
I did it via an environment instead of a commandline option mostly
because `github.com/osbuild/images/osbuild:RunOBuild()` already has
`extraEnv` option.
When developing or rebuilding manifests a lot it is common to want to
checkpoint everything to the store. It seems we all have small shell
scripts hanging around for this.
Let `--checkpoint` take a shell-like glob such as `--checkpoint="*"` to
checkpoint everything.
Note that there's a behavioral change here; previously `osbuild
--checkpoint=a` would error if that specific checkpoint wasn't found.
Now `osbuild` will only error if nothing was selected by the passed
globs.
Integrate the recently added file system cache `FsCache` into our
object store `ObjectStore`. NB: This changes the semantics of it:
previously a call to `ObjectStore.commit` resulted in the object
being in the cache (i/o errors aside). But `FsCache.store`, which
is now the backing store for objects, will only commit objects if
there is enough space left. Thus we cannot rely that objects are
present for reading after a call to `FsCache.store`. To cope with
this we now always copy the object into the cache, even for cases
where we previously moved it: for the case where commit is called
with `object_id` matching `Object.id`, which is the case for when
`commit` is called for last stage in the pipeline. We could keep
this optimization but then we would have to special case it and
not call `commit` for these cases but only after we exported all
objects; or in other words, after we are sure we will never read
from any committed object again. The extra complexity seems not
worth it for the little gain of the optimization.
Convert all the tests for the new semantic and also remove a lot
of them that make no sense under this new paradigm.
Add a new command line option `--cache-max-size` which will set
the maximum size of the cache, if specified.
Now that metadata is stored and can be accessed via `Object.meta`,
read it from the built or stored objects when serializing the
result in the `format.output` functions.
Move the reporting of results into the try-cache and ObjectStore
context. This prepares to use the store during the `fmt.output`
call and possible reporting of store cache usages.
Make sure all --help output is consistent. In this particular case,
each line should consistently start with a lower-case character and
avoid a leading `the`.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
This adds a `osbuild --version` command that prints the current osbuild
version in use. Allows users to confirm their osbuild is up to date
enough to use newer features.
Introduce two new command line arguments, which can be used to
specify which monitor class to use (`--monitor`) and what file
descriptor to use for monitoring (`--monitor-fd`). The latter
defaults to standard out. The monitor class, if not specified,
is depended on the `--json` argument.
Remove the constraint that either checkpoints or the output directory
has to be supplied on the command line. Now that we have `--export`
and on-demand building it is perfectly fine to supply neither an output
directory nor checkpoints and implicitly no --export` which corresponds
to a download only mode.
Use the new Manifest.depsolve function to only build the pipelines that
were explicitly requested and their dependencies, taking into account
what is already present in the store.
Since now not all pipeline will be built, there wont be a result entry
for all the pipelines, thus the format version 2 result formatting was
changed to not require the pipeline to be present in result set.
When building a version 1 manifest, the assembler would always be
exported, even when not requested via the `--export` command line
option. This was done for backwards compatibility so to not break
tools relying on that behavior. The problem is that support for
this uses a completely different code path and might also now be
confusing behavior. Thus remove the implicit and really only ever
export what was explicitly requested by the caller.
Instead of building everything and then failing if an export is
invalid and could not be found, resolve the exports early and
also ensure that if `--export` is given, `--output-directory`
is present too.
Port sources to also use the host services infrastructure that is
used by inputs, devices and mounts. Sources are a bit different
from the other services that they don't run for the duration of
the stage but are run before anything is built. By using the same
infrastructure we re-use the process management and inter process
communcation. Additionally, this will forward all messages from
sources to the existing monitoring framework.
Adapt all existing sources and tests.
This new command line argument can be used to explicitly request
the exporting of built artifcats. They will be store in the
output directory in a subdirectory with the name or id that was
requested.
Now that meta.Index provides a way to detect the format given the
input data, use that method instead of the hard-coded use of the
version 1 format via the input. This should make the main entry
point format independent.
Add a new `info` property that holds the `meta.ModuleInfo` info
for the stage. This gives each instance of a stage access to
meta (or class) information about it, i.e. its schema, docs but,
more importantly, also its name and path to the executable.
Thefore the `name` property is coverted into a transient property
which access the `name` member of `info`.
Change the `formats/v1` load mechanism to carry a new `index`
argument which is used to load the `ModuleInfo` for each stage.
Adapt all tests to load the info as well when creating stages.
Instead of having build pipelines nested within the pipeline it is
the build pipeline for, the nested structure is transferred into a
flat list of pipelines. As a result the recursion is gone and all
the pipelines and trees are build one after the other. This is now
possible since floating objects are kept alive by the store itself
and all trees that are being built are transparently via them.
The immediate result dictionary changed accordingly. To keep the
JSON output of osbuild the same, the result is now routed through
a format specific converter.
Additionally, the v1 format module gained a function to retrieve
the global tree_id and output_id. With the new models those global
ids will go away eventually and thus need to go through the format
specific code.
Instead of passing the store directory to Pipeline.run, pass an
already initialized ObjectStore object. This binds the lifetime
of the store and its (temporary) objects to the run of osbuild
not the run of the pipeline.
This prepares re-using the stores with multiple (non-nested)
pipelines.
Instead of a pipeline, describe now takes a Manifest instance.
The reason is that a manifest fully describes the build, which
includes the sources. Now that the describe function takes the
manifest, the sources can be included as well.
Adapt the tests to refelect that change.
The 'Manifest' class represents what to build and the necessary
sources to do so. For now thus it is just a combination of the
pipeline the source options.
The description of a pipeline is format dependent and thus needs
to be located at the specific format module.
Temporarily remove two tests; they should be added back to a format
specific test suit.
Instead of having the pipeline and the source option as separate
arguments, the load function now takes the full manifest, which
has those two items combined.
Instead of importing the load, load_build functions into the osbuild
namespace and using it via that, use the load function via the module
that provides them, i.e. the formats.v1 module.
The validation of the manifest descritpion is eo ipso format
specific and thus belongs into the format specific module.
Adapt all usages throughout the codebase to directly use the
version 1 specific function.
This was deprecated in favor of always having the source in the
manifest. Remove the command line option and the corresponding
code that would override the sources definitions.
Update the docs accordingly.
osbuild_cli() sometimes returned an exit code, but at the end called
sys.exit() directly. The idea was probably to always return the code
with which the executable should exit.
Make this consistent and call sys.exit() in __main__.py, with the value
returned by osbuild_cli().
Change the default of libdir to /usr/lib/osbuild and
remove redundant logic. Additionally, change how the
python package is detected.
Instead of checking if libdir is None, check if
/usr/lib/osbuild is empty - i.e. if the user has specified
a different directory than the default.
Introduce the concept of pipeline monitoring: A new monitor class is
passed to the pipeline.run() function. The main idea is to separate
the monitoring from the code that builds pipeline. Through the build
process various methods will be called on that object, representing
the different steps and their targets during the build process. This
can be used to fully stream the output of the various stages or just
indicate the start and finish of the individual stages.
This replaces the 'interactive' argument throughout the pipeline
code. The old interactive behavior is replicated via the new
`LogMonitor` class that logs the beginning of stages/assembler,
but also streams all the output of them to stdout.
The non-interactive behavior of not reporting anything is done by
using the `NullMonitor` class, which in turn outputs nothing.
If the user does not specify an output directory or checkpoints
to osbuild, exit successfully without building.
Previously, if a user did not include an output directory or
checkpoints, it would build the manifest and throw out the result.
Returning early will be clearer to the user and avoid wasting work.
Instead of having a another indirection via `main_cli`, directly
use `osbuild_cli` in as main function in `__main__.py`. Also use
that in as the entry point for the generated `osbuild` executable.
Change `osbuild_cli` to be self-contained, i.e. it directly uses
`sys.argv` and `sys.exit`.
The way secrets work has been changed via commit 372b117: instead
of passing them in via the command line, the information how to
obtain secrets are encoded along the sources themselves.
The only stage that still has support for the old style way is the
deprecated org.osbuild.dnf stage, which might be removed in the
near future.
Change the assembler-commit to be conditional on checkpoints, just like
we already do for stages. This means, assembler output is not
automatically committed, but only if you requested so via a checkpoint.
With this in place we can start sharing caches in osbuild-composer. The
only thing in the cache will be sources as well as checkpointed stages.
We can start checkpointing known pipelines and thus make use of the
cache. Furthermore, we can cache sources, as long as we do not fetch an
unbound set of RPMs. However, our RPM set is currently static, so this
should not be an issue. Nevertheless, it is up to Composer to decide
when to enable the cache.
Using `[]` as default value for arguments makes `pylint` complain. The
reason is that it creates an array statically at the time the function
is parsed, rather than dynamically on invocation of the function. This
means, when you append to this array, you change the global instance and
every further invocation of that function works on this modified array.
While our use-cases are safe, this is indeed a common pitfall. Lets
avoid using this and resort to `None` instead.
This silences a lot of warnings from pylint about "dangerous use of []".
Drop the --build-env command-line argument. It is not used by anything.
Furthermore, our manifests now allow embedding build-environments, so
there is little reason to continue supporting this.
In case `--libdir` is not specified on the command line, and thus
`args.libdir` is `None`, pass the standard `/usr/lib/osbuild` path
to the meta.Index constructor. Otherwise no schema information can
be found.
This option will print the manifest in JSON, including all the ids,
to stdout. It will not build the pipeline, but the input manifest
will be validated and if that fails the validation result will be
return in JSON.
Validate the options of stages and assembler of the pipeline
before running it. A validation failure will abort the run.
Errors are printed in human readable unless `--json` is passed;
For each error a human readable message together with a path
to the object with the error is given. The syntax of the path
is such it can be used via the `jq` command to select the item.
Split off the argument parser as well as the manifest parser into
helper functions. Drop the pylint hints from the main function now that
it is considerably smaller.
