debian-forge/NEWS.md

OSBuild - Build-Pipelines for Operating System Artifacts

CHANGES WITH 16:

• Support for ignition: a new org.osbuild.ignition stage has been added together with a new option in the org.osbuild.grub2 stage, called ignition. When used together, a new variable for the kernel command line, called $ignition_firstboot, will exist that will trigger the run of ignition on the first boot of an image.

• A new org.osbuild.copy stage was added that can be used to copy files and directories from an archive to the file system tree. The archive will be fetched via the existing org.osbuild.files source.

• The result of the assembler will now not automatically be committed to the store anymore, but only when requested via --checkpoint; very much like it is already the case for the stages.

• The tree_id and output_id identifiers have been dropped from the osbuild result. This reflects the policy that the internals of the store are private. The --output-directory command line option can be used to obtain the final artifact instead.

• The org.osbuild.files and org.osbuild.ostree sources have been properly documented and the JSON schema for their options was added. osbuild gained support for the validation of the source options in the manifest. As a result the whole manifest is now validated.

• The GPG signature verification of RPMs in the org.osbuild.rpm stage is now optional and opt-in. The GPG key can now also be provided per package.

• The org.osbuild.ostree gained support for pre-populating /var like it is done by anaconda. Also its rootfs option is not required anymore, since in specific cases, like when ignition is being used, the root file system is identified by its label only.

• The common term for Stages, Assemblers and Sources shall from now on be "module". Rename the StageInfo class to ModuleInfo.

• Small bug fixes, including to the org.osbuild.users stage, that now allows the creation of users with uid/gids that are 0 and descriptions and passwords that are empty. The org.osbuild.files source got a bug fix to allow the use of URL format but without specifying the secrets key.

• Numerous small fixes throughout the source code to fix all pylint warnings. These are now also enabled for the source checks.

• Lots of improvements to the test infrastructure and the CI.

Contributions from: Christian Kellner, David Rheinsberg, Jacob Kozol, Major Hayden, Tom Gundersen

— Berlin, 2020-06-04

CHANGES WITH 15:

• A new assembler, org.osbuild.oci-archive, that will turn a tree into an Open Container Initiative Image compliant archive. These archives can be used to run containers via e.g. podman.

• Support for client side certificates to download content from the Red Hat servers: the org.osbuild.files source got support for reading entitlements and pass those certificates along when fetching content, i.e. RPMs.

• A new ManifestPreProcessor (MPP) was added as a new tool located in tools/mpp-depsolve.py. Currently, it can take an existing manifest and dep-solve packages specified via a new mpp-depsolve option in existing org.osbuild.rpm stages. This is now used to generate Fedora 32 based test pipelines.

• The org.osbuild.ostree.commit assembler gained an option to produce a tarball archive instead of emitting the plain OSTree repository.

• Schema validation is now done with the draft 4 validator, and works therefore with pyhthon-jsonschema 2.6.

• The tree_id and output_id fields got dropped from the resulting JSON when inspecting pipelines via osbuild --inspect.

• The --build-env option has been dropped from the command line interface. It was deprecated and not used anymore.

• Tests have been converted to not rely on tree_id and output_id anymore, as they are deprecated and will be removed in the future.

• Lots of other improvements to the test infrastructure and the CI.

• And finally for something meta: this file has been re-formatted to be proper markdown.

Contributions from: Christian Kellner, David Rheinsberg, Jacob Kozol, Major Hayden

— Berlin, 2020-05-20

CHANGES WITH 14:

• Schema validation: The osbuild python library gained support for retrieving the metadata of modules and schema validation. This is being used on each invocation of osbuild in order to validate the manifest. Should the validation fail the build is aborted and validation errors are returned, either in human readable form or in JSON, if --json was specified.

• A --inspect command line option was added for osbuild. Instead of attempting to build the pipeline, the manifest will be printed to stdout in JSON form, including all the calulcated identifiers of stages, the assembler and the tree_id and output_id of the pipeline (and build pipelines). Schema validation will be done and errors will be reported.

• Internally, the buildroot class now uses PYTHONPATH to point to the osbuild module instead of the symlinks or bind-mounts in the individual modules.

• Fixes to the CI and many cleanups to the schemata, sample and test pipelines as a result of the schema validation work.

Contributions from: Christian Kellner, David Rheinsberg, Ondřej Budai

— Berlin, 2020-05-06

CHANGES WITH 13:

• Stage org.osbuild.yum has been dropped. It has been deprecated for some time and org.osbuild.rpm provides a better alternative.

• XZ compression now utilizes all available CPU cores. This affects all stages and assemblers that support XZ compression. It should decrease compression times considerably.

• org.osbuild.grub2 now supports referring to file-systems via a label additionally to a UUID. This affects all places where an existing file-system is referred to. Disk creation still requires a UUID to be provided. org.osbuild.fstab gained similar support.

• RHEL-8.3 is now supported as host system.

• The 'libdir' layout in /usr/lib/osbuild/ has been simplified. Distributions are no longer required to create mount anchors during installation. Instead, all modules (stages, assemblers, sources, and runners) can be copied verbatim from the source tree.

• org.osbuild.grub2 now correctly pads grubenv files to 1024 bytes. This was not done correctly, previously, and caused other parsers to fail.

• The containerization via systemd-nspawn was adjusted to support running in a container. With sufficient privileges, you can now run osbuild pipelines from within a container.

Contributions from: Christian Kellner, David Rheinsberg, Major Hayden

— Berlin, 2020-04-29

CHANGES WITH 12:

• The qemu assembler now supports the VHDX image format. This is the preferred format for AWS targets, so it is a natural fit for our assemblers.

• The grub2 stage now disables the legacy compatibility by default. You have to explicitly enable it in the stage options if you require it.

• Additionally, the grub2 stage now also has a uefi.install option to control whether it installs the UEFI configuration from the build tree into the target tree. Furthermore, a new option called write_defaults controls whether default options are written to /etc (enabled by default).

• The dnf stage was removed. The rpm stage fully replaces all its functionality.

• The fedora27 runner is no longer supported. Fedora 30 is the minimum required host version for Fedora systems.

• Add OSTree integration. This includes multiple stages and sources which allow to export osbuild trees as ostree commits, or import ostree commits into an osbuild pipeline:

  • org.osbuild.rpm-ostree: This stage uses rpm-ostree compose to post-process a tree and prepare it for committing to ostree.

  • org.osbuild.ostree.commit: A new assembler that takes a tree that conforms to the ostree layout and turns it into an ostree commit.

  • org.osbuild.ostree: A new source that provides external ostree commits to a pipeline.

  • org.osbuild.ostree: A new stage that takes an ostree commit and prepares the working directory with its content.

• The osbuild binary now has an --output-directory=DIR argument which allows to specify a directory where to put the output of the pipeline assembler. This is optional for now, but will be made mandatory in the future.

• A new stage called org.osbuild.first-boot allows to control the execution of scripts at the first bootup of the generated images.

Contributions from: Christian Kellner, David Rheinsberg, Major Hayden, Ondřej Budai, Tom Gundersen

— Berlin, 2020-04-15

CHANGES WITH 11:

• Drop support for legacy input: passing in non-manifest style pipelines is now not supported anymore.

• Support for specifying an UUID for partitions when using the GPT partition layout was added to the org.osbuild.qemu assembler.

• Fix a crash in the case the assembler failed, which was caused by cleanup up the object while the object was still being written to.

• Delay the cleanup of the build tree to after the error checking since in the error case there is nothing to clean up and trying to do so will lead to crash.

• objectstore.Object now directly cleans its working tree up, in contrast to relying on the implicit cleanup of TemporaryDirectory. One advantage of this is that the custom cleanup code can handle immutable directories, which Python 3 fails to clean up.

• Drop custom os-release creation from the RHEL 8.2 runner. The issue that made this neccessary got fixed upstream.

• Ensure the build tree is always being built even if there are no stages specified.

• spec file: Do no generate dependencies for the internal files and add NEWS.md to the documentation section.

• The Fedora 30 based aarch64 example was fixed and now builds again.

Contributions from: Christian Kellner, David Rheinsberg, Lars Karlitski, Major Hayden, Martin Sehnoutka, Ondřej Budai

— Berlin, 2020-04-01

CHANGES WITH 10:

• A new man-page osbuild-manifest(5) is available, which describes the input format of the JSON manifest that osbuild expects.

• Man-pages can now be built via make man. This supports SRCDIR and BUILDDIR variables to build out-of-tree.

• Temporary objects in the object-store are now created in .osbuild/tmp/, rather than in the top-level directory. This should help cleaning up temporary objects after a crash. If no osbuild process is running, the tmp/ subdirectory should not exist.

• The final stage of a build-pipeline is no longer automatically committed. You must pass checkpoints via --checkpoint to commit anything to the store.

• Improve curl timeout handling. This should improve osbuild behavior with slow or bad mirrors and make sure operations are retried correctly, or time-out if no progress is made.

Contributions from: Christian Kellner, David Rheinsberg, Lars Karlitski, Major Hayden, Tom Gundersen

— Berlin, 2020-03-18

CHANGES WITH 9:

• The last pipeline stage is no longer automatically committed to the store. This used to be a special case to make things work, but it has now been properly fixed. From now on, if you want a stage committed to the store, you need to pass a --checkpoint option for the stage.

• The runner for the host system is now auto-detected. The runners/org.osbuild.default symlink is now longer required (nor supported).

• A generic runner named org.osbuild.linux was added. This runner uses the default value of ID in /etc/os-release. That is, if the local OS cannot be detected, or if no os-release file is provided, this is the fallback runner that is used. This runner only performs the bare minimum of initialization. It is enough to run the most basic stages on all systems we tested.

• On Archlinux, the generic runner will now be used.

• A new runner for RHEL-8.1 is available.

• The JSON input to osbuild is now a monolithic manifest format which contains all build information. For now, this means the input manifest can contain a pipeline: key with the pipeline definition, as well as a sources: key with external source definitions previously passed via --sources. The old input format is still supported, but will be dropped in the next release.

• The osbuild sources now come with a man-page osbuild(1). Further pages will follow in the future.

Contributions from: Christian Kellner, David Rheinsberg, Jacob Kozol, Lars Karlitski, Major Hayden, Martin Sehnoutka, Tom Gundersen

— Berlin, 2020-03-05

CHANGES BEFORE 9:

• Initial implementation of 'osbuild'.

Contributions from: Brian C. Lane, Christian Kellner, David Rheinsberg, Jacob Kozol, Lars Karlitski, Major Hayden, Martin Sehnoutka, Ondřej Budai, Sehny, Tom Gundersen, Tomas Tomecek, Will Woods