An example that takes an ostree commit and creates a qcow2 image
that is bootable. NB: the commit ids and the corresponding source
will need to be adjusted after having run the pipeline that will
create the ostree commit.
This is a Fedora 31 based example that results in a ostree repo
with a ostree commit. The latter was composed from the file system
tree that was created via the given osbuild stages.
The file `/etc/defaults/grub` sets the defaults that are used by
grub2-mkconfig to (re-)generate the grub config (grub.cfg). This
command is not run by any scripts but by the user directly. On
modern installations (without the grubby-deprecated package)
the kernel is configured via Bootloader Specification snippets
and thus the grub config should not need to be touched at all
under normal circumstances. In the new future the grub2-mkconfig
will be updated to not require GRUB_ENABLE_BLSCFG which should
make the existence `/etc/defaults/grub` even more superfluous.
Additionally, in the future, some images might not contain
the grub2 packages at all.
Add support for copying EFI data from the build root. If
`uefi.install` is set to `true`, `BOOT` and `uefi.vendor`
directories will be copied from the build root. This is
useful for example on OSTree based systems where boot/efi/EFI
is not being populated by an RPM package; but it can be used
also on other systems where it is not desirable to deliver
the EFI data via packages.
Add /boot to be mounted from the build tree into the build root,
because the EFI binaries for grub are stored in there and for
ostree grub2 support those need to be copied too.
Deploying an OSTree commit leads to creation of hardlinks from
the repository to the check out. These will have the correct
SELinux labels, since the files in the repository will have the
correct SELinux labels[1]. But new files are generated in '/etc'
of the new deployment, due to the 3-way configuration merge[2].
Also a new kernel, initramfs and the corresponding Bootloader
Specification entries are created in the global '/boot' dir.
In theory, ostree will set the correct SELinux labels by loading
the SELinux config from the deployment and then setting the
correct security contexts via ostree_sepolicy_setfscreatecon().
But it does so conditionally on is_selinux_enabled(2)[3], which
in our container is FALSE Therefore we have to do the same dance
as ostree does, at least for now, and manually re-label the
affected paths.
[1] Assuming they had the correct label when the commit was made
[2] https://ostree.readthedocs.io/en/latest/manual/deployment/
[3] via ostree_sepolicy_setfscreatecon in ostree-sepolicy.c
line 640 of commit 2c1658538f8fde5813e95e7408d65662a489be91
Add the ability to specify one ore more remotes for the system
repository. The required options for a single remote are its
`name` and the `url`. Optionally one or more branch can be passed
via `branches`. GPG keys can be given via `gpgkeys`; if none are
specified, no gpg verification will be done.
The list of mount points is changed from a list of strings to a
list of objects containing `path` and an optional `mode` value.
The latter can be used to set the mode of the mount point that
will be created in the file system tree. It defaults to 0755,
or 493 in decimal, because JSON does not support octal values.
Instead of taking a raw string for the root file system kernel
option, convert the `rootfs` option to an object that must have
either `uuid` or `label` set. This will then be translated into
the proper kernel command line argument.
Add a new optional `ref` option. When set, a references for the
commit that was pulled with the value of `ref` will be created
in the system ostree repo. It will also be used when deploying
the commit and thus will be set as the origin for it. This is
necessary for updates to work.
Initializes the filesystem via ostree and then pulls a specified
commit and deploys that. Options are `commit`, which is the id
of the commit to pull and deploy, `osname`, which is the name
to be used for the operating system root. The `roofs` together
with `kernel_opts` options are used to build the kernel command
line for the deployment.
Additionally, a `mounts` parameter can be supplied that indicates
file system boundaries. This is needed because ostree uses a hard
link farm, which must not link across said file system boundaries.
Instead of verifying the gpg signature when pull from the actual
remote source into the local cache, verify the commit when it is
being pulled from the local cache into the output directory. This
ensures that the signatures are checked against the provided keys
even when the commit was already in the cache and at that time
the key might have been different.
NB: ostree expects the signature to be present on the remote at
the *target* repository, i.e. in our case the output repository.
The keys are therefore attached to a temporary remote that is
created at the output repository with the same name/id that is
used for the actual remote.
Add a new `gpgkeys` option that, if set, must contain a list of
public keys. These keys will then be used by ostree to verify
signed commits when pulling from the remote. If the `gpgkeys`
option is missing, no verification will be attempted.
This source can be used to fetch ostree commits. The commits are
accessed via their commit is. The only option currently is `url`,
given for each commit, that will be used as the location of the
remote. A cache repository, that will be created if necessary,
acts as an intermediary, so remotes will be added with `name` as
the identifier to it and commits are pulled into that. In the
output directory another repository will be created as 'repo' and
the requested commit pulled into that from the cache repository via
a local pull.
Add a new assembler that takes a file system tree that is already
conforming to the ostree system layout[1], creates a new repository
in archive mode and commits the file system tree to it. Afterwards,
a reference is created with the value supplied in `ref`.
The repository is located at the `/repo` directory and additional
metadata is /compose.json which contain the compose information.
Currently uses rpm-ostree to do the actual committing. In the future
this might change to plain ostree.
[1] https://ostree.readthedocs.io/en/stable/manual/adapting-existing/
A stage that uses rpm-ostree compose to do post-processing of the
tree. Among other things the main steps are:
- moves /etc to /usr/etc
- move /boot to /usr/lib/ostree-boot
- potentially moving /var/lib/rpm
- re-creates the initramfs (dracut)
- stores the treefile at /usr/share/rpm-ostree/treefile.json
- adds altfiles module to nss
- Recompiles SELinux policy (semodule -nB)
- Migrates /usr/etc/{passwd, group} to /usr/lib/
- Postprocess SELinux policy
- Convert /var to tmpfiles.d
- Prepares symlinks
- /usr/local -> /var/usrlocal
- /var/lib/alternatives -> /usr/lib/alternatives
- /var/lib/vagrant -> /usr/lib/vagrant
- copies the rpmdb
Based on commit 1cf0d557ae8059e689b1fed670022727e9842288 (rpm-ostree)
Add a small wrapper around the setfiles(8) utility that can be used
to set the security context fields on one or multiple provided paths,
given a specification. The root of the file system tree can be given
via `root` and all elements of `paths` will be interpreted as
relative to that root.
Add a helper, `parse_config`, to parse a selinux configuration file,
see selinux(8), and return a dictionary containing the configuration
data in key, value pairs. This, in turn, can be fed into the other
helper method, `config_get_policy`, to get the effective policy or
`None` if SELinux is disabled or the policy type is not configured.
Add a new test suite that checks the basic functionality of the
helpers above.
When using rpm-ostree compose, a Treefile[1] controls various
aspects of its behaviour. Since rpm-ostree will, at least in
the beginning, be used to post-process and committing the tree
add a helper class to ease the creation of correct Treefiles.
The docstring of the Treefile contains the information in which
phases ('install', 'postprocess', 'commit') the option is used,
as of rpm-ostree commit 1cf0d557ae8059e689b1fed670022727e9842288
Add basic checks for the ostree.Treefile helper. Some of the
tests require rpm-ostree to be installed.
[1] https://rpm-ostree.readthedocs.io/en/stable/manual/treefile/
Now that stages no longer access the network, drop CA certificate
setup.
In the future, we may want to restrict all network access to the
container, but that requires more work.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This has now been entirely replaced by the rpm stage in all its
users. The dnf stage does not fit very nicely into the osbuild
module, in particular it requires direct network access, which
we would like to avoid.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For the sake of backwards compatibility, legacy support was enabled
by default. Flip this around, so that leaving the parameter out
means disabling it.
This is more intuitive, and will pave the way for dropping support
for the value being a bool in the future.
`osbuild-composer` always passes the argumnet explicitly, though
still always as a boolean.
Signed-off-by: Tom Gundersen <teg@jklm.no>
VHDX is the best format for uploading to AWS, thus this commit adds the
support for it. Pros over other formats supported by AWS:
- vmdk - doesn't work, qemu-img probably needs some special options
- vhd - the image size gets round up (I can get only a >=7GB volume from
a 6GB image)
- ova - just a wrapper over vmdk/vhd/vhdx adding some metadata
- raw - no compression, the images are huge
Also, the format specification is open, therefore I can't see any issues
with it.
Test a deployment of osbuild along with osbuild-composer for each PR.
More testing could be stacked on top.
Signed-off-by: Major Hayden <major@redhat.com>
Causes a problem with ostree-osbuild on CI (travis) otherwise:
Traceback (most recent call last):
File "osbuild-ostree", line 345, in <module>
sys.exit(main())
File "osbuild-ostree", line 337, in main
return build(args)
File "osbuild-ostree", line 257, in build
output_id, commit_id = build_commit(builddir, args)
File "osbuild-ostree", line 162, in build_commit
r = pipeline.run(store.store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 358, in run
r = self.assemble(object_store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 314, in assemble
r = self.assembler.run(input_dir,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 148, in run
osbuild_module_path = os.path.dirname(importlib.util.find_
Make sure we prune the caches after each stage-test to keep our disk
footprint small. This does considerably increase build-times since we
no longer share cached entries. However, the current CI builds simply
run out of disk-space.
Once we use separate output-directories we will be able to drop the
automatic checkpointing from the tests, and thus effectively get the
same behavior. Until then, lets prune the caches explicitly.
The `osbuild` executable requires that the exact same version of the
corresponding python library is installed, but this was not enforced
in the RPM package. Thus a old version of osbuild could be installed
alongside an older version of python3-osbuild, which results in an
osbuild crash (see below).
Therefore, enforce that both installed packages have matching
versions by specifying the exact version for the `python3-osbuild`
dependency of the `osbuild` package.
Example osbuild crash if versions are not matching:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
ws.require(__requires__)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (osbuild 10 (/usr/lib/python3.8/site-packages), Requirement.parse('osbuild==11'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/osbuild", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3252, in <module>
def _initialize_master_working_set():
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 585, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'osbuild==11' distribution was not found and is required by the application
The sources.json file contains repositories for x86_64 and s390,
this patch adds the same for aarch64 Fedora 30 so that the example
pipeline can be build easily.
The "legacy" option in stages/org.osbuild.grub2 defaults to True:
`legacy = options.get("legacy", True)`
If both legacy and uefi is enable, the stage will try to create a hybrid
image, but that fails for aarch64. This patch makes sure we use uefi
mode only which fixed this issue.
I tested the patch by hand together with the fix from this PR:
https://github.com/osbuild/osbuild/pull/308
The GUID Partition Table (GPT) layout supports assigning UUIDs for
individual partitions. Add support for specifying those in the
partition description.
Move the whole result handling of the assembler outside the context
manager; this includes the cleanup of the object in the error case
which would conflict with the ongoing write operation inside the
context manager and thus lead to a crash:
Traceback (most recent call last):
File "/usr/bin/osbuild", line 11, in <module>;
load_entry_point('osbuild==10', 'console_scripts', 'osbuild')()
File "/usr/lib/python3.7/site-packages/osbuild/__main__.py", line 99, in main
secrets=secrets
File "/usr/lib/python3.7/site-packages/osbuild/pipeline.py", line 362, in run
libdir)
File "/usr/lib/python3.7/site-packages/osbuild/pipeline.py", line 324, in assemble
output.cleanup()
File "/usr/lib/python3.7/site-packages/osbuild/objectstore.py", line 160, in cleanup
self._check_writer()
File "/usr/lib/python3.7/site-packages/osbuild/objectstore.py", line 178, in _check_writer
raise ValueError("Write operation is ongoing")
ValueError: Write operation is ongoing
Instead of using the chattr binary, which adds another dependency
use what amounts to ioctl(fd, ,FS_IOC_SETFLAGS, ~FS_IMMUTABLE_FL),
to clear the immutable flag. Constants are taken from linux/fs.h.
The tree, which is created by stages and assemblers, might contain
immutable files, which for Python 3 currently (version 3.8) leads
to errors when the tempfile.TemporaryDirectory is being cleaned up.
Therefore, manually cleanup the tree directory, if it exists, via
shutil.rmtree with a custom onerror handler that also removes the
immutable bit on permission errors.
osbuild can now take only manifests as its input (the legacy input format
was dropped in e48c2f1). This commit changes all remaining occurrences of
"pipeline" to "manifest" when describing the osbuild input.
This drops support for passing in non-manifest style pipelines
directly. It used to be that we directly pass in the pipeline
description, but it got changed to a proper manifest format in:
commit e48c2f178c
Author: Tom Gundersen <teg@jklm.no>
Date: Thu Feb 13 17:44:54 2020 +0100
osbuild: allow the sources to be passed in on stdin
With 2 releases in between, we are now far enough to drop the old
format. All code has been converted, our API guarantee is not in place,
yet, so lets just drop the legacy code and fully commit to the
manifest.
Fixes#265.
Recent qemu version will warn with our current code:
qemu-system-x86_64: -accel kvm:hvf:tcg: Don't use ':' with -accel,
use -M accel=... for now instead
Since this might result in hard-errors, lets just follow the advice and
use the `-M` switch.
RPM is smart about dependencies - it goes over all shebangs of packaged files
and adds Requires to them. Prior this commit osbuild package depended on
python 3.6, because rhel runner has /usr/bin/python3.6 as shebang.
This commit fixes it by turning off dependency generators for our internal
stuff: assemblers, runners and stages. They run in a container, so I think
there's no reason to generate dependencies from them.
