This fixes a bunch of minor pylint warnings:
* Drop unused imports.
* Fix "inline-JSON" formatting.
* Fix space before/after brackets.
* Use `_` for unused variables.
* Break overlong lines.
* Mark unittest as `no-self-use` if applicable.
* Drop spurious newline at end of file.
This fixes 3 things:
* Drop an unused argument from the http server handler.
* Break an overlong `with:` statement.
* Fix indentation where it is wrong.
This drops the `server` alias for `http.server`. There is only a single
caller, so lets just be explicit so the callsite is easier to
understand.
As a side effect, this unifies all the imports, no cherrypicking
anymore.
By default, xz only uses one CPU core even if multiple cores are
available. If xz compression is chosen, allow xz to use all of the
cores available.
Signed-off-by: Major Hayden <major@redhat.com>
This stage has been replaced by the org.osbuild.rpm stage. The
latter does not need access to network due inside the container
due to its use of the osbuild sources API.
This converts all fedora links in our samples to `mirrors.kernel.org`.
This mirror works best from around the world, so lets avoid the wild
mix of local mirrors and instead use kernel.org.
This mirror is also well-managed and properly funded, so we should not
run into too many problems with it.
Comments must not be indented in makefiles, otherwise they might end up
being interpreted as rules. Simply drop the indentation and move it into
the comments itself.
This stage runs a given command only on the first boot of the image,
useful for doing instantiation tasks that can only be done in the
target environment, or that should be done per-instance, rather
than per image.
Ideally we would use systemd's ConditionFirstBoot for this, but that
requires images to ship without an /etc/machine-id, and currently
we only support shipping images with an empty /etc/machine-id.
Changing this would mean dropping /etc/fstab in favor of mounting
the rootfs rw from the initrd. This is likely the right thing to
do regardless, but we would have to audit what other first-boot
services we would end up with pulling in in this case.
Instead we introduce our own flag file /etc/osbuild-first-boot,
and use ConditionPathExists.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add a new output-directory argument which specifies where to store
result objects. For now, this is purely optional and simply copies from
the old `output_id` into the specified directory. This allows a
backwards compatible transition towards removing any external access to
the osbuild cache.
Note that this has still lots of room for improvements:
* We only support assembler-output for now, but we could also easily
support entire trees as output, in case no assembler was selected.
Alternatively, we could introduce a "copy" assembler, that just
outputs the input tree.
* This parameter is optional, but should really be mandatory. There
is little reason to have the default behavior just dropping any
generated content. This would be a breaking change, though.
* We could move data out of a temporary object-store entry, rather
than copy it. But again, for backwards-compatibility, we leave the
latest store-object intact and do not move things out of it.
* We could now transition towards never committing anything to the
store, not even output IDs, unless explicitly checkpointed.
That was mostly prepared already, the one last missing piece was
to use the default grub2 stage (previous commit) and now, with
this commit, to actually install the grub2 legacy bootloader via
the qemu assembler.
An example that takes an ostree commit and creates a qcow2 image
that is bootable. NB: the commit ids and the corresponding source
will need to be adjusted after having run the pipeline that will
create the ostree commit.
This is a Fedora 31 based example that results in a ostree repo
with a ostree commit. The latter was composed from the file system
tree that was created via the given osbuild stages.
The file `/etc/defaults/grub` sets the defaults that are used by
grub2-mkconfig to (re-)generate the grub config (grub.cfg). This
command is not run by any scripts but by the user directly. On
modern installations (without the grubby-deprecated package)
the kernel is configured via Bootloader Specification snippets
and thus the grub config should not need to be touched at all
under normal circumstances. In the new future the grub2-mkconfig
will be updated to not require GRUB_ENABLE_BLSCFG which should
make the existence `/etc/defaults/grub` even more superfluous.
Additionally, in the future, some images might not contain
the grub2 packages at all.
Add support for copying EFI data from the build root. If
`uefi.install` is set to `true`, `BOOT` and `uefi.vendor`
directories will be copied from the build root. This is
useful for example on OSTree based systems where boot/efi/EFI
is not being populated by an RPM package; but it can be used
also on other systems where it is not desirable to deliver
the EFI data via packages.
Add /boot to be mounted from the build tree into the build root,
because the EFI binaries for grub are stored in there and for
ostree grub2 support those need to be copied too.
Deploying an OSTree commit leads to creation of hardlinks from
the repository to the check out. These will have the correct
SELinux labels, since the files in the repository will have the
correct SELinux labels[1]. But new files are generated in '/etc'
of the new deployment, due to the 3-way configuration merge[2].
Also a new kernel, initramfs and the corresponding Bootloader
Specification entries are created in the global '/boot' dir.
In theory, ostree will set the correct SELinux labels by loading
the SELinux config from the deployment and then setting the
correct security contexts via ostree_sepolicy_setfscreatecon().
But it does so conditionally on is_selinux_enabled(2)[3], which
in our container is FALSE Therefore we have to do the same dance
as ostree does, at least for now, and manually re-label the
affected paths.
[1] Assuming they had the correct label when the commit was made
[2] https://ostree.readthedocs.io/en/latest/manual/deployment/
[3] via ostree_sepolicy_setfscreatecon in ostree-sepolicy.c
line 640 of commit 2c1658538f8fde5813e95e7408d65662a489be91
Add the ability to specify one ore more remotes for the system
repository. The required options for a single remote are its
`name` and the `url`. Optionally one or more branch can be passed
via `branches`. GPG keys can be given via `gpgkeys`; if none are
specified, no gpg verification will be done.
The list of mount points is changed from a list of strings to a
list of objects containing `path` and an optional `mode` value.
The latter can be used to set the mode of the mount point that
will be created in the file system tree. It defaults to 0755,
or 493 in decimal, because JSON does not support octal values.
Instead of taking a raw string for the root file system kernel
option, convert the `rootfs` option to an object that must have
either `uuid` or `label` set. This will then be translated into
the proper kernel command line argument.
Add a new optional `ref` option. When set, a references for the
commit that was pulled with the value of `ref` will be created
in the system ostree repo. It will also be used when deploying
the commit and thus will be set as the origin for it. This is
necessary for updates to work.
Initializes the filesystem via ostree and then pulls a specified
commit and deploys that. Options are `commit`, which is the id
of the commit to pull and deploy, `osname`, which is the name
to be used for the operating system root. The `roofs` together
with `kernel_opts` options are used to build the kernel command
line for the deployment.
Additionally, a `mounts` parameter can be supplied that indicates
file system boundaries. This is needed because ostree uses a hard
link farm, which must not link across said file system boundaries.
Instead of verifying the gpg signature when pull from the actual
remote source into the local cache, verify the commit when it is
being pulled from the local cache into the output directory. This
ensures that the signatures are checked against the provided keys
even when the commit was already in the cache and at that time
the key might have been different.
NB: ostree expects the signature to be present on the remote at
the *target* repository, i.e. in our case the output repository.
The keys are therefore attached to a temporary remote that is
created at the output repository with the same name/id that is
used for the actual remote.
Add a new `gpgkeys` option that, if set, must contain a list of
public keys. These keys will then be used by ostree to verify
signed commits when pulling from the remote. If the `gpgkeys`
option is missing, no verification will be attempted.
This source can be used to fetch ostree commits. The commits are
accessed via their commit is. The only option currently is `url`,
given for each commit, that will be used as the location of the
remote. A cache repository, that will be created if necessary,
acts as an intermediary, so remotes will be added with `name` as
the identifier to it and commits are pulled into that. In the
output directory another repository will be created as 'repo' and
the requested commit pulled into that from the cache repository via
a local pull.
Add a new assembler that takes a file system tree that is already
conforming to the ostree system layout[1], creates a new repository
in archive mode and commits the file system tree to it. Afterwards,
a reference is created with the value supplied in `ref`.
The repository is located at the `/repo` directory and additional
metadata is /compose.json which contain the compose information.
Currently uses rpm-ostree to do the actual committing. In the future
this might change to plain ostree.
[1] https://ostree.readthedocs.io/en/stable/manual/adapting-existing/
A stage that uses rpm-ostree compose to do post-processing of the
tree. Among other things the main steps are:
- moves /etc to /usr/etc
- move /boot to /usr/lib/ostree-boot
- potentially moving /var/lib/rpm
- re-creates the initramfs (dracut)
- stores the treefile at /usr/share/rpm-ostree/treefile.json
- adds altfiles module to nss
- Recompiles SELinux policy (semodule -nB)
- Migrates /usr/etc/{passwd, group} to /usr/lib/
- Postprocess SELinux policy
- Convert /var to tmpfiles.d
- Prepares symlinks
- /usr/local -> /var/usrlocal
- /var/lib/alternatives -> /usr/lib/alternatives
- /var/lib/vagrant -> /usr/lib/vagrant
- copies the rpmdb
Based on commit 1cf0d557ae8059e689b1fed670022727e9842288 (rpm-ostree)
Add a small wrapper around the setfiles(8) utility that can be used
to set the security context fields on one or multiple provided paths,
given a specification. The root of the file system tree can be given
via `root` and all elements of `paths` will be interpreted as
relative to that root.
Add a helper, `parse_config`, to parse a selinux configuration file,
see selinux(8), and return a dictionary containing the configuration
data in key, value pairs. This, in turn, can be fed into the other
helper method, `config_get_policy`, to get the effective policy or
`None` if SELinux is disabled or the policy type is not configured.
Add a new test suite that checks the basic functionality of the
helpers above.
When using rpm-ostree compose, a Treefile[1] controls various
aspects of its behaviour. Since rpm-ostree will, at least in
the beginning, be used to post-process and committing the tree
add a helper class to ease the creation of correct Treefiles.
The docstring of the Treefile contains the information in which
phases ('install', 'postprocess', 'commit') the option is used,
as of rpm-ostree commit 1cf0d557ae8059e689b1fed670022727e9842288
Add basic checks for the ostree.Treefile helper. Some of the
tests require rpm-ostree to be installed.
[1] https://rpm-ostree.readthedocs.io/en/stable/manual/treefile/
Now that stages no longer access the network, drop CA certificate
setup.
In the future, we may want to restrict all network access to the
container, but that requires more work.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This has now been entirely replaced by the rpm stage in all its
users. The dnf stage does not fit very nicely into the osbuild
module, in particular it requires direct network access, which
we would like to avoid.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For the sake of backwards compatibility, legacy support was enabled
by default. Flip this around, so that leaving the parameter out
means disabling it.
This is more intuitive, and will pave the way for dropping support
for the value being a bool in the future.
`osbuild-composer` always passes the argumnet explicitly, though
still always as a boolean.
Signed-off-by: Tom Gundersen <teg@jklm.no>
VHDX is the best format for uploading to AWS, thus this commit adds the
support for it. Pros over other formats supported by AWS:
- vmdk - doesn't work, qemu-img probably needs some special options
- vhd - the image size gets round up (I can get only a >=7GB volume from
a 6GB image)
- ova - just a wrapper over vmdk/vhd/vhdx adding some metadata
- raw - no compression, the images are huge
Also, the format specification is open, therefore I can't see any issues
with it.
Test a deployment of osbuild along with osbuild-composer for each PR.
More testing could be stacked on top.
Signed-off-by: Major Hayden <major@redhat.com>
Causes a problem with ostree-osbuild on CI (travis) otherwise:
Traceback (most recent call last):
File "osbuild-ostree", line 345, in <module>
sys.exit(main())
File "osbuild-ostree", line 337, in main
return build(args)
File "osbuild-ostree", line 257, in build
output_id, commit_id = build_commit(builddir, args)
File "osbuild-ostree", line 162, in build_commit
r = pipeline.run(store.store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 358, in run
r = self.assemble(object_store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 314, in assemble
r = self.assembler.run(input_dir,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 148, in run
osbuild_module_path = os.path.dirname(importlib.util.find_
Make sure we prune the caches after each stage-test to keep our disk
footprint small. This does considerably increase build-times since we
no longer share cached entries. However, the current CI builds simply
run out of disk-space.
Once we use separate output-directories we will be able to drop the
automatic checkpointing from the tests, and thus effectively get the
same behavior. Until then, lets prune the caches explicitly.
