Two cleanups for the context-managers we use:
* Use `contextlib.AbstractContextManager` if possible. This class
simply provides a default `__enter__` implementation which just
returns `self`. So use it where applicable.
Additionally, it provides an abstract `__exit__` method and thus
allows static checks for an existance of `__exit__` in the dependent
class. We might use that everywhere, but this is a separate
decision, so not included here.
* Explicitly return `None` from `__exit__`. The python docs state:
If an exception is supplied, and the method wishes to suppress
the exception (i.e., prevent it from being propagated), it
should return a true value. Otherwise, the exception will be
processed normally upon exit from this method.
That is, unless we want exceptions to be suppressed, we should
never return a `truthy` value. The python contextlib suggest using
`None` as a default return value, so lets just do that.
In particular, the explicit `return exc_type is None` that we use
has no effect at all, since it only returns `True` if no exception
was raised.
This commit cleans this up and just follows what the `contextlib`
module does and returns None everywhere (well, it returns nothing
which apparently is the same as returning `None` in python). It is
unlikely that we ever want to suppress any exceptions, anyway.
Make use of the new immutable-flag ioctl helpers. While at it, move the
`chmod` to `fchmod` and re-use the open file-descriptor. Document the
behavior and move the `fchmod` into its own try-block for the same
reasons as the `ioctl` call: We rely on the following unlink() to catch
any errors. Errors in the fixperms() step are non-consequential.
The FS_IOC_{GET,SET}FLAGS ioctl numbers are not stable across different
architectures. Most of them use the asm-generic versions, but ALPHA and
SPARC in particular use completely different IOC number setups (see the
definition of _IOC, _IOR, _IOW, etc. in the kernel).
This commit moves the helpers for `FS_IMMUTABLE_FL` into
`osbuild/util/` and adds explicit tests. This will make sure that we
catch any ioctl mismatches as soon as possible when we run the osbuild
test-suite on other architectures. Until then, we will have to live with
this mismatch.
Move remove_tree() into its own module in `osbuild.util.rmrf`. This way
we can use it in other modules as well, without cross-referencing
internal helpers.
Add a new module that implements a simple JSON communication channel.
This is meant to replace all our hard-coded SOCK_DGRAM code that is
copied all over the place.
This is intentionally left simple. It only supports synchronous
operations, trivial JSON encoding and decoding, and uses a message-based
transport mode.
For historical and occult reasons the grubenv file is, according
to its documentation[1] a 'preallocated 1024-byte file'. The
unused space in the file needs to be filled with '#' as padding,
which tools will count as "free space"[2] and there must not be a
trailing new-line.
Fix our code to do as they say to make grub2-editenv work and in
turn greenboot.
[1] https://www.gnu.org/software/grub/manual/grub/html_node/Environment-block.html
[2] grub-core/lib/envblk.c#L105 (commit 0f102b9844f852d48501d231d32a17e1cc24062d)
We want to run stages and other scripts inside of the nspawn containers
we use to build pipelines. Since our pipelines are meant to be
self-contained, this should imply that the build-root must have osbuild
installed. However, this has not been the case so far for several
reasons including:
1. OSBuild is not packaged for all the build-roots we want to support
and thus we have the chicken-and-egg problem.
2. During testing and development, we want to support using a local
`libdir`.
3. We already provide an API to the container. Importing scripts from
the outside just makes this API bigger, but does not change the
fact that build-roots are not self-contained. Same is true for the
running kernel, and probably much more..
With all this in mind, our strategy probably still is to eventually
package osbuild for the build-root. This would significantly reduce our
API exposure, points-of-failure, and host-reliance. However, this switch
might still be some weeks out.
With this in mind, though, we can expect the ideal setup to have a full
osbuild available in the build-root. Hence, any script we import so far
should be able to access the entire `libdir`. This commit unifies the
libdir handling by installing the symlinks into `libdir` and providing
a single bind-mount of the module-path into `libdir`.
We can always decide to scratch that in the future when we scratch the
libdir-import from the host-root. Until then, I believe this commit
nicely unifies the way we import the module both in a local checkout as
well as in the container.
The mknod() method currently allows passing no dir_fd, in which case an
internal one is opened. This FD is then never closed, though.
Fix this by simply making the dir_fd mandatory. All callers pass it
(there is actually only a single caller), so no need for the fallback.
Allow file systems to be identified via there label in addition to
their uuid; i.e. either `uuid` or `label` must be specified, which
results in either `UUID=<uuid>` or `LABEL=<label>` to end up in the
"fs_spec" field. See also fstab(5).
In addition to support for identifying file-systems via their uuids,
they now can be identified via their label as well. Two new options
are introduce for this: `rootfs` and `bootfs` for the root and boot
file system. The latter is option in the case a separated partition
is used for /boot. Both options are an object that can either have
`uuid` or `label` set. The old uuid based options, `root_fs_uuid` &
`boot_fs_uuid` are still supported for now.
Additionally, remove the `GRUB2_ROOT_FS_UUID` option from the
grubenv file and directly write the root file system identifier into
the grub config file.
Make sure we always pylint all python sources. We currently skip tests
as well as osbuild/util. Fix this by always recursively looking for all
python sources and then linting them.
A few adjustments to the AST check:
* Avoid local imports. Move the AST import to the head of the file.
We will get warnings if it is unused, so it should not get stale
there once we drop the AST hack.
* Avoid `TODO:` since pylint will parse it (parsing comments, yay!)
* Use `isinstance()` for type-checks.
This fixes a bunch of minor pylint warnings:
* Drop unused imports.
* Fix "inline-JSON" formatting.
* Fix space before/after brackets.
* Use `_` for unused variables.
* Break overlong lines.
* Mark unittest as `no-self-use` if applicable.
* Drop spurious newline at end of file.
This fixes 3 things:
* Drop an unused argument from the http server handler.
* Break an overlong `with:` statement.
* Fix indentation where it is wrong.
This drops the `server` alias for `http.server`. There is only a single
caller, so lets just be explicit so the callsite is easier to
understand.
As a side effect, this unifies all the imports, no cherrypicking
anymore.
By default, xz only uses one CPU core even if multiple cores are
available. If xz compression is chosen, allow xz to use all of the
cores available.
Signed-off-by: Major Hayden <major@redhat.com>
This stage has been replaced by the org.osbuild.rpm stage. The
latter does not need access to network due inside the container
due to its use of the osbuild sources API.
This converts all fedora links in our samples to `mirrors.kernel.org`.
This mirror works best from around the world, so lets avoid the wild
mix of local mirrors and instead use kernel.org.
This mirror is also well-managed and properly funded, so we should not
run into too many problems with it.
Comments must not be indented in makefiles, otherwise they might end up
being interpreted as rules. Simply drop the indentation and move it into
the comments itself.
This stage runs a given command only on the first boot of the image,
useful for doing instantiation tasks that can only be done in the
target environment, or that should be done per-instance, rather
than per image.
Ideally we would use systemd's ConditionFirstBoot for this, but that
requires images to ship without an /etc/machine-id, and currently
we only support shipping images with an empty /etc/machine-id.
Changing this would mean dropping /etc/fstab in favor of mounting
the rootfs rw from the initrd. This is likely the right thing to
do regardless, but we would have to audit what other first-boot
services we would end up with pulling in in this case.
Instead we introduce our own flag file /etc/osbuild-first-boot,
and use ConditionPathExists.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add a new output-directory argument which specifies where to store
result objects. For now, this is purely optional and simply copies from
the old `output_id` into the specified directory. This allows a
backwards compatible transition towards removing any external access to
the osbuild cache.
Note that this has still lots of room for improvements:
* We only support assembler-output for now, but we could also easily
support entire trees as output, in case no assembler was selected.
Alternatively, we could introduce a "copy" assembler, that just
outputs the input tree.
* This parameter is optional, but should really be mandatory. There
is little reason to have the default behavior just dropping any
generated content. This would be a breaking change, though.
* We could move data out of a temporary object-store entry, rather
than copy it. But again, for backwards-compatibility, we leave the
latest store-object intact and do not move things out of it.
* We could now transition towards never committing anything to the
store, not even output IDs, unless explicitly checkpointed.
That was mostly prepared already, the one last missing piece was
to use the default grub2 stage (previous commit) and now, with
this commit, to actually install the grub2 legacy bootloader via
the qemu assembler.
An example that takes an ostree commit and creates a qcow2 image
that is bootable. NB: the commit ids and the corresponding source
will need to be adjusted after having run the pipeline that will
create the ostree commit.
This is a Fedora 31 based example that results in a ostree repo
with a ostree commit. The latter was composed from the file system
tree that was created via the given osbuild stages.
The file `/etc/defaults/grub` sets the defaults that are used by
grub2-mkconfig to (re-)generate the grub config (grub.cfg). This
command is not run by any scripts but by the user directly. On
modern installations (without the grubby-deprecated package)
the kernel is configured via Bootloader Specification snippets
and thus the grub config should not need to be touched at all
under normal circumstances. In the new future the grub2-mkconfig
will be updated to not require GRUB_ENABLE_BLSCFG which should
make the existence `/etc/defaults/grub` even more superfluous.
Additionally, in the future, some images might not contain
the grub2 packages at all.
Add support for copying EFI data from the build root. If
`uefi.install` is set to `true`, `BOOT` and `uefi.vendor`
directories will be copied from the build root. This is
useful for example on OSTree based systems where boot/efi/EFI
is not being populated by an RPM package; but it can be used
also on other systems where it is not desirable to deliver
the EFI data via packages.
Add /boot to be mounted from the build tree into the build root,
because the EFI binaries for grub are stored in there and for
ostree grub2 support those need to be copied too.
Deploying an OSTree commit leads to creation of hardlinks from
the repository to the check out. These will have the correct
SELinux labels, since the files in the repository will have the
correct SELinux labels[1]. But new files are generated in '/etc'
of the new deployment, due to the 3-way configuration merge[2].
Also a new kernel, initramfs and the corresponding Bootloader
Specification entries are created in the global '/boot' dir.
In theory, ostree will set the correct SELinux labels by loading
the SELinux config from the deployment and then setting the
correct security contexts via ostree_sepolicy_setfscreatecon().
But it does so conditionally on is_selinux_enabled(2)[3], which
in our container is FALSE Therefore we have to do the same dance
as ostree does, at least for now, and manually re-label the
affected paths.
[1] Assuming they had the correct label when the commit was made
[2] https://ostree.readthedocs.io/en/latest/manual/deployment/
[3] via ostree_sepolicy_setfscreatecon in ostree-sepolicy.c
line 640 of commit 2c1658538f8fde5813e95e7408d65662a489be91
Add the ability to specify one ore more remotes for the system
repository. The required options for a single remote are its
`name` and the `url`. Optionally one or more branch can be passed
via `branches`. GPG keys can be given via `gpgkeys`; if none are
specified, no gpg verification will be done.
The list of mount points is changed from a list of strings to a
list of objects containing `path` and an optional `mode` value.
The latter can be used to set the mode of the mount point that
will be created in the file system tree. It defaults to 0755,
or 493 in decimal, because JSON does not support octal values.
Instead of taking a raw string for the root file system kernel
option, convert the `rootfs` option to an object that must have
either `uuid` or `label` set. This will then be translated into
the proper kernel command line argument.
Add a new optional `ref` option. When set, a references for the
commit that was pulled with the value of `ref` will be created
in the system ostree repo. It will also be used when deploying
the commit and thus will be set as the origin for it. This is
necessary for updates to work.
Initializes the filesystem via ostree and then pulls a specified
commit and deploys that. Options are `commit`, which is the id
of the commit to pull and deploy, `osname`, which is the name
to be used for the operating system root. The `roofs` together
with `kernel_opts` options are used to build the kernel command
line for the deployment.
Additionally, a `mounts` parameter can be supplied that indicates
file system boundaries. This is needed because ostree uses a hard
link farm, which must not link across said file system boundaries.
Instead of verifying the gpg signature when pull from the actual
remote source into the local cache, verify the commit when it is
being pulled from the local cache into the output directory. This
ensures that the signatures are checked against the provided keys
even when the commit was already in the cache and at that time
the key might have been different.
NB: ostree expects the signature to be present on the remote at
the *target* repository, i.e. in our case the output repository.
The keys are therefore attached to a temporary remote that is
created at the output repository with the same name/id that is
used for the actual remote.
Add a new `gpgkeys` option that, if set, must contain a list of
public keys. These keys will then be used by ostree to verify
signed commits when pulling from the remote. If the `gpgkeys`
option is missing, no verification will be attempted.
This source can be used to fetch ostree commits. The commits are
accessed via their commit is. The only option currently is `url`,
given for each commit, that will be used as the location of the
remote. A cache repository, that will be created if necessary,
acts as an intermediary, so remotes will be added with `name` as
the identifier to it and commits are pulled into that. In the
output directory another repository will be created as 'repo' and
the requested commit pulled into that from the cache repository via
a local pull.
Add a new assembler that takes a file system tree that is already
conforming to the ostree system layout[1], creates a new repository
in archive mode and commits the file system tree to it. Afterwards,
a reference is created with the value supplied in `ref`.
The repository is located at the `/repo` directory and additional
metadata is /compose.json which contain the compose information.
Currently uses rpm-ostree to do the actual committing. In the future
this might change to plain ostree.
[1] https://ostree.readthedocs.io/en/stable/manual/adapting-existing/
A stage that uses rpm-ostree compose to do post-processing of the
tree. Among other things the main steps are:
- moves /etc to /usr/etc
- move /boot to /usr/lib/ostree-boot
- potentially moving /var/lib/rpm
- re-creates the initramfs (dracut)
- stores the treefile at /usr/share/rpm-ostree/treefile.json
- adds altfiles module to nss
- Recompiles SELinux policy (semodule -nB)
- Migrates /usr/etc/{passwd, group} to /usr/lib/
- Postprocess SELinux policy
- Convert /var to tmpfiles.d
- Prepares symlinks
- /usr/local -> /var/usrlocal
- /var/lib/alternatives -> /usr/lib/alternatives
- /var/lib/vagrant -> /usr/lib/vagrant
- copies the rpmdb
Based on commit 1cf0d557ae8059e689b1fed670022727e9842288 (rpm-ostree)
