A usual step in creating OS file system trees is to apply the
correct SELinux labels for all files and directories. This is
done by the org.osbuild.selinux stage, which internally uses the
setfiles command in order to do so. The SELiunx policy to be
used for this operation is the one of the newly created system,
not the host one. It therefore can contain labels that are not
known on the host. The kernel will prevent setting invalid,
i.e. unknown, labels unless the caller has the CAP_MAC_ADMIN
capability. By default, setfiles is executed in the setfiles_t
domain, where it lacks that capability. Therefore a custom
osbuild SELinux policy was created, with a special transition
rule that will execute setfiles in the setfiles_mac_t
domain. All stages, sources and assemblers as well as the main
binary are label with the new osbuild_exec_t label.
Additionally, allow a transition from osbuild_t to install_t by
using `anaconda_domtrans_install`, so that ostree and
rpm-ostree, which are labeled as install_exec_t, can transition
to the install_t domain when called form osbuild. Update the
spec file to build the policy and include it in a new
osbuild-selinux sub-package.
Major changes are the support for ignition, that assembler results
wont be automatically committed anymore, a new org.osbuild.copy
stage, the removal of the `tree_id` & `output_id` as well as full
manifest validation due to the addition of source schema validation.
Now that as a result of commit 4d2f15f all symlinks have been
dropped from the individual module paths, the search for module
contents can be simplified again.
Install the schema to %{_datadir}/osbuild/schema and provide a
link from %{pkgdir}/schema to that location so that the osbuild
library can easily access the schemata.
Drop the `osbuild -> ../osbuild` symlink from all module directories.
We now properly initialize the PYTHONPATH to provide the imported
osbuild module from the host environment. Therefore, these links are no
longer needed.
The sources run from the host environment, so they should just pick them
up from the environment the same way osbuild itself does.
We want to run stages and other scripts inside of the nspawn containers
we use to build pipelines. Since our pipelines are meant to be
self-contained, this should imply that the build-root must have osbuild
installed. However, this has not been the case so far for several
reasons including:
1. OSBuild is not packaged for all the build-roots we want to support
and thus we have the chicken-and-egg problem.
2. During testing and development, we want to support using a local
`libdir`.
3. We already provide an API to the container. Importing scripts from
the outside just makes this API bigger, but does not change the
fact that build-roots are not self-contained. Same is true for the
running kernel, and probably much more..
With all this in mind, our strategy probably still is to eventually
package osbuild for the build-root. This would significantly reduce our
API exposure, points-of-failure, and host-reliance. However, this switch
might still be some weeks out.
With this in mind, though, we can expect the ideal setup to have a full
osbuild available in the build-root. Hence, any script we import so far
should be able to access the entire `libdir`. This commit unifies the
libdir handling by installing the symlinks into `libdir` and providing
a single bind-mount of the module-path into `libdir`.
We can always decide to scratch that in the future when we scratch the
libdir-import from the host-root. Until then, I believe this commit
nicely unifies the way we import the module both in a local checkout as
well as in the container.
The `osbuild` executable requires that the exact same version of the
corresponding python library is installed, but this was not enforced
in the RPM package. Thus a old version of osbuild could be installed
alongside an older version of python3-osbuild, which results in an
osbuild crash (see below).
Therefore, enforce that both installed packages have matching
versions by specifying the exact version for the `python3-osbuild`
dependency of the `osbuild` package.
Example osbuild crash if versions are not matching:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
ws.require(__requires__)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (osbuild 10 (/usr/lib/python3.8/site-packages), Requirement.parse('osbuild==11'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/osbuild", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3252, in <module>
def _initialize_master_working_set():
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 585, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'osbuild==11' distribution was not found and is required by the application
RPM is smart about dependencies - it goes over all shebangs of packaged files
and adds Requires to them. Prior this commit osbuild package depended on
python 3.6, because rhel runner has /usr/bin/python3.6 as shebang.
This commit fixes it by turning off dependency generators for our internal
stuff: assemblers, runners and stages. They run in a container, so I think
there's no reason to generate dependencies from them.
Use `make man` rather than hard-coding man-page creation in the
spec-file. Furthermore, install all man-pages, not just the base one.
The commands are adjusted to account for possibly more man-pages being
added. Tree-wide the only place to adjust for new man-pages being added
is the `%files` list in the spec-file.
We already indent the values in the spec-file to all start at the same
column. However, we use different indentation sizes for different
blocks of values. This makes it really confusing to read. Lets use a
consistent indentation and align everything with the main assignments in
the spec-file.
Detect the host dynamically from os-release(5) instead of relying on the
`org.osbuild.host` symlink.
It is awkward to install a symlink that tells osbuild which distro is is
running on, when there is a standard way to detect this.
This makes it easier to run osbuild from sources and removes the need to
include every host in the spec file. The latter became hard to do,
because there's no obvious way to distinguish RHEL minor releases.
This changes the name of the source tarball, but apart from that has
no effect as-is.
However, if the `commit` global is set, then the source tarball is
expected to be the standard git-archive at that commit, and the
release field is set to contain the date and the git sha. This is
useful for when we want to build rpms between releases.
This works analogously to how osbulid-composer currently works.
See <https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation>
for details.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This source adds support for downloaded files. The files are
indexed by their content hash, and the only option is their URL.
The main usecase for this will be downloading rpms. Allowing depsolving
to be done outside of osbuild, network access to be restricted and
downloaded rpms to be reused between runs.
Each source is now passed two additional arguments, a cache directory
and an output directory. Both are in the source's namespace, and
the source is responsible for managing them. Each directory may
contain contents from previous runs, but neither is ever guaranteed
to do so.
Downloaded contents may be saved to the cache and resued between
runs, and the requested content should be written to the output dir.
If secrets are used, the source must only ever write contents to
the output that corresponds to the available secrets (rather than
contents from the cache from previous runs).
Each stage is passed an additional argument, a sources directory.
The directory is read-only, and contains a subdirectory named after
each used source, which will contain the requseted contents when
the `Get()` call returns (if the source uses this functionality).
Based on a patch by Lars Karlitski.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The Fedora 31 and Fedora 32 runners are symlinks but the spec file
only looked for files and not symlinks. Fix that.
Follow up by 6a14ba40f7. Found by
Lars.
osbuild currently throws an error when not passing a build environment
on the command line, because the runner is unset. This is annoying on
hosts which only need a runner set, but no build pipeline.
To simplify running osbuild in this common case, introduce
`org.osbuild.host`, which is a runner that is defined to work on the
host that osbuild is installed on. Use this runner by default and
include a symlink to the right runner in the Fedora and RHEL packages.
Also add `runners/org.osbuild.host` to `.gitignore`, so that developers
can set the symlink when running osbuild from the source directory.
Fixes#171
We've been using a generic `osbuild-run`, which sets up the build
environment (and works around bugs) for all build roots. It is already
getting unwieldy, because it tries to detect the OS for some things it
configures. It's also about to cause problems for RHEL, which doesn't
currently support a python3 shebang without having /etc around.
This patch changes the `build` key in a pipeline to not be a pipeline
itself, but an object with `runner` and `pipeline` keys. `pipeline` is
the build pipeline, as before. `runner` is the name of the runner to
use. Runners are programs in the `runners` subdirectory.
Three runners are included in this patch. They're copies of osbuild-run
for now (except some additions for rhel82). The idea is that each of
them only contains the minimal setup code necessary for an OS, and that
we can review what's needed when updating a build root.
Also modify the `--build-pipeline` command line switch to accept such a
build object (instead of a pipeline) and rename it accordingly, to
`--build-env`.
Correspondingly, `OSBUILD_TEST_BUILD_PIPELINE` → `OSBUILD_TEST_BUILD_ENV`.
_libdir is platform dependant, but that is not what we want because we
would need additional runtime logic to handle platforms. this patch
overrides the defautl location
The repository now contains a Vagrantfile for running the testing script
against an RPM package created locally using `make rpm`. To run this
test use `make vagrant-test`. setup.py was also modified to adhere to
packaging guidelines and not to install system-level executables.
The lincense is now included in the Python package using the MANIFEST.in
file.
