particle-os/debian-koji
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

first pass at converting docs to sphinx-doc reStructured Text

Browse source
This commit is contained in:
Adam Miller 2015-11-03 16:51:25 -06:00 • committed by Mike McLean
parent 05330a2959
commit 3fd4275fe3
24 changed files with 5665 additions and 683 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -2,3 +2,4 @@
*.pyo
tests/test.py
.coverage
docs/build/

321
docs/HOWTO.html
View file

@ -1,321 +0,0 @@
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Koji HOWTO</title>
</head>
<body>
<h1>Introduction</h1>
Koji is a system for building and tracking RPMs. It was designed with the following
features in mind:
<p>
<b>Security</b>
<ul>
<li>New buildroot for each build</li>
<li>nfs is used (mostly) read-only</li>
</ul>
<b>Leverage other software</b>
<ul>
<li>Uses Yum and Mock open-source components</li>
<li>XML-RPC APIs for easy integration with other tools</li>
</ul>
<b>Flexibility</b>
<ul>
<li>rich data model</li>
<li>active code base</li>
</ul>
<b>Usability</b>
<ul>
<li>Web interface with Kerberos authentication</li>
<li>Thin, portable client</li>
<li>Users can create local buildroots</li>
</ul>
<b>Reproducibility</b>
<ul>
<li>Buildroot contents are tracked in the database</li>
<li>Versioned data</li>
</ul>
<p> This HOWTO document covers the basic tasks that a developer needs to be
able to accomplish with Koji.
</p>
<h1>Getting started</h1>
<h2>The web interface</h2>
<p>The primary interface for viewing Koji data is a web application. Most of the interface
is read-only, but if you are logged in (see below) and have sufficient privileges there
are some actions that can be performed though the web. For example:
<ul>
<li>Cancel a build</li>
<li>Resubmit a failed task</li>
</ul>
Those with admin privileges will find additional actions, such as:
<ul>
<li>Create/Edit/Delete a tag</li>
<li>Create/Edit/Delete a target</li>
<li>Enable/Disable a build host</li>
</ul>
<p>The web site utilizes Kerberos authentication. In order to log in you will
need a valid Kerberos ticket and your web browser will need to be configured to send the
Kerberos information to the server.
<p>In Firefox or Mozilla, you will need to use the about:config page to set a few parameters.
Use the search term 'negotiate' to filter the list. Change
network.negotiate-auth.trusted-uris to the domain you want to authenticate against,
e.g .example.com. You can leave network.negotiate-auth.delegation-uris blank, as it
enables Kerberos ticket passing, which is not required. If you do not see those two
config options listed, your version of Firefox or Mozilla may be too old to support
Negotiate authentication, and you should consider upgrading.
<p>In order to obtain a Kerberos ticket, use the kinit command.
<h2>Installing the Koji cli</h2>
<p>There is a single point of entry for most operations. The command is
called 'koji' and is included in the main koji package.
<p>Repos/webpage TBD
<p>
The koji tool authenticates to the central server using Kerberos, so you will need
to have a valid Kerberos ticket to use many features. However, many of the read-only
commands will work without authentication.
<h2>Building a package</h2>
<p>Builds are initiated with the command line tool.
To build a package, the syntax is:</p>
<pre>$ koji build &lt;build target&gt; &lt;cvs URL&gt;</pre>
<p>For example:</p>
<pre>$ koji build dist-fc7-scratch 'cvs://cvs.example.com/cvs/dist?rpms/kernel/FC-7#kernel-2_6_20-1_2925_fc7'</pre>
<p>
The <code>koji build</code> command creates a build task in Koji. By default
the tool will wait
and print status updates until the build completes. You can override this with
the <code>--nowait</code> option. To view other options to the build command use the
<code>--help</code> option.
</p>
<pre>$ koji build --help
</pre>
<h2>Build Options</h2>
<p>
There are a few options to the build command. Here are some more detailed explanations
of them:
</p>
<dl>
<dt>--skip-tag</dt>
<dd>Normally the package is tagged after the build completes. This option causes
the tagging step to be skipped. The package will be in the system, but untagged
(you can later tag it with the tag-pkg command)</dd>
<dt>--scratch</dt>
<dd>This makes the build into a scratch build. The build will not be
imported into the db, it will just be built. The rpms will land under
&lt;topdir&gt;/scratch. Scratch builds are not tracked and can never
be tagged, but can be convenient for testing. Scratch builds are
typically removed from the filesystem after one week.
</dd>
<dt>--nowait</dt>
<dd>As stated above, this prevents the cli from waiting on the build task.</dd>
<dt>--arch-override</dt>
<dd>This option allows you to override the base set of arches to build for.
This option is really only for testing during the beta period, but it may
be retained for scratch builds in the future.</dd>
</dl>
<h2>Build Failures</h2>
<p>If your package fails to build, you will see something like this.</p>
<pre>
420066 buildArch (kernel-2.6.18-1.2739.10.9.el5.jjf.215394.2.src.rpm,
ia64): open (build-1.example.com) -> FAILED: BuildrootError:
error building package (arch ia64), mock exited with status 10
</pre>
<p>You can figure out why the build failed by looking at the log files. If
there is a build.log, start there. Otherwise, look at init.log</p>
<pre>
$ ls -1 &lt;topdir&gt;/work/tasks/420066/*
&lt;topdir&gt;/work/tasks/420066/build.log
&lt;topdir&gt;/work/tasks/420066/init.log
&lt;topdir&gt;/work/tasks/420066/mockconfig.log
&lt;topdir&gt;/work/tasks/420066/root.log
</pre>
<h2>Filing Bugs</h2>
<p>bug tracking TBD
<h1>Koji Architecture</h1>
<h2>Terminology</h2>
In Koji, it is sometimes necessary to distinguish between the a package in general,
a specific build of a package, and the various rpm files created by a build. When
precision is needed, these terms should be interpreted as follows:
<dl>
<dt>Package</dt>
<dd>The name of a source rpm. This refers to the package in general and not
any particular build or subpackage. For example: kernel, glibc, etc.</dd>
<dt>Build</dt>
<dd>A particular build of a package. This refers to the entire build: all arches
and subpackages. For example: kernel-2.6.9-34.EL, glibc-2.3.4-2.19.</dd>
<dt>RPM</dt>
<dd>A particular rpm. A specific arch and subpackage of a build.
For example: kernel-2.6.9-34.EL.x86_64, kernel-devel-2.6.9-34.EL.s390,
glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64</dd>
</dl>
<h2>Koji Components</h2>
Koji is comprised of several components:
<ul>
<li><em>koji-hub</em> is the center of all Koji operations. It is an XML-RPC server
running under mod_python in Apache. koji-hub is passive in that it only
receives XML-RPC calls and relies upon the build daemons and other
components to initiate communication. koji-hub is the only component that
has direct access to the database and is one of the two components that have
write access to the file system.</li>
<li><em>kojid</em> is the build daemon that runs on each of the build machines. Its
primary responsibility is polling for incoming build requests and handling
them accordingly. Koji also has support for tasks other than building.
Creating install images is one example. kojid is responsible for handling
these tasks as well.
<p>kojid uses mock for building. It also creates a fresh buildroot for
every build. kojid is written in Python and communicates with koji-hub via
XML-RPC.</p></li>
<li><em>koji-web</em> is a set of scripts that run in mod_python and use the Cheetah
templating engine to provide an web interface to Koji. koji-web exposes a
lot of information and also provides a means for certain operations, such as
cancelling builds.</li>
<li><em>koji</em> is a CLI written in Python that provides many hooks into
Koji. It allows the user to query much of the data as well as perform
actions such as build initiation.</li>
<li><em>kojirepod</em> is a daemon that keeps the build root repodata
updated.</li>
</ul>
<h2>Package Organization</h2>
<p><i>Tags and Targets</i></p>
<p>Koji organizes packages using tags. In Koji a tag is roughly analogous to
a beehive collection instance, but differ in a number of ways:</p>
<ul>
<li>Tags are tracked in the database but not on disk</li>
<li>Tags support multiple inheritance</li>
<li>Each tag has its own list of valid packages (inheritable)</li>
<li>Package ownership can be set per-tag (inheritable)</li>
<li>Tag inheritance is more configurable</li>
<li>When you build you specify a <i>target</i> rather than a tag</li>
</ul>
<p>
A build target specifies where a package should be built and how it
should be tagged afterwards. This allows target names to remain fixed
as tags change through releases. You can get a full list of build targets
with the following command:</p>
<pre>$ koji list-targets
</pre>
You can see just a single target with the <code>--name</code> option:
<pre>$ koji list-targets --name dist-fc7
Name Buildroot Destination
---------------------------------------------------------------------------------------------
dist-fc7 dist-fc7-build dist-fc7
</pre>
This tells you a build for target dist-fc7 will use a buildroot with packages
from the tag dist-fc7-build and tag the resulting packages as dist-fc7.
<p>
You can get a list of tags with the following command:</p>
<pre>$ koji list-tags
</pre>
<p><i>Package lists</i></p>
<p>
As mentioned above, each tag has its own list of packages that may be placed
in the tag. To see that list for a tag, use the <code>list-pkgs</code> command:</p>
<pre>$ koji list-pkgs --tag dist-fc7
Package Tag Extra Arches Owner
----------------------- ----------------------- ---------------- ----------------
ElectricFence dist-fc6 pmachata
GConf2 dist-fc6 rstrode
lucene dist-fc6 dbhole
lvm2 dist-fc6 lvm-team
ImageMagick dist-fc6 nmurray
m17n-db dist-fc6 majain
m17n-lib dist-fc6 majain
MAKEDEV dist-fc6 clumens
...
</pre>
The first column is the name of the package, the second tells you which tag
the package entry has been inherited from, and the third tells you the owner
of the package.
<p><i>Latest Builds</i></p>
<p>
To see the latest builds for a tag, use the <code>latest-pkg</code> command:</p>
<pre>$ koji latest-pkg --all dist-fc7
Build Tag Built by
---------------------------------------- -------------------- ----------------
ConsoleKit-0.1.0-5.fc7 dist-fc7 davidz
ElectricFence-2.2.2-20.2.2 dist-fc6 jkeating
GConf2-2.16.0-6.fc7 dist-fc7 mclasen
ImageMagick-6.2.8.0-3.fc6.1 dist-fc6-updates nmurray
MAKEDEV-3.23-1.2 dist-fc6 nalin
MySQL-python-1.2.1_p2-2 dist-fc7 katzj
NetworkManager-0.6.5-0.3.cvs20061025.fc7 dist-fc7 caillon
ORBit2-2.14.6-1.fc7 dist-fc7 mclasen
</pre>
The output gives you not only the latest builds, but which tag they have
been inherited from and who built them (note: for builds imported from beehive
the "built by" field may be misleading)
<h2>Exploring Koji</h2>
<p>We've tried to make Koji self-documenting wherever possible. The command
line tool will print a list of valid commands and each command supports
<code>--help</code>. For example:</p>
<pre>
$ koji help
Koji commands are:
build Build a package from source
cancel-task Cancel a task
help List available commands
latest-build Print the latest rpms for a tag
latest-pkg Print the latest builds for a tag
...
$ koji build --help
usage: koji build [options] tag URL
(Specify the --help global option for a list of other help options)
options:
-h, --help show this help message and exit
--skip-tag Do not attempt to tag package
--scratch Perform a scratch build
--nowait Don't wait on build
...
</pre>
<h1>Getting Involved</h1>
If you would like to be more involved with the Koji project...
<p>Project data TBD
</body>
</html>

177
docs/Makefile
View file

@ -1,4 +1,177 @@
install:
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = build
# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
clean:
rm -f *.o *.so *.pyc *~
rm -rf $(BUILDDIR)/*
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Koji.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Koji.qhc"
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/Koji"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Koji"
@echo "# devhelp"
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

76
docs/Migrating_to_1.10.txt
View file

@ -1,76 +0,0 @@
Migrating to Koji 1.10
======================
// asciidoc formatted
The 1.10 release of Koji includes a few changes that you should consider when
migrating.
DB Updates
----------
The new +tag_extra+ table tracks extra data for tags.
There is a new entry in the +channels+ table and some additions and updates to
the +archivetypes+ table.
As in previous releases, we provide a migration script that updates the
database.
# psql koji koji </usr/share/doc/koji-1.10.0/docs/schema-upgrade-1.9-1.10.sql
Command line changes
--------------------
A few commands support new arguments
* maven-build
** --ini : Pass build parameters via a .ini file
** --section : Get build parameters from this section of the .ini
* wrapper-rpm
** --ini : Pass build parameters via a .ini file
** --section : Get build parameters from this section of the .ini
* import
** --link : Attempt to hardlink instead of uploading
* list-tagged
** --latest-n : Only show the latest N builds/rpms
* list-history
** --watch : Monitor history data
* edit-tag
** --extra : Set tag extra option
* list-tasks
** --user : Only tasks for this user
** --arch : Only tasks for this architecture
** --method : Only tasks of this method
** --channel : Only tasks in this channel
** --host : Only tasks for this host
* download-build
** --task-id : Interpret id as a task id
And there are three new commands
* image-build-indirection
* maven-chain
* runroot
Other Configuration changes
---------------------------
The Koji web interface can now treate +extra-footer.html+ as a Cheetah template.
This behavior can be enabled by setting the +LiteralFooter+ option to +False+ in
the kojiweb config.
RPC API Changes
---------------
The +readTaggedBuilds+ and +readTaggedRPMS+ now treat an integer value for the optional
latest argument differently. Before it was simply treated as a boolean flag, which
if true caused the call to return only the latest build for each package. Now, if
the value is a positive integer N, it will return the N latest builds for each
package. The behavior is unchanged for other values.
New rpc calls: +chainMaven+, +buildImageIndirection+, and +mergeScratch+

79
docs/Migrating_to_1.9.txt
View file

@ -1,79 +0,0 @@
Migrating to Koji 1.9
=====================
// asciidoc formatted
The 1.9 release of Koji includes a few changes that you should consider when
migrating.
DB Updates
----------
ImageFactory support introduced some new archive types. These have been added to
the +archivetypes+ table. The inaccurate +vmx+ entry has been removed.
As in previous releases, we provide a migration script that updates the
database.
# psql koji koji </usr/share/doc/koji-1.9.0/docs/schema-upgrade-1.8-1.9.sql
Command line changes
--------------------
The command line interface handles configuration files a little differently. Old
configs should work just fine, but now there are new options and enhancements.
In addition to the main configuration files, the koji cli now checks for
+/etc/koji.conf.d+ and +~/.koji/config.d+ directories and loads any +*.conf+ files
contained within. Also if the user specifies a directory with the +-c/--config+
option, then that directory will be processed similarly.
The command line supports a new +-p/--profile+ option to select alternate configuration
profiles without having to link or rename the koji executable.
The new +image-build+ command is used to generate images using ImageFactory. The older
spin-appliance command is now deprecated.
The +mock-config+ command no longer requires a name argument. You can still specify one
if you want to override the default choice. It also supports new options. The
+--latest+ option causes the resulting mock config to reference the ``latest'' repo (a
varying symlink). The +--target+ option allows generating the config from a target name.
Other command line changes include:
* a new +download-logs+ command
* the +list-groups+ command now accepts event args
* the +taginfo+ command now reports the list of comps groups for the tag
* the fast upload feature is now used automatically if the server supports it
Other Configuration changes
---------------------------
There are also some minor configuration changes in other parts of Koji.
In +kojid+ the time limit for rpm builds is now configurable via the +rpmbuild_timeout+
setting in kojid.conf. The default is 24 hours.
The +koji-gc+ tool supports two new configuration options. The +krbservice+ option allows
you to specify the kerberos service for authentication, and the +email_domain+ option
allows you to specify the email domain for sending gc notices.
The messagebus hub plugin now supports +timeout+ and +heartbeat+ options for the message
bus connection.
RPC API Changes
---------------
Most of these changes are extensions, though some of the host-only call changes are
incompatible.
The +tagHistory+ call accepts a new named boolean option (+active+) to select only
active/inactive entries. It also now reports the additional fields maven_build_id and
win_build_id if builds are maven or win builds respectively.
New rpc calls: +buildImageOz+, +host.completeImageBuild+, and +host.evalPolicy+.
The host-only calls +host.moveImageBuildToScratch+ and +host.importImage+ no longer
accept the +rpm_results+ argument. The rpm results can be embedded in the regular
+results+ argument.

153
docs/Writing_a_plugin.md
View file

@ -1,153 +0,0 @@
# Writing Koji plugins
Depending on what you are trying to do, there are different ways to write a
Koji plugin.
Each is described in this file, by use case.
## Adding new task types
Koji can do several things, for example build RPMs, or live CDs. Those are
types of tasks which Koji knows about.
If you need to do something which Koji does not know yet how to do, you could
create a Koji Builder plugin.
Such a plugin would minimally look like this:
from koji.tasks import BaseTaskHandler
class MyTask(BaseTaskHandler):
Methods = ['mytask']
_taskWeight = 2.0
def handler(self, arg1, arg2, kwarg1=None):
self.logger.debug("Running my task...")
# Here is where you actually do something
A few explanations on what goes on here:
* Your task needs to inherit from `koji.tasks.BaseTaskHandler`
* Your task must have a `Methods` attribute, which is a list of the method
names your task can handle.
* You can specify the weight of your task with the `_taskWeight` attribute.
The more intensive (CPU, IO, ...) your task is, the higher this number
should be.
* The task object has a `logger` attribute, which is a Python logger with the
usual `debug`, `info`, `warning` and `error` methods. The messages you send
with it will end up in the Koji Builder logs (`kojid.log`)
* Your task must have a `handler()` method. That is the method Koji will call
to run your task. It is the method that should actually do what you need. It
can have as many positional and named arguments as you want.
Save your plugin as e.g `mytask.py`, then install it in the Koji Builder
plugins folder: `/usr/lib/koji-builder-plugins/`
Finally, edit the Koji Builder config file, `/etc/kojid/kojid.conf`:
# A space-separated list of plugins to enable
plugins = mytask
Restart the Koji Builder service, and your plugin will be enabled.
You can try running a task from your new task type with the command-line:
$ koji make-task mytask arg1 arg2 kwarg1
## Exporting new API methods over XMLRPC
Koji clients talk to the Koji Hub via an XMLRPC API.
It is sometimes desirable to add to that API, so that clients can request
things Koji does not expose right now.
Such a plugin would minimally look like this:
def mymethod(arg1, arg2, kwarg1=None):
# Here is where you actually do something
mymethod.exported = True
A few explanations on what goes on here:
* Your plugin is just a method, with whatever positional and/or named
arguments you need.
* You must export your method by setting its `exported` attribute to `True`
* The `context.session.assertPerm()` is how you ensure that the
Save your plugin as e.g `mymethod.py`, then install it in the Koji Hub plugins
folder: `/usr/lib/koji-hub-plugins/`
Finally, edit the Koji Hub config file, `/etc/koji-hub/hub.conf`:
# A space-separated list of plugins to enable
Plugins = mymethod
Restart the Koji Hub service, and your plugin will be enabled.
You can try calling the new XMLRPC API with the Python client library:
>>> import koji
>>> session = koji.ClientSession("http://koji/example.org/kojihub")
>>> session.mymethod(arg1, arg2, kwarg1='some value')
### Ensuring the user has the required permissions
If you want your new XMLRPC API to require specific permissions from the user,
all you need to do is add the following to your method:
from koji.context import context
def mymethod(arg1, arg2, kwarg1=None):
context.session.assertPerm("admin")
# Here is where you actually do something
mymethod.exported = True
In the example above, Koji will ensure that the user is an administrator. You
could of course create your own permission, and check for that.
## Running code automatically triggered on events
You might want to run something automatically when something else happens in
Koji.
A typical example is to automatically sign a package right after a build
finished. Another would be to send a notification to a message bus after any
kind of event.
This can be achieved with a plugin, which would look minimally as follows:
from koji.plugin import callback
@callback('preTag', 'postTag')
def mycallback(cbtype, tag, build, user, force=False):
# Here is where you actually do something
A few explanations on what goes on here:
* The `@callback` decorator allows you to declare which events should trigger
your function. You can pass as many as you want. For a list of supported
events, see `koji/plugins.py`.
* The arguments of the function depend on the event you subscribed to. As a
result, you need to know how it will be called by Koji. You probably should
use `*kwargs` to be safe. You can see how callbacks are called in the
`hub/kojihub.py` file, search for calls of the `run_callbacks` function.
Save your plugin as e.g `mycallback.py`, then install it in the Koji Hub
plugins folder: `/usr/lib/koji-hub-plugins`
Finally, edit the Koji Hub config file, `/etc/koji-hub/hub.conf`:
# A space-separated list of plugins to enable
Plugins = mycallback
Restart the Koji Hub service, and your plugin will be enabled.
You can try triggering your callback plugin with the command-line. For
example, if you registered a callback for the `postTag` event, try tagging a
build:
$ koji tag-build mytag mypkg-1.0-1

342
docs/source/HOWTO.rst Normal file
View file

@ -0,0 +1,342 @@
==========
Koji HOWTO
==========
Introduction
============
Koji is a system for building and tracking RPMs. It was designed with
the following features in mind:
**Security**
- New buildroot for each build
- nfs is used (mostly) read-only
**Leverage other software**
- Uses Yum and Mock open-source components
- XML-RPC APIs for easy integration with other tools
**Flexibility**
- rich data model
- active code base
**Usability**
- Web interface with Kerberos authentication
- Thin, portable client
- Users can create local buildroots
**Reproducibility**
- Buildroot contents are tracked in the database
- Versioned data
This HOWTO document covers the basic tasks that a developer needs to be
able to accomplish with Koji.
Getting started
===============
The web interface
-----------------
The primary interface for viewing Koji data is a web application. Most
of the interface is read-only, but if you are logged in (see below) and
have sufficient privileges there are some actions that can be performed
though the web. For example:
- Cancel a build
- Resubmit a failed task
Those with admin privileges will find additional actions, such as:
- Create/Edit/Delete a tag
- Create/Edit/Delete a target
- Enable/Disable a build host
The web site utilizes Kerberos authentication. In order to log in you
will need a valid Kerberos ticket and your web browser will need to be
configured to send the Kerberos information to the server.
In Firefox or Mozilla, you will need to use the about:config page to set
a few parameters. Use the search term 'negotiate' to filter the list.
Change network.negotiate-auth.trusted-uris to the domain you want to
authenticate against, e.g .example.com. You can leave
network.negotiate-auth.delegation-uris blank, as it enables Kerberos
ticket passing, which is not required. If you do not see those two
config options listed, your version of Firefox or Mozilla may be too old
to support Negotiate authentication, and you should consider upgrading.
In order to obtain a Kerberos ticket, use the kinit command.
Installing the Koji cli
-----------------------
There is a single point of entry for most operations. The command is
called 'koji' and is included in the main koji package.
Repos/webpage TBD
The koji tool authenticates to the central server using Kerberos, so you
will need to have a valid Kerberos ticket to use many features. However,
many of the read-only commands will work without authentication.
Building a package
------------------
Builds are initiated with the command line tool. To build a package, the
syntax is:
::
$ koji build <build target> <cvs URL>
For example:
::
$ koji build dist-fc7-scratch 'cvs://cvs.example.com/cvs/dist?rpms/kernel/FC-7#kernel-2_6_20-1_2925_fc7'
The ``koji build`` command creates a build task in Koji. By default the
tool will wait and print status updates until the build completes. You
can override this with the ``--nowait`` option. To view other options to
the build command use the ``--help`` option.
::
$ koji build --help
Build Options
-------------
There are a few options to the build command. Here are some more
detailed explanations of them:
``--skip-tag``
Normally the package is tagged after the build completes. This
option causes the tagging step to be skipped. The package will be in
the system, but untagged (you can later tag it with the tag-pkg
command)
``--scratch``
This makes the build into a scratch build. The build will not be
imported into the db, it will just be built. The rpms will land
under <topdir>/scratch. Scratch builds are not tracked and can never
be tagged, but can be convenient for testing. Scratch builds are
typically removed from the filesystem after one week.
``--nowait``
As stated above, this prevents the cli from waiting on the build
task.
``--arch-override``
This option allows you to override the base set of arches to build
for. This option is really only for testing during the beta period,
but it may be retained for scratch builds in the future.
Build Failures
--------------
If your package fails to build, you will see something like this.
::
420066 buildArch (kernel-2.6.18-1.2739.10.9.el5.jjf.215394.2.src.rpm,
ia64): open (build-1.example.com) -> FAILED: BuildrootError:
error building package (arch ia64), mock exited with status 10
You can figure out why the build failed by looking at the log files. If
there is a build.log, start there. Otherwise, look at init.log
::
$ ls -1 <topdir>/work/tasks/420066/*
<topdir>/work/tasks/420066/build.log
<topdir>/work/tasks/420066/init.log
<topdir>/work/tasks/420066/mockconfig.log
<topdir>/work/tasks/420066/root.log
Filing Bugs
-----------
bug tracking TBD
Koji Architecture
=================
Terminology
-----------
In Koji, it is sometimes necessary to distinguish between the a package
in general, a specific build of a package, and the various rpm files
created by a build. When precision is needed, these terms should be
interpreted as follows:
Package
The name of a source rpm. This refers to the package in general and
not any particular build or subpackage. For example: kernel, glibc,
etc.
Build
A particular build of a package. This refers to the entire build:
all arches and subpackages. For example: kernel-2.6.9-34.EL,
glibc-2.3.4-2.19.
RPM
A particular rpm. A specific arch and subpackage of a build. For
example: kernel-2.6.9-34.EL.x86\_64, kernel-devel-2.6.9-34.EL.s390,
glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64
Koji Components
---------------
Koji is comprised of several components:
- **koji-hub** is the center of all Koji operations. It is an XML-RPC
server running under mod\_python in Apache. koji-hub is passive in
that it only receives XML-RPC calls and relies upon the build daemons
and other components to initiate communication. koji-hub is the only
component that has direct access to the database and is one of the
two components that have write access to the file system.
- **kojid** is the build daemon that runs on each of the build machines.
Its primary responsibility is polling for incoming build requests and
handling them accordingly. Koji also has support for tasks other than
building. Creating install images is one example. kojid is
responsible for handling these tasks as well.
kojid uses mock for building. It also creates a fresh buildroot for
every build. kojid is written in Python and communicates with
koji-hub via XML-RPC.
- **koji-web** is a set of scripts that run in mod\_python and use the
Cheetah templating engine to provide an web interface to Koji.
koji-web exposes a lot of information and also provides a means for
certain operations, such as cancelling builds.
- **koji** is a CLI written in Python that provides many hooks into Koji.
It allows the user to query much of the data as well as perform
actions such as build initiation.
- **kojirepod** is a daemon that keeps the build root repodata updated.
Package Organization
--------------------
**Tags and Targets**
Koji organizes packages using tags. In Koji a tag is roughly analogous
to a beehive collection instance, but differ in a number of ways:
- Tags are tracked in the database but not on disk
- Tags support multiple inheritance
- Each tag has its own list of valid packages (inheritable)
- Package ownership can be set per-tag (inheritable)
- Tag inheritance is more configurable
- When you build you specify a *target* rather than a tag
A build target specifies where a package should be built and how it
should be tagged afterwards. This allows target names to remain fixed as
tags change through releases. You can get a full list of build targets
with the following command:
::
$ koji list-targets
You can see just a single target with the ``--name`` option:
::
$ koji list-targets --name dist-fc7
Name Buildroot Destination
---------------------------------------------------------------------------------------------
dist-fc7 dist-fc7-build dist-fc7
This tells you a build for target dist-fc7 will use a buildroot with
packages from the tag dist-fc7-build and tag the resulting packages as
dist-fc7.
You can get a list of tags with the following command:
::
$ koji list-tags
*Package lists*
As mentioned above, each tag has its own list of packages that may be
placed in the tag. To see that list for a tag, use the ``list-pkgs``
command:
::
$ koji list-pkgs --tag dist-fc7
Package Tag Extra Arches Owner
----------------------- ----------------------- ---------------- ----------------
ElectricFence dist-fc6 pmachata
GConf2 dist-fc6 rstrode
lucene dist-fc6 dbhole
lvm2 dist-fc6 lvm-team
ImageMagick dist-fc6 nmurray
m17n-db dist-fc6 majain
m17n-lib dist-fc6 majain
MAKEDEV dist-fc6 clumens
...
The first column is the name of the package, the second tells you which
tag the package entry has been inherited from, and the third tells you
the owner of the package.
**Latest Builds**
To see the latest builds for a tag, use the ``latest-pkg`` command:
::
$ koji latest-pkg --all dist-fc7
Build Tag Built by
---------------------------------------- -------------------- ----------------
ConsoleKit-0.1.0-5.fc7 dist-fc7 davidz
ElectricFence-2.2.2-20.2.2 dist-fc6 jkeating
GConf2-2.16.0-6.fc7 dist-fc7 mclasen
ImageMagick-6.2.8.0-3.fc6.1 dist-fc6-updates nmurray
MAKEDEV-3.23-1.2 dist-fc6 nalin
MySQL-python-1.2.1_p2-2 dist-fc7 katzj
NetworkManager-0.6.5-0.3.cvs20061025.fc7 dist-fc7 caillon
ORBit2-2.14.6-1.fc7 dist-fc7 mclasen
The output gives you not only the latest builds, but which tag they have
been inherited from and who built them (note: for builds imported from
beehive the "built by" field may be misleading)
Exploring Koji
--------------
We've tried to make Koji self-documenting wherever possible. The command
line tool will print a list of valid commands and each command supports
``--help``. For example:
::
$ koji help
Koji commands are:
build Build a package from source
cancel-task Cancel a task
help List available commands
latest-build Print the latest rpms for a tag
latest-pkg Print the latest builds for a tag
...
$ koji build --help
usage: koji build [options] tag URL
(Specify the --help global option for a list of other help options)
options:
-h, --help show this help message and exit
--skip-tag Do not attempt to tag package
--scratch Perform a scratch build
--nowait Don't wait on build
...
Getting Involved
================
If you would like to be more involved with the Koji project...
Project data TBD

261
docs/source/conf.py Normal file
View file

@ -0,0 +1,261 @@
# -*- coding: utf-8 -*-
#
# Koji documentation build configuration file, created by
# sphinx-quickstart on Tue Nov 3 12:11:18 2015.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'Koji'
copyright = u'2015, Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '1.10.0'
# The full version, including alpha/beta/rc tags.
release = '1.10.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'default'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'Kojidoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'Koji.tex', u'Koji Documentation',
u'Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'koji', u'Koji Documentation',
[u'Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller'], 1)
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'Koji', u'Koji Documentation',
u'Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller', 'Koji', 'One line description of project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
#texinfo_no_detailmenu = False

256
docs/source/defining_hub_policies.rst Normal file
View file

@ -0,0 +1,256 @@
=====================
Defining Hub Policies
=====================
Defining a policy on the hub allows you fine control over certain activities
in the system. At present, policy allows you to control:
* tag/untag/move operations
* allowing builds from srpm
* allowing builds from expired repos
* managing the package list for a tag
* managing which channel a task goes to
In the future, we expect to add more policy hooks for controlling more aspects
of the system.
Policy configuration is optional. If you don't define one, then by default:
* tag/untag/move operations are governed by tag locks/permissions
* builds from srpm are only allowed for admins
* builds from expired repos are only allowed for admins
* only admins may modify package lists
* tasks go to the default channel
Configuration
=============
The hub policy is configured in the ``hub.conf`` file, which is an ini-style
configuration file. Policies are defined in the section named ``[policy]``.
Each ``name = value`` pair defines the policy of that name. With multiple line
policies, successive lines should be indented so that the parser treats them
as part of the whole.
Consider the following simple (and strict) example:
::
[policy]
tag =
has_perm admin :: allow
tag *-candidate :: allow
all :: deny
This policy section defines a single policy (named 'tag'). The policy is a
series of rules, one per line. The rule lines must be indented. Each rule is
a test and an action, separated by a double colon. The valid actions for
current policies are 'allow' and 'deny'. There are many tests available,
though not all of them are applicable for all policies. Each test is specified
by giving the name of the test followed by any arguments the test accepts.
Each rule in the policy is checked until a match is found. Upon finding a
match, the action is applied. Our example above limits non-admins to tags
ending in -candidate.
Getting a bit more complicated
The example above is very simple. The policy syntax also supports compound
tests, negated tests, and nested tests. Consider the following example:
::
[policy]
tag =
buildtag *epel* :: {
tag *epel* !! deny
}
tag *-updates :: {
operation move :: {
fromtag *-updates-candidate :: allow
fromtag *-updates-testing :: allow
all :: deny
}
operation tag && hastag *-updates-candidate *-updates-testing :: deny
}
all :: allow
This policy sets up some rules concerning tags ending in -updates and tags
containing epel, but is otherwise permissive.
The first nested rule limits builds built from a tag matching ``epel`` to only
such tags. Note the use of !! instead of :: negates the test.
For tags matching ``*-updates``, a particular work-flow is enforced. Moving is
only allowed if the move is coming from a tag matching ``*-updates-candidate``
or ``*-updates-testing``. Conversely, a basic tag operation (not a move) is
denied if the build also has such a tag (the policy requires a move instead).
General format
==============
The general form of a basic policy line is one of the following
::
test [params] [&& test [params] ...] :: action-if-true
test [params] [&& test [params] ...] !! action-if-false
And for nested rules:
::
test [params] [&& ...] [::|!!] {
test [params] [&& ...] [::|!!] action
test [params] [&& ...] [::|!!] {
...
}
}
Note that each closing brace must be on a line by itself.
Using ``!!`` instead of ``::`` negates the entire test.
Tests can only be joined with &&, the syntax does not support ``||``.
Available policies
==================
The system currently looks for the following policies
* tag : checked during tag/untag/move operations
* build_from_srpm : checked when a build from srpm (not an SCM reference) is
requested.
* build_from_repo_id : checked when a build from a specified repo id is
requested
* package_list : checked when the package list for a tag is modified
* channel : consulted when a task is created
These policies are set by assigning a rule set to the given name in the policy
section.
Note that the use of tag policies does not bypass tag locks or permissions
Note that an admin can bypass the tag policy by using ``--force``.
Actions
=======
Most of the policies are simply allow/deny policies. They have two possible
actions: ``allow`` or ``deny``.
The channel policy is used to determine the channel for a task. It supports
the following actions:
``use <channel>``
* use the given channel
``req``
* use the requested channel
* generally this means the default, though some calls allow the client to
request a channel
``parent``
* use the parent's channel
* only valid for child tasks
* recommend using the ``is_child_task`` test to be sure
Available tests
===============
``true``
* always true. no arguments
``all``
* an alias of true
``false``
* always false. no arguments
``none``
* an alias of false
``operation``
* for tag operations, the operation is one of: tag, untag, move. This test
checks its arguments against the name of the operation and returns true if
there is a match. Accepts glob patterns.
* only applicable to the tag policy
``package``
* Matches its arguments against the package name. Accepts glob patterns.
``tag``
* matches its arguments against the tag name. Accepts glob patterns.
* for move operations, the tag name tested is the destination tag (see
fromtag)
* for untag operations, the tag name is null and this test will always be
false (see fromtag)
* for the build_from_* policies, tests the destination tag for the build
(which will be null is --skip-tag is used)
``fromtag``
* matches against the tag name that a build is leaving. Accepts glob
patterns
* for tag operations, the tag name is null and this test will always be
false
* for move operations, the tag name test is the one that the build is
moving from
* for untag operations, tests the tag the build is being removed from
* only applicable to the tag policy
``hastag``
* checks the current tags for the build in question against the arguments.
``buildtag``
* checks the build tag name against the arguments
* for the build_from_* policies the build tag is determined by the build
target requested
* for the tag policies, determines the build tag from the build data,
which will by null for imported builds
``skip_tag``
* checks to see if the --skip-tag option was used
* only applicable to the build_from_* policies
``imported``
* checks to see if the build in question was imported
* takes no arguments
* true if any of the component rpms in the build lacks buildroot data
* only applicable to the tag policy
``is_build_owner``
* Check if requesting user owns the build (not the same as package
ownership)
* take no arguments
``user_in_group``
* matches the users groups against the arguments
* true if user is in /any/ matching group
``has_perm``
* matches the user's permissions against the arguments
* true is user has /any/ matching permission
``source``
* test the build source against the arguments
* for the build_from_* policies, this is the source specified for the build
* for the tag policy, this comes from the task corresponding to the build
(and will be null for imported builds)
``policy``
* takes a single argument, which is the name of another policy to check
* checks the named policy. true if the resulting action is one of: yes,
true, allow
* additional policies are defined in the [policy] section, just like the
others
``is_new_package``
* true if the package being added is new to the system
* intended for use with the package_list policy
``is_child_task``
* true if the task is a child task
* for use with the channel policy
``method``
* matches the task method name agaist glob pattern(s)
* true if the method name matches any of the patterns
* for use with the channel policy
``user``
* checks the username against glob patterns
* true if any pattern matches
* the user matched is the user performing the action

136
docs/source/external_repo_server_bootstrap.rst Normal file
View file

@ -0,0 +1,136 @@
====================================
External Repository Server Bootstrap
====================================
Bootstrapping a new External Repo Koji build environment
========================================================
These are the steps involved in pointing a new Koji server at external
repositories so that it can be used for building. This assumes that the Koji
hub is up, appropriate authentication methods have been configured, the Koji
repo administration daemon (``kojira``) is properly configured and running,
and at least one Koji builder (``kojid``) is properly configured and running.
All koji cli commands assume that the user is a Koji ``admin``. If you need
help with these tasks, see the :doc:`ServerHowTo <server_howto>`.
* Create a new tag. ::
$ koji add-tag dist-foo
* Create a build tag with the desired arches, and the previously created tag
as a parent. ::
$ koji add-tag --parent dist-foo --arches "i386 x86_64 ppc ppc64" dist-foo-build
* Add an external repository to your build tag. Koji substitutes $arch for the
arches in your build tag. The ``$`` needs to be escaped for the shell to send
it though without interpreting it. See below for some additional examples of
external repo URLs. ::
$ koji add-external-repo -t dist-foo-build dist-foo-external-repo http://repo-server.example.com/path/to/repo/for/foo/\$arch/
.. note::
If you are adding multiple external repos, koji assigns a priority to each
repo in FIFO order. This may cause updated packages to not be visible if a
repo with older packages is ranked at a higher priority (lower numeric
value). Use the ``-p`` flag to set specific repo priorities.
.. note::
This uses $arch NOT $basearch
* Create a build target that includes the tags you've already created. ::
$ koji add-target dist-foo dist-foo-build
* Create a ''build'' and ''srpm-build'' group associated with your build tag. ::
$ koji add-group dist-foo-build build
$ koji add-group dist-foo-build srpm-build
* Populate the ``build`` and ``srpm-build`` group with packages that will be
installed into the minimal buildroot. You can find out what the is in the
current groups for Fedora by running ``koji list-groups dist-f9-build``
against the Fedora Koji instance. This is probably a good starting point for
your minimal buildroot and srpm creation buildroot. If you are rebuilding
Fedora packages, it is recommended that you add all packages listed in the
Fedora Koji instance. ::
$ koji add-group-pkg dist-foo-build build <pkg1> <pkg2> .....
* Add packages that you intend to build to your tag. ::
$ koji add-pkg --owner <kojiuser> dist-foo <pkg1> <pkg2> .....
* Wait for the repo to regenerate, and you should now be able to run a build
successfully.
Regenerating your repo
======================
koji doesn't monitor external repositories for changes. new repositories
will be generated when packages you build land in a tag that populates
the buildroot or you manually regenerate the repository. you should be
sure to regularly regenerate the repositories manually to pick up
updates.
::
$ koji regen-repo dist-foo-build
Examples of urls to use for external Repositories
=================================================
all these examples use mirrors.kernel.org please find the closest mirror
to yourself. Note that the Fedora minimal buildroots download ~100Mb
then build dependencies on top. these are downloaded each build you can
save alot of network bandwidth by using a local mirror or running
through a caching proxy.
NOTE: this uses $arch **NOT** $basearch
Fedora 10
---------
::
http://mirrors.kernel.org/fedora/releases/10/Everything/\$arch/os/
http://mirrors.kernel.org/fedora/updates/10/\$arch/
CentOS 5 and EPEL
-----------------
::
http://mirrors.kernel.org/centos/5/os/\$arch/
http://mirrors.kernel.org/centos/5/updates/\$arch/
http://mirrors.kernel.org/fedora-epel/5/\$arch/
Example tags and targets
========================
In the simplest setup, where you just want to build against what is
available in the external repositories, you may want to go with a simple
layout of *dist-f\ **X**-build* tags inheriting one another, and
*dist-f\ **X**-updates* tags and targets that inherit the
*dist-f\ **X**-build* tag and have external repos attached to them. This
way, a *dist-f\ **Y**-build* or *dist-f\ **Y**-updates* tag will not
automatically inherit the external repos of your *dist-f\ **X*** tags.
Tags
----
::
dist-f10-updates - This is where the external repos for f10 release and f10 updates are attached
`- dist-f10-build - This is the f10 build target with the 'build' and 'srpm-build' group inherited from dist-f9-build,
| so that your buildroot gets populated but you do not have to maintain these groups for each
| seperate release.
`- dist-f9-build - etc.
`- dist-f8-build - etc.
Targets
-------
Each *dist-f\ **X**-build* tag has a *dist-f\ **X**-updates* child tag,
and each *dist-f\ **X**-updates* tag has a corresponding
*dist-f\ **X**-updates-candidate* build target.

943
docs/source/image_build.rst Normal file
View file

@ -0,0 +1,943 @@
=======================
Building Images in Koji
=======================
Image Building
==============
Koji is capable of building many different types of images or appliances. They
broadly fall into a few categories: LiveCDs, Disk Images, and Containers. All
types of images end up with a Name-Version-Release just like an RPM build.
What you need to provide Koji to perform a build varies by category, and the
specifics will be explained below.
For additional questions and information, ask around in ``#koji`` on FreeNode
IRC and sign up to the right mailing lists as the `Koji project website`_
dictates.
Kickstart First
===============
No matter which type of image you want to build, you need to feed Koji a
`kickstart`_ file, so we touch on how to create them and how to get them to Koji
first. Once you have a kickstart to try out, look over the kickstart-specific
caveats for each image type in the later sections.
Kickstart Reference
-------------------
If you're new to automated installations, you should read the `Anaconda
Kickstart Guide`_ first.
Getting your Kickstart to Koji
------------------------------
For scratch builds, you can get away with just using the ``--kickstart``
parameter, which accepts a path (relative or absolute) to your kickstart file
on disk. For non-scratch builds though, the kickstart file needs to live in a
remote Source Control Manager (SCM) such as git or Subversion. To access that,
you need to pass the ``--ksurl`` parameter, which accepts a wacky string in
this form:
::
git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7
Here we pointed to a repository hosted at
``git://git.fedorahosted.org/git/spin-kickstarts.git``. Note the ``?`` in
there, this delimiter separates the host URL with a directory structure to
look in for the kickstart file. In other words, if you were to clone the
repository, you should expect to see these (sub)directories laid out below the
root of the clone. The important thing to realize is the behavior of
``--kickstart`` changes if ``--ksurl`` is specified. Instead of indicating a
path to a kickstart file, you just indicate a filename. The file itself is not
specified in the fragment of the string between ``?`` and ``#`` that is passed
to ``--ksurl``, Koji still uses the ``--kickstart`` option for that. After the
directory structure comes a ``#``, which indicates the start of the commit ID.
This is the commit Koji will reset the repository to after cloning. If you
created a git repository which had just the spec file template in it, your SCM
URL would have the ``?`` and ``#`` right next to each other.
If we passed ``--ksurl`` the string above, and gave
``--kickstart "server-ec2.ks"``, then Koji would do the following:
* Clone the repository locally
* Call ``git reset --hard 68c40eb7``
* Search in the "fedora22" subdirectory of the clone for a file called
``server-ec2.ks``, and pass that to Anaconda
* Perform an automated install
.. _building-livecds:
Building LiveCDs
================
Let's describe building LiveCDs and the mechanics behind it.
Getting Started
---------------
What you need before proceeding:
* a name for your LiveCD, and a version number
* a Koji build target
* what architectures you want
* a flattened kickstart file
* you may also need yum repositories generated by release engineering if you
require signed packages be installed into the LiveCD
* the livecd permission in Koji
The Koji command that creates a LiveCD is called ``spin-livecd``. It calls out
to `livecd-tools`_ in a mock chroot to construct the LiveCD. To run a LiveCD
build, use the spin-livecd command like so:
::
$ koji spin-livecd --release 4 fedora-workstation 23 f23-image-build fedora-workstation.ks
In this example a LiveCD will be created with the N-V-R of
fedora-workstation-23-4. If ``--release`` was not included an incrementing
value would be computed by Koji instead. spin-livecd takes a minimum of 5
arguments:
Name
the name of the image, without versioning information. Examples could be
``fedora`` or ``fedora-workstation``.
Version
an arbitrary version string. For Fedora images, this usually matches the
release version, such as 21, 22, or 23.
Build Target
just like RPM builds Koji must be told which target to use. This controls
what tag the image will be tagged into, and what packages are available to
install into the image.
Architecture
only x86_64 or i386 is supported
Kickstart File
this is a recipe file that performs drives the construction of the image.
Use ``--help`` to discover more options to ``spin-livecd``. You can override
the Release field with ``--release``, or ``--repo`` to override what yum repo
is used to provide packages to install to the image. Note that regardless of
what repo you use, it must contain RPMs built by Koji, otherwise you will get
an error. (unless you are building a scratch image)
Mechanics
---------
The way LiveCDs are created in Koji is fairly straightforward. A chroot is
initialized and populated with the packages and their dependencies from the
``livecd-build`` package group. Next, the kickstart file is copied into it if
it was provided from local storage. If not, it is checked out into it from an
SCM. It is then modified to use the repo associated with the build tag for the
target specified in the command unless the ``--repo`` option was given. Both
the original and the modified kickstart files are saved as part of the output
for the task for later review. A `livecd-creator`_ command is executed using the
mock('--chroot', ...) method.
.. note::
This process runs as root. This produces the desired image which is
uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.
.. _caveats-for-livecds:
Caveats for LiveCDs
-------------------
There are some known caveats with using the spin-livecd command that users
should be aware of.
%include macros in the kickstart
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A word of caution about kickstart files and the ``%include`` macro.
`livecd-creator`_ is smart enough to search the current directory of the
submitted kickstart file if it has %include macros. If the kickstart specified
to koji is from local storage, only that kickstart file will be copied into
the chroot, and this creates a problem if it has ``%include`` macros, because
the other kickstart files it needs will be inaccessible. This issue is not
present when the kickstart file is retrieved from a remote SCM (such as the
spin-kickstarts git repo), because the entire repository is checked out.
Presumably it will include any other kickstart files the specified one is
including in the same directory. A workaround for the issue would be to use
``ksflatten`` (from pykickstart) on kickstart files with ``%include`` macros
that are going to be submitted to koji from the user's local disk.
Package Groups in the Kickstart File
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Package Groups in the kickstart file cause a problem if the Koji repos do not
define them, which they most likely don't since Koji's comps.xml is based on
the "groups" set up from the CLI. `livecd-creator`_'s behavior is to ignore
package groups that are not defined in the repo it is using, so this can be
troublesome when creating the image since packages could be left out. There
are a couple possible workarounds:
* do not use package groups in the kickstart file and just specify a huge list
of packages
* use ``--repo`` and specify a repo that does have a comps.xml that defines
the groups it uses
Only Include RPMs Built in Koji
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The image building tasks will fail if your image tries to include a package
that was not built in your build system. This is because the package does not
have any origin information stored in Koji's database. The repos defined in the
kickstart will automatically be overridden with the repo of the build tag for
the build target, unless you use the ``--repo`` option. Since only packages
you have built (or include from an external repo) should be there, you should
never have this problem unless you use ``--repo``.
No Signed or Debuginfo RPMs in Koji's Build Tags
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you need signed RPMs or debuginfo RPMs, you will run into trouble because
Koji does not keep those in its build tag repos. The only work around for this
is to create a repo yourself that includes these RPMs and then use ``--repo``.
This will force the image to take RPMs from that repo. Remember, the task will
fail if Koji detects RPMs were installed that were not built in the build
system.
%post Section in Kickstart
^^^^^^^^^^^^^^^^^^^^^^^^^^
While livecd-tools does support building on SELinux disabled hosts, you can run
into denials when booting if you create and use new files in the ``%post``
section of your kickstart file. If you do, you should either set the labels
appropriately at the end of the ``%post`` section, or instigate an autorelabel
at boot time.
Troubleshooting
---------------
If your build fails, you will be notified on the command line. In the output
will be a URL to the Koji UI, visit that and click on the red subtask link.
From that page review ``root.log`` and ``livecd.log`` for errors. Often errors
are caused by packages being missing, or malformed kickstart files. The log
files are at the bottom of the page. If you're stuck, contact Release
Engineering.
Build System Preparation
------------------------
This section assumes you have know-how required to install and configure a new
instance of Koji, and that you have already done so. You can learn how to do so
:doc:`here <server_howto>` if you need to. Please ensure you are using the
latest version of the software and that your database schema is updated as
well. You should also have some familiarity with how `livecd-creator`_ works.
This section only covers preparation for LiveCD builds.
Follow this procedure step by step to get things prepared they way they need to
be.
#. ``koji add-host-to-channel <your-host> livecd``
add a builder to the livecd channel
#. ``koji grant-permission livecd <user>``
grant the permission to build an image type to a user. This step is
optional since admins have all permissions.
#. You will need a tag and target to build the images from. The yum repo
generated for the build tag of the target is what Koji will use to populate
the LiveCDs with by default. (the alternative is to use the ``--repo``
option, more on that later)
#. ``koji add-group <build-tag> livecd-build``
add the livecd-build group
#. ``koji add-group-pkg <build-tag> livecd-build <pkg> ...``
add packages to the livecd-build group. These package lists vary has
packages and dependencies change. As of October, 2015 for Fedora 23 the
needed packages for each image type are:
* bash, coreutils, fedora-logos, fedora-release, livecd-tools,
policycoreutils, python-dbus, sed, selinux-policy-targeted,
shadow-utils, squashfs-tools, sssd-client, tar, unzip, util-linux,
which, yum
Building Disk Images
====================
Disk images are files that represent virtual disks. They have a partition table
and filesystems on them, and are available in a variety of formats: qcow2,
vmdk, ova, Hyper-V, raw, "base" container images, and more.
Getting Started
---------------
What you need:
* a name for your LiveCD, and a version number
* what architectures you want
* a Koji build target
* kickstart file
* installation tree
* you may also need yum repositories generated by Rel-Eng if you require signed
packages be installed into the image
The Koji command to build a disk image is called ``image-build``. The
``image-build`` command uses `ImageFactory`_ and `Oz`_ to start a VM guest and
perform an automated Anaconda installation. Here is a (lengthy) example for
building a disk image.
::
$ koji image-build --repo 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' --kickstart fedora-server.ks --scratch --distro Fedora-22 --format qcow2 fedora-server-kvm 22 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' x86_64
This example builds a scratch qcow2 disk image using packages from an
additional yum repository. Without this option the yum repo to populate the
build root would be used instead. If this was the first image with the N-V of
fedora-server-kvm-22, then the N-V-R would be fedora-server-kvm-22-1, because
Koji uses an incrementing number for the release if you do not provide one.
Like all Koji commands, use ``--help`` to see more options that are available.
For Docker, Koji only supports Base Images right now using a kickstart file as
described above. In the future it will support layered images, but not before
some Docker requirements are met, and Koji is maintaining a Registry of its
own. This scoping effort is ongoing.
``image-build`` takes a minimum of 5 positional arguments, and 2 options must
be specified. They are reviewed in the list below, with the positional
arguments first.
Name
the name of the image, without versioning information. Examples could be
``fedora-server`` or ``fedora-workstation``.
Version
an arbitrary version string. For Fedora images, this usually matches the
release version, such as 22 or 23.
Build Target
just like RPM builds Koji must be told which target to use. This controls
what tag the image will be tagged into, and what packages are available to
install into the image.
Installation Tree URL
this is a URL to a location you can install an operating system from. It is
the same place you would direct a PXE-booted system to go. In 99% of cases
this location is provided by Release Engineering. It should have an
"isolinux" subdirectory and yum metadata somewhere within.
Architecture
only x86_64 or i386 is supported, and you can specify both on the command
line. This will cause two subtasks to be run, allowing you to build for
both arches in parallel. If either fail, the whole build will fail.
Kickstart File
this is a recipe file that performs drives the construction of the image.
Pass in the path to a kickstart file, which must be flattened.
Kickstart URL
in a non-scratch build, you'll need this too. For more details, see the
Getting your Kickstart to Koji section.
Distro
a string that indicates what OS is being built. These always follow the
convention of "Fedora-X", where X is the release number.
Since this command can get very long, a configuration file can drive the task
as well, using the ``--config`` option. It accepts a path to a configuration
file written in the Python ConfigParser format (like a Windows .ini). The
options are all named the same with one caveat, see below. Here's what one
could look like:
::
[image-build]
name = fedora-server-docker
version = 22
target = f22-candidate
install_tree = http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
arches = x86_64
format = qcow2,rhevm-ova,vsphere-ova
distro = Fedora-22
repo = http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
disk_size = 20
ksversion = DEVEL
kickstart = fedora-22-server-docker.ks
ksurl = git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7
specfile = git://git.fedorahosted.org/git/spin-kickstarts.git?spec_templates/fedora22#68c40eb7
A few notes on the syntax:
* it allows for comments too, the lines start with a hash (#)
* options on the command line that can be used multiple times can accept
values here as comma-separated strings
* options with a hyphen need to use an underscore instead. ``--disk-size`` for
example would be disk_size in the config file
OVA Features
------------
If you're building OVAs, either for RHEVM or vSphere, you can specify OVA
options with a special section in the configuration file. It looks something
like this:
::
[ova-options]
vsphere_product_version=22
rhevm_description=Fedora Cloud 22
vsphere_product_vendor_name=Fedora Project
ovf_memory_mb=6144
rhevm_default_display_type=1
vsphere_product_name=Fedora Cloud 22
ovf_cpu_count=4
rhevm_os_descriptor=Fedora-22
or this:
::
[ova-options]
vsphere_ova_format = vagrant-virtualbox
rhevm_ova_format = vagrant-libvirt
vagrant_sync_directory = /home/vagrant/sync
The second one is actually the secret sauce for generating an image for use in
Vagrant. At this time, you would need rename the image file extension from .ova
to .box, but otherwise this should work fine.
Kickstart Preparation
---------------------
Kickstarts for the image-build command have some specific requirements which
are covered in this section.
Required Kickstart Arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Anaconda of course requires many commands to be defined in the kickstart file.
If you're starting from scratch you should review the reference linked above,
or use an existing kickstart file in the spin-kickstarts git repo. It is
critically important that the installation be completely automated, if
Anaconda has to prompt for input for any reason, the build will fail because
you cannot send input to the guest. Some of the kickstart commands are optional
to Anaconda, but are required in Koji for your build to succeed. Here's the
list and the reasons why.
zerombr
You must tell Anaconda to wipe out the MBR in the virtual block device, if
you don't Anaconda will ask you.
clearpart --all --initlabel
Anaconda has to be told to wipe out all data on the virtual block device we
install on otherwise it will ask for confirmation to do so. Since it is
blank anyway this is harmless.
reboot
When the installation completes, the guest is rebooted. `ImageFactory`_ is
specifically looking for this behavior to conclude the installation
completed. Anaconda's default behavior is to wait for a key press to reboot
the system, but this is impossible from outside of Koji.
locking the root account
You have to lock the root account (rootpw --lock) or create a non-root user
(user), otherwise Anaconda will prompt for one.
Do not use the url command
The repo commands are overridden by Koji to point to internal Koji repos,
or what you specified on the command line with ``--repo``, it does not
override the url command if you provided it. Anaconda has a behavior where
it will prefer packages from the repositories given with the url command
over those with the repo command, and this is generally not what you want.
If Koji sees an RPM was installed that was not built in the system, it will
fail the build.
Recommended Kickstart Arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Often you want a ``%post`` section in your kickstart to perform
post-installation configuration steps. Review that section of the reference
and note that you can specify ``--log`` and ``--interpreter``. Both of these
are recommended (but not required) to assist with the development and debugging
process. Here are some other recommendations:
* You probably want the network to use dhcp, sshd to be started, and port 22
opened in the firewall to allow access as well.
* If you're building an image that will be shipped with a product, SELinux
should be enabled.
* Images that will be used in cloud deployments like OpenStack or EC2 should
have ``cloud-init`` in the package list.
* It is discouraged to have root passwords in plaintext in your kickstart file.
* If your %post section is written in bash, consider setting -x.
* For images that have multiple partitions, use the ``--asprimary`` option for
the part command that defines the root file system. This will ensure it is
the first partition on the image, which is a requirement in some cloud
environments like EC2.
Troubleshooting
---------------
If your disk image build fails, follow the link in the command line output
that takes you to the task page in the Koji web UI. Click on the failed
createImage subtask in red. On that page review the screenshot.ppm file if it
was provided, or oz.log. Most failures are from Anaconda rejecting a malformed
kickstart file, which will be indicated in the screenshot. Your installation
must be completely automatic, there can be no interactivity at all, otherwise
Anaconda will sit there indefinitely until Koji (actually ImageFactory) kills
the task.
It is very easy to write a kickstart file with bugs or that results in a system
that does not boot. This section will present a series of questions to ask
yourself and examples to help diagnose where the problem lies. Once you know
that, it should be easier to understand what you can do to inspect further.
There are 4 steps in the process:
#. create a guest
#. perform an automated installation in the guest
#. boot the guest and extract the list of installed RPMs
#. upload and archive the disk image of the guest
Is it a problem with guest creation?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There have been unusual cases where libvirt, ImageFactory, or Oz was
misconfigured and guests could not be started properly. A misconfiguration with
Puppet or whatever Fedora Infrastructure is up to can cause this. So far the
errors have been clear in the task output, look either in the results string
or oz.log. The bad news is that in this case you really can only inform Rel-Eng
about the issue and wait for a resolution. The good news is these cases are
very rare.
Did the installation fail?
^^^^^^^^^^^^^^^^^^^^^^^^^^
The Anaconda installation can fail for many reasons: missing packages, network
problems, or syntax errors in %post. Tasks will also fail if Anaconda prompts
for input for any reason. If Koji detects a lack of disk activity in the guest
for more than 5 minutes, it will fail the build and tear down the guest.
Looking in oz.log may have the answer: dracut, anaconda, and yum logs are all
printed there.
These sorts of failures often have a screenshot taken and saved with the task
output called screenshot.ppm. Viewing this will usually tell you what Anaconda
is complaining about if the installer detected an issue or prompted for input.
The string in the results output that says "No disk activity in 300 seconds,
failing." This almost always means Anaconda hit an issue and either gave up or
waited.
If Anaconda claims it is missing packages, confirm they exist in the repos you
are using with ``--repo``, if you are using that option. If you are not,
confirm the builds you expect are in the tag inheritance for the target you are
running. This is a lot like checking whether an RPM will build against the
right libraries, except we're building an image instead.
If you get the rare Anaconda dialog box that says something like "An unexpected
error occurred", try using the ``text`` command in kickstart, which will have
Anaconda boot in text mode. Sometimes the Python traceback (or whatever the
error condition is) will be printed there. I have also seen cases where
text-mode yields a black screen, but booting in graphical mode (the default)
does produce a useful dialog box. Issues like this stem from syntax errors in
the kickstart file, or bugs in pykickstart itself. If you think it is a
pykickstart bug, then someone in Rel-Eng needs to update pykickstart on the
builders.
Did the guest boot?
^^^^^^^^^^^^^^^^^^^
Koji waits 5 minutes for a guest to boot in this step. It unfortunately does
not give a lot of insight to why a guest may not boot, so these are a tougher
class of issues to work through. You can usually answer this question by
looking in results string. If you see "Timed out waiting for guest to boot",
then this is your problem. You can also confirm this in oz.log.
For now, the best way to investigate an issue like this is to drive a guest
installation locally using something like Gnome's Virtual Machine Manager
(VMM). The steps to perform are:
* Select a Network Install
* For the Operating System Install URL use the same one you gave to Koji. It
will be something like
http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/
* Set the Kickstart URL to where your kickstart file is. You may need to make
it available over http.
* Bump the memory to 2048M for good measure
* Launch the guest and let it complete installation
* Open a VNC session and watch what happens when the guest attempts to boot.
If the console is not providing enough information, we have to get more
creative. Anaconda supports starting an SSH daemon while the installation is
happening with the sshpw command in kickstart. Set that and comment out the
reboot command. This will let the installation complete locally and wait for a
keystroke to reboot the guest. At this point you should be able to ssh in and
inspect the environment to figure out what is going on. You should also
consider making use of the ``--log`` option to %post so that output from the
script is saved somewhere.
Another option would be to scp logs and other files off of the guest as part of
the ``%post`` script.
Other Guest Misconfigurations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If the guest boots but you're having problems accessing it I'd suggest
following same procedure as when the guest fails to boot. This could be a
result of firewall misconfigurations or SSH not being available for some
reason. Usually in this case the build is succeeding in Koji, but there's
something still fundamentally broken in the image. If the issue is something
you can investigate while the guest is online (you can log in), then I'd
suggest importing it locally using the libvirt.xml and the disk image provided
in Koji's task output.
You can also do investigative work in an offline mode by mounting the image
locally or using something like libguestfs to poke around without starting the
guest. The fast, dirty way to do it is by mounting it. This can often pollute
your guest environment. Here's how to do it:
* Download the image from Koji
* If the image format is not raw, you have to convert it first with qemu-img.
Something like:
::
$ qemu-img convert -O raw <image-file> <output-file></pre>
* Now mount it up using loopback devices. (as root) If your image has multiple
partitions in it, you may need to pass in a different mapped loopback device
like ``loop0p2``. Whichever one you think is the root partition or has the
issue you're trying to fix.
::
# kpartx -av <raw-image>
# mount -o loop /dev/mapper/loop0p1 /mnt/my_directory
Hopefully at this point you figure out the issue. To tear down the image you'll
run commands as root like so:
::
# umount /mnt/my_directory
# dmsetup remove loop0p1
# losetup -d /dev/loop0
Again, if you used different loopback devices, substitute those in to the
dmsetup and losetup commands.
Build System Preparation
------------------------
Follow this guide if you're a Koji admin and would like to enable image
building or want to set up some testing before enabling the integration.
When moving to ImageFactory to do image builds Koji lost the ability to easily
reproduce the build environment for images the way we do for RPMs using Mock.
This section will document how to set up an image builder for Koji. This is a
lengthy task, it will take first-timers about a week to have a useful instance,
and it is painful because it requires a bare metal system be provisioned since
`ImageFactory`_ provisions VMs to build the image. There is a significant
performance penalty for using nested virtualization.
Follow the steps below to set up your builder.
.. note::
You do not have to stand up a complete Koji instance to test the way Koji
builds images. However, if you want to test image builds with an accurate
representation of how Koji does it, or you want to test code changes
related to image builds in Koji, you should follow all the steps below.
ImageFactory/Oz Preparation
^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. Provision a system with at least 4G of memory with the current release of
RHEL 6 or later. Sometimes builders lag behind a month or two before taking
in updates, but this will still get you pretty close to where you want to
be. For Fedora, use the latest release.
#. Install the following packages to the builder.
#. oz
#. imagefactory
#. imagefactory-plugins-TinMan
#. imagefactory-plugins-vSphere
#. imagefactory-plugins-ovfcommon
#. imagefactory-plugins-docker
#. imagefactory-plugins
#. imagefactory-plugins-OVA
#. imagefactory-plugins-RHEVM
#. python-psphere => 0.5
#. VMDKStream => 0.2
#. pykickstart
#. Edit ``/etc/oz/oz.cfg``, and set the memory value in the ``[libvirt]``
section to at least 2048. Set ``safe_generation`` under ``[icicle]`` to yes.
#. Run: ``mkdir -p ~root/.psphere/templates``, and then copy the following
code into ``~root/.psphere/config.yaml``. Do not worry about the server,
username, and password credentials; they are not used anywhere.
::
general:
server: 10.16.120.224
username: Administrator
password: whatever
template_dir: ~/.psphere/templates/
logging:
destination: ~/.psphere/psphere.log
level: DEBUG # DEBUG, INFO, etc
Start up the services and ImageFactory/Oz should be ready to go. You should
read more about `how to use Oz`_ and `how to use ImageFactory`_. If you want
to try calling `ImageFactory`_ as if from a Koji Builder (but not set up a
whole Koji instance), you can use the code below to emulate that. If you want
to test the Koji integration with a full Koji instance, proceed to the next
section instead.
::
#!/usr/bin/python -tt
import logging
import os.path
import random
import sys
from imgfac.BuildDispatcher import BuildDispatcher
from imgfac.PluginManager import PluginManager
from imgfac.ReservationManager import ReservationManager
plugin_mgr = PluginManager('/etc/imagefactory/plugins.d')
plugin_mgr.load()
from imgfac.ApplicationConfiguration import ApplicationConfiguration
# logging
handler = logging.StreamHandler(sys.stdout)
tlog = logging.getLogger()
tlog.setLevel(logging.DEBUG)
tlog.addHandler(handler)
# configuration
ks = open('oztest.ks').read()
workdir = '/tmp/koji/test'
config = {
#Oz specific
'oz_data_dir': os.path.join(workdir, 'oz_data'),
'oz_screenshot_dir': os.path.join(workdir, 'oz_screenshots'),
#IF specific
'imgdir': os.path.join(workdir, 'scratch_images'),
'tmpdir': os.path.join(workdir, 'oz-tmp'),
'verbose' : True,
'timeout': 3600,
'output': 'log',
'raw': False,
'debug': True,
'image_manager': 'file',
'plugins': '/etc/imagefactory/plugins.d',
'tdl_require_root_pw': False,
'image_manager_args': {
'storage_path': os.path.join(workdir, 'output_image')},
}
random.seed() # necessary to ensure a unique mac address
rm = ReservationManager()
rm._listen_port = random.randint(rm.MIN_PORT, rm.MAX_PORT)
ApplicationConfiguration(configuration=config)
params = {'install_script': ks}
template = """<template>
<name>test-appliance</name>
<os>
<name>Fedora</name>
<version>22</version>
<arch>x86_64</arch>
<install type='url'>
<url>http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/</url>
</install>
<icicle>
<extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>
</icicle>
</os>
<description>test-appliance OS</description>
<disk>
<size>16G</size>
</disk>
</template>
"""
bd = BuildDispatcher()
# build the image
base = bd.builder_for_base_image(template, parameters=params)
base.base_thread.join()
tlog.removeHandler(handler)
This script is run as root with no arguments. It uses ``oztest.ks`` in your
local directory as the kickstart you want to try to use. The URL in the XML
template is where the RPM packages will be installed from, and what the guest
will be booted with.
Koji Preparation
^^^^^^^^^^^^^^^^
#. :doc:`Install Koji <server_howto>` if you need it. This will easily be the
most time-consuming part of the process; my first time took 3 days to get
it working properly. Follow the guide closely, and go with the SSL
authentication method. SSL is a lot easier to set up locally. You will need
to install every Koji component (except koji-vmd) on the same system.
Proceed to the next step after you've had a successful Kojira repository
generated.
#. At this point, you have a system that should be ready to build images. We
just have to do some Koji configuration so that your instance is pulling
content from Koji. Replace the base tag names with whatever fits your
conventions.
#. Add tags, tag inheritance, new pkg (with pkg owner), new external repo, and regen the repo
::
koji add-tag fedora22
koji add-tag jay-fedora22
koji add-tag jay-fedora22-override --parent jay-1-fedora22
koji add-tag jay-fedora22-build --arches x86_64 --parent jay-fedora22-override
koji add-tag jay-fedora22-candidate --parent jay-fedora22
koji add-tag-inheritance --priority 40 jay-fedora22-build fedora22
koji add-pkg --owner kojiadmin jay-fedora22 fedora-server-ec2 fedora-server-kvm
koji add-external-repo -t fedora::20 fedora22 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'
koji add-target jay-fedora22-candidate jay-fedora22-build
koji regen-repo jay-fedora22-build
#. Grab a kickstart file from an image task in Koji that relates to what you want to test.
#. Finally, kick off a build!
::
koji image-build fedora-server-ec2 22 --distro Fedora-22 jay-fedora22-candidate --kickstart fedora-server-starter-ec2.ks 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'
Building Appliances
===================
This section is here for the sake of legacy. Unless you are trying to build
ARM images, you should use the image-build command described in the previous
section.
.. note::
The spin-appliance command, described herein, is deprecated.
Getting Started
---------------
Here's what you need before proceeding:
* a name for your Appliance, and a version number
* a Koji build target
* what architectures you want
* a flattened kickstart file
* you may also need yum repositories generated by release engineering if you
require signed packages be installed into the appliance
* the appliance permission in Koji
The Koji command that creates an Appliance is called ``spin-appliance``. It
calls out to appliance-creator in a mock chroot to construct the Appliance.
To run an Appliance build, use the spin-appliance command like so:
::
$ koji spin-appliance --release 4 fedora-workstation 23 f23-build fedora-workstation.ks
In this example an Appliance will be created with the N-V-R of
``fedora-workstation-23-4``. If ``--release`` was not included an incrementing
value would be computed by Koji instead. spin-appliance takes a minimum of 5
arguments:
Name
the name of the image, without versioning information. Examples could be
``fedora`` or ``fedora-workstation``.
Version
an arbitrary version string. For Fedora images, this usually matches the
release version, such as 21, 22, or 23.
Build Target
just like RPM builds Koji must be told which target to use. This controls
what tag the image will be tagged into, and what packages are available to
install into the image.
Architecture
only arm, x86_64, or i386 are supported
Kickstart File
this is a recipe file that performs drives the construction of the image.
Use ``--help`` to discover more options to spin-appliance. You can override the
Release field with ``--release``, or ``--repo`` to override what yum repo is
used to provide packages to install to the image. Note that regardless of what
repo you use, it must contain RPMs built by Koji, otherwise you will get an
error. (unless you are building a scratch image)
Mechanics
---------
The way Appliances are created in Koji is the same as :ref:`building-livecds`.
Caveats for Appliances
----------------------
Caveats for using ``spin-appliance`` are the same as using ``spin-livecd`` to
:ref:`caveats-for-livecds`.
Troubleshooting
---------------
If your build fails, you will be notified on the command line. In the output
will be a URL to the Koji UI, visit that and click on the red subtask link.
From that page review ``root.log`` and ``appliance.log`` for errors. Often
errors are caused by packages being missing, or malformed kickstart files. The
log files are at the bottom of the page. If you're stuck, contact Release
Engineering.
Build System Preparation
------------------------
This section assumes you have know-how required to install and configure a new
instance of Koji, and that you have already done so. You can learn how to do so
:doc:`here <server_howto>` if you need to. Please ensure you are using the
latest version of the software and that your database schema is updated as
well. You should also have some familiarity with how `appliance-creator`_
works. This section only covers preparation for Appliance builds.
Follow this procedure step by step to get things prepared they way they need
to be.
#. Add a builder to the appliance channel
::
koji add-host-to-channel <your-host> appliance
#. Grant the permission to build an appliance to a user. This step is optional
since admins have all permissions.
::
koji grant-permission appliance <user>
#. You will need a tag and target to build the images from. The yum repo
generated for the build tag of the target is what Koji will use to populate
the Appliances with by default. (the alternative is to use the ``--repo``
option, more on that later)
#. Add the appliance-build group
::
koji add-group <build-tag> appliance-build``
#. Add packages to the appliance-build group. These package lists vary has
packages and dependencies change. As of October, 2015 for Fedora 24 the
needed packages for appliances:
* appliance-tools, bash, coreutils, grub, parted, perl, policycoreutils,
selinux-policy, shadow-utils, sssd-client
::
koji add-group-pkg <build-tag> appliance-build <pkg> ...
.. _Koji project website: https://fedorahosted.org/koji/wiki
.. _kickstart:
https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst
.. _Anaconda Kickstart Guide:
https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst
.. _livecd-tools: https://github.com/rhinstaller/livecd-tools/
.. _livecd-creator: https://fedoraproject.org/wiki/FedoraLiveCD
.. _ImageFactory: http://imgfac.org/
.. _Oz: http://github.com/clalancette/oz
.. _how to use Oz: https://github.com/clalancette/oz/wiki
.. _how to use ImageFactory: http://imgfac.org/documentation/
.. _appliance-creator: http://fedoraproject.org/wiki/Features/ApplianceTools

279
docs/source/index.rst Normal file
View file

@ -0,0 +1,279 @@
.. Koji documentation master file, created by
sphinx-quickstart on Tue Nov 3 12:11:18 2015.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
================================
Welcome to Koji's documentation!
================================
Koji
====
Koji is the software that builds `RPM packages for the Fedora project`_. It
uses `Mock`_ to create chroot environments to perform builds. To download the
source code, report bugs, join the mailing list etc., see the `Koji project
website`_.
Contents
========
.. toctree::
:maxdepth: 2
HOWTO
image_build
server_howto
defining_hub_policies
external_repo_server_bootstrap
image_build
misc
migrations
runs_here
server_bootstrap
server_howto
using_the_koji_build_system
writing_a_plugin
writing_koji_code
HowTos
======
Setting up and using Koji on Fedora:
* :doc:`Using the Koji build system <using_the_koji_build_system>`
* :doc:`Run Your Own Koji Build Server <server_howto>`
* :doc:`Building Images in Koji <image_build>`
* :doc:`Defining hub policies <defining_hub_policies>`
Koji Architecture
=================
Terminology
-----------
In Koji it is sometimes necessary to distinguish between a package in general,
a specific build of a package, and the various rpm files created by a build.
When precision is needed, these terms should be interpreted as follows:
Package
The name of a source rpm. This refers to the package in general and not
any particular build or subpackage. For example: kernel, glibc, etc.
Build
A particular build of a package. This refers to the entire build: all
arches and subpackages. For example: kernel-2.6.9-34.EL, glibc-2.3.4-2.19.
RPM
A particular rpm. A specific arch and subpackage of a build. For example:
kernel-2.6.9-34.EL.x86_64, kernel-devel-2.6.9-34.EL.s390,
glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64
Koji Components
===============
Koji is comprised of several components:
Koji-Hub
--------
koji-hub is the center of all Koji operations. It is an XML-RPC server running
under mod_wsgi in Apache. koji-hub is passive in that it only receives XML-RPC
calls and relies upon the build daemons and other components to initiate
communication. koji-hub is the only component that has direct access to the
database and is one of the two components that have write access to the file
system.
Kojid
-----
kojid is the build daemon that runs on each of the build machines. Its primary
responsibility is polling for incoming build requests and handling them
accordingly. Essentially kojid asks koji-hub for work. Koji also has support
for tasks other than building. Creating install images is one example. kojid
is responsible for handling these tasks as well. kojid uses mock for building.
It also creates a fresh buildroot for every build. kojid is written in Python
and communicates with koji-hub via XML-RPC.
Koji-Web
--------
koji-web is a set of scripts that run in mod_wsgi and use the Cheetah
templating engine to provide a web interface to Koji. It acts as a client to
koji-hub providing a visual interface to perform a limited amount of
administration. koji-web exposes a lot of information and also provides a means
for certain operations, such as cancelling builds.
Koji-client
-----------
koji-client is a CLI written in Python that provides many hooks into Koji. It
allows the user to query much of the data as well as perform actions such as
adding users and initiating build requests.
Kojira
------
kojira is a daemon that keeps the build root repodata updated. It is
responsible for removing redundant build roots and cleaning up after a build
request is completed.
Package Organization
====================
Tags and Targets
----------------
Koji organizes packages using tags:
* Tags are tracked in the database but not on disk
* Tags support multiple inheritance
* Each tag has its own list of valid packages (inheritable)
* Package ownership can be set per-tag (inheritable)
* Tag inheritance is more configurable
* When you build you specify a target rather than a tag
A build target specifies where a package should be built and how it should be
tagged afterwards. This allows target names to remain fixed as tags change
through releases. You can get a full list of build targets with the following
command:
::
$ koji list-targets
You can see just a single target with the --name option:
::
$ koji list-targets --name dist-fc7
Name Buildroot Destination
---------------------------------------------------------------------------------------------
dist-fc7 dist-fc7-build dist-fc7
This tells you a build for target dist-fc7 will use a buildroot with packages
from the tag dist-fc7-build and tag the resulting packages as dist-fc7.
You can get a list of tags with the following command:
::
$ koji list-tags
Package lists
-------------
As mentioned above, each tag has its own list of packages that may be placed in
the tag. To see that list for a tag, use the list-pkgs command:
::
$ koji list-pkgs --tag dist-fc7
Package Tag Extra Arches Owner
----------------------- ----------------------- ---------------- ----------------
ElectricFence dist-fc6 pmachata
GConf2 dist-fc6 rstrode
lucene dist-fc6 dbhole
lvm2 dist-fc6 lvm-team
ImageMagick dist-fc6 nmurray
m17n-db dist-fc6 majain
m17n-lib dist-fc6 majain
MAKEDEV dist-fc6 clumens
[...]
The first column is the name of the package, the second tells you which tag the
package entry has been inherited from, and the third tells you the owner of the
package.
Latest Builds
-------------
To see the latest builds for a tag, use the latest-pkg command:
::
$ koji latest-pkg --all dist-fc7
Build Tag Built by
---------------------------------------- -------------------- ----------------
ConsoleKit-0.1.0-5.fc7 dist-fc7 davidz
ElectricFence-2.2.2-20.2.2 dist-fc6 jkeating
GConf2-2.16.0-6.fc7 dist-fc7 mclasen
ImageMagick-6.2.8.0-3.fc6.1 dist-fc6-updates nmurray
MAKEDEV-3.23-1.2 dist-fc6 nalin
MySQL-python-1.2.1_p2-2 dist-fc7 katzj
NetworkManager-0.6.5-0.3.cvs20061025.fc7 dist-fc7 caillon
ORBit2-2.14.6-1.fc7 dist-fc7 mclasen
The output gives you not only the latest builds, but which tag they have been
inherited from and who built them (note: for builds imported from beehive the
"built by" field may be misleading).
Documentation
-------------
We've tried to make Koji self-documenting wherever possible. The command line
tool will print a list of valid commands and each command supports --help.
For example:
::
$ koji help
Koji commands are:
build Build a package from source
cancel-task Cancel a task
help List available commands
latest-build Print the latest rpms for a tag
latest-pkg Print the latest builds for a tag
[...]
::
$ koji build --help
usage: koji build [options] tag URL
(Specify the --help global option for a list of other help options)
options:
-h, --help show this help message and exit
--skip-tag Do not attempt to tag package
--scratch Perform a scratch build
--nowait Don't wait on build
[...]
You can see administrator-only command help with --admin. Most users will
never use these additional commands, but if you're setting up your own Koji
system, you may find them very useful.
::
$ koji help --admin
Available commands:
add-external-repo Create an external repo and/or add one to a tag
add-group Add a group to a tag
add-group-pkg Add a package to a group's package listing
[...]
Koji Deployments
================
Koji is also known to be used in many places, and we :doc:`track them on this
page <runs_here>`. Feel free to add your entry. There is no additional
obligation to you for doing so. :)
Koji Contributor Guides
=======================
If you're interested in submitting patches, writing documentation, or filing
bugs this section is for you. In time this will be the best place to learn
how to get involved.
* :doc:`Getting Started as a Developer <writing_koji_code>`
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
.. _Mock: http://fedoraproject.org/wiki/Projects/Mock
.. _RPM packages for the Fedora project: http://koji.fedoraproject.org/koji/
.. _Koji project website: https://fedorahosted.org/koji/wiki

85
docs/source/migrating_to_1.10.rst Normal file
View file

@ -0,0 +1,85 @@
Migrating to Koji 1.10
======================
.. reStructured Text formatted
The 1.10 release of Koji includes a few changes that you should consider when
migrating.
DB Updates
----------
The new ``tag_extra`` table tracks extra data for tags.
There is a new entry in the ``channels`` table and some additions and updates to
the ``archivetypes`` table.
As in previous releases, we provide a migration script that updates the
database.
::
# psql koji koji </usr/share/doc/koji-1.10.0/docs/schema-upgrade-1.9-1.10.sql
Command line changes
--------------------
A few commands support new arguments
``maven-build``
* ``--ini`` : Pass build parameters via a .ini file
* ``--section`` : Get build parameters from this section of the .ini
``wrapper-rpm``
* ``--ini`` : Pass build parameters via a .ini file
* ``--section`` : Get build parameters from this section of the .ini
``import``
* ``--link`` : Attempt to hardlink instead of uploading
``list-tagged``
* ``--latest-n``: Only show the latest N builds/rpms
``list-history``
* ``--watch`` : Monitor history data
``edit-tag``
* ``--extra`` : Set tag extra option
``list-tasks``
* ``--user`` : Only tasks for this user
* ``--arch`` : Only tasks for this architecture
* ``--method`` : Only tasks of this method
* ``--channel`` : Only tasks in this channel
* ``--host`` : Only tasks for this host
``download-build``
* ``--task-id`` : Interpret id as a task id
And there are three new commands
* ``image-build-indirection``
* ``maven-chain``
* ``runroot``
Other Configuration changes
---------------------------
The Koji web interface can now treate ``extra-footer.html`` as a Cheetah
template. This behavior can be enabled by setting the ``LiteralFooter`` option
to ``False`` in the kojiweb config.
RPC API Changes
---------------
The ``readTaggedBuilds`` and ``readTaggedRPMS`` now treat an integer value for
the optional latest argument differently. Before it was simply treated as a
boolean flag, which if true caused the call to return only the latest build for
each package. Now, if the value is a positive integer N, it will return the N
latest builds for each package. The behavior is unchanged for other values.
New rpc calls: ``chainMaven``, ``buildImageIndirection``, and ``mergeScratch``

60
docs/Migrating_to_1.7.txt → docs/source/migrating_to_1.7.rst
View file

@ -1,29 +1,31 @@
Migrating to Koji 1.7
=====================
// asciidoc formatted
.. reStructured Text formatted
The 1.7 release of Koji contains changes that will require a little extra
work when updating. These changes are:
- DB schema updates to support storage volumes
- The change from mod_python to mod_wsgi
- The introduction of a separate configuration file for koji-web
- Changes to url options
* DB schema updates to support storage volumes
* The change from mod_python to mod_wsgi
* The introduction of a separate configuration file for koji-web
* Changes to url options
DB Schema Updates
-----------------
The 1.7 release adds two new tables to the database. The +volume+ table tracks
the names of available storage volumes, and the +tag_updates+ table tracks
The 1.7 release adds two new tables to the database. The ``volume`` table tracks
the names of available storage volumes, and the ``tag_updates`` table tracks
changes to tags that are not easily calculated from other tables. There is
also a new field in the +build+ table, +volume_id+, which indicates which
also a new field in the ``build`` table, ``volume_id``, which indicates which
volume a build is stored on.
As in previous releases, we provide a migration script that updates the
database.
# psql koji koji </usr/share/doc/koji-1.7.0/docs/schema-upgrade-1.5-1.7.sql
::
# psql koji koji </usr/share/doc/koji-1.7.0/docs/schema-upgrade-1.5-1.7.sql
mod_python and mod_wsgi
@ -35,7 +37,7 @@ Koji administrators can opt to stay on mod_python for now, but some minor
configuration changes will be required.
Migrating to mod_wsgi
~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^
The mod_wsgi package is now required for both koji-hub and koji-web. Folks
running RHEL5 can find mod_wsgi in EPEL.
@ -44,15 +46,15 @@ You will need to adjust your http config for both koji-hub and koji-web. Our
example config files default to mod_wsgi. To adapt your existing config, you
will need to:
- For both the koji-hub and koji-web/scripts directories:
* add +Options ExecCGI+
* change +SetHandler+ from mod_python to wsgi-script
- Ensure that the koji-web Alias points to wsgi_publisher.py
- If you have not already, migrate all koji-hub PythonOptions to hub.conf
- Migrate all koji-web PythonOptions to web.conf (see later section)
* For both the koji-hub and koji-web/scripts directories:
* add ``Options ExecCGI``
* change ``SetHandler`` from mod_python to wsgi-script
* Ensure that the koji-web Alias points to wsgi_publisher.py
* If you have not already, migrate all koji-hub PythonOptions to hub.conf
* Migrate all koji-web PythonOptions to web.conf (see later section)
Staying on mod_python
~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^
Support for mod_python is _deprecated_ and will disappear in a future version
of Koji.
@ -64,8 +66,8 @@ The koji-hub http config should continue to function without modification.
The koji-web http config will, at minimum, require the following changes:
- Ensure that the koji-web +Alias+ points to wsgi_publisher.py
- Change koji-web's +PythonHandler+ setting to wsgi_publisher
* Ensure that the koji-web ``Alias`` points to wsgi_publisher.py
* Change koji-web's ``PythonHandler`` setting to wsgi_publisher
Our example http configurations contain commented examples of mod_python
configuration.
@ -81,7 +83,9 @@ Starting with version 1.7, koji-web uses a separate configuration file, rather
than PythonOptions embedded in the httpd config. The location of the new file
is:
/etc/kojiweb/web.conf
::
/etc/kojiweb/web.conf
The web.conf file is an ini-style configuration file. Options should be placed
in the [web] section. All previous options accepted via PythonOptions are
@ -89,21 +93,25 @@ accepted in web.conf. Please see the example web.conf file.
Custom Config File Location
~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The location of web.conf can be specified in the httpd configuration. To
specify the location under mod_wsgi, use:
SetEnv koji.web.ConfigFile /path/to/web.conf
::
SetEnv koji.web.ConfigFile /path/to/web.conf
Under mod_python, use:
PythonOption koji.web.ConfigFile /path/to/web.conf
::
PythonOption koji.web.ConfigFile /path/to/web.conf
If you opt to stay on mod_python, the server will continue to process the old
PythonOptions. To ease migration, it does so by default unless the
koji.web.ConfigFile PythonOption is specified. In order to use web.conf under
mod_python, you _must_ specify koji.web.ConfigFile in your http config.
mod_python, you *must* specify koji.web.ConfigFile in your http config.
We strongly recommend moving to web.conf. The server will issue a warning at
startup if web.conf is not in use.
@ -127,13 +135,13 @@ Additional Notes
----------------
Split Storage
~~~~~~~~~~~~~
^^^^^^^^^^^^^
Apart from the schema changes, no other migration steps are required for the
split storage feature. By default, builds are stored in the normal location.
Web Themes
~~~~~~~~~~
^^^^^^^^^^
Using the old method (httpd aliases for koji static content) should continue
to work. For (brief) instructions on the new method, see the README file under

66
docs/Migrating_to_1.8.txt → docs/source/migrating_to_1.8.rst
View file

@ -1,7 +1,7 @@
Migrating to Koji 1.8
=====================
// asciidoc formatted
.. reStructured Text formatted
The 1.8 release of Koji refactors how images (livecd and appliance) are stored
in the database and on disc. These changes will require a little extra work
@ -15,21 +15,23 @@ DB Schema Updates
-----------------
Previous to 1.8, images were stored in separately from other builds, both in
the database and on disc. The new schema adds new tables: +image_builds+,
+image_listing+, and +image_archives+.
the database and on disc. The new schema adds new tables: ``image_builds``,
``image_listing``, and ``image_archives``.
The following tables are now obsolete: +imageinfo+ and +imageinfo_listing+.
The following tables are now obsolete: ``imageinfo`` and ``imageinfo_listing``.
However you should not drop these tables until you have migrated your image
data.
As in previous releases, we provide a migration script that updates the
database.
# psql koji koji </usr/share/doc/koji-1.8.0/docs/schema-upgrade-1.7-1.8.sql
::
# psql koji koji </usr/share/doc/koji-1.8.0/docs/schema-upgrade-1.7-1.8.sql
Note that the SQL script does not (and can not) automatically migrate your old
image data to the new tables. After applying the schema changes, you can
migrate old images using the +migrateImage+ hub call. This method is necessary
migrate old images using the ``migrateImage`` hub call. This method is necessary
because the new schema requires each image to have a name, version, and release
value. The values for name and version cannot be automatically guessed.
@ -38,16 +40,20 @@ Migrating your old images
-------------------------
If you have old images, you can migrate them to the new system using the
+migrateImage+ hub call. This call requires admin privilege and must also be
enabled with the +EnableImageMigration+ configuration option in +hub.conf+.
``migrateImage`` hub call. This call requires admin privilege and must also be
enabled with the ``EnableImageMigration`` configuration option in ``hub.conf``.
The signature of the call is:
migrateImage(old_image_id, name, version)
::
migrateImage(old_image_id, name, version)
This call can made from the command line:
# koji call migrateImage 45 my_livecd 1.1
::
# koji call migrateImage 45 my_livecd 1.1
Cleaning up
@ -62,29 +68,33 @@ migrated any desired images.
Removing the old data is simply a matter of dropping tables and deleting files.
koji=> DROP TABLE imageinfo_listing;
koji=> DROP TABLE imageinfo;
# rm -rf /mnt/koji/images
::
koji=> DROP TABLE imageinfo_listing;
koji=> DROP TABLE imageinfo;
# rm -rf /mnt/koji/images
Command line changes
--------------------
For clarity and consistency, all of the +-pkg+ commands have been renamed to
+-build+ commands.
For clarity and consistency, all of the ``-pkg`` commands have been renamed to
``-build`` commands.
latest-pkg -> latest-build
move-pkg -> move-build
tag-pkg -> tag-build
untag-pkg -> untag-build
::
latest-pkg -> latest-build
move-pkg -> move-build
tag-pkg -> tag-build
untag-pkg -> untag-build
For backwards compatibility, the old commands names are also recognized.
A new command has been added, +remove-pkg+.
A new command has been added, ``remove-pkg``.
Several commands have been modified to support images.
The +spin-livecd+ and +spin-appliance+ commands now require additional
The ``spin-livecd`` and ``spin-appliance`` commands now require additional
arguments. These arguments specify the name and version to use for the image.
@ -93,20 +103,24 @@ New kojira options
The following options are new to kojira:
max_delete_processes
max_repo_tasks_maven
::
max_delete_processes
max_repo_tasks_maven
Previously, kojira ran as a single process and repo deletions could potentially
slow things down (particularly for Maven-enabled repos). Now kojira spawns
a separate process to handle these deletions. The +max_delete_processes+
a separate process to handle these deletions. The ``max_delete_processes``
determines how many such processes it will launch at one time.
When Maven-enabled repos are in use, they can potentially take a very long time
to regenerate. If a number of these pile up it can severely slow down
regeneration of non-Maven repos. The +max_repo_tasks_maven+ limits how many
regeneration of non-Maven repos. The ``max_repo_tasks_maven`` limits how many
Maven repos kojira will attempt to regenerate at once.
Also the following kojira option has been removed:
prune_batch_size
::
prune_batch_size

86
docs/source/migrating_to_1.9.rst Normal file
View file

@ -0,0 +1,86 @@
Migrating to Koji 1.9
=====================
.. reStructured Text formatted
The 1.9 release of Koji includes a few changes that you should consider when
migrating.
DB Updates
----------
ImageFactory support introduced some new archive types. These have been added to
the ``archivetypes`` table. The inaccurate ``vmx`` entry has been removed.
As in previous releases, we provide a migration script that updates the
database.
::
# psql koji koji </usr/share/doc/koji-1.9.0/docs/schema-upgrade-1.8-1.9.sql
Command line changes
--------------------
The command line interface handles configuration files a little differently. Old
configs should work just fine, but now there are new options and enhancements.
In addition to the main configuration files, the koji cli now checks for
``/etc/koji.conf.d`` and ``~/.koji/config.d`` directories and loads any
``*.conf`` files contained within. Also if the user specifies a directory with
the ``-c/--config`` option, then that directory will be processed similarly.
The command line supports a new ``-p/--profile`` option to select alternate
configuration profiles without having to link or rename the koji executable.
The new ``image-build`` command is used to generate images using ImageFactory.
The older spin-appliance command is now deprecated.
The ``mock-config`` command no longer requires a name argument. You can still
specify one
if you want to override the default choice. It also supports new options. The
``--latest`` option causes the resulting mock config to reference the
``latest`` repo (a varying symlink). The ``--target`` option allows generating
the config from a target name.
Other command line changes include:
* a new ``download-logs`` command
* the ``list-groups`` command now accepts event args
* the ``taginfo`` command now reports the list of comps groups for the tag
* the fast upload feature is now used automatically if the server supports it
Other Configuration changes
---------------------------
There are also some minor configuration changes in other parts of Koji.
In ``kojid`` the time limit for rpm builds is now configurable via the
``rpmbuild_timeout`` setting in ``kojid.conf``. The default is 24 hours.
The ``koji-gc`` tool supports two new configuration options. The ``krbservice``
option allows you to specify the kerberos service for authentication, and the
``email_domain`` option allows you to specify the email domain for sending gc
notices.
The messagebus hub plugin now supports ``timeout`` and ``heartbeat`` options
for the message bus connection.
RPC API Changes
---------------
Most of these changes are extensions, though some of the host-only call
changes are incompatible.
The ``tagHistory`` call accepts a new named boolean option (``active``) to
select only active/inactive entries. It also now reports the additional fields
``maven_build_id`` and ``win_build_id`` if builds are maven or win builds
respectively.
New rpc calls: ``buildImageOz``, ``host.completeImageBuild``, and
``host.evalPolicy``.
The host-only calls ``host.moveImageBuildToScratch`` and ``host.importImage``
no longer accept the ``rpm_results`` argument. The rpm results can be embedded
in the regular ``results`` argument.

11
docs/source/migrations.rst Normal file
View file

@ -0,0 +1,11 @@
==========
Migrations
==========
.. toctree::
:maxdepth: 1
migrating_to_1.10
migrating_to_1.9
migrating_to_1.8
migrating_to_1.7

191
docs/source/misc.rst Normal file
View file

@ -0,0 +1,191 @@
========================
Koji Miscellaneous Notes
========================
Notes and miscellaneous details about Koji
==========================================
This document is intended to illuminate some of the small quirks that
you may encounter while running your own koji server.
Koji CLI
--------
Getting Help
~~~~~~~~~~~~
- There are multiple ways to receive help from the koji command, each
of them gives you something a little different:
- List of user commands:
::
koji help
- Command-specific help:
::
koji [command] --help
- List of admin commands:
::
koji help --admin
Building from SRPM
~~~~~~~~~~~~~~~~~~
Koji only allows admins access to build directly from srpm. It expects
normal users to build out of an SCM.
Building from SCM
~~~~~~~~~~~~~~~~~
SCM Layout
^^^^^^^^^^
When building out of an SCM, Koji expects there will be a Makefile in
the project root that has a 'sources' target. Koji will call 'make
sources' on the checked out files.
This target simply needs to download all the sources for the SRPM that
are not already included in the SCM repository.
SCM URI Structure
^^^^^^^^^^^^^^^^^
In Fedora, this detail is typically handled by the fedpkg tool. Outside
of Fedora, you may need to know how to manually submit builds from an
SCM to koji.
Koji accepts an SCM URI in this format:
::
koji build [tag] [scheme]://[user]@[hostname]/[path/to/repository]?[path/to/project]#[revision]
Note the division between repository path and project path. During setup
of kojid, the allowed\_scms parameter is configured in
/etc/kojid/kojid.conf. This value of this parameter should match the
path to the repository.
In some SCM configurations, there isn't a difference between repository
path and project path. In these cases, it should be understood that
dividing your SCM path into two components, URI path and URI query, can
seem somewhat arbitrary. An easy way to remember this detail is that the
path specified in allowed\_scms is the portion of your SCM path that
goes before the URI query, any sub-directories not specified in
allowed\_scms is given to koji as the URI query.
Koji tasks
----------
BuildNotifications
~~~~~~~~~~~~~~~~~~
- Koji sends build notifications to the package owner and the user who
submitted the build. For BuildNotifications to work successfully, the
package owner's username needs to match a valid username for an
e-mail address, because kojihub sends to
username@domain\_in\_kojihub.conf.
- For SSL authentication, this means that your CN must be valid as
the user portion of an e-mail address.
Koji server administration
--------------------------
Importing an rpm without srpm
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are running a private koji instance, you may run into a situation
where you need to integrate a proprietary rpm into your build system. To
do this, it is similar to the package imports you did when bootstrapping
your koji instance:
::
koji import --create-build [package]
Multi-arch builds
~~~~~~~~~~~~~~~~~
There are some packages that need to build across multiple arches. For
example, the kernel package no longer builds an i386 kernel, kernels are
built on i586 and above. To instruct koji to build these additional
arches, use this command:
::
koji set-pkg-arches [options] <arches> <tag> <package> [<package2> ...]
Note: is a single entity, so denote multiple arches within quotes (e.g.
'i386 i586 i686').
Manually submitting a task
~~~~~~~~~~~~~~~~~~~~~~~~~~
Occasionally, you may need to manually submit a task to koji. One likely
usage is to manually trigger a createRepo. To do this, use this command:
::
koji make-task [options] <arg1> [<arg2>...]
The make-task command is a bit of under-documented black magic. Task
parameters are defined in kojihub.py. The easiest way I have found to
figure out the right incantations for make-task is to query the *task*
table in the koji database directly. Find a similar task to the one you
want to create, and look in the request field for the parameters the
task used, and mimic those.
So, citing the createRepo case above, here is an example:
::
kojiadmin@localhost$ koji make-task --channel=createrepo --priority=15 \
newRepo dist-foo-build
Managing your tags
~~~~~~~~~~~~~~~~~~
Occasionally an unwanted package or version of a package will be built
by koji. Don't fret. There are two mechanisms to handle rescinding a
package or specific package version.
- To remove a specific version of a package, you can untag it:
::
koji untag-build [options] <tag> <pkg> [<pkg>...]
supports either %name or %name-%version-%release
- To remove all versions of a package, you can untag it as above or you
can administratively block it from being listed in a tag:
::
koji block-pkg [options] tag package [package2 ...]
Spec file processing
--------------------
Macro processing
~~~~~~~~~~~~~~~~
Macros in the spec file are expanded before Requires and BuildRequires
are processed. If there are any custom macros in the spec file, the
package that drops those macros into /etc/rpm must be tagged under your
dist-build tag
%dist tags
^^^^^^^^^^
For packages that incorporate the %dist tags in their filename, they
expect %dist to be defined in /etc/rpm/macros.dist, which was added in
Fedora 7. For building on RHEL5/FC6 and earlier, koji needs the
`http://buildsys.fedoraproject.org/buildgroups/
buildsys-macros <http://buildsys.fedoraproject.org/buildgroups/ buildsys-macros>`__
package tagged under the dist-build tag.

67
docs/source/runs_here.rst Normal file
View file

@ -0,0 +1,67 @@
=========
Runs Here
=========
List of locations that Koji is currently in use
===============================================
Please add yourself here if you run a local version of Koji. Please also
describe what you are using it for.
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| # | Project | Site Name | Used for |
+=========+============================================================================+===================================================================+===========================================================================================================================================================================================================================================================================+
| 0 | Grid and HPC Operations Center of A.U.Th | Not-Public | Building packages for local fabric needs. Maintenance of RPM repositories. Monitoring, messaging and information system software for `LCG <http://cern.ch/lcg>`__. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1 | pc V Pharmacy | Not-public | Compiling proprietary software across Fedora and RHEL. Both the software and any dependencies that are not available in Fedora and/or RHEL that may also be proprietary. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2 | CERN | \* | CERN no longer runs their own Koji, but does make daily use of the one run by the Scientific Computing Center of A.U.Th. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 3 | `fedora.danny.cz <http://fedora.danny.cz/danny>`__ | the hub is not public yet | http://sharkcz.livejournal.com/3988.html |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 4 | `Ergo Project <http://www.ergo-project.org>`__ | `Koji Web Interface <http://koji.ergo-project.org>`__ | Fast-Track repositories for Enterprise Linux and Fedora, FTBFS, `Blog posts on Koji <http://planet.ergo-project.org/category/tags/koji>`__ |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 5 | `NetNix Estonia OÜ <http://netnix.ee/>`__ | Internal | Packaging commercial and in-house software for RHEL/CentOS/Fedora support. Multiple instances installed, locally and customer sites. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 6 | `Kolab Systems AG <http://www.kolabsys.com>`__ | Not Public (yet) | Packaging for ISV products |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7 | `Lebanon Evangelical School for Boys and Girls <http://www.lesbg.com>`__ | http://koji.lesbg.com/koji | Used to provide configuration RPMS for our three schools |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 8 | `Messinet Secure Services <http://messinet.com>`__ | http://messinet.com/koji | Building and packaging to support local infrastructure as well as incorporate software not packaged at Fedora or RPMFusion. Most packaged software is available to all. See `Messinet Secure Services Fedora RPM Repository <http://messinet.com/rpms/>`__ for details. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 9 | `ACOSS France <http://www.acoss.fr/>`__ | Not-public | Building and packaging on top of RHEL and CentOS 6 for newer software version. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10 | `Virtual Bridges, Inc. <http://www.vbridges.com/>`__ | Not-public | Building `VERDE LEAF <http://www.vbridges.com/products/verde/verde-leaf/>`__ virtualization platform/distribution. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 11 | Caltech | http://koji.hep.caltech.edu/koji/ | multi-university CERN-based project |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 12 | Spacewalk | http://koji.spacewalkproject.org/koji/ | building Spacewalk for RHEL 5, 6, and Fedora 14, 15 |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 13 | `National Nanotechnology Network Grid <http://www.ngrid.ru/>`__ | http://koji.ngrid.ru/koji/ | GridNNN middleware |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| \| 14 | TomTom | non-public | used to build rpms that aren't available in the RHEL repos or EPEL |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| \| 15 | Amazon | non-public | used to build RPMS for the Amazon Linux AMI |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| \| 16 | Open Science Grid | http://koji-hub.batlab.org/koji/ | Used to build RPMs for the Open Science Grid software project. Software built here is deployed across 100+ universities and labs across the U.S. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| \| 17 | RussianFedora | http://koji.russianfedora.ru | Building add-on packages for the RFRemix. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| \| 18 | Scientific Linux | Not-public (yet) | Building Scientific Linux. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| \| 19 | Katello | http://koji.katello.org/koji/ | Building packages for Katello project |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| \| 20 | NIWA | non-public | Building packages for climate and forecasting project |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 21 | `Xenith Consulting Limited <http://www.xenithconsulting.com>`__ | Not-Public | Banking, Telecommunications and Embedded Systems |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 22 | Abril Comunicações | non-public | Media / Entertainment Company. Building RPM packages for the Digital products |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 23 | The FireServer Project | http://www.fireserver.com.br | Building packages for FireServer Project |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 24 | Seneca Centre for Development of Open Technology | http://koji.pidora.ca (info at http://pidora.ca) | Pidora (Fedora Remix for Raspberry Pi) and some other projects. |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 25 | OpenVZ/Virtuozzo | `https://openvz.org/Virtuozzo <https://openvz.org/Virtuozzo>`__ | Building OpenVZ packages |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 26 | Cloud Linux | `http://koji.cloudlinux.com/ <http://koji.cloudlinux.com/>`__ | Building Cloud Linux packages |
+---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

95
docs/source/server_bootstrap.rst Normal file
View file

@ -0,0 +1,95 @@
=====================
Koji Server Bootstrap
=====================
Bootstrapping a new Koji build environment
==========================================
These are the steps involved in populating a new Koji server with
packages so that it can be used for building. This assumes that the Koji
hub is up, appropriate authentication methods have been configured, the
Koji repo administration daemon (``kojira``) is properly configured and
running, and at least one Koji builder (``kojid``) is properly
configured and running. All koji cli commands assume that the user is a
Koji *admin*. If you need help with these tasks, see the
`ServerHowTo <Koji/ServerHowTo>`__ .
- Download all source rpms and binary rpms for the arches you're
interested in
- Import all source rpms
::
$ koji import /path/to/package1.src.rpm /path/to/package2.src.rpm ...
If the files are on the same volume as /mnt/koji, you can use
``koji import --link``, which hardlinks the files into place, avoiding
the need to upload them to the hub and **very significantly** increasing
import speed. When using ``--link``, you must run as root. It is
**highly** recommended that you use ``--link``.
- Import all binary rpms using the same method as above
- Create a new tag
::
$ koji add-tag dist-foo
- Tag all of the packages you just imported into the tag you just
created
You can use ``koji list-untagged`` to get a list of all of the packages
you just imported.
::
$ koji list-pkgs --quiet | xargs koji add-pkg --owner <kojiuser> dist-foo
$ koji list-untagged | xargs -n 1 koji call tagBuildBypass dist-foo
We call the *tagBuildBypass* method instead of using ``koji tag-pkg``
because it doesn't require the builders to process *tagBuild* tasks one
at a time, but does the tagging directly. This will save a significant
amount of time, especially when tagging a large number of packages.
- Create a build tag with the desired arches, and the previously
created tag as a parent
::
$ koji add-tag --parent dist-foo --arches "i386 x86_64 ppc ppc64" dist-foo-build
- Create a build target that includes the tags you've already created
::
$ koji add-target dist-foo dist-foo-build
- Create a *build* group associated with your build tag
::
$ koji add-group dist-foo-build build
- Populate the *build* group with packages that will be installed into
the minimal buildroot
You can find out what the current build group for Fedora is by running
``koji -s <nowiki>http://$ARCH.koji.fedoraproject.org/kojihub</nowiki> list-groups f17-build``
against the Fedora Koji instance for your $ARCH. This is probably a good
starting point for your minimal buildroot.
::
$ koji add-group-pkg dist-foo-build build pkg1
$ koji add-group-pkg dist-foo-build build pkg2
- regenerate the repo
::
$ koji regen-repo dist-foo-build
- Wait for the repo to regenerate, and you should now be able to run a
build successfully.

1298
docs/source/server_howto.rst Normal file
View file

File diff suppressed because it is too large Load diff

544
docs/source/using_the_koji_build_system.rst Normal file
View file

@ -0,0 +1,544 @@
===========================
Using the koji build system
===========================
Using Koji in Fedora
====================
The `Koji Build System <Koji>`__ is Fedora's RPM buildsystem. Packagers
use the koji client to request package builds and get information about
the buildsystem. Koji runs on top of Mock to build RPM packages for
specific architectures and ensure that they build correctly.
There is also a `simplified Chinese
edition <Zh/使用Koji编译打包系统>`__.
Installing Koji
---------------
Installing the Koji CLI
^^^^^^^^^^^^^^^^^^^^^^^
Everything you need to use Koji (and be a Fedora contributor) can be
installed in a single step:
::
dnf install fedora-packager
fedora-packager provides useful scripts to help maintain and setup your
koji environment. Additionally, it includes dependencies on the Koji
CLI, so it will be installed when you install ``fedora-packager``. The
command is called ``koji`` and is included in the main koji package. By
default the koji tool authenticates to the central server using
Kerberos. However SSL and username/password authentications are
available. You will need to have a valid authentication token to use
many features. However, many of the read-only commands will work without
authentication.
Fedora Account System (FAS2) Setup
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order to interface with the koji server, maintainers will need to run
::
/usr/bin/fedora-packager-setup
Each user on a system will need to run fedora-packager-setup if they
wish to use Koji to build Fedora packages. Each user has their own
certificates that authenticate them.
.. raw:: mediawiki
{{admon/tip|Plague users rejoice!|For existing users of plague (the old build system that preceded Koji), <code>fedora-packager-setup</code> will use your existing certificates. If you did not have plague before, it will get the server CA certs and tell you where to get your user cert.}}
Fedora Certificates
'''''''''''''''''''
Koji uses three certificates:
``~/.fedora.cert`` (specific to the Fedora Maintainer) : This cert is
generated from running ``fedora-cert -n``. It should have been generated
when you became maintainer. You may need to refresh it when it expires
by running ``fedora-cert -n`` again. You can check if it has expired
with ``fedora-cert -v``.
the following are downloaded automatically by fedora-packager-setup and
dont need to be manually setup
``~/.fedora-upload-ca.cert`` (The certificate for the Certificate
Authority used to sign the user keys.) : It can be manually downloaded
from
`here <https://admin.fedoraproject.org/accounts/fedora-upload-ca.cert>`__
or ``fedora-packager-setup or fedora-cert -n`` should fetch it. using
the CLI is prefered.
``~/.fedora-server-ca.cert`` (The certificate for the Certificate
Authority used to sign the build system's server keys.) : It can be
downloaded manually from
`here <https://admin.fedoraproject.org/accounts/fedora-server-ca.cert>`__
or ``fedora-packager-setup`` should fetch it. This certificate may also
be needed to let `https koji <https://koji.fedroraproject.org>`__ URLs
resolve without untrusted-CA warnings.
.. raw:: mediawiki
{{admon/warning|RHEL6 cert bug|If you're using RHEL6, an incompatibility between RHEL6's openssl and nss causes certificates downloaded from fas to fail to work with some fedpkg tools. [[https://bugzilla.redhat.com/show_bug.cgi?id=631000 Bug 631000 rhel6 openssl creates PKCS#8 encoded PEM RSA private key files, nss can't read them]]. The cert can be made compatible using this command: <code>(openssl x509 -in ~/.fedora.cert -text; echo; openssl rsa -in ~/.fedora.cert) > fedora.cert.new</code>}}
Koji Config
^^^^^^^^^^^
The global local client configuration file for koji is
``/etc/koji.conf``. You should not need to change this from the defaults
for building Fedora packages.These will allow you to use the primary
build system as well as secondary arch build systems.
The web interface
-----------------
.. raw:: mediawiki
{{admon/tip|Optional|The web interface is optional. You may skip to the
next section if you like.}}
The primary interface for viewing Koji data is a web application. It is
available at http://koji.fedoraproject.org/koji/ . Most of the interface
is read-only, but with sufficient privileges, you can log in and perform
some additional actions. For example:
- Cancel a build
- Resubmit a failed task
- Setup a notification
Those with admin privileges will find additional actions, such as:
- Create/Edit/Delete a tag
- Create/Edit/Delete a target
- Enable/Disable a build host
The web site utilizes SSL authentication. In order to log in you will
need a valid SSL certificate and your web browser will need to be
configured to trust the SSL cert. Instructions on how to do this are
printed when running ``fedora-packager-setup --with-browser-cert``.
.. raw:: mediawiki
{{admon/warning|Using the certificate directly downloaded from the FAS web
interface|If you have generated and downloaded the certificate
<code>~/.fedora.cert</code> directly from FAS using the form referenced
above, you need to convert it into a format that the browser can understand
using the following command:
<code>openssl pkcs12 -export -in ~/.fedora.cert -CAfile ~/.fedora-upload-ca.cert -out ~/fedora-browser-cert.p12</code>,
where <code>.fedora-upload-ca.cert</code> can be downloaded from the URL
referenced above.}}
Installing SSL Certificates in Firefox
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. raw:: mediawiki
{{admon/note|Optional|You only need to check these instructions if you are intending to authenticate with the web interface with Firefox. Authenticating with the web interface is optional.}}
Once you have created your FAS account, generated your certificate in
the form posted in the link above and ran
``fedora-packager-setup --with-browser-cert``, you will need to import
it into your web browser. You can do this in Firefox by doing the
following:
1. Launch Firefox and click on the **Edit** menu from the toolbar
2. Select **Preferences** in the sub-menu which appears.
3. This should open the **Preferences** window where you can switch to
the **Advanced** section
4. In the **Advanced** section switch to the **Encryption** tab
5. Click on the **View Certificates** button and the Certificates window
will appear
6. Switch to the **Your Certificates** tab and click on the **Import**
button
7. Point to where your Fedora Certificate is located and click **Open**
(fedora-packager-setup will have told you where it was saved and will
have asked you to set a password for the cert)
You should now be able to see your Fedora Certificate listed under
**Your Certificates** and you should be able to authenticate with the
koji web interface.
Installing SSL Certificates in Chromium
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. raw:: mediawiki
{{admon/note|Optional|You only need to check these instructions if you are intending to authenticate with the web interface with Chromium. Authenticating with the web interface is optional.}}
Chromium uses the NSS Shared DB, you will need the nss-tools package
installed.
::
pk12util -d sql:$HOME/.pki/nssdb -i fedora-browser-cert.p12
Notifications
^^^^^^^^^^^^^
When authenticated with the Koji web interface, you can setup a
notification requests to make sure you do not miss when a package you
care about gets built. Login and scroll to the bottom of the page, there
you should find a *`Add a
notification <https://koji.fedoraproject.org/koji/notificationcreate>`__*
link and a list of your configured notifications.
Building with fedpkg targets
----------------------------
Every push is automatically tagged via git. All you have done to build
the package is to run,
::
fedpkg build
This will trigger a build request for the branch. Easy!
It is also possible to target a specific koji tag as follows:
::
fedpkg build --target TARGET
for example, if building on rawhide against a special tag created by
rel-eng for updating API for many packages, e.g. ``dist-f14-python`` you
would use the following:
::
fedpkg build --target 'dist-f14-python'
Chained builds
^^^^^^^^^^^^^^
.. raw:: mediawiki
{{Admon/warning | chain-builds only work when building on the devel/ branch (aka rawhide). To chain-build packages to update a released OS version, [https://fedoraproject.org/wiki/Bodhi/BuildRootOverrides set up an override using bodhi] requesting packages to be included in the proper buildroot.}}
Sometimes you want to make sure than one build succeeded before
launching the next one, for example when you want to rebuild a package
against a just rebuilt dependency. In that case you can use a chain
build with:
``fedpkg chain-build libwidget libgizmo``
The current package is added to the end of the CHAIN list. Colons (:)
can be used in the CHAIN parameter to define groups of packages.
Packages in any single group will be built in parallel and all packages
in a group must build successfully and populate the repository before
the next group will begin building. For example:
``fedpkg chain-build libwidget libaselib : libgizmo :``
will cause libwidget and libaselib to be built in parallel, followed by
libgizmo and then the currect directory package. If no groups are
defined, packages will be built sequentially.
If a build fail, following builds are cancelled but the builds that
already succeeded are pushed to the repository.
Scratch Builds
--------------
Sometimes it is useful to be able to build a package against the
buildroot but without actually including it in the release. This is
called a scratch build. The following section covers using koji directly
as well as the fedpkg tool to do scratch builds. To create a scratch
build from changes you haven't committed, do the following:
::
rpmbuild -bs foo.spec
koji build --scratch rawhide foo.srpm
From the latest git commit:
::
koji build --scratch rawhide 'git url'
Warning: Scratch builds will *not* work correctly if your .spec file
does something different depending on the value of %fedora, %fc9, and so
on. Macro values like these are set by the *builder*, not by koji, so
the value of %fedora will be for whatever created the source RPM, and
*not* what it's being built on. Non-scratch builds get around this by
first re-building the source RPM.
If you have committed the changes to git and you are in the current
branch, you can do a scratch build with fedpkg tool which wraps the koji
command line tool with the appropriate options:
::
fedpkg scratch-build
if you want to do a scratch build for a specific architecture, you can
type:
::
fedpkg scratch-build-<archs>
can be a comma separated list of several architectures.
finally is possible to combine the scratch-build command with a specific
koji tag in the form:
::
fedpkg scratch-build --target TARGET
fedpkg scratch-build --help or koji build --help for more information.
Build Failures
--------------
If your package fails to build, you will see something like this:
::
420066 buildArch kernel-2.6.18-1.2739.10.9.el5.jjf.215394.2.src.rpm,
ia64): open (build-1.example.com) -> FAILED: BuildrootError:
error building package (arch ia64), mock exited with status 10
You can figure out why the build failed by looking at the log files. If
there is a build.log, start there. Otherwise, look at init.log.
Logs can be found via the web interface in the Task pages for the failed
task. Alternatively the koji client can be used to view the logs via the
``watch-logs`` command. See the help output for more details.
Advanced use of Koji
--------------------
We've tried to make Koji self-documenting wherever possible. The command
line tool will print a list of valid commands and each command supports
--help. For example:
::
$ koji help
Koji commands are:
build Build a package from source
cancel-task Cancel a task
help List available commands
latest-build Print the latest rpms for a tag
latest-pkg Print the latest builds for a tag
[...]
::
$ koji build --help
usage: koji build [options] tag URL
(Specify the --help global option for a list of other help options)
options:
-h, --help show this help message and exit
--skip-tag Do not attempt to tag package
--scratch Perform a scratch build
--nowait Don't wait on build
[...]
Using koji to generate a mock config to replicate a buildroot
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
koji can be used to replicate a build root for local debugging
::
koji mock-config --help
Usage: koji mock-config [options] name
(Specify the --help global option for a list of other help options)
Options:
-h, --help show this help message and exit
--arch=ARCH Specify the arch
--tag=TAG Create a mock config for a tag
--task=TASK Duplicate the mock config of a previous task
--buildroot=BUILDROOT
Duplicate the mock config for the specified buildroot
id
--mockdir=DIR Specify mockdir
--topdir=DIR Specify topdir
--topurl=URL url under which Koji files are accessible
--distribution=DISTRIBUTION
Change the distribution macro
-o FILE Output to a file
for example to get the latest buildroot for dist-f12-build run
::
koji mock-config --tag dist-f12-build --arch=x86_64 --topurl=http://kojipkgs.fedoraproject.org/ dist-f12
you will need to pass in --topurl=http://kojipkgs.fedoraproject.org/ to
any mock-config command to get a working mock-config from fedoras koji.
Using Koji to control tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^
List tasks:
::
koji list-tasks
List only tasks requested by you:
::
koji list-tasks --mine
requeue an already-processed task: general syntax is: koji resubmit
[options] taskID
::
koji resubmit 3
Building a Package with the command-line tool
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Instead of using the fedpkg target, you can also directly use the
command\_line tool, koji.
To build a package, the syntax is:
::
$ koji build <build target> <git URL>
For example:
::
$ koji build dist-f14 'git url'
The koji build command creates a build task in Koji. By default the tool
will wait and print status updates until the build completes. You can
override this with the --nowait option.
.. raw:: html
</pre>
NOTE: For fedora koji, the git url MUST be based on
pkgs.fedoraproject.org. Other arbitrary git repos cannot be used for
builds.
Koji tags and packages organization
-----------------------------------
Terminology
^^^^^^^^^^^
In Koji, it is sometimes necessary to distinguish between a package in
general, a specific build of a package, and the various rpm files
created by a build. When precision is needed, these terms should be
interpreted as follows:
- Package: The name of a source rpm. This refers to the package in
general and not any particular build or subpackage. For example:
kernel, glibc, etc.
- Build: A particular build of a package. This refers to the entire
build: all arches and subpackages. For example: kernel-2.6.9-34.EL,
glibc-2.3.4-2.19.
- RPM: A particular rpm. A specific arch and subpackage of a build. For
example: kernel-2.6.9-34.EL.x86\_64, kernel-devel-2.6.9-34.EL.s390,
glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64
Tags and targets
^^^^^^^^^^^^^^^^
Koji organizes packages using tags. In Koji a tag is roughly a
collection of packages:
- Tags support inheritance
- Each tag has its own list of valid packages (inheritable)
- Package ownership can be set per-tag (inheritable)
- When you build you specify a target rather than a tag
A build target specifies where a package should be built and how it
should be tagged afterwards. This allows target names to remain fixed as
tags change through releases.
Koji commands for tags
^^^^^^^^^^^^^^^^^^^^^^
Targets
'''''''
You can get a full list of build targets with the following command:
::
$ koji list-targets
You can see just a single target with the --name option:
::
$ koji list-targets --name dist-f14
Name Buildroot Destination
---------------------------------------------------------------------------------------------
dist-f14 dist-f14-build dist-f14
This tells you a build for target dist-f14 will use a buildroot with
packages from the tag dist-f14-build and tag the resulting packages as
dist-f14.
Watch out: You probably don't want to build against dist-rawhide. If
Fedora N is the latest one out, to build to the next one, choose
dist-f{N+1}.
Tags
''''
You can get a list of tags with the following command:
::
$ koji list-tags
Packages
''''''''
As mentioned above, each tag has its own list of packages that may be
placed in the tag. To see that list for a tag, use the list-pkgs
command:
::
$ koji list-pkgs --tag dist-f14
The first column is the name of the package, the second tells you which
tag the package entry has been inherited from, and the third tells you
the owner of the package.
Latest Builds
'''''''''''''
To see the latest builds for a tag, use the latest-pkg command:
::
$ koji latest-pkg --all dist-f14
The output gives you not only the latest builds, but which tag they have
been inherited from and who built them.
`Category:Package Maintainers <Category:Package Maintainers>`__

190
docs/source/writing_a_plugin.rst Normal file
View file

@ -0,0 +1,190 @@
=========================
How to write Koji plugins
=========================
Writing Koji plugins
====================
Depending on what you are trying to do, there are different ways to
write a Koji plugin.
Each is described in this file, by use case.
Adding new task types
---------------------
Koji can do several things, for example build RPMs, or live CDs. Those
are types of tasks which Koji knows about.
If you need to do something which Koji does not know yet how to do, you
could create a Koji Builder plugin.
Such a plugin would minimally look like this:
::
from koji.tasks import BaseTaskHandler
class MyTask(BaseTaskHandler):
Methods = ['mytask']
_taskWeight = 2.0
def handler(self, arg1, arg2, kwarg1=None):
self.logger.debug("Running my task...")
# Here is where you actually do something
A few explanations on what goes on here:
- Your task needs to inherit from ``koji.tasks.BaseTaskHandler``
- Your task must have a ``Methods`` attribute, which is a list of the
method names your task can handle.
- You can specify the weight of your task with the ``_taskWeight``
attribute. The more intensive (CPU, IO, ...) your task is, the higher
this number should be.
- The task object has a ``logger`` attribute, which is a Python logger
with the usual ``debug``, ``info``, ``warning`` and ``error``
methods. The messages you send with it will end up in the Koji
Builder logs (``kojid.log``)
- Your task must have a ``handler()`` method. That is the method Koji
will call to run your task. It is the method that should actually do
what you need. It can have as many positional and named arguments as
you want.
Save your plugin as e.g ``mytask.py``, then install it in the Koji
Builder plugins folder: ``/usr/lib/koji-builder-plugins/``
Finally, edit the Koji Builder config file, ``/etc/kojid/kojid.conf``:
::
# A space-separated list of plugins to enable
plugins = mytask
Restart the Koji Builder service, and your plugin will be enabled.
You can try running a task from your new task type with the
command-line:
::
$ koji make-task mytask arg1 arg2 kwarg1
Exporting new API methods over XMLRPC
-------------------------------------
Koji clients talk to the Koji Hub via an XMLRPC API.
It is sometimes desirable to add to that API, so that clients can
request things Koji does not expose right now.
Such a plugin would minimally look like this:
::
def mymethod(arg1, arg2, kwarg1=None):
# Here is where you actually do something
mymethod.exported = True
A few explanations on what goes on here:
- Your plugin is just a method, with whatever positional and/or named
arguments you need.
- You must export your method by setting its ``exported`` attribute to
``True``
- The ``context.session.assertPerm()`` is how you ensure that the
Save your plugin as e.g ``mymethod.py``, then install it in the Koji Hub
plugins folder: ``/usr/lib/koji-hub-plugins/``
Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``:
::
# A space-separated list of plugins to enable
Plugins = mymethod
Restart the Koji Hub service, and your plugin will be enabled.
You can try calling the new XMLRPC API with the Python client library:
::
>>> import koji
>>> session = koji.ClientSession("http://koji/example.org/kojihub")
>>> session.mymethod(arg1, arg2, kwarg1='some value')
Ensuring the user has the required permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want your new XMLRPC API to require specific permissions from the
user, all you need to do is add the following to your method:
::
from koji.context import context
def mymethod(arg1, arg2, kwarg1=None):
context.session.assertPerm("admin")
# Here is where you actually do something
mymethod.exported = True
In the example above, Koji will ensure that the user is an
administrator. You could of course create your own permission, and check
for that.
Running code automatically triggered on events
----------------------------------------------
You might want to run something automatically when something else
happens in Koji.
A typical example is to automatically sign a package right after a build
finished. Another would be to send a notification to a message bus after
any kind of event.
This can be achieved with a plugin, which would look minimally as
follows:
::
from koji.plugin import callback
@callback('preTag', 'postTag')
def mycallback(cbtype, tag, build, user, force=False):
# Here is where you actually do something
A few explanations on what goes on here:
- The ``@callback`` decorator allows you to declare which events should
trigger your function. You can pass as many as you want. For a list
of supported events, see ``koji/plugins.py``.
- The arguments of the function depend on the event you subscribed to.
As a result, you need to know how it will be called by Koji. You
probably should use ``*kwargs`` to be safe. You can see how callbacks
are called in the ``hub/kojihub.py`` file, search for calls of the
``run_callbacks`` function.
Save your plugin as e.g ``mycallback.py``, then install it in the Koji
Hub plugins folder: ``/usr/lib/koji-hub-plugins``
Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``:
::
# A space-separated list of plugins to enable
Plugins = mycallback
Restart the Koji Hub service, and your plugin will be enabled.
You can try triggering your callback plugin with the command-line. For
example, if you registered a callback for the ``postTag`` event, try
tagging a build:
::
$ koji tag-build mytag mypkg-1.0-1

631
docs/source/writing_koji_code.rst Normal file
View file

@ -0,0 +1,631 @@
=================
Writing Koji Code
=================
Getting Started Hacking on Koji
===============================
This page gives an overview of the Koji code and then describes what
needs to change if you want to add a new type of task. A new task could
be for a new content type, or assembling the results of multiple builds
together, or something else that helps your workflow. New contributors
to Koji should leave this page knowing where to begin and have enough
understanding of Koji's architecture to be able to estimate how much
work is still ahead of them.
Task Flow
=========
A task starts with a user submitting it with the Koji client, which is a
command line interface. This contacts the hub, an apache-based server
application. It leaves a row in the database that represents a "free"
task, one that has not been assigned to a builder. Periodically, the
builders asynchronously ping the hub asking if there are any tasks
available, and at some point one will be given the new task. The hub
marks this in the database, and the builder begins executing the task (a
build).
Upon completion, the builder uploads the results to the hub, including
logs, binaries, environment information, and whatever else the task
handler for the build dictated. The hub moves the results to a permanent
shared storage solution, and marks the task as completed (or failed).
During this whole time, the webUI can be used to check up on progress.
So the flow of work is:
::
Client -> Hub -> Builder -> Hub
If you wanted to add a new build type or task that was tightly
integrated in Koji's data model, you would need to modify the CLI, Hub,
Builder, and WebUI at a minimum. Alternatively, you could do this with a
plugin, which is far simpler but less flexible.
Component Overview
==================
Koji is comprised of several components, this section goes into details
for each one, and what you potentially may need to change. Every
component is written in Python, so you will need to know that language
beyond a beginner level.
Koji-client
-----------
koji-client is a command line interface that provides many hooks into
Koji. It allows the user to query much of the data as well as perform
actions such as adding users and initiating build requests.
Option Handling
~~~~~~~~~~~~~~~
The code is in ``cli/koji``. It uses ``OptionParsers`` extensively with
interspersed arguments disabled. That means these two commands are not
interpreted the same:
::
$ koji -u admin -p password tag-build some-tag --force some-build
$ koji tag-build -u admin -p password some-tag --force some-build
The second one will generate an error, because -u and -p are not options
for tag-build, they must show up before that because they are global
options that can be used with any subcommand. There will be two
``OptionParsers`` used with each command. The first is used to pick up
arguments to ``koji`` itself, and the second for the subcommand
specified. When the first one executes (see ``get_options()``) it will
figure out the subcommand and come up with a function name based on it.
The convention is to prepend the word ``handle_`` before it, and change
all hyphens to underscores. If a command does not require an account
with Koji, the function handle will prepended with ``anon_handle_``
instead. The code will dynamically call the derived function handle
which is where the second ``OptionParser`` is used to parse the
remaining options. To have your code log into Koji (you're writing a
handle\_ function), use the ``activate_session`` function. All function
signatures in the client code will get a session object, which is your
interface to the hub.
Profiles
~~~~~~~~
It is possible to run the Koji client with different configuration
profiles so that you can interact with multiple Koji instances easily.
The ``--profile`` option to the Koji command itself enables this. You
should have a ``~/.koji/config`` already, if not just copy from
``/etc/koji.conf`` to get a start. The profile command accepts an
argument that matches a section in that config file. So if your config
file had this:
::
[Fedora]
authtype = ssl
server = http://koji.fedoraproject.org/kojihub
topdir = /mnt/koji
weburl = http://koji.fedoraproject.org/koji
#pkgurl = http://koji.fedoraproject.org/packages
cert = ~/.fedora.cert
ca = ~/.fedora-upload-ca.cert
serverca = ~/.fedora-server-ca.cert
[MyKoji]
server = http://koji.mydomain.com/kojihub
authtype = kerberos
topdir = /mnt/koji
weburl = http://koji.mydomain.com/koji
topurl = http://download.mydomain.com/kojifiles
you could pass Fedora or MyKoji to --profile.
Creating Tasks
~~~~~~~~~~~~~~
Once options are processed and understood, a task needs to be created on
the hub so that a builder can come along and take it. This is
accomplished with the ``maskTask`` method (defined on the Hub, so call
it on the ``session`` object). The name of the task should match the
name given to the task handler in the builder, which is explained later
on.
Be sure to process the channel, priority, background, and watch/nowatch
parameters too, which should be available to most new tasks. They'll be
buried in the first argument to your handler function, which captures
the options passed to the base Koji command.
If the client needs to make locally-available artifacts (config files,
sources, kickstarts) accessible to the builder, it must be uploaded to
the hub. This is the case with uploading SRPMs or kickstarts. You can
easily upload this content with the ``session.uploadWrapper`` method.
You can create progress bars as necessary with this snippet:
::
if _running_in_bg() or task_opts.noprogress:
callback = None
else:
callback = _progress_callback
serverdir = _unique_path('cli-image') # create a unique path on the hub
session.uploadWrapper(somefile, serverdir, callback=callback)
Task Arguments
~~~~~~~~~~~~~~
If you define a new task for Koji, you'll want the task submission
output to have the options ordered usefully. This output is
automatically generated, but sometimes it does not capture the more
important arguments you want displayed.
::
Created task 10001810
Watching tasks (this may be safely interrupted)...
10001810 thing (noarch): free
10001810 thing (noarch): free -> closed
0 free 0 open 1 done 0 failed
10001810 thing (noarch) completed successfully
In this (fake) example, you can see that "noarch" is the only option
being displayed, but maybe you want something more than just the task
architecture displayed, like some other options that were passed in. You
can fix this behavior in ``koji/__init__.py`` in the \_taskLabel
function. Here you can define the string(s) to display when Koji
receives status on a task. That is the return value.
Koji-Hub
--------
koji-hub is the center of all Koji operations. It is an XML-RPC server
running under mod\_wsgi in Apache. koji-hub is passive in that it only
receives XML-RPC calls and relies upon the build daemons and other
components to initiate communication. koji-hub is the only component
that has direct access to the database and is one of the two components
that have write access to the file system. If you want to make changes
to the webUI (new pages or themes), you are looking in the wrong
section, there is a separate component for that.
Implementation Details
~~~~~~~~~~~~~~~~~~~~~~
The **hub/kojihub.py** file is where the server-side code lives. If you
need to fix any server problems or want to add any new tasks, you will
need to modify this file. Changes to the database schema will almost
certainly require code changes too. This file gets deployed to
**/usr/share/koji-hub/kojihub.py**, whenever you make changes to that
remember to restart **httpd**. Also there are cases where httpd looks
for an existing .pyc file and takes it as-is, instead of re-compiling it
when the code is changed.
In the code there are two large classes: **RootExports** and
**HostExports**. RootExports exposes methods using XMLRPC for any client
that connects to the server. The Koji CLI makes use of this quite a bit.
If you want to expose a new API to any remote system, add your code
here. The HostExports class does the same thing except it will ensure
the requests are only coming from builders. Attempting to use an API
exposed here with the CLI will fail. If your work requires the builders
to call a new API, you should implement it here. Any other function
defined in this file is inaccessible by remote hosts. It is generally a
good practice to have the exposed APIs do very little work, and pass off
control to internal functions to do the heavy lifting.
Database Interactions
~~~~~~~~~~~~~~~~~~~~~
Database interactions are done with raw query strings, not with any kind
of modern ORM. Consider using context objects from the Koji contexts
library for thread-safe interactions. The database schema is captured in
the **docs** directory in the root of a git clone. A visualization of
the schema is not available at the time of this writing.
If you plan to introduce schema changes, please update both
``schema.sql`` and provide a migration script if necessary.
Troubleshooting
~~~~~~~~~~~~~~~
The hub runs in an Apache service, so you will need to look in Apache
logs for error messages if you are encountering 500 errors or the
service is failing to start. Specifically you want to check in:
- /var/log/httpd/error\_log
- /var/log/httpd/ssl\_error\_log
If you need more specific tracebacks and debugging data, consider
changing the debugging setting in **/etc/koji-hub/hub.conf**. Be advised
the hub is very verbose with this setting on, your logs will take up
gigabytes of space within several days.
Kojid
-----
kojid is the build daemon that runs on each of the build machines. Its
primary responsibility is polling for incoming build requests and
handling them accordingly. Essentially kojid asks koji-hub for work.
Koji also has support for tasks other than building. Creating install
images is one example. kojid is responsible for handling these tasks as
well. kojid uses mock for building. It also creates a fresh buildroot
for every build. kojid is written in Python and communicates with
koji-hub via XML-RPC.
Implementation Details
~~~~~~~~~~~~~~~~~~~~~~
The daemon runs as a service on a host that is traditionally not the
same as the hub or webUI. This is a good security practice because the
service runs as root, and executes untrusted code to produce builds on a
regular basis. Keeping the Hub separate limits the damage a malicious
package can do to the build system as a whole. For the same reason, the
filesystem that the hub keeps built software on should be mounted
Read-Only on the build host. It should call APIs on the hub that are
exposed through the ``HostExports`` class in the hub code. Whenever the
builder accepts a task, it forks a process to carry out the build.
An initscript/unit-file is available for kojid, so it can be stopped and
started like a normal service. Remember to do this when you deploy
changes!
TaskHandlers
^^^^^^^^^^^^
All tasks in kojid have a ``TaskHandler`` class that defines what to do
when the task is picked up from the hub. The base class is defined in
``koji/tasks.py`` where a lot of useful utility methods are available.
An example is ``uploadFile``, which is used to upload logs and built
binaries from a completed build to the hub since the shared filesystem
is read only.
The daemon code lives in ``builder/kojid``, which is deployed to
/usr/sbin/kojid. In there you'll notice that each task handler class has
a ``Methods`` member and ``_taskWeight`` member. These must be defined,
and the former is used to match the name of a waiting task (on the hub)
with the task handler code to execute. Each task handler object must
have a ``handler`` method defined, which is the entry point for the
forked process when a builder accepts a task.
Tasks can have subtasks, which is a typical model when a build can be
run on multiple architectures. In this case, developers should write 2
task handlers: one handles the build for exact one architecture, and one
that assembles the results of those tasks into a single build, and sends
status information to the hub. You can think of the latter handler as
the parent task.
All task handler objects have a ``session`` object defined, which is the
interface to use for communications with the hub. So, parent tasks
should kick off child tasks using the session object's subtask method
(which is part of HostExports). It should then call ``self.wait`` with
``all=True`` to wait for the results of the child tasks.
Here's a stub of what a new build task might look like:
::
class BuildThingTask(BaseTaskHandler):
Methods = ['thing']
_taskWeight = 0.5
def handler(self, a, b, arches, options):
subtasks = {}
for arch in arches:
subtasks[arch] = session.host.subtask(method='thingArch', a, b, arch)
results = self.wait(subtasks.values(), all=True)
# parse results and put rows in database
# put files in their final resting place
return 'Build successful'
class BuildThingArchTask(BaseTaskHandler):
Methods = ['thingArch']
_taskWeight = 2.0
def handler(self, a, b, arch):
# do the build, capture results in a variable
self.uploadFile('/path/to/some/log')
self.uploadFile('/path/to/binary/file')
return result
Source Control Managers
^^^^^^^^^^^^^^^^^^^^^^^
If you your build needs to check out code from a Source Control Manager
(SCM) such as git or subversion, you can use SCM objects defined in
``koji/daemon.py``. They take a specially formed URL as an argument to
the constructor. Here's an example use. The second line is important, it
makes sure the SCM is in the whitelist of SCMs allowed in
``/etc/kojid/kojid.conf``.
::
scm = SCM(url)
scm.assert_allowed(self.options.allowed_scms)
directory = scm.checkout('/checkout/path', session, uploaddir, logfile)
Checking out takes 4 arguments: where to checkout, a session object
(which is how authentication is handled), a directory to upload the log
to, and a string representing the log file name. Using this method Koji
will checkout (or clone) a remote repository and upload a log of the
standard output to the task results.
Build Root Objects
^^^^^^^^^^^^^^^^^^
It is encouraged to build software in mock chroots if appropriate. That
way Koji can easily track precise details about the environment in which
the build was executed. In ``builder/kojid`` a BuildRoot class is
defined, which provides an interface to execute mock commands. Here's an
example of their use:
::
broot = BuildRoot(self.session, self.options, build_tag, arch, self.id)
A session object, task options, and a build tag should be passed in
as-is. You should also specify the architecture and the task ID. If you
ever need to pass in specialized options to mock, look in the
ImageTask.makeImgBuildRoot method to see how they are defined and passed
in to the BuildRoot constructor.
Troubleshooting
~~~~~~~~~~~~~~~
The daemon writes a log file to ``/var/log/kojid.log``. Debugging output
can be turned on in ``/etc/kojid/kojid.conf``.
Koji-Web
--------
koji-web is a set of scripts that run in mod\_wsgi and use the Cheetah
templating engine to provide a web interface to Koji. It acts as a
client to koji-hub providing a visual interface to perform a limited
amount of administration. koji-web exposes a lot of information and also
provides a means for certain operations, such as cancelling builds.
The web pages are derived from Cheetah templates, the syntax of which
you can read up on
`here <http://cheetahtemplate.org/docs/users_guide_html/>`__. These
templates are the ``chtml`` files sitting in ``www/kojiweb``. You'll
notice quickly that these templates are referencing variables, but where
do they come from?
The ``www/kojiweb/index.py`` file provides them. There are several
functions named after the templates they support, and in each one a
dictionary called ``values`` is populated. This is how data is gathered
about the task, build, archive, or whatever the page is about. Take your
time with ``taskinfo.chtml`` in particular, as the conditionals there
have gotten quite long. If you are adding a new task to Koji, you will
need to extend this at a minimum. A new type of build task would require
this, and possibly another that is specific to viewing the archived
information about the build. (taskinfo vs. buildinfo)
If your web page needs to display the contents of a list or dictionary,
use the ``$printMap`` function to help with that. It is often sensible
to define a function that easily prints options and values in a
dictionary. An example of this is in taskinfo.chtml.
::
#def printOpts($opts)
#if $opts
<strong>Options:</strong><br/>
$printMap($opts, '&nbsp;&nbsp;')
#end if
#end def
Finally, if you need to expand the drop-down menus of "method" types
when searching for tasks in the WebUI, you will need to add them to the
``_TASKS`` list in ``www/kojiweb/index.py``. Add values where
appropriate to ``_TOPLEVEL_TASKS`` and ``_PARENT_TASKS`` as well so that
parent-child relationships show up correctly too.
Remember whenever you update a template or index.py, you will need to
deploy and restart apache/httpd!
Troubleshooting
~~~~~~~~~~~~~~~
Like the hub, this component is backed by apache, so you should follow
the same techniques for debugging Koji-Web as
`Koji-Hub <#Troubleshooting>`__.
Kojira
------
kojira is a daemon that keeps the build root repodata updated. It is
responsible for removing redundant build roots and cleaning up after a
build request is completed.
Building and Deploying Changes
==============================
The root of the git clone for Koji code contains a ``Makefile`` that has
a few targets to make building and deployment a little easier. Among
them are:
- tarball: create a bz2 tarball that could be consumed in an rpm build
- rpm: create Koji rpms. The NVRs will be defined by the spec file,
which is also in the same directory. The results will appear in a
``noarch`` directory.
- test-rpm: like rpm, but append the Release field with a date and time
stamp for easy upgrade-deployment
Plugins
=======
This section is copied from the ``docs/Writing_a_plugin.md`` file.
Koji supports different types of plugins, three of which are captured
here. Depending on what you are trying to do, there are different ways
to write a Koji plugin.
Koji Builder Plugins
--------------------
Koji can do several things, for example build RPMs, or live CDs. Those
are types of tasks which Koji knows about. If you need to do something
which Koji does not know yet how to do, you could create a Koji Builder
plugin. Such a plugin would minimally look like this:
::
from koji.tasks import BaseTaskHandler
class MyTask(BaseTaskHandler):
Methods = ['mytask']
_taskWeight = 2.0
def handler(self, arg1, arg2, kwarg1=None):
self.logger.debug("Running my task...")
# Here is where you actually do something
A few explanations on what goes on here:
- Your task needs to inherit from \`koji.tasks.BaseTaskHandler\`
- Your task must have a \`Methods\` attribute, which is a list of the
method names your task can handle.
- You can specify the weight of your task with the \`\_taskWeight\`
attribute. The more intensive (CPU, IO, ...) your task is, the higher
this number should be.
- The task object has a ``logger`` attribute, which is a Python logger
with the usual \`debug\`, \`info\`, \`warning\` and \`error\`
methods. The messages you send with it will end up in the Koji
Builder log.
- Your task must have a \`handler()\` method. That is the method Koji
will call to run your task. It is the method that should actually do
what you need. It can have as many positional and named arguments as
you want.
Save your plugin as e.g ``mytask.py``, then install it in the Koji
Builder plugins folder: ``/usr/lib/koji-builder-plugins/``. Finally,
edit the Koji Builder config file, ``/etc/kojid/kojid.conf``:
::
# A space-separated list of plugins to enable
plugins = mytask
Restart the Koji Builder service, and your plugin will be enabled. You
can try running a task from your new task type with the command-line:
``$ koji make-task mytask arg1 arg2 kwarg1``
Hub Plugins
-----------
Koji clients talk to the Koji Hub via an XMLRPC API. It is sometimes
desirable to add to that API, so that clients can request things Koji
does not expose right now. Such a plugin would minimally look like this:
::
def mymethod(arg1, arg2, kwarg1=None):
# Here is where you actually do something
mymethod.exported = True
What's happening?
- Your plugin is just a method, with whatever positional and/or named
arguments you need.
- You must export your method by setting its ``exported`` attribute to
``True``
- The ``context.session.assertPerm()`` is how you ensure that the
correct permissions are available.
Save your plugin as e.g \`mymethod.py\`, then install it in the Koji Hub
plugins folder, which is ``/usr/lib/koji-hub-plugins/``
Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``:
::
# A space-separated list of plugins to enable
Plugins = mymethod
Restart the Koji Hub service, and your plugin will be enabled. You can
try calling the new XMLRPC API with the Python client library:
::
>>> import koji
>>> session = koji.ClientSession("http://koji/example.org/kojihub")
>>> session.mymethod(arg1, arg2, kwarg1='some value')
If you want your new XMLRPC API to require specific permissions from the
user, all you need to do is add the following to your method:
::
from koji.context import context
def mymethod(arg1, arg2, kwarg1=None):
context.session.assertPerm("admin")
# Here is where you actually do something
mymethod.exported = True
In the example above, Koji will ensure that the user is an
administrator. You could of course create your own permission, and check
for that.
Event Plugin
------------
You might want to run something automatically when something else
happens in Koji. A typical example is to automatically sign a package
right after a build finished. Another would be to send a notification to
a message bus after any kind of event.
This can be achieved with a plugin too, which would look minimally as
follows:
::
from koji.plugin import callback
@callback('preTag', 'postTag')
def mycallback(cbtype, tag, build, user, force=False):
# Here is where you actually do something
So what is this doing?
- The ``@callback`` decorator allows you to declare which events should
trigger your function. You can pass as many as you want. For a list
of supported events, see ``koji/plugins.py``.
- The arguments of the function depend on the event you subscribed to.
As a result, you need to know how it will be called by Koji. You
probably should use ``*kwargs`` to be safe. You can see how callbacks
are called in the ``hub/kojihub.py`` file, search for calls of the
``run_callbacks`` function.
Save your plugin as e.g ``mycallback.py``, then install it in the Koji
Hub plugins folder: ``/usr/lib/koji-hub-plugins``
Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``:
::
# A space-separated list of plugins to enable
Plugins = mycallback
Restart the Koji Hub service, and your plugin will be enabled. You can
try triggering your callback plugin with the command-line. For example,
if you registered a callback for the ``postTag`` event, try tagging a
build: ``$ koji tag-build mytag mypkg-1.0-1``
Patch Review
============
If you have a patch to submit, please send it to
koji-devel@lists.fedoraproject.org. Here are some guidelines on
producing preferable patches.
- Please do not "patch bomb". Keep them manageable, perhaps providing a
patch for each component that is getting changes.
- Adhere to `PEP8 <https://www.python.org/dev/peps/pep-0008/>`__
- Provide patches a feature at a time, not a pile of several. Please
follow the same practice for bugs.
- Consider using ``git-send-email``, or just attach the patches to the
email
- Patches should apply cleanly to HEAD of the intended branch
- Please maintain backward-compatibility up to RHEL 5 (which means
Python 2.4)
There are only 3-5 people with commit access that regularly watch the
mailing list, so responses may take a couple days if we're all occupied.