diff --git a/.gitignore b/.gitignore index 846fc378..e28848bf 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ *.pyo tests/test.py .coverage +docs/build/ diff --git a/docs/HOWTO.html b/docs/HOWTO.html deleted file mode 100644 index 2abb3e04..00000000 --- a/docs/HOWTO.html +++ /dev/null @@ -1,321 +0,0 @@ - - - Koji HOWTO - - -

Introduction

- - Koji is a system for building and tracking RPMs. It was designed with the following - features in mind: - -

- Security -

- - Leverage other software - - - Flexibility - - - Usability - - - Reproducibility - - - -

This HOWTO document covers the basic tasks that a developer needs to be - able to accomplish with Koji. -

- -

Getting started

- -

The web interface

-

The primary interface for viewing Koji data is a web application. Most of the interface - is read-only, but if you are logged in (see below) and have sufficient privileges there - are some actions that can be performed though the web. For example: -

- Those with admin privileges will find additional actions, such as: - - -

The web site utilizes Kerberos authentication. In order to log in you will - need a valid Kerberos ticket and your web browser will need to be configured to send the - Kerberos information to the server. - -

In Firefox or Mozilla, you will need to use the about:config page to set a few parameters. - Use the search term 'negotiate' to filter the list. Change - network.negotiate-auth.trusted-uris to the domain you want to authenticate against, - e.g .example.com. You can leave network.negotiate-auth.delegation-uris blank, as it - enables Kerberos ticket passing, which is not required. If you do not see those two - config options listed, your version of Firefox or Mozilla may be too old to support - Negotiate authentication, and you should consider upgrading. - -

In order to obtain a Kerberos ticket, use the kinit command. - - -

Installing the Koji cli

-

There is a single point of entry for most operations. The command is - called 'koji' and is included in the main koji package. - -

Repos/webpage TBD - -

- The koji tool authenticates to the central server using Kerberos, so you will need - to have a valid Kerberos ticket to use many features. However, many of the read-only - commands will work without authentication. - -

Building a package

-

Builds are initiated with the command line tool. - To build a package, the syntax is:

- 
$ koji build <build target> <cvs URL>
- -

For example:

- 
$ koji build dist-fc7-scratch 'cvs://cvs.example.com/cvs/dist?rpms/kernel/FC-7#kernel-2_6_20-1_2925_fc7'
-

- The koji build command creates a build task in Koji. By default - the tool will wait - and print status updates until the build completes. You can override this with - the --nowait option. To view other options to the build command use the - --help option. -

- - 
$ koji build --help
-
- -

Build Options

-

- There are a few options to the build command. Here are some more detailed explanations - of them: -

- -
-
--skip-tag
-
Normally the package is tagged after the build completes. This option causes - the tagging step to be skipped. The package will be in the system, but untagged - (you can later tag it with the tag-pkg command)
-
--scratch
-
This makes the build into a scratch build. The build will not be - imported into the db, it will just be built. The rpms will land under - <topdir>/scratch. Scratch builds are not tracked and can never - be tagged, but can be convenient for testing. Scratch builds are - typically removed from the filesystem after one week. -
-
--nowait
-
As stated above, this prevents the cli from waiting on the build task.
-
--arch-override
-
This option allows you to override the base set of arches to build for. - This option is really only for testing during the beta period, but it may - be retained for scratch builds in the future.
-
- -

Build Failures

-

If your package fails to build, you will see something like this.

- 
-      420066 buildArch (kernel-2.6.18-1.2739.10.9.el5.jjf.215394.2.src.rpm,
-      ia64): open (build-1.example.com) -> FAILED: BuildrootError:
-      error building package (arch ia64), mock exited with status 10
-
- -

You can figure out why the build failed by looking at the log files. If - there is a build.log, start there. Otherwise, look at init.log

- - 
-      $ ls -1 <topdir>/work/tasks/420066/*
-      <topdir>/work/tasks/420066/build.log
-      <topdir>/work/tasks/420066/init.log
-      <topdir>/work/tasks/420066/mockconfig.log
-      <topdir>/work/tasks/420066/root.log
-
- -

Filing Bugs

- -

bug tracking TBD - -

Koji Architecture

- -

Terminology

- - In Koji, it is sometimes necessary to distinguish between the a package in general, - a specific build of a package, and the various rpm files created by a build. When - precision is needed, these terms should be interpreted as follows: - -
-
Package
-
The name of a source rpm. This refers to the package in general and not - any particular build or subpackage. For example: kernel, glibc, etc.
-
Build
-
A particular build of a package. This refers to the entire build: all arches - and subpackages. For example: kernel-2.6.9-34.EL, glibc-2.3.4-2.19.
-
RPM
-
A particular rpm. A specific arch and subpackage of a build. - For example: kernel-2.6.9-34.EL.x86_64, kernel-devel-2.6.9-34.EL.s390, - glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64
-
- - -

Koji Components

- - Koji is comprised of several components: - - - -

Package Organization

-

Tags and Targets

-

Koji organizes packages using tags. In Koji a tag is roughly analogous to - a beehive collection instance, but differ in a number of ways:

- -

- A build target specifies where a package should be built and how it - should be tagged afterwards. This allows target names to remain fixed - as tags change through releases. You can get a full list of build targets - with the following command:

- 
$ koji list-targets
-
- You can see just a single target with the --name option: - 
$ koji list-targets --name dist-fc7
-Name                           Buildroot                      Destination
----------------------------------------------------------------------------------------------
-dist-fc7                       dist-fc7-build                 dist-fc7
-
- This tells you a build for target dist-fc7 will use a buildroot with packages - from the tag dist-fc7-build and tag the resulting packages as dist-fc7. -

- You can get a list of tags with the following command:

- 
$ koji list-tags
-
-

Package lists

-

- As mentioned above, each tag has its own list of packages that may be placed - in the tag. To see that list for a tag, use the list-pkgs command:

- 
$ koji list-pkgs --tag dist-fc7
-Package                 Tag                     Extra Arches     Owner
------------------------ ----------------------- ---------------- ----------------
-ElectricFence           dist-fc6                                 pmachata
-GConf2                  dist-fc6                                 rstrode
-lucene                  dist-fc6                                 dbhole
-lvm2                    dist-fc6                                 lvm-team
-ImageMagick             dist-fc6                                 nmurray
-m17n-db                 dist-fc6                                 majain
-m17n-lib                dist-fc6                                 majain
-MAKEDEV                 dist-fc6                                 clumens
-...
-
- The first column is the name of the package, the second tells you which tag - the package entry has been inherited from, and the third tells you the owner - of the package. -

Latest Builds

-

- To see the latest builds for a tag, use the latest-pkg command:

- 
$ koji latest-pkg --all dist-fc7
-Build                                     Tag                   Built by
-----------------------------------------  --------------------  ----------------
-ConsoleKit-0.1.0-5.fc7                    dist-fc7              davidz
-ElectricFence-2.2.2-20.2.2                dist-fc6              jkeating
-GConf2-2.16.0-6.fc7                       dist-fc7              mclasen
-ImageMagick-6.2.8.0-3.fc6.1               dist-fc6-updates      nmurray
-MAKEDEV-3.23-1.2                          dist-fc6              nalin
-MySQL-python-1.2.1_p2-2                   dist-fc7              katzj
-NetworkManager-0.6.5-0.3.cvs20061025.fc7  dist-fc7              caillon
-ORBit2-2.14.6-1.fc7                       dist-fc7              mclasen
-
- The output gives you not only the latest builds, but which tag they have - been inherited from and who built them (note: for builds imported from beehive - the "built by" field may be misleading) - - -

Exploring Koji

- -

We've tried to make Koji self-documenting wherever possible. The command - line tool will print a list of valid commands and each command supports - --help. For example:

- - 
-$ koji help
-Koji commands are:
-        build                Build a package from source
-        cancel-task          Cancel a task
-        help                 List available commands
-        latest-build         Print the latest rpms for a tag
-        latest-pkg           Print the latest builds for a tag
-...
-$ koji build --help
-usage: koji build [options] tag URL
-(Specify the --help global option for a list of other help options)
-
-options:
-  -h, --help            show this help message and exit
-  --skip-tag            Do not attempt to tag package
-  --scratch             Perform a scratch build
-  --nowait              Don't wait on build
-...
-
- -

Getting Involved

- - If you would like to be more involved with the Koji project... - -

Project data TBD - - - diff --git a/docs/Makefile b/docs/Makefile index 43d8d68c..5709e83c 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -1,4 +1,177 @@ -install: +# Makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +PAPER = +BUILDDIR = build + +# User-friendly check for sphinx-build +ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) +$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) +endif + +# Internal variables. +PAPEROPT_a4 = -D latex_paper_size=a4 +PAPEROPT_letter = -D latex_paper_size=letter +ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source +# the i18n builder cannot share the environment and doctrees with the others +I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source + +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext + +help: + @echo "Please use \`make ' where is one of" + @echo " html to make standalone HTML files" + @echo " dirhtml to make HTML files named index.html in directories" + @echo " singlehtml to make a single large HTML file" + @echo " pickle to make pickle files" + @echo " json to make JSON files" + @echo " htmlhelp to make HTML files and a HTML help project" + @echo " qthelp to make HTML files and a qthelp project" + @echo " devhelp to make HTML files and a Devhelp project" + @echo " epub to make an epub" + @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" + @echo " latexpdf to make LaTeX files and run them through pdflatex" + @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" + @echo " text to make text files" + @echo " man to make manual pages" + @echo " texinfo to make Texinfo files" + @echo " info to make Texinfo files and run them through makeinfo" + @echo " gettext to make PO message catalogs" + @echo " changes to make an overview of all changed/added/deprecated items" + @echo " xml to make Docutils-native XML files" + @echo " pseudoxml to make pseudoxml-XML files for display purposes" + @echo " linkcheck to check all external links for integrity" + @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: - rm -f *.o *.so *.pyc *~ + rm -rf $(BUILDDIR)/* + +html: + $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." + +dirhtml: + $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." + +singlehtml: + $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml + @echo + @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." + +pickle: + $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle + @echo + @echo "Build finished; now you can process the pickle files." + +json: + $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json + @echo + @echo "Build finished; now you can process the JSON files." + +htmlhelp: + $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp + @echo + @echo "Build finished; now you can run HTML Help Workshop with the" \ + ".hhp project file in $(BUILDDIR)/htmlhelp." + +qthelp: + $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp + @echo + @echo "Build finished; now you can run "qcollectiongenerator" with the" \ + ".qhcp project file in $(BUILDDIR)/qthelp, like this:" + @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Koji.qhcp" + @echo "To view the help file:" + @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Koji.qhc" + +devhelp: + $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp + @echo + @echo "Build finished." + @echo "To view the help file:" + @echo "# mkdir -p $$HOME/.local/share/devhelp/Koji" + @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Koji" + @echo "# devhelp" + +epub: + $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub + @echo + @echo "Build finished. The epub file is in $(BUILDDIR)/epub." + +latex: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo + @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." + @echo "Run \`make' in that directory to run these through (pdf)latex" \ + "(use \`make latexpdf' here to do that automatically)." + +latexpdf: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through pdflatex..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +latexpdfja: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through platex and dvipdfmx..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +text: + $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text + @echo + @echo "Build finished. The text files are in $(BUILDDIR)/text." + +man: + $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man + @echo + @echo "Build finished. The manual pages are in $(BUILDDIR)/man." + +texinfo: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo + @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." + @echo "Run \`make' in that directory to run these through makeinfo" \ + "(use \`make info' here to do that automatically)." + +info: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo "Running Texinfo files through makeinfo..." + make -C $(BUILDDIR)/texinfo info + @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." + +gettext: + $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale + @echo + @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." + +changes: + $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes + @echo + @echo "The overview file is in $(BUILDDIR)/changes." + +linkcheck: + $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck + @echo + @echo "Link check complete; look for any errors in the above output " \ + "or in $(BUILDDIR)/linkcheck/output.txt." + +doctest: + $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest + @echo "Testing of doctests in the sources finished, look at the " \ + "results in $(BUILDDIR)/doctest/output.txt." + +xml: + $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml + @echo + @echo "Build finished. The XML files are in $(BUILDDIR)/xml." + +pseudoxml: + $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml + @echo + @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." diff --git a/docs/Migrating_to_1.10.txt b/docs/Migrating_to_1.10.txt deleted file mode 100644 index e934706c..00000000 --- a/docs/Migrating_to_1.10.txt +++ /dev/null @@ -1,76 +0,0 @@ -Migrating to Koji 1.10 -====================== - -// asciidoc formatted - -The 1.10 release of Koji includes a few changes that you should consider when -migrating. - -DB Updates ----------- - -The new +tag_extra+ table tracks extra data for tags. - -There is a new entry in the +channels+ table and some additions and updates to -the +archivetypes+ table. - -As in previous releases, we provide a migration script that updates the -database. - - # psql koji koji >> import koji - >>> session = koji.ClientSession("http://koji/example.org/kojihub") - >>> session.mymethod(arg1, arg2, kwarg1='some value') - -### Ensuring the user has the required permissions - -If you want your new XMLRPC API to require specific permissions from the user, -all you need to do is add the following to your method: - - from koji.context import context - - def mymethod(arg1, arg2, kwarg1=None): - context.session.assertPerm("admin") - - # Here is where you actually do something - - mymethod.exported = True - -In the example above, Koji will ensure that the user is an administrator. You -could of course create your own permission, and check for that. - -## Running code automatically triggered on events - -You might want to run something automatically when something else happens in -Koji. - -A typical example is to automatically sign a package right after a build -finished. Another would be to send a notification to a message bus after any -kind of event. - -This can be achieved with a plugin, which would look minimally as follows: - - from koji.plugin import callback - - @callback('preTag', 'postTag') - def mycallback(cbtype, tag, build, user, force=False): - # Here is where you actually do something - -A few explanations on what goes on here: - -* The `@callback` decorator allows you to declare which events should trigger - your function. You can pass as many as you want. For a list of supported - events, see `koji/plugins.py`. -* The arguments of the function depend on the event you subscribed to. As a - result, you need to know how it will be called by Koji. You probably should - use `*kwargs` to be safe. You can see how callbacks are called in the - `hub/kojihub.py` file, search for calls of the `run_callbacks` function. - -Save your plugin as e.g `mycallback.py`, then install it in the Koji Hub -plugins folder: `/usr/lib/koji-hub-plugins` - -Finally, edit the Koji Hub config file, `/etc/koji-hub/hub.conf`: - - # A space-separated list of plugins to enable - Plugins = mycallback - -Restart the Koji Hub service, and your plugin will be enabled. - -You can try triggering your callback plugin with the command-line. For -example, if you registered a callback for the `postTag` event, try tagging a -build: - - $ koji tag-build mytag mypkg-1.0-1 diff --git a/docs/source/HOWTO.rst b/docs/source/HOWTO.rst new file mode 100644 index 00000000..ab7dec2a --- /dev/null +++ b/docs/source/HOWTO.rst @@ -0,0 +1,342 @@ +========== +Koji HOWTO +========== + +Introduction +============ + +Koji is a system for building and tracking RPMs. It was designed with +the following features in mind: + +**Security** + +- New buildroot for each build +- nfs is used (mostly) read-only + +**Leverage other software** + +- Uses Yum and Mock open-source components +- XML-RPC APIs for easy integration with other tools + +**Flexibility** + +- rich data model +- active code base + +**Usability** + +- Web interface with Kerberos authentication +- Thin, portable client +- Users can create local buildroots + +**Reproducibility** + +- Buildroot contents are tracked in the database +- Versioned data + +This HOWTO document covers the basic tasks that a developer needs to be +able to accomplish with Koji. + +Getting started +=============== + +The web interface +----------------- + +The primary interface for viewing Koji data is a web application. Most +of the interface is read-only, but if you are logged in (see below) and +have sufficient privileges there are some actions that can be performed +though the web. For example: + +- Cancel a build +- Resubmit a failed task + +Those with admin privileges will find additional actions, such as: + +- Create/Edit/Delete a tag +- Create/Edit/Delete a target +- Enable/Disable a build host + +The web site utilizes Kerberos authentication. In order to log in you +will need a valid Kerberos ticket and your web browser will need to be +configured to send the Kerberos information to the server. + +In Firefox or Mozilla, you will need to use the about:config page to set +a few parameters. Use the search term 'negotiate' to filter the list. +Change network.negotiate-auth.trusted-uris to the domain you want to +authenticate against, e.g .example.com. You can leave +network.negotiate-auth.delegation-uris blank, as it enables Kerberos +ticket passing, which is not required. If you do not see those two +config options listed, your version of Firefox or Mozilla may be too old +to support Negotiate authentication, and you should consider upgrading. + +In order to obtain a Kerberos ticket, use the kinit command. + +Installing the Koji cli +----------------------- + +There is a single point of entry for most operations. The command is +called 'koji' and is included in the main koji package. + +Repos/webpage TBD + +The koji tool authenticates to the central server using Kerberos, so you +will need to have a valid Kerberos ticket to use many features. However, +many of the read-only commands will work without authentication. + +Building a package +------------------ + +Builds are initiated with the command line tool. To build a package, the +syntax is: + +:: + + $ koji build + +For example: + +:: + + $ koji build dist-fc7-scratch 'cvs://cvs.example.com/cvs/dist?rpms/kernel/FC-7#kernel-2_6_20-1_2925_fc7' + +The ``koji build`` command creates a build task in Koji. By default the +tool will wait and print status updates until the build completes. You +can override this with the ``--nowait`` option. To view other options to +the build command use the ``--help`` option. + +:: + + $ koji build --help + +Build Options +------------- + +There are a few options to the build command. Here are some more +detailed explanations of them: + +``--skip-tag`` + Normally the package is tagged after the build completes. This + option causes the tagging step to be skipped. The package will be in + the system, but untagged (you can later tag it with the tag-pkg + command) +``--scratch`` + This makes the build into a scratch build. The build will not be + imported into the db, it will just be built. The rpms will land + under /scratch. Scratch builds are not tracked and can never + be tagged, but can be convenient for testing. Scratch builds are + typically removed from the filesystem after one week. +``--nowait`` + As stated above, this prevents the cli from waiting on the build + task. +``--arch-override`` + This option allows you to override the base set of arches to build + for. This option is really only for testing during the beta period, + but it may be retained for scratch builds in the future. + +Build Failures +-------------- + +If your package fails to build, you will see something like this. + +:: + + 420066 buildArch (kernel-2.6.18-1.2739.10.9.el5.jjf.215394.2.src.rpm, + ia64): open (build-1.example.com) -> FAILED: BuildrootError: + error building package (arch ia64), mock exited with status 10 + +You can figure out why the build failed by looking at the log files. If +there is a build.log, start there. Otherwise, look at init.log + +:: + + $ ls -1 /work/tasks/420066/* + /work/tasks/420066/build.log + /work/tasks/420066/init.log + /work/tasks/420066/mockconfig.log + /work/tasks/420066/root.log + + +Filing Bugs +----------- + +bug tracking TBD + +Koji Architecture +================= + +Terminology +----------- + +In Koji, it is sometimes necessary to distinguish between the a package +in general, a specific build of a package, and the various rpm files +created by a build. When precision is needed, these terms should be +interpreted as follows: + +Package + The name of a source rpm. This refers to the package in general and + not any particular build or subpackage. For example: kernel, glibc, + etc. +Build + A particular build of a package. This refers to the entire build: + all arches and subpackages. For example: kernel-2.6.9-34.EL, + glibc-2.3.4-2.19. +RPM + A particular rpm. A specific arch and subpackage of a build. For + example: kernel-2.6.9-34.EL.x86\_64, kernel-devel-2.6.9-34.EL.s390, + glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64 + +Koji Components +--------------- + +Koji is comprised of several components: + +- **koji-hub** is the center of all Koji operations. It is an XML-RPC + server running under mod\_python in Apache. koji-hub is passive in + that it only receives XML-RPC calls and relies upon the build daemons + and other components to initiate communication. koji-hub is the only + component that has direct access to the database and is one of the + two components that have write access to the file system. +- **kojid** is the build daemon that runs on each of the build machines. + Its primary responsibility is polling for incoming build requests and + handling them accordingly. Koji also has support for tasks other than + building. Creating install images is one example. kojid is + responsible for handling these tasks as well. + + kojid uses mock for building. It also creates a fresh buildroot for + every build. kojid is written in Python and communicates with + koji-hub via XML-RPC. + +- **koji-web** is a set of scripts that run in mod\_python and use the + Cheetah templating engine to provide an web interface to Koji. + koji-web exposes a lot of information and also provides a means for + certain operations, such as cancelling builds. +- **koji** is a CLI written in Python that provides many hooks into Koji. + It allows the user to query much of the data as well as perform + actions such as build initiation. +- **kojirepod** is a daemon that keeps the build root repodata updated. + +Package Organization +-------------------- + +**Tags and Targets** + +Koji organizes packages using tags. In Koji a tag is roughly analogous +to a beehive collection instance, but differ in a number of ways: + +- Tags are tracked in the database but not on disk +- Tags support multiple inheritance +- Each tag has its own list of valid packages (inheritable) +- Package ownership can be set per-tag (inheritable) +- Tag inheritance is more configurable +- When you build you specify a *target* rather than a tag + +A build target specifies where a package should be built and how it +should be tagged afterwards. This allows target names to remain fixed as +tags change through releases. You can get a full list of build targets +with the following command: + +:: + + $ koji list-targets + +You can see just a single target with the ``--name`` option: + +:: + + $ koji list-targets --name dist-fc7 + Name Buildroot Destination + --------------------------------------------------------------------------------------------- + dist-fc7 dist-fc7-build dist-fc7 + +This tells you a build for target dist-fc7 will use a buildroot with +packages from the tag dist-fc7-build and tag the resulting packages as +dist-fc7. + +You can get a list of tags with the following command: + +:: + + $ koji list-tags + +*Package lists* + +As mentioned above, each tag has its own list of packages that may be +placed in the tag. To see that list for a tag, use the ``list-pkgs`` +command: + +:: + + $ koji list-pkgs --tag dist-fc7 + Package Tag Extra Arches Owner + ----------------------- ----------------------- ---------------- ---------------- + ElectricFence dist-fc6 pmachata + GConf2 dist-fc6 rstrode + lucene dist-fc6 dbhole + lvm2 dist-fc6 lvm-team + ImageMagick dist-fc6 nmurray + m17n-db dist-fc6 majain + m17n-lib dist-fc6 majain + MAKEDEV dist-fc6 clumens + ... + +The first column is the name of the package, the second tells you which +tag the package entry has been inherited from, and the third tells you +the owner of the package. + +**Latest Builds** + +To see the latest builds for a tag, use the ``latest-pkg`` command: + +:: + + $ koji latest-pkg --all dist-fc7 + Build Tag Built by + ---------------------------------------- -------------------- ---------------- + ConsoleKit-0.1.0-5.fc7 dist-fc7 davidz + ElectricFence-2.2.2-20.2.2 dist-fc6 jkeating + GConf2-2.16.0-6.fc7 dist-fc7 mclasen + ImageMagick-6.2.8.0-3.fc6.1 dist-fc6-updates nmurray + MAKEDEV-3.23-1.2 dist-fc6 nalin + MySQL-python-1.2.1_p2-2 dist-fc7 katzj + NetworkManager-0.6.5-0.3.cvs20061025.fc7 dist-fc7 caillon + ORBit2-2.14.6-1.fc7 dist-fc7 mclasen + +The output gives you not only the latest builds, but which tag they have +been inherited from and who built them (note: for builds imported from +beehive the "built by" field may be misleading) + +Exploring Koji +-------------- + +We've tried to make Koji self-documenting wherever possible. The command +line tool will print a list of valid commands and each command supports +``--help``. For example: + +:: + + $ koji help + Koji commands are: + build Build a package from source + cancel-task Cancel a task + help List available commands + latest-build Print the latest rpms for a tag + latest-pkg Print the latest builds for a tag + ... + $ koji build --help + usage: koji build [options] tag URL + (Specify the --help global option for a list of other help options) + + options: + -h, --help show this help message and exit + --skip-tag Do not attempt to tag package + --scratch Perform a scratch build + --nowait Don't wait on build + ... + +Getting Involved +================ + +If you would like to be more involved with the Koji project... + +Project data TBD diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 00000000..c3945fd8 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,261 @@ +# -*- coding: utf-8 -*- +# +# Koji documentation build configuration file, created by +# sphinx-quickstart on Tue Nov 3 12:11:18 2015. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys +import os + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +#sys.path.insert(0, os.path.abspath('.')) + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +#needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.doctest', +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix of source filenames. +source_suffix = '.rst' + +# The encoding of source files. +#source_encoding = 'utf-8-sig' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Koji' +copyright = u'2015, Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = '1.10.0' +# The full version, including alpha/beta/rc tags. +release = '1.10.0' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +#language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +#today = '' +# Else, today_fmt is used as the format for a strftime call. +#today_fmt = '%B %d, %Y' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +exclude_patterns = [] + +# The reST default role (used for this markup: `text`) to use for all +# documents. +#default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +#add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +#add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +#show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +#modindex_common_prefix = [] + +# If true, keep warnings as "system message" paragraphs in the built documents. +#keep_warnings = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +html_theme = 'default' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +#html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +#html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# " v documentation". +#html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +#html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +#html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +#html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# Add any extra paths that contain custom files (such as robots.txt or +# .htaccess) here, relative to this directory. These files are copied +# directly to the root of the documentation. +#html_extra_path = [] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +#html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +#html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +#html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +#html_additional_pages = {} + +# If false, no module index is generated. +#html_domain_indices = True + +# If false, no index is generated. +#html_use_index = True + +# If true, the index is split into individual pages for each letter. +#html_split_index = False + +# If true, links to the reST sources are added to the pages. +#html_show_sourcelink = True + +# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. +#html_show_sphinx = True + +# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. +#html_show_copyright = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +#html_use_opensearch = '' + +# This is the file name suffix for HTML files (e.g. ".xhtml"). +#html_file_suffix = None + +# Output file base name for HTML help builder. +htmlhelp_basename = 'Kojidoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { +# The paper size ('letterpaper' or 'a4paper'). +#'papersize': 'letterpaper', + +# The font size ('10pt', '11pt' or '12pt'). +#'pointsize': '10pt', + +# Additional stuff for the LaTeX preamble. +#'preamble': '', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + ('index', 'Koji.tex', u'Koji Documentation', + u'Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +#latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +#latex_use_parts = False + +# If true, show page references after internal links. +#latex_show_pagerefs = False + +# If true, show URL addresses after external links. +#latex_show_urls = False + +# Documents to append as an appendix to all manuals. +#latex_appendices = [] + +# If false, no module index is generated. +#latex_domain_indices = True + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + ('index', 'koji', u'Koji Documentation', + [u'Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller'], 1) +] + +# If true, show URL addresses after external links. +#man_show_urls = False + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + ('index', 'Koji', u'Koji Documentation', + u'Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McLeod, Ralph Bean, Adam Miller', 'Koji', 'One line description of project.', + 'Miscellaneous'), +] + +# Documents to append as an appendix to all manuals. +#texinfo_appendices = [] + +# If false, no module index is generated. +#texinfo_domain_indices = True + +# How to display URL addresses: 'footnote', 'no', or 'inline'. +#texinfo_show_urls = 'footnote' + +# If true, do not generate a @detailmenu in the "Top" node's menu. +#texinfo_no_detailmenu = False diff --git a/docs/source/defining_hub_policies.rst b/docs/source/defining_hub_policies.rst new file mode 100644 index 00000000..0c35b1f3 --- /dev/null +++ b/docs/source/defining_hub_policies.rst @@ -0,0 +1,256 @@ +===================== +Defining Hub Policies +===================== + +Defining a policy on the hub allows you fine control over certain activities +in the system. At present, policy allows you to control: +* tag/untag/move operations +* allowing builds from srpm +* allowing builds from expired repos +* managing the package list for a tag +* managing which channel a task goes to + +In the future, we expect to add more policy hooks for controlling more aspects +of the system. + +Policy configuration is optional. If you don't define one, then by default: +* tag/untag/move operations are governed by tag locks/permissions +* builds from srpm are only allowed for admins +* builds from expired repos are only allowed for admins +* only admins may modify package lists +* tasks go to the default channel + +Configuration +============= + +The hub policy is configured in the ``hub.conf`` file, which is an ini-style +configuration file. Policies are defined in the section named ``[policy]``. +Each ``name = value`` pair defines the policy of that name. With multiple line +policies, successive lines should be indented so that the parser treats them +as part of the whole. + +Consider the following simple (and strict) example: + +:: + + [policy] + tag = + has_perm admin :: allow + tag *-candidate :: allow + all :: deny + +This policy section defines a single policy (named 'tag'). The policy is a +series of rules, one per line. The rule lines must be indented. Each rule is +a test and an action, separated by a double colon. The valid actions for +current policies are 'allow' and 'deny'. There are many tests available, +though not all of them are applicable for all policies. Each test is specified +by giving the name of the test followed by any arguments the test accepts. + +Each rule in the policy is checked until a match is found. Upon finding a +match, the action is applied. Our example above limits non-admins to tags +ending in -candidate. + +Getting a bit more complicated +The example above is very simple. The policy syntax also supports compound +tests, negated tests, and nested tests. Consider the following example: + +:: + + [policy] + tag = + buildtag *epel* :: { + tag *epel* !! deny + } + tag *-updates :: { + operation move :: { + fromtag *-updates-candidate :: allow + fromtag *-updates-testing :: allow + all :: deny + } + operation tag && hastag *-updates-candidate *-updates-testing :: deny + } + all :: allow + +This policy sets up some rules concerning tags ending in -updates and tags +containing epel, but is otherwise permissive. + +The first nested rule limits builds built from a tag matching ``epel`` to only +such tags. Note the use of !! instead of :: negates the test. + +For tags matching ``*-updates``, a particular work-flow is enforced. Moving is +only allowed if the move is coming from a tag matching ``*-updates-candidate`` +or ``*-updates-testing``. Conversely, a basic tag operation (not a move) is +denied if the build also has such a tag (the policy requires a move instead). + +General format +============== +The general form of a basic policy line is one of the following + +:: + + test [params] [&& test [params] ...] :: action-if-true + test [params] [&& test [params] ...] !! action-if-false + +And for nested rules: + +:: + + test [params] [&& ...] [::|!!] { + test [params] [&& ...] [::|!!] action + test [params] [&& ...] [::|!!] { + ... + } + } + +Note that each closing brace must be on a line by itself. +Using ``!!`` instead of ``::`` negates the entire test. +Tests can only be joined with &&, the syntax does not support ``||``. + +Available policies +================== +The system currently looks for the following policies + +* tag : checked during tag/untag/move operations +* build_from_srpm : checked when a build from srpm (not an SCM reference) is + requested. +* build_from_repo_id : checked when a build from a specified repo id is + requested +* package_list : checked when the package list for a tag is modified +* channel : consulted when a task is created + +These policies are set by assigning a rule set to the given name in the policy +section. + +Note that the use of tag policies does not bypass tag locks or permissions + +Note that an admin can bypass the tag policy by using ``--force``. + +Actions +======= + +Most of the policies are simply allow/deny policies. They have two possible +actions: ``allow`` or ``deny``. + +The channel policy is used to determine the channel for a task. It supports +the following actions: + +``use `` + * use the given channel + +``req`` + * use the requested channel + * generally this means the default, though some calls allow the client to + request a channel + +``parent`` + * use the parent's channel + * only valid for child tasks + * recommend using the ``is_child_task`` test to be sure + +Available tests +=============== +``true`` + * always true. no arguments + +``all`` + * an alias of true + +``false`` + * always false. no arguments + +``none`` + * an alias of false + +``operation`` + * for tag operations, the operation is one of: tag, untag, move. This test + checks its arguments against the name of the operation and returns true if + there is a match. Accepts glob patterns. + * only applicable to the tag policy + +``package`` + * Matches its arguments against the package name. Accepts glob patterns. + +``tag`` + * matches its arguments against the tag name. Accepts glob patterns. + * for move operations, the tag name tested is the destination tag (see + fromtag) + * for untag operations, the tag name is null and this test will always be + false (see fromtag) + * for the build_from_* policies, tests the destination tag for the build + (which will be null is --skip-tag is used) + +``fromtag`` + * matches against the tag name that a build is leaving. Accepts glob + patterns + * for tag operations, the tag name is null and this test will always be + false + * for move operations, the tag name test is the one that the build is + moving from + * for untag operations, tests the tag the build is being removed from + * only applicable to the tag policy + +``hastag`` + * checks the current tags for the build in question against the arguments. + +``buildtag`` + + * checks the build tag name against the arguments + * for the build_from_* policies the build tag is determined by the build + target requested + * for the tag policies, determines the build tag from the build data, + which will by null for imported builds + +``skip_tag`` + * checks to see if the --skip-tag option was used + * only applicable to the build_from_* policies + +``imported`` + * checks to see if the build in question was imported + * takes no arguments + * true if any of the component rpms in the build lacks buildroot data + * only applicable to the tag policy + +``is_build_owner`` + * Check if requesting user owns the build (not the same as package + ownership) + * take no arguments + +``user_in_group`` + * matches the users groups against the arguments + * true if user is in /any/ matching group + +``has_perm`` + * matches the user's permissions against the arguments + * true is user has /any/ matching permission + +``source`` + * test the build source against the arguments + * for the build_from_* policies, this is the source specified for the build + * for the tag policy, this comes from the task corresponding to the build + (and will be null for imported builds) + +``policy`` + * takes a single argument, which is the name of another policy to check + * checks the named policy. true if the resulting action is one of: yes, + true, allow + * additional policies are defined in the [policy] section, just like the + others + +``is_new_package`` + * true if the package being added is new to the system + * intended for use with the package_list policy + +``is_child_task`` + * true if the task is a child task + * for use with the channel policy + +``method`` + * matches the task method name agaist glob pattern(s) + * true if the method name matches any of the patterns + * for use with the channel policy + +``user`` + * checks the username against glob patterns + * true if any pattern matches + * the user matched is the user performing the action + diff --git a/docs/source/external_repo_server_bootstrap.rst b/docs/source/external_repo_server_bootstrap.rst new file mode 100644 index 00000000..488f8ed8 --- /dev/null +++ b/docs/source/external_repo_server_bootstrap.rst @@ -0,0 +1,136 @@ +==================================== +External Repository Server Bootstrap +==================================== + +Bootstrapping a new External Repo Koji build environment +======================================================== + +These are the steps involved in pointing a new Koji server at external +repositories so that it can be used for building. This assumes that the Koji +hub is up, appropriate authentication methods have been configured, the Koji +repo administration daemon (``kojira``) is properly configured and running, +and at least one Koji builder (``kojid``) is properly configured and running. +All koji cli commands assume that the user is a Koji ``admin``. If you need +help with these tasks, see the :doc:`ServerHowTo `. + +* Create a new tag. :: + + $ koji add-tag dist-foo + +* Create a build tag with the desired arches, and the previously created tag + as a parent. :: + + $ koji add-tag --parent dist-foo --arches "i386 x86_64 ppc ppc64" dist-foo-build + +* Add an external repository to your build tag. Koji substitutes $arch for the + arches in your build tag. The ``$`` needs to be escaped for the shell to send + it though without interpreting it. See below for some additional examples of + external repo URLs. :: + + $ koji add-external-repo -t dist-foo-build dist-foo-external-repo http://repo-server.example.com/path/to/repo/for/foo/\$arch/ + +.. note:: + If you are adding multiple external repos, koji assigns a priority to each + repo in FIFO order. This may cause updated packages to not be visible if a + repo with older packages is ranked at a higher priority (lower numeric + value). Use the ``-p`` flag to set specific repo priorities. + +.. note:: + This uses $arch NOT $basearch + +* Create a build target that includes the tags you've already created. :: + + $ koji add-target dist-foo dist-foo-build + +* Create a ''build'' and ''srpm-build'' group associated with your build tag. :: + + $ koji add-group dist-foo-build build + $ koji add-group dist-foo-build srpm-build + +* Populate the ``build`` and ``srpm-build`` group with packages that will be + installed into the minimal buildroot. You can find out what the is in the + current groups for Fedora by running ``koji list-groups dist-f9-build`` + against the Fedora Koji instance. This is probably a good starting point for + your minimal buildroot and srpm creation buildroot. If you are rebuilding + Fedora packages, it is recommended that you add all packages listed in the + Fedora Koji instance. :: + + $ koji add-group-pkg dist-foo-build build ..... + +* Add packages that you intend to build to your tag. :: + + $ koji add-pkg --owner dist-foo ..... + +* Wait for the repo to regenerate, and you should now be able to run a build + successfully. + +Regenerating your repo +====================== + +koji doesn't monitor external repositories for changes. new repositories +will be generated when packages you build land in a tag that populates +the buildroot or you manually regenerate the repository. you should be +sure to regularly regenerate the repositories manually to pick up +updates. + +:: + + $ koji regen-repo dist-foo-build + +Examples of urls to use for external Repositories +================================================= + +all these examples use mirrors.kernel.org please find the closest mirror +to yourself. Note that the Fedora minimal buildroots download ~100Mb +then build dependencies on top. these are downloaded each build you can +save alot of network bandwidth by using a local mirror or running +through a caching proxy. + +NOTE: this uses $arch **NOT** $basearch + +Fedora 10 +--------- + +:: + + http://mirrors.kernel.org/fedora/releases/10/Everything/\$arch/os/ + http://mirrors.kernel.org/fedora/updates/10/\$arch/ + +CentOS 5 and EPEL +----------------- + +:: + + http://mirrors.kernel.org/centos/5/os/\$arch/ + http://mirrors.kernel.org/centos/5/updates/\$arch/ + http://mirrors.kernel.org/fedora-epel/5/\$arch/ + +Example tags and targets +======================== + +In the simplest setup, where you just want to build against what is +available in the external repositories, you may want to go with a simple +layout of *dist-f\ **X**-build* tags inheriting one another, and +*dist-f\ **X**-updates* tags and targets that inherit the +*dist-f\ **X**-build* tag and have external repos attached to them. This +way, a *dist-f\ **Y**-build* or *dist-f\ **Y**-updates* tag will not +automatically inherit the external repos of your *dist-f\ **X*** tags. + +Tags +---- + +:: + + dist-f10-updates - This is where the external repos for f10 release and f10 updates are attached + `- dist-f10-build - This is the f10 build target with the 'build' and 'srpm-build' group inherited from dist-f9-build, + | so that your buildroot gets populated but you do not have to maintain these groups for each + | seperate release. + `- dist-f9-build - etc. + `- dist-f8-build - etc. + +Targets +------- + +Each *dist-f\ **X**-build* tag has a *dist-f\ **X**-updates* child tag, +and each *dist-f\ **X**-updates* tag has a corresponding +*dist-f\ **X**-updates-candidate* build target. diff --git a/docs/source/image_build.rst b/docs/source/image_build.rst new file mode 100644 index 00000000..0debe9f1 --- /dev/null +++ b/docs/source/image_build.rst @@ -0,0 +1,943 @@ +======================= +Building Images in Koji +======================= + +Image Building +============== + +Koji is capable of building many different types of images or appliances. They +broadly fall into a few categories: LiveCDs, Disk Images, and Containers. All +types of images end up with a Name-Version-Release just like an RPM build. +What you need to provide Koji to perform a build varies by category, and the +specifics will be explained below. + +For additional questions and information, ask around in ``#koji`` on FreeNode +IRC and sign up to the right mailing lists as the `Koji project website`_ +dictates. + +Kickstart First +=============== + +No matter which type of image you want to build, you need to feed Koji a +`kickstart`_ file, so we touch on how to create them and how to get them to Koji +first. Once you have a kickstart to try out, look over the kickstart-specific +caveats for each image type in the later sections. + +Kickstart Reference +------------------- + +If you're new to automated installations, you should read the `Anaconda +Kickstart Guide`_ first. + +Getting your Kickstart to Koji +------------------------------ + +For scratch builds, you can get away with just using the ``--kickstart`` +parameter, which accepts a path (relative or absolute) to your kickstart file +on disk. For non-scratch builds though, the kickstart file needs to live in a +remote Source Control Manager (SCM) such as git or Subversion. To access that, +you need to pass the ``--ksurl`` parameter, which accepts a wacky string in +this form: + +:: + + git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7 + +Here we pointed to a repository hosted at +``git://git.fedorahosted.org/git/spin-kickstarts.git``. Note the ``?`` in +there, this delimiter separates the host URL with a directory structure to +look in for the kickstart file. In other words, if you were to clone the +repository, you should expect to see these (sub)directories laid out below the +root of the clone. The important thing to realize is the behavior of +``--kickstart`` changes if ``--ksurl`` is specified. Instead of indicating a +path to a kickstart file, you just indicate a filename. The file itself is not +specified in the fragment of the string between ``?`` and ``#`` that is passed +to ``--ksurl``, Koji still uses the ``--kickstart`` option for that. After the +directory structure comes a ``#``, which indicates the start of the commit ID. +This is the commit Koji will reset the repository to after cloning. If you +created a git repository which had just the spec file template in it, your SCM +URL would have the ``?`` and ``#`` right next to each other. + +If we passed ``--ksurl`` the string above, and gave +``--kickstart "server-ec2.ks"``, then Koji would do the following: + +* Clone the repository locally +* Call ``git reset --hard 68c40eb7`` +* Search in the "fedora22" subdirectory of the clone for a file called + ``server-ec2.ks``, and pass that to Anaconda +* Perform an automated install + +.. _building-livecds: + +Building LiveCDs +================ + +Let's describe building LiveCDs and the mechanics behind it. + +Getting Started +--------------- + +What you need before proceeding: + +* a name for your LiveCD, and a version number +* a Koji build target +* what architectures you want +* a flattened kickstart file +* you may also need yum repositories generated by release engineering if you + require signed packages be installed into the LiveCD +* the livecd permission in Koji + +The Koji command that creates a LiveCD is called ``spin-livecd``. It calls out +to `livecd-tools`_ in a mock chroot to construct the LiveCD. To run a LiveCD +build, use the spin-livecd command like so: + +:: + +$ koji spin-livecd --release 4 fedora-workstation 23 f23-image-build fedora-workstation.ks + +In this example a LiveCD will be created with the N-V-R of +fedora-workstation-23-4. If ``--release`` was not included an incrementing +value would be computed by Koji instead. spin-livecd takes a minimum of 5 +arguments: + +Name + the name of the image, without versioning information. Examples could be + ``fedora`` or ``fedora-workstation``. + +Version + an arbitrary version string. For Fedora images, this usually matches the + release version, such as 21, 22, or 23. + +Build Target + just like RPM builds Koji must be told which target to use. This controls + what tag the image will be tagged into, and what packages are available to + install into the image. + +Architecture + only x86_64 or i386 is supported + +Kickstart File + this is a recipe file that performs drives the construction of the image. + +Use ``--help`` to discover more options to ``spin-livecd``. You can override +the Release field with ``--release``, or ``--repo`` to override what yum repo +is used to provide packages to install to the image. Note that regardless of +what repo you use, it must contain RPMs built by Koji, otherwise you will get +an error. (unless you are building a scratch image) + +Mechanics +--------- + +The way LiveCDs are created in Koji is fairly straightforward. A chroot is +initialized and populated with the packages and their dependencies from the +``livecd-build`` package group. Next, the kickstart file is copied into it if +it was provided from local storage. If not, it is checked out into it from an +SCM. It is then modified to use the repo associated with the build tag for the +target specified in the command unless the ``--repo`` option was given. Both +the original and the modified kickstart files are saved as part of the output +for the task for later review. A `livecd-creator`_ command is executed using the +mock('--chroot', ...) method. + +.. note:: + This process runs as root. This produces the desired image which is + uploaded to /mnt/koji/images//$imageID if it is not a scratch image. + +.. _caveats-for-livecds: + +Caveats for LiveCDs +------------------- + +There are some known caveats with using the spin-livecd command that users +should be aware of. + +%include macros in the kickstart +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A word of caution about kickstart files and the ``%include`` macro. +`livecd-creator`_ is smart enough to search the current directory of the +submitted kickstart file if it has %include macros. If the kickstart specified +to koji is from local storage, only that kickstart file will be copied into +the chroot, and this creates a problem if it has ``%include`` macros, because +the other kickstart files it needs will be inaccessible. This issue is not +present when the kickstart file is retrieved from a remote SCM (such as the +spin-kickstarts git repo), because the entire repository is checked out. +Presumably it will include any other kickstart files the specified one is +including in the same directory. A workaround for the issue would be to use +``ksflatten`` (from pykickstart) on kickstart files with ``%include`` macros +that are going to be submitted to koji from the user's local disk. + +Package Groups in the Kickstart File +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Package Groups in the kickstart file cause a problem if the Koji repos do not +define them, which they most likely don't since Koji's comps.xml is based on +the "groups" set up from the CLI. `livecd-creator`_'s behavior is to ignore +package groups that are not defined in the repo it is using, so this can be +troublesome when creating the image since packages could be left out. There +are a couple possible workarounds: + +* do not use package groups in the kickstart file and just specify a huge list + of packages +* use ``--repo`` and specify a repo that does have a comps.xml that defines + the groups it uses + +Only Include RPMs Built in Koji +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The image building tasks will fail if your image tries to include a package +that was not built in your build system. This is because the package does not +have any origin information stored in Koji's database. The repos defined in the +kickstart will automatically be overridden with the repo of the build tag for +the build target, unless you use the ``--repo`` option. Since only packages +you have built (or include from an external repo) should be there, you should +never have this problem unless you use ``--repo``. + +No Signed or Debuginfo RPMs in Koji's Build Tags +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you need signed RPMs or debuginfo RPMs, you will run into trouble because +Koji does not keep those in its build tag repos. The only work around for this +is to create a repo yourself that includes these RPMs and then use ``--repo``. +This will force the image to take RPMs from that repo. Remember, the task will +fail if Koji detects RPMs were installed that were not built in the build +system. + +%post Section in Kickstart +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +While livecd-tools does support building on SELinux disabled hosts, you can run +into denials when booting if you create and use new files in the ``%post`` +section of your kickstart file. If you do, you should either set the labels +appropriately at the end of the ``%post`` section, or instigate an autorelabel +at boot time. + +Troubleshooting +--------------- + +If your build fails, you will be notified on the command line. In the output +will be a URL to the Koji UI, visit that and click on the red subtask link. +From that page review ``root.log`` and ``livecd.log`` for errors. Often errors +are caused by packages being missing, or malformed kickstart files. The log +files are at the bottom of the page. If you're stuck, contact Release +Engineering. + +Build System Preparation +------------------------ + +This section assumes you have know-how required to install and configure a new +instance of Koji, and that you have already done so. You can learn how to do so +:doc:`here ` if you need to. Please ensure you are using the +latest version of the software and that your database schema is updated as +well. You should also have some familiarity with how `livecd-creator`_ works. +This section only covers preparation for LiveCD builds. + +Follow this procedure step by step to get things prepared they way they need to +be. + +#. ``koji add-host-to-channel livecd`` + add a builder to the livecd channel +#. ``koji grant-permission livecd `` + grant the permission to build an image type to a user. This step is + optional since admins have all permissions. +#. You will need a tag and target to build the images from. The yum repo + generated for the build tag of the target is what Koji will use to populate + the LiveCDs with by default. (the alternative is to use the ``--repo`` + option, more on that later) +#. ``koji add-group livecd-build`` + add the livecd-build group +#. ``koji add-group-pkg livecd-build ...`` + add packages to the livecd-build group. These package lists vary has + packages and dependencies change. As of October, 2015 for Fedora 23 the + needed packages for each image type are: + + * bash, coreutils, fedora-logos, fedora-release, livecd-tools, + policycoreutils, python-dbus, sed, selinux-policy-targeted, + shadow-utils, squashfs-tools, sssd-client, tar, unzip, util-linux, + which, yum + + +Building Disk Images +==================== + +Disk images are files that represent virtual disks. They have a partition table +and filesystems on them, and are available in a variety of formats: qcow2, +vmdk, ova, Hyper-V, raw, "base" container images, and more. + +Getting Started +--------------- + +What you need: + +* a name for your LiveCD, and a version number +* what architectures you want +* a Koji build target +* kickstart file +* installation tree +* you may also need yum repositories generated by Rel-Eng if you require signed + packages be installed into the image + +The Koji command to build a disk image is called ``image-build``. The +``image-build`` command uses `ImageFactory`_ and `Oz`_ to start a VM guest and +perform an automated Anaconda installation. Here is a (lengthy) example for +building a disk image. + +:: + + $ koji image-build --repo 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' --kickstart fedora-server.ks --scratch --distro Fedora-22 --format qcow2 fedora-server-kvm 22 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' x86_64 + +This example builds a scratch qcow2 disk image using packages from an +additional yum repository. Without this option the yum repo to populate the +build root would be used instead. If this was the first image with the N-V of +fedora-server-kvm-22, then the N-V-R would be fedora-server-kvm-22-1, because +Koji uses an incrementing number for the release if you do not provide one. +Like all Koji commands, use ``--help`` to see more options that are available. + +For Docker, Koji only supports Base Images right now using a kickstart file as +described above. In the future it will support layered images, but not before +some Docker requirements are met, and Koji is maintaining a Registry of its +own. This scoping effort is ongoing. + +``image-build`` takes a minimum of 5 positional arguments, and 2 options must +be specified. They are reviewed in the list below, with the positional +arguments first. + +Name + the name of the image, without versioning information. Examples could be + ``fedora-server`` or ``fedora-workstation``. + +Version + an arbitrary version string. For Fedora images, this usually matches the + release version, such as 22 or 23. + +Build Target + just like RPM builds Koji must be told which target to use. This controls + what tag the image will be tagged into, and what packages are available to + install into the image. + +Installation Tree URL + this is a URL to a location you can install an operating system from. It is + the same place you would direct a PXE-booted system to go. In 99% of cases + this location is provided by Release Engineering. It should have an + "isolinux" subdirectory and yum metadata somewhere within. + +Architecture + only x86_64 or i386 is supported, and you can specify both on the command + line. This will cause two subtasks to be run, allowing you to build for + both arches in parallel. If either fail, the whole build will fail. + +Kickstart File + this is a recipe file that performs drives the construction of the image. + Pass in the path to a kickstart file, which must be flattened. + +Kickstart URL + in a non-scratch build, you'll need this too. For more details, see the + Getting your Kickstart to Koji section. + +Distro + a string that indicates what OS is being built. These always follow the + convention of "Fedora-X", where X is the release number. + +Since this command can get very long, a configuration file can drive the task +as well, using the ``--config`` option. It accepts a path to a configuration +file written in the Python ConfigParser format (like a Windows .ini). The +options are all named the same with one caveat, see below. Here's what one +could look like: + +:: + + [image-build] + name = fedora-server-docker + version = 22 + target = f22-candidate + install_tree = http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/ + arches = x86_64 + + format = qcow2,rhevm-ova,vsphere-ova + distro = Fedora-22 + repo = http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/ + disk_size = 20 + + ksversion = DEVEL + kickstart = fedora-22-server-docker.ks + ksurl = git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7 + specfile = git://git.fedorahosted.org/git/spin-kickstarts.git?spec_templates/fedora22#68c40eb7 + +A few notes on the syntax: + +* it allows for comments too, the lines start with a hash (#) +* options on the command line that can be used multiple times can accept + values here as comma-separated strings +* options with a hyphen need to use an underscore instead. ``--disk-size`` for + example would be disk_size in the config file + +OVA Features +------------ + +If you're building OVAs, either for RHEVM or vSphere, you can specify OVA +options with a special section in the configuration file. It looks something +like this: + +:: + + [ova-options] + vsphere_product_version=22 + rhevm_description=Fedora Cloud 22 + vsphere_product_vendor_name=Fedora Project + ovf_memory_mb=6144 + rhevm_default_display_type=1 + vsphere_product_name=Fedora Cloud 22 + ovf_cpu_count=4 + rhevm_os_descriptor=Fedora-22 + +or this: + +:: + + [ova-options] + vsphere_ova_format = vagrant-virtualbox + rhevm_ova_format = vagrant-libvirt + vagrant_sync_directory = /home/vagrant/sync + +The second one is actually the secret sauce for generating an image for use in +Vagrant. At this time, you would need rename the image file extension from .ova +to .box, but otherwise this should work fine. + +Kickstart Preparation +--------------------- + +Kickstarts for the image-build command have some specific requirements which +are covered in this section. + +Required Kickstart Arguments +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Anaconda of course requires many commands to be defined in the kickstart file. +If you're starting from scratch you should review the reference linked above, +or use an existing kickstart file in the spin-kickstarts git repo. It is +critically important that the installation be completely automated, if +Anaconda has to prompt for input for any reason, the build will fail because +you cannot send input to the guest. Some of the kickstart commands are optional +to Anaconda, but are required in Koji for your build to succeed. Here's the +list and the reasons why. + +zerombr + You must tell Anaconda to wipe out the MBR in the virtual block device, if + you don't Anaconda will ask you. + +clearpart --all --initlabel + Anaconda has to be told to wipe out all data on the virtual block device we + install on otherwise it will ask for confirmation to do so. Since it is + blank anyway this is harmless. + +reboot + When the installation completes, the guest is rebooted. `ImageFactory`_ is + specifically looking for this behavior to conclude the installation + completed. Anaconda's default behavior is to wait for a key press to reboot + the system, but this is impossible from outside of Koji. + +locking the root account + You have to lock the root account (rootpw --lock) or create a non-root user + (user), otherwise Anaconda will prompt for one. + +Do not use the url command + The repo commands are overridden by Koji to point to internal Koji repos, + or what you specified on the command line with ``--repo``, it does not + override the url command if you provided it. Anaconda has a behavior where + it will prefer packages from the repositories given with the url command + over those with the repo command, and this is generally not what you want. + If Koji sees an RPM was installed that was not built in the system, it will + fail the build. + +Recommended Kickstart Arguments +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Often you want a ``%post`` section in your kickstart to perform +post-installation configuration steps. Review that section of the reference +and note that you can specify ``--log`` and ``--interpreter``. Both of these +are recommended (but not required) to assist with the development and debugging +process. Here are some other recommendations: + +* You probably want the network to use dhcp, sshd to be started, and port 22 + opened in the firewall to allow access as well. +* If you're building an image that will be shipped with a product, SELinux + should be enabled. +* Images that will be used in cloud deployments like OpenStack or EC2 should + have ``cloud-init`` in the package list. +* It is discouraged to have root passwords in plaintext in your kickstart file. +* If your %post section is written in bash, consider setting -x. +* For images that have multiple partitions, use the ``--asprimary`` option for + the part command that defines the root file system. This will ensure it is + the first partition on the image, which is a requirement in some cloud + environments like EC2. + +Troubleshooting +--------------- + +If your disk image build fails, follow the link in the command line output +that takes you to the task page in the Koji web UI. Click on the failed +createImage subtask in red. On that page review the screenshot.ppm file if it +was provided, or oz.log. Most failures are from Anaconda rejecting a malformed +kickstart file, which will be indicated in the screenshot. Your installation +must be completely automatic, there can be no interactivity at all, otherwise +Anaconda will sit there indefinitely until Koji (actually ImageFactory) kills +the task. + +It is very easy to write a kickstart file with bugs or that results in a system +that does not boot. This section will present a series of questions to ask +yourself and examples to help diagnose where the problem lies. Once you know +that, it should be easier to understand what you can do to inspect further. + +There are 4 steps in the process: + +#. create a guest +#. perform an automated installation in the guest +#. boot the guest and extract the list of installed RPMs +#. upload and archive the disk image of the guest + +Is it a problem with guest creation? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There have been unusual cases where libvirt, ImageFactory, or Oz was +misconfigured and guests could not be started properly. A misconfiguration with +Puppet or whatever Fedora Infrastructure is up to can cause this. So far the +errors have been clear in the task output, look either in the results string +or oz.log. The bad news is that in this case you really can only inform Rel-Eng +about the issue and wait for a resolution. The good news is these cases are +very rare. + +Did the installation fail? +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Anaconda installation can fail for many reasons: missing packages, network +problems, or syntax errors in %post. Tasks will also fail if Anaconda prompts +for input for any reason. If Koji detects a lack of disk activity in the guest +for more than 5 minutes, it will fail the build and tear down the guest. +Looking in oz.log may have the answer: dracut, anaconda, and yum logs are all +printed there. + +These sorts of failures often have a screenshot taken and saved with the task +output called screenshot.ppm. Viewing this will usually tell you what Anaconda +is complaining about if the installer detected an issue or prompted for input. +The string in the results output that says "No disk activity in 300 seconds, +failing." This almost always means Anaconda hit an issue and either gave up or +waited. + +If Anaconda claims it is missing packages, confirm they exist in the repos you +are using with ``--repo``, if you are using that option. If you are not, +confirm the builds you expect are in the tag inheritance for the target you are +running. This is a lot like checking whether an RPM will build against the +right libraries, except we're building an image instead. + +If you get the rare Anaconda dialog box that says something like "An unexpected +error occurred", try using the ``text`` command in kickstart, which will have +Anaconda boot in text mode. Sometimes the Python traceback (or whatever the +error condition is) will be printed there. I have also seen cases where +text-mode yields a black screen, but booting in graphical mode (the default) +does produce a useful dialog box. Issues like this stem from syntax errors in +the kickstart file, or bugs in pykickstart itself. If you think it is a +pykickstart bug, then someone in Rel-Eng needs to update pykickstart on the +builders. + +Did the guest boot? +^^^^^^^^^^^^^^^^^^^ + +Koji waits 5 minutes for a guest to boot in this step. It unfortunately does +not give a lot of insight to why a guest may not boot, so these are a tougher +class of issues to work through. You can usually answer this question by +looking in results string. If you see "Timed out waiting for guest to boot", +then this is your problem. You can also confirm this in oz.log. + +For now, the best way to investigate an issue like this is to drive a guest +installation locally using something like Gnome's Virtual Machine Manager +(VMM). The steps to perform are: + +* Select a Network Install +* For the Operating System Install URL use the same one you gave to Koji. It + will be something like + http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/ +* Set the Kickstart URL to where your kickstart file is. You may need to make + it available over http. +* Bump the memory to 2048M for good measure +* Launch the guest and let it complete installation +* Open a VNC session and watch what happens when the guest attempts to boot. + +If the console is not providing enough information, we have to get more +creative. Anaconda supports starting an SSH daemon while the installation is +happening with the sshpw command in kickstart. Set that and comment out the +reboot command. This will let the installation complete locally and wait for a +keystroke to reboot the guest. At this point you should be able to ssh in and +inspect the environment to figure out what is going on. You should also +consider making use of the ``--log`` option to %post so that output from the +script is saved somewhere. + +Another option would be to scp logs and other files off of the guest as part of +the ``%post`` script. + +Other Guest Misconfigurations +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If the guest boots but you're having problems accessing it I'd suggest +following same procedure as when the guest fails to boot. This could be a +result of firewall misconfigurations or SSH not being available for some +reason. Usually in this case the build is succeeding in Koji, but there's +something still fundamentally broken in the image. If the issue is something +you can investigate while the guest is online (you can log in), then I'd +suggest importing it locally using the libvirt.xml and the disk image provided +in Koji's task output. + +You can also do investigative work in an offline mode by mounting the image +locally or using something like libguestfs to poke around without starting the +guest. The fast, dirty way to do it is by mounting it. This can often pollute +your guest environment. Here's how to do it: + +* Download the image from Koji +* If the image format is not raw, you have to convert it first with qemu-img. + Something like: + :: + + $ qemu-img convert -O raw + +* Now mount it up using loopback devices. (as root) If your image has multiple + partitions in it, you may need to pass in a different mapped loopback device + like ``loop0p2``. Whichever one you think is the root partition or has the + issue you're trying to fix. + :: + + # kpartx -av + # mount -o loop /dev/mapper/loop0p1 /mnt/my_directory + +Hopefully at this point you figure out the issue. To tear down the image you'll +run commands as root like so: + +:: + + # umount /mnt/my_directory + # dmsetup remove loop0p1 + # losetup -d /dev/loop0 + +Again, if you used different loopback devices, substitute those in to the +dmsetup and losetup commands. + +Build System Preparation +------------------------ + +Follow this guide if you're a Koji admin and would like to enable image +building or want to set up some testing before enabling the integration. + +When moving to ImageFactory to do image builds Koji lost the ability to easily +reproduce the build environment for images the way we do for RPMs using Mock. +This section will document how to set up an image builder for Koji. This is a +lengthy task, it will take first-timers about a week to have a useful instance, +and it is painful because it requires a bare metal system be provisioned since +`ImageFactory`_ provisions VMs to build the image. There is a significant +performance penalty for using nested virtualization. + +Follow the steps below to set up your builder. + +.. note:: + You do not have to stand up a complete Koji instance to test the way Koji + builds images. However, if you want to test image builds with an accurate + representation of how Koji does it, or you want to test code changes + related to image builds in Koji, you should follow all the steps below. + +ImageFactory/Oz Preparation +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Provision a system with at least 4G of memory with the current release of + RHEL 6 or later. Sometimes builders lag behind a month or two before taking + in updates, but this will still get you pretty close to where you want to + be. For Fedora, use the latest release. +#. Install the following packages to the builder. + #. oz + #. imagefactory + #. imagefactory-plugins-TinMan + #. imagefactory-plugins-vSphere + #. imagefactory-plugins-ovfcommon + #. imagefactory-plugins-docker + #. imagefactory-plugins + #. imagefactory-plugins-OVA + #. imagefactory-plugins-RHEVM + #. python-psphere => 0.5 + #. VMDKStream => 0.2 + #. pykickstart +#. Edit ``/etc/oz/oz.cfg``, and set the memory value in the ``[libvirt]`` + section to at least 2048. Set ``safe_generation`` under ``[icicle]`` to yes. +#. Run: ``mkdir -p ~root/.psphere/templates``, and then copy the following + code into ``~root/.psphere/config.yaml``. Do not worry about the server, + username, and password credentials; they are not used anywhere. + + :: + + general: + server: 10.16.120.224 + username: Administrator + password: whatever + template_dir: ~/.psphere/templates/ + logging: + destination: ~/.psphere/psphere.log + level: DEBUG # DEBUG, INFO, etc + +Start up the services and ImageFactory/Oz should be ready to go. You should +read more about `how to use Oz`_ and `how to use ImageFactory`_. If you want +to try calling `ImageFactory`_ as if from a Koji Builder (but not set up a +whole Koji instance), you can use the code below to emulate that. If you want +to test the Koji integration with a full Koji instance, proceed to the next +section instead. + +:: + + #!/usr/bin/python -tt + + import logging + import os.path + import random + import sys + + from imgfac.BuildDispatcher import BuildDispatcher + from imgfac.PluginManager import PluginManager + from imgfac.ReservationManager import ReservationManager + plugin_mgr = PluginManager('/etc/imagefactory/plugins.d') + plugin_mgr.load() + from imgfac.ApplicationConfiguration import ApplicationConfiguration + + # logging + handler = logging.StreamHandler(sys.stdout) + tlog = logging.getLogger() + tlog.setLevel(logging.DEBUG) + tlog.addHandler(handler) + + # configuration + ks = open('oztest.ks').read() + workdir = '/tmp/koji/test' + config = { + #Oz specific + 'oz_data_dir': os.path.join(workdir, 'oz_data'), + 'oz_screenshot_dir': os.path.join(workdir, 'oz_screenshots'), + #IF specific + 'imgdir': os.path.join(workdir, 'scratch_images'), + 'tmpdir': os.path.join(workdir, 'oz-tmp'), + 'verbose' : True, + 'timeout': 3600, + 'output': 'log', + 'raw': False, + 'debug': True, + 'image_manager': 'file', + 'plugins': '/etc/imagefactory/plugins.d', + 'tdl_require_root_pw': False, + 'image_manager_args': { + 'storage_path': os.path.join(workdir, 'output_image')}, + } + random.seed() # necessary to ensure a unique mac address + rm = ReservationManager() + rm._listen_port = random.randint(rm.MIN_PORT, rm.MAX_PORT) + ApplicationConfiguration(configuration=config) + params = {'install_script': ks} + template = """ + """ + bd = BuildDispatcher() + + # build the image + base = bd.builder_for_base_image(template, parameters=params) + base.base_thread.join() + tlog.removeHandler(handler) + +This script is run as root with no arguments. It uses ``oztest.ks`` in your +local directory as the kickstart you want to try to use. The URL in the XML +template is where the RPM packages will be installed from, and what the guest +will be booted with. + +Koji Preparation +^^^^^^^^^^^^^^^^ + +#. :doc:`Install Koji ` if you need it. This will easily be the + most time-consuming part of the process; my first time took 3 days to get + it working properly. Follow the guide closely, and go with the SSL + authentication method. SSL is a lot easier to set up locally. You will need + to install every Koji component (except koji-vmd) on the same system. + Proceed to the next step after you've had a successful Kojira repository + generated. +#. At this point, you have a system that should be ready to build images. We + just have to do some Koji configuration so that your instance is pulling + content from Koji. Replace the base tag names with whatever fits your + conventions. + + #. Add tags, tag inheritance, new pkg (with pkg owner), new external repo, and regen the repo + + :: + + koji add-tag fedora22 + koji add-tag jay-fedora22 + koji add-tag jay-fedora22-override --parent jay-1-fedora22 + koji add-tag jay-fedora22-build --arches x86_64 --parent jay-fedora22-override + koji add-tag jay-fedora22-candidate --parent jay-fedora22 + koji add-tag-inheritance --priority 40 jay-fedora22-build fedora22 + koji add-pkg --owner kojiadmin jay-fedora22 fedora-server-ec2 fedora-server-kvm + koji add-external-repo -t fedora::20 fedora22 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' + koji add-target jay-fedora22-candidate jay-fedora22-build + koji regen-repo jay-fedora22-build + #. Grab a kickstart file from an image task in Koji that relates to what you want to test. + #. Finally, kick off a build! + + :: + + koji image-build fedora-server-ec2 22 --distro Fedora-22 jay-fedora22-candidate --kickstart fedora-server-starter-ec2.ks 'http://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' + + +Building Appliances +=================== + +This section is here for the sake of legacy. Unless you are trying to build +ARM images, you should use the image-build command described in the previous +section. + +.. note:: + The spin-appliance command, described herein, is deprecated. + +Getting Started +--------------- + +Here's what you need before proceeding: + +* a name for your Appliance, and a version number +* a Koji build target +* what architectures you want +* a flattened kickstart file +* you may also need yum repositories generated by release engineering if you + require signed packages be installed into the appliance +* the appliance permission in Koji + +The Koji command that creates an Appliance is called ``spin-appliance``. It +calls out to appliance-creator in a mock chroot to construct the Appliance. +To run an Appliance build, use the spin-appliance command like so: + +:: + + $ koji spin-appliance --release 4 fedora-workstation 23 f23-build fedora-workstation.ks + +In this example an Appliance will be created with the N-V-R of +``fedora-workstation-23-4``. If ``--release`` was not included an incrementing +value would be computed by Koji instead. spin-appliance takes a minimum of 5 +arguments: + +Name + the name of the image, without versioning information. Examples could be + ``fedora`` or ``fedora-workstation``. + +Version + an arbitrary version string. For Fedora images, this usually matches the + release version, such as 21, 22, or 23. + +Build Target + just like RPM builds Koji must be told which target to use. This controls + what tag the image will be tagged into, and what packages are available to + install into the image. + +Architecture + only arm, x86_64, or i386 are supported + +Kickstart File + this is a recipe file that performs drives the construction of the image. + +Use ``--help`` to discover more options to spin-appliance. You can override the +Release field with ``--release``, or ``--repo`` to override what yum repo is +used to provide packages to install to the image. Note that regardless of what +repo you use, it must contain RPMs built by Koji, otherwise you will get an +error. (unless you are building a scratch image) + +Mechanics +--------- + +The way Appliances are created in Koji is the same as :ref:`building-livecds`. + +Caveats for Appliances +---------------------- + +Caveats for using ``spin-appliance`` are the same as using ``spin-livecd`` to +:ref:`caveats-for-livecds`. + +Troubleshooting +--------------- + +If your build fails, you will be notified on the command line. In the output +will be a URL to the Koji UI, visit that and click on the red subtask link. +From that page review ``root.log`` and ``appliance.log`` for errors. Often +errors are caused by packages being missing, or malformed kickstart files. The +log files are at the bottom of the page. If you're stuck, contact Release +Engineering. + +Build System Preparation +------------------------ + +This section assumes you have know-how required to install and configure a new +instance of Koji, and that you have already done so. You can learn how to do so +:doc:`here ` if you need to. Please ensure you are using the +latest version of the software and that your database schema is updated as +well. You should also have some familiarity with how `appliance-creator`_ +works. This section only covers preparation for Appliance builds. + +Follow this procedure step by step to get things prepared they way they need +to be. + +#. Add a builder to the appliance channel + :: + + koji add-host-to-channel appliance + +#. Grant the permission to build an appliance to a user. This step is optional + since admins have all permissions. + + :: + + koji grant-permission appliance + +#. You will need a tag and target to build the images from. The yum repo + generated for the build tag of the target is what Koji will use to populate + the Appliances with by default. (the alternative is to use the ``--repo`` + option, more on that later) + +#. Add the appliance-build group + + :: + + koji add-group appliance-build`` + +#. Add packages to the appliance-build group. These package lists vary has + packages and dependencies change. As of October, 2015 for Fedora 24 the + needed packages for appliances: + + * appliance-tools, bash, coreutils, grub, parted, perl, policycoreutils, + selinux-policy, shadow-utils, sssd-client + + :: + + koji add-group-pkg appliance-build ... + +.. _Koji project website: https://fedorahosted.org/koji/wiki +.. _kickstart: + https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst +.. _Anaconda Kickstart Guide: + https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst +.. _livecd-tools: https://github.com/rhinstaller/livecd-tools/ +.. _livecd-creator: https://fedoraproject.org/wiki/FedoraLiveCD +.. _ImageFactory: http://imgfac.org/ +.. _Oz: http://github.com/clalancette/oz +.. _how to use Oz: https://github.com/clalancette/oz/wiki +.. _how to use ImageFactory: http://imgfac.org/documentation/ +.. _appliance-creator: http://fedoraproject.org/wiki/Features/ApplianceTools diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 00000000..a82566c3 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,279 @@ +.. Koji documentation master file, created by + sphinx-quickstart on Tue Nov 3 12:11:18 2015. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +================================ +Welcome to Koji's documentation! +================================ + +Koji +==== + +Koji is the software that builds `RPM packages for the Fedora project`_. It +uses `Mock`_ to create chroot environments to perform builds. To download the +source code, report bugs, join the mailing list etc., see the `Koji project +website`_. + +Contents +======== + +.. toctree:: + :maxdepth: 2 + + HOWTO + image_build + server_howto + defining_hub_policies + external_repo_server_bootstrap + image_build + misc + migrations + runs_here + server_bootstrap + server_howto + using_the_koji_build_system + writing_a_plugin + writing_koji_code + +HowTos +====== + +Setting up and using Koji on Fedora: + +* :doc:`Using the Koji build system ` +* :doc:`Run Your Own Koji Build Server ` +* :doc:`Building Images in Koji ` +* :doc:`Defining hub policies ` + +Koji Architecture +================= + +Terminology +----------- +In Koji it is sometimes necessary to distinguish between a package in general, +a specific build of a package, and the various rpm files created by a build. +When precision is needed, these terms should be interpreted as follows: + +Package + The name of a source rpm. This refers to the package in general and not + any particular build or subpackage. For example: kernel, glibc, etc. + +Build + A particular build of a package. This refers to the entire build: all + arches and subpackages. For example: kernel-2.6.9-34.EL, glibc-2.3.4-2.19. + +RPM + A particular rpm. A specific arch and subpackage of a build. For example: + kernel-2.6.9-34.EL.x86_64, kernel-devel-2.6.9-34.EL.s390, + glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64 + +Koji Components +=============== + +Koji is comprised of several components: + +Koji-Hub +-------- +koji-hub is the center of all Koji operations. It is an XML-RPC server running +under mod_wsgi in Apache. koji-hub is passive in that it only receives XML-RPC +calls and relies upon the build daemons and other components to initiate +communication. koji-hub is the only component that has direct access to the +database and is one of the two components that have write access to the file +system. + +Kojid +----- +kojid is the build daemon that runs on each of the build machines. Its primary +responsibility is polling for incoming build requests and handling them +accordingly. Essentially kojid asks koji-hub for work. Koji also has support +for tasks other than building. Creating install images is one example. kojid +is responsible for handling these tasks as well. kojid uses mock for building. +It also creates a fresh buildroot for every build. kojid is written in Python +and communicates with koji-hub via XML-RPC. + +Koji-Web +-------- +koji-web is a set of scripts that run in mod_wsgi and use the Cheetah +templating engine to provide a web interface to Koji. It acts as a client to +koji-hub providing a visual interface to perform a limited amount of +administration. koji-web exposes a lot of information and also provides a means +for certain operations, such as cancelling builds. + +Koji-client +----------- +koji-client is a CLI written in Python that provides many hooks into Koji. It +allows the user to query much of the data as well as perform actions such as +adding users and initiating build requests. + +Kojira +------ +kojira is a daemon that keeps the build root repodata updated. It is +responsible for removing redundant build roots and cleaning up after a build +request is completed. + +Package Organization +==================== + +Tags and Targets +---------------- + +Koji organizes packages using tags: + +* Tags are tracked in the database but not on disk +* Tags support multiple inheritance +* Each tag has its own list of valid packages (inheritable) +* Package ownership can be set per-tag (inheritable) +* Tag inheritance is more configurable +* When you build you specify a target rather than a tag + +A build target specifies where a package should be built and how it should be +tagged afterwards. This allows target names to remain fixed as tags change +through releases. You can get a full list of build targets with the following +command: + +:: + + $ koji list-targets + +You can see just a single target with the --name option: + +:: + + $ koji list-targets --name dist-fc7 + + Name Buildroot Destination + --------------------------------------------------------------------------------------------- + dist-fc7 dist-fc7-build dist-fc7 + +This tells you a build for target dist-fc7 will use a buildroot with packages +from the tag dist-fc7-build and tag the resulting packages as dist-fc7. + +You can get a list of tags with the following command: + +:: + + $ koji list-tags + +Package lists +------------- + +As mentioned above, each tag has its own list of packages that may be placed in +the tag. To see that list for a tag, use the list-pkgs command: + +:: + + $ koji list-pkgs --tag dist-fc7 + + Package Tag Extra Arches Owner + ----------------------- ----------------------- ---------------- ---------------- + ElectricFence dist-fc6 pmachata + GConf2 dist-fc6 rstrode + lucene dist-fc6 dbhole + lvm2 dist-fc6 lvm-team + ImageMagick dist-fc6 nmurray + m17n-db dist-fc6 majain + m17n-lib dist-fc6 majain + MAKEDEV dist-fc6 clumens + [...] + +The first column is the name of the package, the second tells you which tag the +package entry has been inherited from, and the third tells you the owner of the +package. + +Latest Builds +------------- + +To see the latest builds for a tag, use the latest-pkg command: + +:: + + $ koji latest-pkg --all dist-fc7 + + Build Tag Built by + ---------------------------------------- -------------------- ---------------- + ConsoleKit-0.1.0-5.fc7 dist-fc7 davidz + ElectricFence-2.2.2-20.2.2 dist-fc6 jkeating + GConf2-2.16.0-6.fc7 dist-fc7 mclasen + ImageMagick-6.2.8.0-3.fc6.1 dist-fc6-updates nmurray + MAKEDEV-3.23-1.2 dist-fc6 nalin + MySQL-python-1.2.1_p2-2 dist-fc7 katzj + NetworkManager-0.6.5-0.3.cvs20061025.fc7 dist-fc7 caillon + ORBit2-2.14.6-1.fc7 dist-fc7 mclasen + +The output gives you not only the latest builds, but which tag they have been +inherited from and who built them (note: for builds imported from beehive the +"built by" field may be misleading). + +Documentation +------------- + +We've tried to make Koji self-documenting wherever possible. The command line +tool will print a list of valid commands and each command supports --help. +For example: + +:: + + $ koji help + + Koji commands are: + build Build a package from source + cancel-task Cancel a task + help List available commands + latest-build Print the latest rpms for a tag + latest-pkg Print the latest builds for a tag + [...] + +:: + + $ koji build --help + + usage: koji build [options] tag URL + (Specify the --help global option for a list of other help options) + + options: + -h, --help show this help message and exit + --skip-tag Do not attempt to tag package + --scratch Perform a scratch build + --nowait Don't wait on build + [...] + +You can see administrator-only command help with --admin. Most users will +never use these additional commands, but if you're setting up your own Koji +system, you may find them very useful. + +:: + + $ koji help --admin + Available commands: + add-external-repo Create an external repo and/or add one to a tag + add-group Add a group to a tag + add-group-pkg Add a package to a group's package listing + [...] + +Koji Deployments +================ + +Koji is also known to be used in many places, and we :doc:`track them on this +page `. Feel free to add your entry. There is no additional +obligation to you for doing so. :) + +Koji Contributor Guides +======================= + +If you're interested in submitting patches, writing documentation, or filing +bugs this section is for you. In time this will be the best place to learn +how to get involved. + +* :doc:`Getting Started as a Developer ` + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` + +.. _Mock: http://fedoraproject.org/wiki/Projects/Mock +.. _RPM packages for the Fedora project: http://koji.fedoraproject.org/koji/ +.. _Koji project website: https://fedorahosted.org/koji/wiki diff --git a/docs/source/migrating_to_1.10.rst b/docs/source/migrating_to_1.10.rst new file mode 100644 index 00000000..8002abbe --- /dev/null +++ b/docs/source/migrating_to_1.10.rst @@ -0,0 +1,85 @@ +Migrating to Koji 1.10 +====================== + +.. reStructured Text formatted + +The 1.10 release of Koji includes a few changes that you should consider when +migrating. + +DB Updates +---------- + +The new ``tag_extra`` table tracks extra data for tags. + +There is a new entry in the ``channels`` table and some additions and updates to +the ``archivetypes`` table. + +As in previous releases, we provide a migration script that updates the +database. + +:: + + # psql koji koji DROP TABLE imageinfo_listing; - koji=> DROP TABLE imageinfo; - # rm -rf /mnt/koji/images +:: + + koji=> DROP TABLE imageinfo_listing; + koji=> DROP TABLE imageinfo; + # rm -rf /mnt/koji/images Command line changes -------------------- -For clarity and consistency, all of the +-pkg+ commands have been renamed to -+-build+ commands. +For clarity and consistency, all of the ``-pkg`` commands have been renamed to +``-build`` commands. - latest-pkg -> latest-build - move-pkg -> move-build - tag-pkg -> tag-build - untag-pkg -> untag-build +:: + + latest-pkg -> latest-build + move-pkg -> move-build + tag-pkg -> tag-build + untag-pkg -> untag-build For backwards compatibility, the old commands names are also recognized. -A new command has been added, +remove-pkg+. +A new command has been added, ``remove-pkg``. Several commands have been modified to support images. -The +spin-livecd+ and +spin-appliance+ commands now require additional +The ``spin-livecd`` and ``spin-appliance`` commands now require additional arguments. These arguments specify the name and version to use for the image. @@ -93,20 +103,24 @@ New kojira options The following options are new to kojira: - max_delete_processes - max_repo_tasks_maven +:: + + max_delete_processes + max_repo_tasks_maven Previously, kojira ran as a single process and repo deletions could potentially slow things down (particularly for Maven-enabled repos). Now kojira spawns -a separate process to handle these deletions. The +max_delete_processes+ +a separate process to handle these deletions. The ``max_delete_processes`` determines how many such processes it will launch at one time. When Maven-enabled repos are in use, they can potentially take a very long time to regenerate. If a number of these pile up it can severely slow down -regeneration of non-Maven repos. The +max_repo_tasks_maven+ limits how many +regeneration of non-Maven repos. The ``max_repo_tasks_maven`` limits how many Maven repos kojira will attempt to regenerate at once. Also the following kojira option has been removed: - prune_batch_size +:: + + prune_batch_size diff --git a/docs/source/migrating_to_1.9.rst b/docs/source/migrating_to_1.9.rst new file mode 100644 index 00000000..fbfb27c2 --- /dev/null +++ b/docs/source/migrating_to_1.9.rst @@ -0,0 +1,86 @@ +Migrating to Koji 1.9 +===================== + +.. reStructured Text formatted + +The 1.9 release of Koji includes a few changes that you should consider when +migrating. + +DB Updates +---------- + +ImageFactory support introduced some new archive types. These have been added to +the ``archivetypes`` table. The inaccurate ``vmx`` entry has been removed. + +As in previous releases, we provide a migration script that updates the +database. + +:: + + # psql koji koji [ ...] + +Note: is a single entity, so denote multiple arches within quotes (e.g. +'i386 i586 i686'). + +Manually submitting a task +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Occasionally, you may need to manually submit a task to koji. One likely +usage is to manually trigger a createRepo. To do this, use this command: + +:: + + koji make-task [options] [...] + +The make-task command is a bit of under-documented black magic. Task +parameters are defined in kojihub.py. The easiest way I have found to +figure out the right incantations for make-task is to query the *task* +table in the koji database directly. Find a similar task to the one you +want to create, and look in the request field for the parameters the +task used, and mimic those. + +So, citing the createRepo case above, here is an example: + +:: + + kojiadmin@localhost$ koji make-task --channel=createrepo --priority=15 \ + newRepo dist-foo-build + +Managing your tags +~~~~~~~~~~~~~~~~~~ + +Occasionally an unwanted package or version of a package will be built +by koji. Don't fret. There are two mechanisms to handle rescinding a +package or specific package version. + +- To remove a specific version of a package, you can untag it: + +:: + + koji untag-build [options] [...] + + supports either %name or %name-%version-%release + +- To remove all versions of a package, you can untag it as above or you + can administratively block it from being listed in a tag: + +:: + + koji block-pkg [options] tag package [package2 ...] + +Spec file processing +-------------------- + +Macro processing +~~~~~~~~~~~~~~~~ + +Macros in the spec file are expanded before Requires and BuildRequires +are processed. If there are any custom macros in the spec file, the +package that drops those macros into /etc/rpm must be tagged under your +dist-build tag + +%dist tags +^^^^^^^^^^ + +For packages that incorporate the %dist tags in their filename, they +expect %dist to be defined in /etc/rpm/macros.dist, which was added in +Fedora 7. For building on RHEL5/FC6 and earlier, koji needs the +`http://buildsys.fedoraproject.org/buildgroups/ +buildsys-macros `__ +package tagged under the dist-build tag. diff --git a/docs/source/runs_here.rst b/docs/source/runs_here.rst new file mode 100644 index 00000000..12c4aa2c --- /dev/null +++ b/docs/source/runs_here.rst @@ -0,0 +1,67 @@ +========= +Runs Here +========= + +List of locations that Koji is currently in use +=============================================== + +Please add yourself here if you run a local version of Koji. Please also +describe what you are using it for. + ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| # | Project | Site Name | Used for | ++=========+============================================================================+===================================================================+===========================================================================================================================================================================================================================================================================+ +| 0 | Grid and HPC Operations Center of A.U.Th | Not-Public | Building packages for local fabric needs. Maintenance of RPM repositories. Monitoring, messaging and information system software for `LCG `__. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 1 | pc V Pharmacy | Not-public | Compiling proprietary software across Fedora and RHEL. Both the software and any dependencies that are not available in Fedora and/or RHEL that may also be proprietary. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2 | CERN | \* | CERN no longer runs their own Koji, but does make daily use of the one run by the Scientific Computing Center of A.U.Th. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 3 | `fedora.danny.cz `__ | the hub is not public yet | http://sharkcz.livejournal.com/3988.html | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 4 | `Ergo Project `__ | `Koji Web Interface `__ | Fast-Track repositories for Enterprise Linux and Fedora, FTBFS, `Blog posts on Koji `__ | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 5 | `NetNix Estonia OÜ `__ | Internal | Packaging commercial and in-house software for RHEL/CentOS/Fedora support. Multiple instances installed, locally and customer sites. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 6 | `Kolab Systems AG `__ | Not Public (yet) | Packaging for ISV products | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 7 | `Lebanon Evangelical School for Boys and Girls `__ | http://koji.lesbg.com/koji | Used to provide configuration RPMS for our three schools | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 8 | `Messinet Secure Services `__ | http://messinet.com/koji | Building and packaging to support local infrastructure as well as incorporate software not packaged at Fedora or RPMFusion. Most packaged software is available to all. See `Messinet Secure Services Fedora RPM Repository `__ for details. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 9 | `ACOSS France `__ | Not-public | Building and packaging on top of RHEL and CentOS 6 for newer software version. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 10 | `Virtual Bridges, Inc. `__ | Not-public | Building `VERDE LEAF `__ virtualization platform/distribution. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 11 | Caltech | http://koji.hep.caltech.edu/koji/ | multi-university CERN-based project | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 12 | Spacewalk | http://koji.spacewalkproject.org/koji/ | building Spacewalk for RHEL 5, 6, and Fedora 14, 15 | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 13 | `National Nanotechnology Network Grid `__ | http://koji.ngrid.ru/koji/ | GridNNN middleware | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| \| 14 | TomTom | non-public | used to build rpms that aren't available in the RHEL repos or EPEL | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| \| 15 | Amazon | non-public | used to build RPMS for the Amazon Linux AMI | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| \| 16 | Open Science Grid | http://koji-hub.batlab.org/koji/ | Used to build RPMs for the Open Science Grid software project. Software built here is deployed across 100+ universities and labs across the U.S. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| \| 17 | RussianFedora | http://koji.russianfedora.ru | Building add-on packages for the RFRemix. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| \| 18 | Scientific Linux | Not-public (yet) | Building Scientific Linux. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| \| 19 | Katello | http://koji.katello.org/koji/ | Building packages for Katello project | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| \| 20 | NIWA | non-public | Building packages for climate and forecasting project | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 21 | `Xenith Consulting Limited `__ | Not-Public | Banking, Telecommunications and Embedded Systems | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 22 | Abril Comunicações | non-public | Media / Entertainment Company. Building RPM packages for the Digital products | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 23 | The FireServer Project | http://www.fireserver.com.br | Building packages for FireServer Project | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 24 | Seneca Centre for Development of Open Technology | http://koji.pidora.ca (info at http://pidora.ca) | Pidora (Fedora Remix for Raspberry Pi) and some other projects. | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 25 | OpenVZ/Virtuozzo | `https://openvz.org/Virtuozzo `__ | Building OpenVZ packages | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 26 | Cloud Linux | `http://koji.cloudlinux.com/ `__ | Building Cloud Linux packages | ++---------+----------------------------------------------------------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ diff --git a/docs/source/server_bootstrap.rst b/docs/source/server_bootstrap.rst new file mode 100644 index 00000000..86c35887 --- /dev/null +++ b/docs/source/server_bootstrap.rst @@ -0,0 +1,95 @@ +===================== +Koji Server Bootstrap +===================== + +Bootstrapping a new Koji build environment +========================================== + +These are the steps involved in populating a new Koji server with +packages so that it can be used for building. This assumes that the Koji +hub is up, appropriate authentication methods have been configured, the +Koji repo administration daemon (``kojira``) is properly configured and +running, and at least one Koji builder (``kojid``) is properly +configured and running. All koji cli commands assume that the user is a +Koji *admin*. If you need help with these tasks, see the +`ServerHowTo `__ . + +- Download all source rpms and binary rpms for the arches you're + interested in + +- Import all source rpms + +:: + + $ koji import /path/to/package1.src.rpm /path/to/package2.src.rpm ... + +If the files are on the same volume as /mnt/koji, you can use +``koji import --link``, which hardlinks the files into place, avoiding +the need to upload them to the hub and **very significantly** increasing +import speed. When using ``--link``, you must run as root. It is +**highly** recommended that you use ``--link``. + +- Import all binary rpms using the same method as above + +- Create a new tag + +:: + + $ koji add-tag dist-foo + +- Tag all of the packages you just imported into the tag you just + created + +You can use ``koji list-untagged`` to get a list of all of the packages +you just imported. + +:: + + $ koji list-pkgs --quiet | xargs koji add-pkg --owner dist-foo + $ koji list-untagged | xargs -n 1 koji call tagBuildBypass dist-foo + +We call the *tagBuildBypass* method instead of using ``koji tag-pkg`` +because it doesn't require the builders to process *tagBuild* tasks one +at a time, but does the tagging directly. This will save a significant +amount of time, especially when tagging a large number of packages. + +- Create a build tag with the desired arches, and the previously + created tag as a parent + +:: + + $ koji add-tag --parent dist-foo --arches "i386 x86_64 ppc ppc64" dist-foo-build + +- Create a build target that includes the tags you've already created + +:: + + $ koji add-target dist-foo dist-foo-build + +- Create a *build* group associated with your build tag + +:: + + $ koji add-group dist-foo-build build + +- Populate the *build* group with packages that will be installed into + the minimal buildroot + +You can find out what the current build group for Fedora is by running +``koji -s http://$ARCH.koji.fedoraproject.org/kojihub list-groups f17-build`` +against the Fedora Koji instance for your $ARCH. This is probably a good +starting point for your minimal buildroot. + +:: + + $ koji add-group-pkg dist-foo-build build pkg1 + $ koji add-group-pkg dist-foo-build build pkg2 + +- regenerate the repo + +:: + + $ koji regen-repo dist-foo-build + +- Wait for the repo to regenerate, and you should now be able to run a + build successfully. diff --git a/docs/source/server_howto.rst b/docs/source/server_howto.rst new file mode 100644 index 00000000..026720d3 --- /dev/null +++ b/docs/source/server_howto.rst @@ -0,0 +1,1298 @@ +============= +Server How To +============= + +Setting Up a Koji Build System +============================== + +The Koji components may **live** on separate resources as long as all resources +are able to communicate. This document will cover how to setup each service +individually, however, all services may **live** on the same resource. + +Knowledge Prerequisites +======================= + +* Basic understanding of SSL and authentication via certificates and/or + Kerberos credentials +* Basic knowledge about creating a database in PostgreSQL and importing a schema +* Working with psql +* Basic knowledge about Apache configuration +* Basic knowledge about `dnf`_/`yum`_/`createrepo`_/`mock`_ - else you'll not + be able to debug problems! +* Basic knowledge about using command line +* Basic knowledge about RPM building +* Simple usage of the Koji client +* For an overview of yum, mock, Koji (and all its subcomponents), mash, and how + they all work together, see the excellent slides put together by `Steve + Traylen at CERN `_. + +Package Prerequisites +===================== + +On the server (koji-hub/koji-web) +--------------------------------- +* httpd +* mod_ssl +* postgresql-server +* mod_wsgi + +On the builder (koji-builder) +----------------------------- +* mock +* setarch (for some archs you'll require a patched version) +* rpm-build +* createrepo + +A note on filesystem space +========================== +Koji will consume copious amounts of disk space under the primary KojiDir +directory (as set in the kojihub.conf file - defaults to ``/mnt/koji``). +However, as koji makes use of mock on the backend to actually create build +roots and perform the builds in those build roots, it might come to a surprise +to users that a running koji server will consume large amounts of disk space +under ``/var/lib/mock`` and ``/var/cache/mock`` as well. Users should either +plan the disk and filesystem allocations for this, or plan to modify the +default mock build directory in the kojid.conf file. If you change the +location, ensure that the new directories are owned by the group "mock" and +have 02755 permission. + +Koji Authentication Selection +============================= +Koji primarily supports Kerberos and SSL Certificate authentication. For basic +koji command line access, plain user/pass combinations are possible. However, +kojiweb does **not** support plain user/pass authentication and once either +Kerberos or SSL Certificate authentication is enabled for kojiweb, the plain +user/pass method will stop working entirely. For this reason we encourage +skipping the plain user/pass method altogether and properly configuring either +Kerberos or SSL Certification authentication from the start. + +The decision on how to authenticate users will affect all other actions you +take in setting up koji. For this reason it is a decision best made up front. + +For Kerberos authentication + a working Kerberos environment (the user is assumed to either already have + this or know how to set it up themselves, instructions for it are not + included here) and the Kerberos credentials of the initial admin user will + be necessary to bootstrap the user database. + +For SSL authentication + SSL certificates for the xmlrpc server, for the various koji components, + and one for the admin user will need to be setup (the user need not know + how to create certificate chains already, we include the instructions for + this below). + +Setting up SSL Certificates for authentication +---------------------------------------------- + +Certificate generation +^^^^^^^^^^^^^^^^^^^^^^ +Create the ``/etc/pki/koji`` directory and copy-and-paste the ssl.cnf listed +here, and save it in the new directory. This configuration file is used along +with the ``openssl`` command to generate the SSL certificates for the various +koji components. + +``ssl.cnf`` + +:: + + 

+    HOME                    = .
+    RANDFILE                = .rand
+
+    [ca] 
+    default_ca              = ca_default
+
+    [ca_default] 
+    dir                     = .
+    certs                   = $dir/certs
+    crl_dir                 = $dir/crl
+    database                = $dir/index.txt
+    new_certs_dir           = $dir/newcerts
+    certificate             = $dir/%s_ca_cert.pem
+    private_key             = $dir/private/%s_ca_key.pem
+    serial                  = $dir/serial
+    crl                     = $dir/crl.pem
+    x509_extensions         = usr_cert
+    name_opt                = ca_default
+    cert_opt                = ca_default
+    default_days            = 3650
+    default_crl_days        = 30
+    default_md              = sha256
+    preserve                = no
+    policy                  = policy_match
+
+    [policy_match] 
+    countryName             = match
+    stateOrProvinceName     = match
+    organizationName        = match
+    organizationalUnitName  = optional
+    commonName              = supplied
+    emailAddress            = optional
+
+    [req] 
+    default_bits            = 1024
+    default_keyfile         = privkey.pem
+    distinguished_name      = req_distinguished_name
+    attributes              = req_attributes
+    x509_extensions         = v3_ca # The extentions to add to the self signed cert
+    string_mask             = MASK:0x2002
+
+    [req_distinguished_name] 
+    countryName                     = Country Name (2 letter code)
+    countryName_default             = AT
+    countryName_min                 = 2
+    countryName_max                 = 2
+    stateOrProvinceName             = State or Province Name (full name)
+    stateOrProvinceName_default     = Vienna
+    localityName                    = Locality Name (eg, city)
+    localityName_default            = Vienna
+    0.organizationName              = Organization Name (eg, company)
+    0.organizationName_default      = My company
+    organizationalUnitName          = Organizational Unit Name (eg, section)
+    commonName                      = Common Name (eg, your name or your server\'s hostname)
+    commonName_max                  = 64
+    emailAddress                    = Email Address
+    emailAddress_max                = 64
+
+    [req_attributes] 
+    challengePassword               = A challenge password
+    challengePassword_min           = 4
+    challengePassword_max           = 20
+    unstructuredName                = An optional company name
+
+    [usr_cert] 
+    basicConstraints                = CA:FALSE
+    nsComment                       = "OpenSSL Generated Certificate"
+    subjectKeyIdentifier            = hash
+    authorityKeyIdentifier          = keyid,issuer:always
+
+    [v3_ca] 
+    subjectKeyIdentifier            = hash
+    authorityKeyIdentifier          = keyid:always,issuer:always
+    basicConstraints                = CA:true
+
+Although it is not required, it is recommended that you edit the default values
+in the ``[req_distinguished_name]`` section of the configuration to match the
+information for your own server. This will allow you to accept most of the
+default values when generating certificates later. The other sections can be
+left unedited.
+
+Generate CA
+^^^^^^^^^^^
+
+The CA is the Certificate Authority.  It's the key/cert pair used to sign all
+the other certificate requests.  When configuring the various koji components,
+both the client CA and the server CA will be a copy of the CA generated here.
+The CA certificate will be placed in the ``/etc/pki/koji`` directory and the
+certificates for the other components will be placed in the
+``/etc/pki/koji/certs`` directory. The ``index.txt`` file which is created is
+a database of the certificates generated and can be used to view the
+information for any of the certificates simply by viewing the contents of
+``index.txt``.
+
+::
+
+    cd /etc/pki/koji/
+    mkdir {certs,private,confs}
+    touch index.txt
+    echo 01 > serial
+    openssl genrsa -out private/koji_ca_cert.key 2048
+    openssl req -config ssl.cnf -new -x509 -days 3650 -key private/koji_ca_cert.key \
+    -out koji_ca_cert.crt -extensions v3_ca
+
+The last command above will ask you to confirm a number of items about the
+certificate you are generating. Presumably you already edited the defaults for
+the country, state/province, locale, and organization in the ``ssl.cnf`` file
+and you only needed to hit enter. It's the organizational unit and the common
+name that we will be changing in the various certs we create. For the CA
+itself, these fields don't have a hard requirement. One suggestion for this
+certificate is to use the FQDN of the server.
+
+If you are trying to automate this process via a configuration management
+tool, you can create the cert in one command with a line like this:
+
+::
+
+    openssl req -config ssl.cnf -new -x509 \
+    -subj "/C=US/ST=Oregon/L=Portland/O=IT/CN=koji.example.com" \
+    -days 3650 -key private/koji_ca_cert.key -out koji_ca_cert.crt -extensions v3_ca
+
+Generate the koji component certificates and the admin certificate
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Each koji component needs its own certificate to identify it. Two of the
+certificates (kojihub and kojiweb) are used as server side certificates that
+authenticate the server to the client. For this reason, you want the common
+name on both of those certs to be the fully qualified domain name of the web
+server they are running on so that clients don't complain about the common
+name and the server not being the same. You can set the OU for these two
+certificates to be kojihub and kojiweb for identification purposes.
+
+For the other certificates (kojira, kojid, the initial admin account, and all
+user certificates), the cert is used to authenticate the client to the server.
+The common name for these certs should be set to the login name for that
+specific component. For example the common name for the kojira cert should be
+set to kojira so that it matches the username. The reason for this is that the
+common name of the cert will be matched to the corresponding user name in the
+koji database. If there is not a username in the database which matches the CN
+of the cert the client will not be authenticated and access will be denied.
+
+When you later use ``koji add-host`` to add a build machine into the koji
+database, it creates a user account for that host even though the user account
+doesn't appear in the user list.  The user account created must match the
+common name of the certificate which that component uses to authenticate with
+the server. When creating the kojiweb certificate, you'll want to remember
+exactly what values you enter for each field as you'll have to regurgitate
+those into the /etc/koji-hub/hub.conf file as the ProxyDNs entry.
+
+When you need to create multiple certificates it may be convenient to create a
+loop or a script like the on listed below and run the script to create the
+certificates. You can simply adjust the number of kojibuilders and the name of
+the admin account as you see fit. For much of this guide, the admin account is
+called ``kojiadmin``.
+
+::
+
+    #!/bin/bash
+    # if you change your certificate authority name to something else you will
+    # need to change the caname value to reflect the change.
+    caname=koji
+
+    # user is equal to parameter one or the first argument when you actually
+    # run the script
+    user=$1
+
+    openssl genrsa -out private/${user}.key 2048
+    cat ssl.cnf | sed 's/insert_hostname/'${user}'/'> ssl2.cnf
+    openssl req -config ssl2.cnf -new -nodes -out certs/${user}.csr -key private/${user}.key
+    openssl ca -config ssl2.cnf -keyfile private/${caname}_ca_cert.key -cert ${caname}_ca_cert.crt \
+        -out certs/${user}.crt -outdir certs -infiles certs/${user}.csr
+    cat certs/${user}.crt private/${user}.key > ${user}.pem
+    mv ssl2.cnf confs/${user}-ssl.cnf
+
+Generate a PKCS12 user certificate (for web browser)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+This is only required for user certificates.
+
+::
+
+    openssl pkcs12 -export -inkey private/${user}.key -in certs/${user}.crt \
+        -CAfile ${caname}_ca_cert.crt -out certs/${user}_browser_cert.p12
+
+When generating certs for a user, the user will need the ``${user}.pem``, the
+``${caname}_ca_cert.crt``, and the ``${user}_browser_cert.p12`` files which
+were generated above.  The ${user}.pem file would normally be installed as
+``~/.fedora.cert``, the ``${caname}_ca_cert.crt`` file would be installed as
+both ``~/.fedora-upload-ca.cert`` and ``~/.fedora-server-ca.cert``, and the
+user would import the ``${user}_brower_cert.p12`` into their web browser as a
+personal certificate.
+
+Copy certificates into ~/.koji for kojiadmin
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You're going to want to be able to send admin commands to the kojihub. In order
+to do so, you'll need to use the newly created certificates to authenticate
+with the hub. Create the kojiadmin user then copy the certificates for the koji
+CA and the kojiadmin user to ``~/.koji``:
+
+::
+
+    kojiadmin@localhost$ mkdir ~/.koji
+    kojiadmin@localhost$ cp /etc/pki/koji/kojiadmin.pem ~/.koji/client.crt   # NOTE: It is IMPORTANT you use the PEM and NOT the CRT
+    kojiadmin@localhost$ cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/clientca.crt
+    kojiadmin@localhost$ cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/serverca.crt
+
+.. note::
+    See /etc/koji.conf for the current system wide koji client configuration.
+    Copy /etc/koji.conf to ~/.koji/config if you wish to change the config on a
+    per user basis.
+
+Setting up Kerberos for authentication
+--------------------------------------
+
+The initial configuration of a kerberos service is outside the scope of this
+document, however there are a few specific things required by koji.
+
+DNS
+^^^
+
+The koji builders (kojid) use DNS to find the kerberos servers for any given
+realm.
+
+::
+
+    _kerberos._udp    IN SRV  10 100 88 kerberos.EXAMPLE.COM.
+
+The trailing dot denotes DNS root and is needed if FQDN is used.
+
+
+Principals and Keytabs
+^^^^^^^^^^^^^^^^^^^^^^
+
+It should be noted that in general you will need to use the fully qualified
+domain name of the hosts when generating the keytabs for services.
+
+You will need the following principals extracted to a keytab for a fully
+kerberized configuration, the requirement for a host key for the koji-hub is
+currently hard coded into the koji client.
+
+``host/kojihub@EXAMPLE.COM``
+    Used by the koji-hub server when communicating with the koji client
+
+``HTTP/kojiweb@EXAMPLE.COM``
+    Used by the koji-web server when performing a negotiated Kerberos
+    authentication with a web browser. This is a service principal for
+    Apache's mod_auth_kerb.
+
+``koji/kojiweb@EXAMPLE.COM``
+    Used by the koji-web server during communications with the koji-hub. This
+    is a user principal that will authenticate koji-web to Kerberos as
+    "koji/kojiweb@EXAMPLE.COM". Koji-web will proxy the mod_auth_kerb user
+    information to koji-hub (the ProxyPrincipals koji-hub config
+    option).
+
+``koji/kojira@EXAMPLE.COM``
+    Used by the kojira server during communications with the koji-hub
+
+``compile/builder1@EXAMPLE.COM``
+    Used on builder1 to communicate with the koji-hub
+
+PostgreSQL Server
+=================
+
+Once the authentication scheme has been setup your will need to install and
+configure a PostgreSQL server and prime the database which will be used to hold
+the koji users.
+
+Configuration Files
+-------------------
+
+* ``/var/lib/pgsql/data/pg_hba.conf``
+* ``/var/lib/pgsql/data/postgresql.conf``
+
+Install PostgreSQL
+------------------
+
+In Fedora 22 and Later versions use `dnf`_:
+
+::
+
+    # dnf install postgresql-server
+
+Or `yum`_ in Fedora 21 and earlier versions: 
+
+::
+
+    # yum install postgresql-server
+
+Initialize PostgreSQL DB:
+-------------------------
+
+The following commands will initialize PostgreSQL and will start the database service
+
+::
+
+    root@localhost$ su - postgres -c "PGDATA=/var/lib/pgsql/data initdb"
+    root@localhost$ systemctl enable postgresql
+    root@localhost$ systemctl start postgresql
+
+Setup User Accounts:
+--------------------
+
+The following commands will setup the ``koji`` account and assign it a password
+
+::
+
+    root@localhost$ useradd koji
+    root@localhost$ passwd koji
+
+Setup PostgreSQL and populate schema:
+-------------------------------------
+
+The following commands will:
+
+* create the koji user within PostgreSQL
+* create the koji database within PostgreSQL
+* set a password for the koji user
+* create the koji schema using the provided
+  ``/usr/share/doc/koji*/docs/schema.sql`` file
+
+::
+
+    root@localhost$ su - postgres
+    postgres@localhost$ createuser --no-superuser --no-createrole --no-createdb koji
+    postgres@localhost$ createdb -O koji koji
+    postgres@localhost$ psql -c "alter user koji with encrypted password 'mypassword';"
+    postgres@localhost$ logout
+    root@localhost$ su - koji
+    koji@localhost$ psql koji koji < /usr/share/doc/koji*/docs/schema.sql
+    koji@localhost$ exit
+
+.. note::
+    When issuing the command to import the psql schema into the new database it
+    is important to ensure that the directory path
+    /usr/share/doc/koji*/docs/schema.sql remains intact and is not resolved to
+    a specific version of koji. In test it was discovered that when the path is
+    resolved to a specific version of koji then not all of the tables were
+    created correctly.
+
+.. note::
+    When issuing the command to import the psql schema into the new database it
+    is important to ensure that you are logged in as the koji database owner.
+    This will ensure all objects are owned by the koji database user. Upgrades
+    may be difficult if this was not done correctly.
+
+Authorize Koji-web and Koji-hub resources
+-----------------------------------------
+
+.. note::
+    In this example, Koji-web and Koji-hub are running on localhost.
+
+``/var/lib/pgsql/data/pg_hba.conf``
+    These settings need to be valid and inline with other services
+    configurations. Please note, the first matching auth line is used so this
+    line must be above any other potential matches. Add:
+
+    ::
+
+        #TYPE   DATABASE    USER    CIDR-ADDRESS      METHOD
+        host    koji        koji    127.0.0.1/32      trust
+        host    koji        koji     ::1/128          trust
+
+    It may also be necessary to add an entry for your machine's external IP
+    address:
+
+    ::
+
+        host    koji        koji    $IP_ADDRESS/32    trust
+
+    You can also use UNIX socket access. The DBHost variable must be unset to
+    use this method. Add:
+
+    ::
+
+        local   koji        apache                            trust
+        local   koji        koji                              trust
+
+    .. note::
+        To enforce password based logins to the database, change trust to md5.
+
+    ::
+
+        #TYPE   DATABASE    USER    CIDR-ADDRESS      METHOD
+        host    koji        koji    127.0.0.1/32      md5
+        host    koji        koji     ::1/128          md5
+        host    koji        koji    $IP_ADDRESS/32    md5
+
+    Make auth changes live:
+    You must reload the PostgreSQL configuration for these changes to become active.
+
+    ::
+
+        root@localhost$ systemctl reload postgresql
+
+Bootstrapping the initial koji admin user into the PostgreSQL database
+----------------------------------------------------------------------
+
+The initial admin user must be manually added to the user database using sql
+commands.  Once added and given admin privilege, you may add additional users
+and change privileges of those users via the koji command line tool's
+administrative commands.
+
+However, if you decided to use the simple user/pass method of authentication,
+then any password setting/changing must be done manually via sql commands as
+there is no password manipulation support exposed through the koji tools.
+
+The sql commands you need to use vary by authentication mechanism.
+
+Set User/Password Authentication
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+    root@localhost$ su - koji
+    koji@localhost$ psql
+    koji=> insert into users (name, password, status, usertype) values ('admin-user-name', 'admin-password-in-plain-text', 0, 0);
+
+Kerberos authentication
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The process is very similar to user/pass except you would replace the first
+insert above with this:
+
+::
+
+    root@localhost$ su - koji
+    koji@localhost$ psql
+    koji=> insert into users (name, krb_principal, status, usertype) values ('admin-user-name', 'admin@EXAMPLE.COM', 0, 0);
+
+SSL Certificate authentication
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There is no need for either a password or a Kerberos principal, so this will
+suffice:
+
+::
+
+    root@localhost$ su - koji
+    koji@localhost$ psql
+    koji=> insert into users (name, status, usertype) values ('admin-user-name', 0, 0);
+
+Give yourself admin permissions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following command will give the user admin permissions. In order to do
+this you will need to know the ID of the user.
+
+::
+
+    koji=> insert into user_perms (user_id, perm_id, creator_id) values (, 1, );
+
+.. note::
+    If you do not know the ID of the admin user, you can get the ID by running the query:
+
+::
+
+    koji=> select * from users;
+
+You can't actually log in and perform any actions until kojihub is up and
+running in your web server.  In order to get to that point you still need to
+complete the authentication setup and the kojihub configuration. If you wish
+to access koji via a web browser, you will also need to get kojiweb up and
+running.
+
+Set Database To Listen On All Addresses
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The koji-hub service will attempt to connect to the database server in the
+manner you configure.  If you use the system hostname, then the database will
+need to be avalible on that address.  To configure this please perform the
+following:
+
+#.  Edit /var/lib/pgsql/data/postgresql.conf
+#.  Set listen_address
+
+    ::
+
+        listen_addresses = '*'
+#.  Reload the postgresql service
+
+    ::
+        systemctl restart postgresql
+
+Koji Hub
+========
+
+.. note::
+    Koji 1.7 and greater uses mod_wsgi.  Any mod_python configurations will
+    need to be migrated.
+
+Koji-hub is the center of all Koji operations. It is an XML-RPC server running
+under mod_wsgi in the Apache httpd. koji-hub is passive in that it only
+receives XML-RPC calls and relies upon the build daemons and other components
+to initiate communication. Koji-hub is the only component that has direct
+access to the database and is one of the two components that have write access
+to the file system.
+
+Configuration Files
+-------------------
+
+* ``/etc/koji-hub/hub.conf``
+* ``/etc/httpd/conf/httpd.conf``
+* ``/etc/httpd/conf.d/kojihub.conf``
+* ``/etc/httpd/conf.d/ssl.conf`` (when using ssl auth)
+
+Install koji-hub
+----------------
+
+In Fedora 22 and later versions use `dnf`_: 
+
+::
+
+    # dnf install koji-hub httpd mod_ssl
+
+Or with `yum`_ in Fedora 21 and earlier versions:
+
+::
+
+    # yum install koji-hub httpd mod_ssl
+
+Required Configuration
+----------------------
+
+/etc/httpd/conf/httpd.conf
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The apache web server has two places that it sets maximum requests a server
+will handle before the server restarts. The xmlrpc interface in kojihub is a
+python application, and processes can sometimes grow outrageously large when it
+doesn't reap memory often enough. As a result, it is strongly recommended that
+you set both instances of MaxRequestsPerChild in httpd.conf to something
+reasonable in order to prevent the server from becoming overloaded and crashing
+(at 100 the httpd processes will grow to about 75MB resident set size before
+respawning).
+
+::
+
+    
+    ...
+    MaxRequestsPerChild  100
+    
+    
+    ...
+    MaxRequestsPerChild  100
+    
+
+/etc/httpd/conf.d/kojihub.conf
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The koji-hub package provides this configuration file. You will need to modify
+it based on your authentication type. Instructions are contained within the
+file and should be simple to follow.
+
+/etc/httpd/conf.d/ssl.conf
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If using SSL you will also need to add the needed SSL options for apache. These
+options should point to where the certificates are located on the hub.
+
+::
+
+    SSLCertificateFile /etc/pki/koji/certs/kojihub.crt
+    SSLCertificateKeyFile /etc/pki/koji/certs/kojihub.key
+    SSLCertificateChainFile /etc/pki/koji/koji_ca_cert.crt
+    SSLCACertificateFile /etc/pki/koji/koji_ca_cert.crt
+    SSLVerifyClient require
+    SSLVerifyDepth  10
+
+/etc/koji-hub/hub.conf
+^^^^^^^^^^^^^^^^^^^^^^
+
+This file contains the configuration information for the hub. You will need to
+edit this configuration to point Koji Hub to the database you are using and to
+setup Koji Hub to utilize the authentication scheme you selected in the
+beginning.
+
+::
+
+    DBName = koji
+    DBUser = koji
+    DBPass = mypassword
+    DBHost = db.example.com
+    KojiDir = /mnt/koji
+    LoginCreatesUser = On
+    KojiWebURL = http://kojiweb.example.com/koji
+
+If kojihub is running on the same server as the koji db, then DBHost should be
+set to 127.0.0.1
+
+Authentication Configuration
+----------------------------
+
+/etc/koji-hub/hub.conf
+^^^^^^^^^^^^^^^^^^^^^^
+
+If using Kerberos, these settings need to be valid and inline with other
+services configurations.
+
+::
+
+    AuthPrincipal host/kojihub@EXAMPLE.COM
+    AuthKeytab /etc/koji.keytab
+    ProxyPrincipals koji/kojiweb@EXAMPLE.COM
+    HostPrincipalFormat compile/%s@EXAMPLE.COM
+
+If using SSL auth, these settings need to be valid and inline with other
+services configurations for kojiweb to allow logins.
+
+ProxyDNs should be set to the DN of the kojiweb certificate.  The exact format
+depends on your mod_ssl version.
+
+For mod_ssl < 2.3.11 use:
+
+::
+
+    DNUsernameComponent = CN
+    ProxyDNs = /C=US/ST=Massachusetts/O=Example Org/OU=kojiweb/CN=example/emailAddress=example@example.com
+
+However, for mod_ssl >= 2.3.11 use:
+
+::
+
+    DNUsernameComponent = CN
+    ProxyDNs = CN=example.com,OU=kojiweb,O=Example Org,ST=Massachusetts,C=US
+
+.. note::
+    More details on this format change, including handling of special
+    characters, can be found in the `Apache mod_ssl documentation`_.  See
+    LegacyDNStringFormat there.
+
+Koji filesystem skeleton
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Above in the ``kojihub.conf`` file we set KojiDir to ``/mnt/koji``.  For
+certain reasons, if you change this, you should make a symlink from
+``/mnt/koji`` to the new location (note: this is a bug and should be fixed
+eventually).  However, before other parts of koji will operate properly, we
+need to create a skeleton filesystem structure for koji as well as make the
+file area owned by apache so that the xmlrpc interface can write to it as
+needed.
+
+::
+
+    cd /mnt
+    mkdir koji
+    cd koji
+    mkdir {packages,repos,work,scratch}
+    chown apache.apache *
+
+SELinux Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+If running in Enforcing mode
+    * you will need to allow apache to connect to the postgreSQL server
+    * you will need to allow apache to write some files to disk
+
+Even if you are not currently running in Enforcing mode, it is still
+recommended to configure the SELinux settings so that there are no future
+issues with SELinux if Enforcing mode is enabled later on.
+
+::
+
+    root@localhost$ setsebool -P httpd_can_network_connect_db=1 allow_httpd_anon_write=1
+    root@localhost$ chcon -R -t public_content_rw_t /mnt/koji/*
+
+If you've placed ``/mnt/koji`` on an NFS share you may also need to set
+``httpd_use_nfs``.
+
+Check Your Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+At this point, you can now restart apache and you should have at least minimal
+operation.  The admin user should be able to connect via the command line
+client, add new users, etc.  It's possible at this time to undertake initial
+administrative steps such as adding users and hosts to the koji database.
+
+So we will need a working client to test with.
+
+Koji cli - The standard client
+==============================
+
+The koji cli is the standard client. It can perform most tasks and is essential
+to the successful use of any koji environment.
+
+Ensure that your client is configured to work with your server. The system-wide
+koji client configuration file is ``/etc/koji.conf``, and the user-specific one
+is in ``~/.koji/config``. You may also use the ``-c`` option when using the
+Koji client to specify an alternative configuration file.
+
+If you are using SSL for authentication, you will need to edit the Koji client
+configuration to tell it which URLs to use for the various Koji components and
+where their SSL certificates can be found.
+
+For a simple test, all we need is the ``server`` and authentication sections.
+
+::
+
+    [koji]
+
+    ;url of XMLRPC server
+    server = http://koji-hub.example.com/kojihub
+
+    ;url of web interface
+    weburl = http://koji-web.example.com/koji
+
+    ;url of package download site
+    topurl = http://koji-filesystem.example.com/kojifiles
+
+    ;path to the koji top directory
+    topdir = /mnt/koji
+
+    ; configuration for Kerberos authentication
+
+    ;the service name of the principal being used by the hub
+    ;krbservice = host
+
+    ; configuration for SSL athentication
+
+    ;client certificate
+    cert = ~/.koji/client.crt
+
+    ;certificate of the CA that issued the client certificate
+    ca = ~/.koji/clientca.crt
+
+    ;certificate of the CA that issued the HTTP server certificate
+    serverca = ~/.koji/serverca.crt
+
+The following command will test your login to the hub:
+
+::
+
+    root@localhost$ koji moshimoshi
+
+Koji Web - Interface for the Masses
+===================================
+
+.. note::
+    Koji 1.7 and greater uses mod_wsgi.  Any mod_python configurations will
+    need to be migrated.
+
+Koji-web is a set of scripts that run in mod_wsgi and use the Cheetah
+templating engine to provide an web interface to Koji. koji-web exposes a lot
+of information and also provides a means for certain operations, such as
+cancelling builds.
+
+Configuration Files
+-------------------
+
+* ``/etc/httpd/conf.d/kojiweb.conf``
+* ``/etc/httpd/conf.d/ssl.conf``
+* ``/etc/kojiweb/web.conf``
+
+Install Koji-Web
+----------------
+
+For Fedora 22 and later versions use `dnf`_:
+
+::
+
+    # dnf install koji-web mod_ssl
+
+Or with `yum`_ for Fedora 21 and earlier versions:
+
+::
+
+    # yum install koji-web mod_ssl
+
+Required Configuration
+----------------------
+
+/etc/httpd/conf.d/kojiweb.conf
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The koji-web package provides this configuration file. You will need to modify
+it based on your authentication type. Instructions are contained within the
+file and should be simple to follow.
+
+.. note::
+    RHEL 5's mod_python publisher does not support non-basic auth, so Kerberos
+    authentication does not currently work in EPEL 5's Koji-web. See
+    `BZ #682319 `_.
+
+/etc/httpd/conf.d/ssl.conf
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you are using SSL you will need to add the needed SSL options for apache.
+
+::
+
+    SSLVerifyClient require
+    SSLVerifyDepth  10
+
+/etc/kojiweb/web.conf
+^^^^^^^^^^^^^^^^^^^^^
+
+You will need to edit the kojiweb configuration file to tell kojiweb which URLs
+it should use to access the hub, the koji packages and its own web interface.
+You will also need to tell kojiweb where it can find the SSL certificates for
+each of these components. If you are using SSL authentication, the "WebCert"
+line below must contain both the public **and** private key. You will also want
+to change the last line in the example below to a unique password.
+
+::
+
+    [web]
+    SiteName = koji
+    # KojiTheme = 
+
+    # Necessary urls
+    KojiHubURL = https://koji-hub.example.com/kojihub
+    KojiFilesURL = http://koji-filesystem.example.com/kojifiles
+
+    ## Kerberos authentication options
+    ; WebPrincipal = koji/web@EXAMPLE.COM
+    ; WebKeytab = /etc/httpd.keytab
+    ; WebCCache = /var/tmp/kojiweb.ccache
+
+    ## SSL authentication options
+    ; WebCert = /etc/pki/koji/koji-web.pem
+    ; ClientCA = /etc/pki/koji/ca_cert.crt
+    ; KojiHubCA = /etc/pki/koji/ca_cert.crt
+
+    LoginTimeout = 72
+
+    # This must be set before deployment
+    #Secret = CHANGE_ME
+
+    LibPath = /usr/share/koji-web/lib
+
+Filesystem Configuration
+------------------------
+
+You'll need to make ``/mnt/koji/`` web-accessible, either here, on the hub, or
+on another web server altogether.
+
+This URL will go into various clients such as:
+* ``/etc/kojiweb/web.conf`` as KojiFilesURL
+* ``/etc/kojid/kojid.conf`` as topurl
+* ``/etc/koji.conf`` as topurl
+
+::
+
+    Alias /kojifiles/ /mnt/koji/
+    
+        Options Indexes
+        AllowOverride None
+        # Apache < 2.4
+        #   Order allow,deny
+        #   Allow from all
+        # Apache >= 2.4
+        Require all granted
+    
+
+Wherever you configure this, please go back and set it correctly in
+``/etc/kojiweb/web.conf`` now.
+
+Web interface now operational
+-----------------------------
+
+At this point you should be able to point your web browser at the kojiweb URL
+and be presented with the koji interface.  Many operations should work in read
+only mode at this point, and any configured users should be able to log in.
+
+Koji Daemon - Builder
+=====================
+
+Kojid is the build daemon that runs on each of the build machines. Its primary
+responsibility is polling for incoming build requests and handling them
+accordingly. Koji also has support for tasks other than building such as
+creating livecd images or raw disk images, and kojid is responsible for
+handling these tasks as well. The kojid service uses mock for creating pristine
+build environments and creates a fresh one for every build, ensuring that
+artifacts of build processes cannot contaminate each other. All of kojid is
+written in Python and communicates with koji-hub via XML-RPC.
+
+Configuration Files
+-------------------
+
+* ``/etc/kojid/kojid.conf`` - Koji Daemon Configuration
+* ``/etc/sysconfig/kojid`` - Koji Daemon Switches
+
+Install kojid
+-------------
+
+For Fedora 22 and later versions use `dnf`_:
+
+::
+
+    # dnf install koji-builder
+
+Or with `yum`_ in Fedora 21 and earlier versions:
+
+::
+
+    # yum install koji-builder
+
+Required Configuration
+----------------------
+
+Add the host entry for the koji builder to the database
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You will now need to add the koji builder to the database so that they can be
+utilized by kojij hub. Make sure you do this before you start kojid for the
+first time, or you'll need to manually remove entries from the sessions and
+users table before it can be run successfully.
+
+::
+
+    kojiadmin@localhost$ koji add-host kojibuilder1.example.com i386 x86_64
+
+The first argument used after the ``add-host`` command should the hostname of
+the builder. The second argument is used to specify the architecture which the
+builder uses.
+
+
+/etc/kojid/kojid.conf
+^^^^^^^^^^^^^^^^^^^^^
+
+The configuration file for each koji builder must be edited so that the line
+below points to the URL for the koji hub. The user tag must also be edited to
+point to the username used to add the koji builder.
+
+::
+
+    ; The URL for the xmlrpc server
+    server=http://hub.example.com/kojihub
+
+    ; the username has to be the same as what you used with add-host
+    ; in this example follow as below
+    user = kojibuilder1.example.com
+
+The koji filesystem may also be needed over http.  Set this as it was
+configured about.
+
+::
+
+    # The URL for the file access
+    topurl=http://koji-filesystem.example.com/kojifiles
+
+This item may be changed, but may not be the same as KojiDir on the
+``kojihub.conf`` file (although it can be something under KojiDir, just not
+the same as KojiDir)
+
+::
+
+    ; The directory root for temporary storage
+    workdir=/tmp/koji
+
+The root of the koji build directory (ie, ``/mnt/koji``) must be mounted on the
+builder. A Read-Only NFS mount is the easiest way to handle this.
+
+::
+
+    # The directory root where work data can be found from the koji hub
+    topdir=/mnt/koji
+
+Authentication Configuration (SSL certificates)
+-----------------------------------------------
+
+/etc/kojid/kojid.conf
+^^^^^^^^^^^^^^^^^^^^^
+
+If you are using SSL, these settings need to be edited to point to the
+certificates you generated at the beginning of the setup process.
+
+::
+
+    ;client certificate
+    ; This should reference the builder certificate we created on the kojihub CA, for kojibuilder1.example.com
+    ; ALSO NOTE: This is the PEM file, NOT the crt
+    cert = /etc/kojid/kojid.pem
+
+    ;certificate of the CA that issued the client certificate
+    ca = /etc/kojid/koji_ca_cert.crt
+
+    ;certificate of the CA that issued the HTTP server certificate
+    serverca = /etc/kojid/koji_ca_cert.crt
+
+It is important to note that if your builders are hosted on seperate machines
+from koji hub and koji web, you will need to scp the certificates mentioned in
+the above configuration file from the ``/etc/kojid/`` directory on koji hub to
+the ``/etc/koji/`` directory on the local machine so that the builder can be
+authenticated.
+
+Authentication Configuration (Kerberos)
+---------------------------------------
+
+/etc/kojid/kojid.conf
+^^^^^^^^^^^^^^^^^^^^^
+
+If using Kerberos, these settings need to be valid and inline with other
+services configurations.
+
+::
+
+    ; the username has to be the same as what you used with add-host
+    ;user =
+
+    host_principal_format=compile/%s@EXAMPLE.COM
+
+By default it will look for the Kerberos keytab in ``/etc/kojid/kojid.keytab``
+
+.. note::
+    Kojid will not attempt kerberos authentication to the koji-hub unless the
+    username field is commented out
+
+Optional Configuration SourceCodeManagement
+-------------------------------------------
+
+/etc/kojid/kojid.conf
+^^^^^^^^^^^^^^^^^^^^^
+
+The pattern is as follows:
+    hostname:/path/match:checkout /common?, hostname2:/path/match:checkout /common?
+
+``checkout /common? is 'true' unless set to false``
+
+::
+
+    allowed_scms=scm-server.example.com:/repo/base/repos*/:false
+
+Once the code is checked out kojid will run the following:
+
+::
+
+    make sources
+    mv *.spec /SPECS
+    mv * /SOURCES
+    rpmbuild -bs /SPECS/*.spec
+
+Add the host to the createrepo channel
+--------------------------------------
+
+Channels are a way to control which builders process which tasks.  By default
+hosts are added to the ''default'' channel.  At least some build hosts also
+needs to be added to the ''createrepo'' channel so there will be someone to
+process repo creation tasks initiated by kojira.
+
+::
+
+    kojiadmin@localhost$ koji add-host-to-channel kojibuilder1.example.com createrepo
+
+A note on capacity
+------------------
+
+The default capacity of a host added to the host database is 2. This means that
+once the load average on that machine exceeds 2, kojid will not accept any
+additional tasks. This is separate from the maxjobs item in the configuration
+file. Before kojid will accept a job, it must pass both the test to ensure the
+load average is below capacity and that the current number of jobs it is
+already processing is less than maxjobs. However, in today's modern age of quad
+core and higher CPUs, a load average of 2 is generally insufficient to fully
+utilize hardware.
+
+::
+
+    koji edit-host --capacity=16 kojibuilder1.example.com
+
+The koji-web interface also offers the ability to edit this value to admin
+accounts.
+
+Start Kojid
+-----------
+
+Once the builder has been added to the database you must start kojid
+
+::
+
+    root@localhost$ service kojid start
+
+Check ``/var/log/kojid.log`` to verify that kojid has started successfully. If
+the log does not show any errors then the koji builder should be up and ready.
+You can check this by pointing your web browser to the web interface and
+clicking on the hosts tab. This will show you a list of builders in the
+database and the status of each builder.
+
+Kojira - Dnd|Yum repository creation and maintenance
+====================================================
+
+Configuration Files
+-------------------
+
+* ``/etc/kojira/kojira.conf`` - Kojira Daemon Configuration
+* ``/etc/sysconfig/kojira`` - Kojira Daemon Switches
+
+Install kojira
+---------------
+For Fedora 22 and later versions use `dnf`_:
+
+::
+
+    # dnf install koji-utils
+
+Or with `yum`_ in Fedora 21 and earlier versions:
+
+::
+
+    # yum install koji-utils
+
+Required Configuration
+----------------------
+
+Add the user entry for the kojira user
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The kojira user requires the ``repo`` permission to function.
+
+::
+
+    kojiadmin@localhost$ koji add-user kojira
+    kojiadmin@localhost$ koji grant-permission repo kojira
+
+``/etc/kojira/kojira.conf``
+    This needs to point at your koji-hub.
+
+    ::
+
+        ; The URL for the xmlrpc server
+        server=http://koji-hub.example.com/kojihub
+
+
+Additional Notes
+^^^^^^^^^^^^^^^^
+* Kojira needs read-write access to ``/mnt/koji``.
+* There should only be one instance of kojira running at any given time.
+* It is not recommended that kojira run on the builders, as builders only
+  should require read-only access to ``/mnt/koji``.
+* Kojira may need to be restarted when new tags are added in order to detect
+  those tags correctly.
+
+Authentication Configuration
+----------------------------
+
+/etc/kojira/kojira.conf
+^^^^^^^^^^^^^^^^^^^^^^^
+
+**If using SSL,** these settings need to be valid.
+
+::
+
+    ;client certificate
+    ; This should reference the kojira certificate we created above
+    cert = /etc/pki/koji/kojira.pem
+
+    ;certificate of the CA that issued the client certificate
+    ca = /etc/pki/koji/koji_ca_cert.crt
+
+    ;certificate of the CA that issued the HTTP server certificate
+    serverca = /etc/pki/koji/koji_ca_cert.crt
+
+**If using Kerberos,** these settings need to be valid.
+
+::
+
+    ; For Kerberos authentication
+    ; the principal to connect with
+    principal=koji/kojira@EXAMPLE.COM
+    ; The location of the keytab for the principal above
+    keytab=/etc/kojira.keytab
+
+``/etc/sysconfig/kojira``
+    The local user kojira runs as needs to be able to read and write to
+    ``/mnt/koji/repos/``. If the volume that directory resides on is
+    root-squashed or otherwise unmodifiable by root, you can set ``RUNAS=`` to
+    a user that has the required privileges.
+
+Start Kojira
+------------
+
+::
+
+    root@localhost$ service kojira start
+
+Check ``/var/log/kojira/kojira.log`` to verify that kojira has started
+successfully.
+
+Bootstrapping the Koji build environment
+========================================
+
+For instructions on importing packages and preparing Koji to run builds, see
+:doc:`Server Bootstrap `.
+
+For instructions on using External Repos and preparing Koji to run builds, see
+:doc:`External Repo Server Bootstrap `.
+
+Useful scripts and config files for setting up a Koji instance are available
+`here `_.
+
+Minutia and Miscellany
+======================
+Please see :doc:`KojiMisc ` for additional details and notes about
+operating a koji server.
+
+.. _dnf: https://fedoraproject.org/wiki/Dnf
+.. _yum: https://fedoraproject.org/wiki/Yum
+.. _createrepo: http://createrepo.baseurl.org/
+.. _mock: https://fedoraproject.org/wiki/Mock
+.. _Apache mod_ssl documentation:
+    https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#ssloptions 
diff --git a/docs/source/using_the_koji_build_system.rst b/docs/source/using_the_koji_build_system.rst
new file mode 100644
index 00000000..ebf7aa22
--- /dev/null
+++ b/docs/source/using_the_koji_build_system.rst
@@ -0,0 +1,544 @@
+===========================
+Using the koji build system
+===========================
+
+
+Using Koji in Fedora
+====================
+
+The `Koji Build System `__ is Fedora's RPM buildsystem. Packagers
+use the koji client to request package builds and get information about
+the buildsystem. Koji runs on top of Mock to build RPM packages for
+specific architectures and ensure that they build correctly.
+
+There is also a `simplified Chinese
+edition `__.
+
+Installing Koji
+---------------
+
+Installing the Koji CLI
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Everything you need to use Koji (and be a Fedora contributor) can be
+installed in a single step:
+
+::
+
+    dnf install fedora-packager
+
+fedora-packager provides useful scripts to help maintain and setup your
+koji environment. Additionally, it includes dependencies on the Koji
+CLI, so it will be installed when you install ``fedora-packager``. The
+command is called ``koji`` and is included in the main koji package. By
+default the koji tool authenticates to the central server using
+Kerberos. However SSL and username/password authentications are
+available. You will need to have a valid authentication token to use
+many features. However, many of the read-only commands will work without
+authentication.
+
+Fedora Account System (FAS2) Setup
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to interface with the koji server, maintainers will need to run
+
+::
+
+    /usr/bin/fedora-packager-setup
+
+Each user on a system will need to run fedora-packager-setup if they
+wish to use Koji to build Fedora packages. Each user has their own
+certificates that authenticate them.
+
+.. raw:: mediawiki
+
+   {{admon/tip|Plague users rejoice!|For existing users of plague (the old build system that preceded Koji), fedora-packager-setup will use your existing certificates.  If you did not have plague before, it will get the server CA certs and tell you where to get your user cert.}}
+
+Fedora Certificates
+'''''''''''''''''''
+
+Koji uses three certificates:
+
+``~/.fedora.cert`` (specific to the Fedora Maintainer) : This cert is
+generated from running ``fedora-cert -n``. It should have been generated
+when you became maintainer. You may need to refresh it when it expires
+by running ``fedora-cert -n`` again. You can check if it has expired
+with ``fedora-cert -v``.
+
+the following are downloaded automatically by fedora-packager-setup and
+dont need to be manually setup
+
+``~/.fedora-upload-ca.cert`` (The certificate for the Certificate
+Authority used to sign the user keys.) : It can be manually downloaded
+from
+`here `__
+or ``fedora-packager-setup or fedora-cert -n`` should fetch it. using
+the CLI is prefered.
+``~/.fedora-server-ca.cert`` (The certificate for the Certificate
+Authority used to sign the build system's server keys.) : It can be
+downloaded manually from
+`here `__
+or ``fedora-packager-setup`` should fetch it. This certificate may also
+be needed to let `https koji `__ URLs
+resolve without untrusted-CA warnings.
+
+.. raw:: mediawiki
+
+   {{admon/warning|RHEL6 cert bug|If you're using RHEL6, an incompatibility between RHEL6's openssl and nss causes certificates downloaded from fas to fail to work with some fedpkg tools.  [[https://bugzilla.redhat.com/show_bug.cgi?id=631000 Bug 631000 rhel6 openssl creates PKCS#8 encoded PEM RSA private key files, nss can't read them]].  The cert can be made compatible using this command: (openssl x509 -in ~/.fedora.cert -text; echo; openssl rsa -in ~/.fedora.cert) > fedora.cert.new}}
+
+Koji Config
+^^^^^^^^^^^
+
+The global local client configuration file for koji is
+``/etc/koji.conf``. You should not need to change this from the defaults
+for building Fedora packages.These will allow you to use the primary
+build system as well as secondary arch build systems.
+
+The web interface
+-----------------
+
+.. raw:: mediawiki
+
+   {{admon/tip|Optional|The web interface is optional.  You may skip to the
+   next section if you like.}}
+
+The primary interface for viewing Koji data is a web application. It is
+available at http://koji.fedoraproject.org/koji/ . Most of the interface
+is read-only, but with sufficient privileges, you can log in and perform
+some additional actions. For example:
+
+-  Cancel a build
+-  Resubmit a failed task
+-  Setup a notification
+
+Those with admin privileges will find additional actions, such as:
+
+-  Create/Edit/Delete a tag
+-  Create/Edit/Delete a target
+-  Enable/Disable a build host
+
+The web site utilizes SSL authentication. In order to log in you will
+need a valid SSL certificate and your web browser will need to be
+configured to trust the SSL cert. Instructions on how to do this are
+printed when running ``fedora-packager-setup --with-browser-cert``.
+
+.. raw:: mediawiki
+
+   {{admon/warning|Using the certificate directly downloaded from the FAS web
+   interface|If you have generated and downloaded the certificate
+   ~/.fedora.cert directly from FAS using the form referenced
+   above, you need to convert it into a format that the browser can understand
+   using the following command:
+   openssl pkcs12 -export -in ~/.fedora.cert -CAfile ~/.fedora-upload-ca.cert -out ~/fedora-browser-cert.p12,
+   where .fedora-upload-ca.cert can be downloaded from the URL
+   referenced above.}}
+
+Installing SSL Certificates in Firefox
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. raw:: mediawiki
+
+   {{admon/note|Optional|You only need to check these instructions if you are intending to authenticate with the web interface with Firefox.  Authenticating with the web interface is optional.}}
+
+Once you have created your FAS account, generated your certificate in
+the form posted in the link above and ran
+``fedora-packager-setup --with-browser-cert``, you will need to import
+it into your web browser. You can do this in Firefox by doing the
+following:
+
+1. Launch Firefox and click on the **Edit** menu from the toolbar
+
+2. Select **Preferences** in the sub-menu which appears.
+
+3. This should open the **Preferences** window where you can switch to
+the **Advanced** section
+
+4. In the **Advanced** section switch to the **Encryption** tab
+
+5. Click on the **View Certificates** button and the Certificates window
+will appear
+
+6. Switch to the **Your Certificates** tab and click on the **Import**
+button
+
+7. Point to where your Fedora Certificate is located and click **Open**
+(fedora-packager-setup will have told you where it was saved and will
+have asked you to set a password for the cert)
+
+You should now be able to see your Fedora Certificate listed under
+**Your Certificates** and you should be able to authenticate with the
+koji web interface.
+
+Installing SSL Certificates in Chromium
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. raw:: mediawiki
+
+   {{admon/note|Optional|You only need to check these instructions if you are intending to authenticate with the web interface with Chromium.  Authenticating with the web interface is optional.}}
+
+Chromium uses the NSS Shared DB, you will need the nss-tools package
+installed.
+
+::
+
+    pk12util -d sql:$HOME/.pki/nssdb -i fedora-browser-cert.p12
+
+Notifications
+^^^^^^^^^^^^^
+
+When authenticated with the Koji web interface, you can setup a
+notification requests to make sure you do not miss when a package you
+care about gets built. Login and scroll to the bottom of the page, there
+you should find a *`Add a
+notification `__*
+link and a list of your configured notifications.
+
+Building with fedpkg targets
+----------------------------
+
+Every push is automatically tagged via git. All you have done to build
+the package is to run,
+
+::
+
+    fedpkg build
+
+This will trigger a build request for the branch. Easy!
+
+It is also possible to target a specific koji tag as follows:
+
+::
+
+    fedpkg build --target TARGET
+
+for example, if building on rawhide against a special tag created by
+rel-eng for updating API for many packages, e.g. ``dist-f14-python`` you
+would use the following:
+
+::
+
+    fedpkg build --target 'dist-f14-python'
+
+Chained builds
+^^^^^^^^^^^^^^
+
+.. raw:: mediawiki
+
+   {{Admon/warning | chain-builds only work when building on the devel/ branch (aka rawhide).  To chain-build packages to update a released OS version, [https://fedoraproject.org/wiki/Bodhi/BuildRootOverrides set up an override using bodhi] requesting packages to be included in the proper buildroot.}}
+
+Sometimes you want to make sure than one build succeeded before
+launching the next one, for example when you want to rebuild a package
+against a just rebuilt dependency. In that case you can use a chain
+build with:
+
+``fedpkg chain-build libwidget libgizmo``
+
+The current package is added to the end of the CHAIN list. Colons (:)
+can be used in the CHAIN parameter to define groups of packages.
+Packages in any single group will be built in parallel and all packages
+in a group must build successfully and populate the repository before
+the next group will begin building. For example:
+
+``fedpkg chain-build libwidget libaselib : libgizmo :``
+
+will cause libwidget and libaselib to be built in parallel, followed by
+libgizmo and then the currect directory package. If no groups are
+defined, packages will be built sequentially.
+
+If a build fail, following builds are cancelled but the builds that
+already succeeded are pushed to the repository.
+
+Scratch Builds
+--------------
+
+Sometimes it is useful to be able to build a package against the
+buildroot but without actually including it in the release. This is
+called a scratch build. The following section covers using koji directly
+as well as the fedpkg tool to do scratch builds. To create a scratch
+build from changes you haven't committed, do the following:
+
+::
+
+    rpmbuild -bs foo.spec
+    koji build --scratch rawhide foo.srpm
+
+From the latest git commit:
+
+::
+
+    koji build --scratch rawhide 'git url'
+
+Warning: Scratch builds will *not* work correctly if your .spec file
+does something different depending on the value of %fedora, %fc9, and so
+on. Macro values like these are set by the *builder*, not by koji, so
+the value of %fedora will be for whatever created the source RPM, and
+*not* what it's being built on. Non-scratch builds get around this by
+first re-building the source RPM.
+
+If you have committed the changes to git and you are in the current
+branch, you can do a scratch build with fedpkg tool which wraps the koji
+command line tool with the appropriate options:
+
+::
+
+    fedpkg scratch-build
+
+if you want to do a scratch build for a specific architecture, you can
+type:
+
+::
+
+    fedpkg scratch-build-
+
+ can be a comma separated list of several architectures.
+
+finally is possible to combine the scratch-build command with a specific
+koji tag in the form:
+
+::
+
+    fedpkg scratch-build --target TARGET
+
+fedpkg scratch-build --help or koji build --help for more information.
+
+Build Failures
+--------------
+
+If your package fails to build, you will see something like this:
+
+::
+
+    420066 buildArch kernel-2.6.18-1.2739.10.9.el5.jjf.215394.2.src.rpm,
+    ia64): open (build-1.example.com) -> FAILED: BuildrootError:
+    error building package (arch ia64), mock exited with status 10
+
+You can figure out why the build failed by looking at the log files. If
+there is a build.log, start there. Otherwise, look at init.log.
+
+Logs can be found via the web interface in the Task pages for the failed
+task. Alternatively the koji client can be used to view the logs via the
+``watch-logs`` command. See the help output for more details.
+
+Advanced use of Koji
+--------------------
+
+We've tried to make Koji self-documenting wherever possible. The command
+line tool will print a list of valid commands and each command supports
+--help. For example:
+
+::
+
+    $ koji help
+
+    Koji commands are:
+    build                Build a package from source
+    cancel-task          Cancel a task
+    help                 List available commands
+    latest-build         Print the latest rpms for a tag
+    latest-pkg           Print the latest builds for a tag
+    [...] 
+
+::
+
+    $ koji build --help
+
+    usage: koji build [options]  tag URL
+    (Specify the --help global option for a list of other help options)
+
+    options:
+    -h, --help            show this help message and exit
+    --skip-tag            Do not attempt to tag package
+    --scratch             Perform a scratch build
+    --nowait              Don't wait on build
+    [...] 
+
+Using koji to generate a mock config to replicate a buildroot
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+koji can be used to replicate a build root for local debugging
+
+::
+
+    koji mock-config --help
+    Usage: koji mock-config [options] name
+    (Specify the --help global option for a list of other help options)
+
+    Options:
+      -h, --help            show this help message and exit
+      --arch=ARCH           Specify the arch
+      --tag=TAG             Create a mock config for a tag
+      --task=TASK           Duplicate the mock config of a previous task
+      --buildroot=BUILDROOT
+                            Duplicate the mock config for the specified buildroot
+                            id
+      --mockdir=DIR         Specify mockdir
+      --topdir=DIR          Specify topdir
+      --topurl=URL          url under which Koji files are accessible
+      --distribution=DISTRIBUTION
+                            Change the distribution macro
+      -o FILE               Output to a file
+
+for example to get the latest buildroot for dist-f12-build run
+
+::
+
+    koji mock-config --tag dist-f12-build --arch=x86_64 --topurl=http://kojipkgs.fedoraproject.org/ dist-f12
+
+you will need to pass in --topurl=http://kojipkgs.fedoraproject.org/ to
+any mock-config command to get a working mock-config from fedoras koji.
+
+Using Koji to control tasks
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+List tasks:
+
+::
+
+    koji list-tasks
+
+List only tasks requested by you:
+
+::
+
+    koji list-tasks --mine
+
+requeue an already-processed task: general syntax is: koji resubmit
+[options] taskID
+
+::
+
+    koji resubmit 3
+
+Building a Package with the command-line tool
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Instead of using the fedpkg target, you can also directly use the
+command\_line tool, koji.
+
+To build a package, the syntax is:
+
+::
+
+    $ koji build  
+
+For example:
+
+::
+
+    $ koji build dist-f14 'git url'
+
+The koji build command creates a build task in Koji. By default the tool
+will wait and print status updates until the build completes. You can
+override this with the --nowait option.
+
+.. raw:: html
+
+
+ +NOTE: For fedora koji, the git url MUST be based on +pkgs.fedoraproject.org. Other arbitrary git repos cannot be used for +builds. + +Koji tags and packages organization +----------------------------------- + +Terminology +^^^^^^^^^^^ + +In Koji, it is sometimes necessary to distinguish between a package in +general, a specific build of a package, and the various rpm files +created by a build. When precision is needed, these terms should be +interpreted as follows: + +- Package: The name of a source rpm. This refers to the package in + general and not any particular build or subpackage. For example: + kernel, glibc, etc. +- Build: A particular build of a package. This refers to the entire + build: all arches and subpackages. For example: kernel-2.6.9-34.EL, + glibc-2.3.4-2.19. +- RPM: A particular rpm. A specific arch and subpackage of a build. For + example: kernel-2.6.9-34.EL.x86\_64, kernel-devel-2.6.9-34.EL.s390, + glibc-2.3.4-2.19.i686, glibc-common-2.3.4-2.19.ia64 + +Tags and targets +^^^^^^^^^^^^^^^^ + +Koji organizes packages using tags. In Koji a tag is roughly a +collection of packages: + +- Tags support inheritance +- Each tag has its own list of valid packages (inheritable) +- Package ownership can be set per-tag (inheritable) +- When you build you specify a target rather than a tag + +A build target specifies where a package should be built and how it +should be tagged afterwards. This allows target names to remain fixed as +tags change through releases. + +Koji commands for tags +^^^^^^^^^^^^^^^^^^^^^^ + +Targets +''''''' + +You can get a full list of build targets with the following command: + +:: + + $ koji list-targets + +You can see just a single target with the --name option: + +:: + + $ koji list-targets --name dist-f14 + + Name Buildroot Destination + --------------------------------------------------------------------------------------------- + dist-f14 dist-f14-build dist-f14 + +This tells you a build for target dist-f14 will use a buildroot with +packages from the tag dist-f14-build and tag the resulting packages as +dist-f14. + +Watch out: You probably don't want to build against dist-rawhide. If +Fedora N is the latest one out, to build to the next one, choose +dist-f{N+1}. + +Tags +'''' + +You can get a list of tags with the following command: + +:: + + $ koji list-tags + +Packages +'''''''' + +As mentioned above, each tag has its own list of packages that may be +placed in the tag. To see that list for a tag, use the list-pkgs +command: + +:: + + $ koji list-pkgs --tag dist-f14 + +The first column is the name of the package, the second tells you which +tag the package entry has been inherited from, and the third tells you +the owner of the package. + +Latest Builds +''''''''''''' + +To see the latest builds for a tag, use the latest-pkg command: + +:: + + $ koji latest-pkg --all dist-f14 + +The output gives you not only the latest builds, but which tag they have +been inherited from and who built them. + +`Category:Package Maintainers `__ diff --git a/docs/source/writing_a_plugin.rst b/docs/source/writing_a_plugin.rst new file mode 100644 index 00000000..3611302b --- /dev/null +++ b/docs/source/writing_a_plugin.rst @@ -0,0 +1,190 @@ +========================= +How to write Koji plugins +========================= + + +Writing Koji plugins +==================== + +Depending on what you are trying to do, there are different ways to +write a Koji plugin. + +Each is described in this file, by use case. + +Adding new task types +--------------------- + +Koji can do several things, for example build RPMs, or live CDs. Those +are types of tasks which Koji knows about. + +If you need to do something which Koji does not know yet how to do, you +could create a Koji Builder plugin. + +Such a plugin would minimally look like this: + +:: + + from koji.tasks import BaseTaskHandler + + class MyTask(BaseTaskHandler): + Methods = ['mytask'] + _taskWeight = 2.0 + + def handler(self, arg1, arg2, kwarg1=None): + self.logger.debug("Running my task...") + + # Here is where you actually do something + +A few explanations on what goes on here: + +- Your task needs to inherit from ``koji.tasks.BaseTaskHandler`` +- Your task must have a ``Methods`` attribute, which is a list of the + method names your task can handle. +- You can specify the weight of your task with the ``_taskWeight`` + attribute. The more intensive (CPU, IO, ...) your task is, the higher + this number should be. +- The task object has a ``logger`` attribute, which is a Python logger + with the usual ``debug``, ``info``, ``warning`` and ``error`` + methods. The messages you send with it will end up in the Koji + Builder logs (``kojid.log``) +- Your task must have a ``handler()`` method. That is the method Koji + will call to run your task. It is the method that should actually do + what you need. It can have as many positional and named arguments as + you want. + +Save your plugin as e.g ``mytask.py``, then install it in the Koji +Builder plugins folder: ``/usr/lib/koji-builder-plugins/`` + +Finally, edit the Koji Builder config file, ``/etc/kojid/kojid.conf``: + +:: + + # A space-separated list of plugins to enable + plugins = mytask + +Restart the Koji Builder service, and your plugin will be enabled. + +You can try running a task from your new task type with the +command-line: + +:: + + $ koji make-task mytask arg1 arg2 kwarg1 + +Exporting new API methods over XMLRPC +------------------------------------- + +Koji clients talk to the Koji Hub via an XMLRPC API. + +It is sometimes desirable to add to that API, so that clients can +request things Koji does not expose right now. + +Such a plugin would minimally look like this: + +:: + + def mymethod(arg1, arg2, kwarg1=None): + # Here is where you actually do something + + mymethod.exported = True + +A few explanations on what goes on here: + +- Your plugin is just a method, with whatever positional and/or named + arguments you need. +- You must export your method by setting its ``exported`` attribute to + ``True`` +- The ``context.session.assertPerm()`` is how you ensure that the + +Save your plugin as e.g ``mymethod.py``, then install it in the Koji Hub +plugins folder: ``/usr/lib/koji-hub-plugins/`` + +Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``: + +:: + + # A space-separated list of plugins to enable + Plugins = mymethod + +Restart the Koji Hub service, and your plugin will be enabled. + +You can try calling the new XMLRPC API with the Python client library: + +:: + + >>> import koji + >>> session = koji.ClientSession("http://koji/example.org/kojihub") + >>> session.mymethod(arg1, arg2, kwarg1='some value') + +Ensuring the user has the required permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want your new XMLRPC API to require specific permissions from the +user, all you need to do is add the following to your method: + +:: + + from koji.context import context + + def mymethod(arg1, arg2, kwarg1=None): + context.session.assertPerm("admin") + + # Here is where you actually do something + + mymethod.exported = True + +In the example above, Koji will ensure that the user is an +administrator. You could of course create your own permission, and check +for that. + +Running code automatically triggered on events +---------------------------------------------- + +You might want to run something automatically when something else +happens in Koji. + +A typical example is to automatically sign a package right after a build +finished. Another would be to send a notification to a message bus after +any kind of event. + +This can be achieved with a plugin, which would look minimally as +follows: + +:: + + from koji.plugin import callback + + @callback('preTag', 'postTag') + def mycallback(cbtype, tag, build, user, force=False): + # Here is where you actually do something + +A few explanations on what goes on here: + +- The ``@callback`` decorator allows you to declare which events should + trigger your function. You can pass as many as you want. For a list + of supported events, see ``koji/plugins.py``. +- The arguments of the function depend on the event you subscribed to. + As a result, you need to know how it will be called by Koji. You + probably should use ``*kwargs`` to be safe. You can see how callbacks + are called in the ``hub/kojihub.py`` file, search for calls of the + ``run_callbacks`` function. + +Save your plugin as e.g ``mycallback.py``, then install it in the Koji +Hub plugins folder: ``/usr/lib/koji-hub-plugins`` + +Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``: + +:: + + # A space-separated list of plugins to enable + Plugins = mycallback + +Restart the Koji Hub service, and your plugin will be enabled. + +You can try triggering your callback plugin with the command-line. For +example, if you registered a callback for the ``postTag`` event, try +tagging a build: + +:: + + $ koji tag-build mytag mypkg-1.0-1 diff --git a/docs/source/writing_koji_code.rst b/docs/source/writing_koji_code.rst new file mode 100644 index 00000000..4dbf7929 --- /dev/null +++ b/docs/source/writing_koji_code.rst @@ -0,0 +1,631 @@ +================= +Writing Koji Code +================= + +Getting Started Hacking on Koji +=============================== + + +This page gives an overview of the Koji code and then describes what +needs to change if you want to add a new type of task. A new task could +be for a new content type, or assembling the results of multiple builds +together, or something else that helps your workflow. New contributors +to Koji should leave this page knowing where to begin and have enough +understanding of Koji's architecture to be able to estimate how much +work is still ahead of them. + +Task Flow +========= + +A task starts with a user submitting it with the Koji client, which is a +command line interface. This contacts the hub, an apache-based server +application. It leaves a row in the database that represents a "free" +task, one that has not been assigned to a builder. Periodically, the +builders asynchronously ping the hub asking if there are any tasks +available, and at some point one will be given the new task. The hub +marks this in the database, and the builder begins executing the task (a +build). + +Upon completion, the builder uploads the results to the hub, including +logs, binaries, environment information, and whatever else the task +handler for the build dictated. The hub moves the results to a permanent +shared storage solution, and marks the task as completed (or failed). +During this whole time, the webUI can be used to check up on progress. +So the flow of work is: + +:: + + Client -> Hub -> Builder -> Hub + +If you wanted to add a new build type or task that was tightly +integrated in Koji's data model, you would need to modify the CLI, Hub, +Builder, and WebUI at a minimum. Alternatively, you could do this with a +plugin, which is far simpler but less flexible. + +Component Overview +================== + +Koji is comprised of several components, this section goes into details +for each one, and what you potentially may need to change. Every +component is written in Python, so you will need to know that language +beyond a beginner level. + +Koji-client +----------- + +koji-client is a command line interface that provides many hooks into +Koji. It allows the user to query much of the data as well as perform +actions such as adding users and initiating build requests. + +Option Handling +~~~~~~~~~~~~~~~ + +The code is in ``cli/koji``. It uses ``OptionParsers`` extensively with +interspersed arguments disabled. That means these two commands are not +interpreted the same: + +:: + + $ koji -u admin -p password tag-build some-tag --force some-build + $ koji tag-build -u admin -p password some-tag --force some-build + +The second one will generate an error, because -u and -p are not options +for tag-build, they must show up before that because they are global +options that can be used with any subcommand. There will be two +``OptionParsers`` used with each command. The first is used to pick up +arguments to ``koji`` itself, and the second for the subcommand +specified. When the first one executes (see ``get_options()``) it will +figure out the subcommand and come up with a function name based on it. + +The convention is to prepend the word ``handle_`` before it, and change +all hyphens to underscores. If a command does not require an account +with Koji, the function handle will prepended with ``anon_handle_`` +instead. The code will dynamically call the derived function handle +which is where the second ``OptionParser`` is used to parse the +remaining options. To have your code log into Koji (you're writing a +handle\_ function), use the ``activate_session`` function. All function +signatures in the client code will get a session object, which is your +interface to the hub. + +Profiles +~~~~~~~~ + +It is possible to run the Koji client with different configuration +profiles so that you can interact with multiple Koji instances easily. +The ``--profile`` option to the Koji command itself enables this. You +should have a ``~/.koji/config`` already, if not just copy from +``/etc/koji.conf`` to get a start. The profile command accepts an +argument that matches a section in that config file. So if your config +file had this: + +:: + + [Fedora] + authtype = ssl + server = http://koji.fedoraproject.org/kojihub + topdir = /mnt/koji + weburl = http://koji.fedoraproject.org/koji + #pkgurl = http://koji.fedoraproject.org/packages + cert = ~/.fedora.cert + ca = ~/.fedora-upload-ca.cert + serverca = ~/.fedora-server-ca.cert + + [MyKoji] + server = http://koji.mydomain.com/kojihub + authtype = kerberos + topdir = /mnt/koji + weburl = http://koji.mydomain.com/koji + topurl = http://download.mydomain.com/kojifiles + +you could pass Fedora or MyKoji to --profile. + +Creating Tasks +~~~~~~~~~~~~~~ + +Once options are processed and understood, a task needs to be created on +the hub so that a builder can come along and take it. This is +accomplished with the ``maskTask`` method (defined on the Hub, so call +it on the ``session`` object). The name of the task should match the +name given to the task handler in the builder, which is explained later +on. + +Be sure to process the channel, priority, background, and watch/nowatch +parameters too, which should be available to most new tasks. They'll be +buried in the first argument to your handler function, which captures +the options passed to the base Koji command. + +If the client needs to make locally-available artifacts (config files, +sources, kickstarts) accessible to the builder, it must be uploaded to +the hub. This is the case with uploading SRPMs or kickstarts. You can +easily upload this content with the ``session.uploadWrapper`` method. +You can create progress bars as necessary with this snippet: + +:: + + if _running_in_bg() or task_opts.noprogress: + callback = None + else: + callback = _progress_callback + serverdir = _unique_path('cli-image') # create a unique path on the hub + session.uploadWrapper(somefile, serverdir, callback=callback) + +Task Arguments +~~~~~~~~~~~~~~ + +If you define a new task for Koji, you'll want the task submission +output to have the options ordered usefully. This output is +automatically generated, but sometimes it does not capture the more +important arguments you want displayed. + +:: + + Created task 10001810 + Watching tasks (this may be safely interrupted)... + 10001810 thing (noarch): free + 10001810 thing (noarch): free -> closed + 0 free 0 open 1 done 0 failed + + 10001810 thing (noarch) completed successfully + +In this (fake) example, you can see that "noarch" is the only option +being displayed, but maybe you want something more than just the task +architecture displayed, like some other options that were passed in. You +can fix this behavior in ``koji/__init__.py`` in the \_taskLabel +function. Here you can define the string(s) to display when Koji +receives status on a task. That is the return value. + +Koji-Hub +-------- + +koji-hub is the center of all Koji operations. It is an XML-RPC server +running under mod\_wsgi in Apache. koji-hub is passive in that it only +receives XML-RPC calls and relies upon the build daemons and other +components to initiate communication. koji-hub is the only component +that has direct access to the database and is one of the two components +that have write access to the file system. If you want to make changes +to the webUI (new pages or themes), you are looking in the wrong +section, there is a separate component for that. + +Implementation Details +~~~~~~~~~~~~~~~~~~~~~~ + +The **hub/kojihub.py** file is where the server-side code lives. If you +need to fix any server problems or want to add any new tasks, you will +need to modify this file. Changes to the database schema will almost +certainly require code changes too. This file gets deployed to +**/usr/share/koji-hub/kojihub.py**, whenever you make changes to that +remember to restart **httpd**. Also there are cases where httpd looks +for an existing .pyc file and takes it as-is, instead of re-compiling it +when the code is changed. + +In the code there are two large classes: **RootExports** and +**HostExports**. RootExports exposes methods using XMLRPC for any client +that connects to the server. The Koji CLI makes use of this quite a bit. +If you want to expose a new API to any remote system, add your code +here. The HostExports class does the same thing except it will ensure +the requests are only coming from builders. Attempting to use an API +exposed here with the CLI will fail. If your work requires the builders +to call a new API, you should implement it here. Any other function +defined in this file is inaccessible by remote hosts. It is generally a +good practice to have the exposed APIs do very little work, and pass off +control to internal functions to do the heavy lifting. + +Database Interactions +~~~~~~~~~~~~~~~~~~~~~ + +Database interactions are done with raw query strings, not with any kind +of modern ORM. Consider using context objects from the Koji contexts +library for thread-safe interactions. The database schema is captured in +the **docs** directory in the root of a git clone. A visualization of +the schema is not available at the time of this writing. + +If you plan to introduce schema changes, please update both +``schema.sql`` and provide a migration script if necessary. + +Troubleshooting +~~~~~~~~~~~~~~~ + +The hub runs in an Apache service, so you will need to look in Apache +logs for error messages if you are encountering 500 errors or the +service is failing to start. Specifically you want to check in: + +- /var/log/httpd/error\_log +- /var/log/httpd/ssl\_error\_log + +If you need more specific tracebacks and debugging data, consider +changing the debugging setting in **/etc/koji-hub/hub.conf**. Be advised +the hub is very verbose with this setting on, your logs will take up +gigabytes of space within several days. + +Kojid +----- + +kojid is the build daemon that runs on each of the build machines. Its +primary responsibility is polling for incoming build requests and +handling them accordingly. Essentially kojid asks koji-hub for work. +Koji also has support for tasks other than building. Creating install +images is one example. kojid is responsible for handling these tasks as +well. kojid uses mock for building. It also creates a fresh buildroot +for every build. kojid is written in Python and communicates with +koji-hub via XML-RPC. + +Implementation Details +~~~~~~~~~~~~~~~~~~~~~~ + +The daemon runs as a service on a host that is traditionally not the +same as the hub or webUI. This is a good security practice because the +service runs as root, and executes untrusted code to produce builds on a +regular basis. Keeping the Hub separate limits the damage a malicious +package can do to the build system as a whole. For the same reason, the +filesystem that the hub keeps built software on should be mounted +Read-Only on the build host. It should call APIs on the hub that are +exposed through the ``HostExports`` class in the hub code. Whenever the +builder accepts a task, it forks a process to carry out the build. + +An initscript/unit-file is available for kojid, so it can be stopped and +started like a normal service. Remember to do this when you deploy +changes! + +TaskHandlers +^^^^^^^^^^^^ + +All tasks in kojid have a ``TaskHandler`` class that defines what to do +when the task is picked up from the hub. The base class is defined in +``koji/tasks.py`` where a lot of useful utility methods are available. +An example is ``uploadFile``, which is used to upload logs and built +binaries from a completed build to the hub since the shared filesystem +is read only. + +The daemon code lives in ``builder/kojid``, which is deployed to +/usr/sbin/kojid. In there you'll notice that each task handler class has +a ``Methods`` member and ``_taskWeight`` member. These must be defined, +and the former is used to match the name of a waiting task (on the hub) +with the task handler code to execute. Each task handler object must +have a ``handler`` method defined, which is the entry point for the +forked process when a builder accepts a task. + +Tasks can have subtasks, which is a typical model when a build can be +run on multiple architectures. In this case, developers should write 2 +task handlers: one handles the build for exact one architecture, and one +that assembles the results of those tasks into a single build, and sends +status information to the hub. You can think of the latter handler as +the parent task. + +All task handler objects have a ``session`` object defined, which is the +interface to use for communications with the hub. So, parent tasks +should kick off child tasks using the session object's subtask method +(which is part of HostExports). It should then call ``self.wait`` with +``all=True`` to wait for the results of the child tasks. + +Here's a stub of what a new build task might look like: + +:: + + class BuildThingTask(BaseTaskHandler): + Methods = ['thing'] + _taskWeight = 0.5 + + def handler(self, a, b, arches, options): + subtasks = {} + for arch in arches: + subtasks[arch] = session.host.subtask(method='thingArch', a, b, arch) + results = self.wait(subtasks.values(), all=True) + # parse results and put rows in database + # put files in their final resting place + return 'Build successful' + + class BuildThingArchTask(BaseTaskHandler): + Methods = ['thingArch'] + _taskWeight = 2.0 + + def handler(self, a, b, arch): + # do the build, capture results in a variable + self.uploadFile('/path/to/some/log') + self.uploadFile('/path/to/binary/file') + return result + +Source Control Managers +^^^^^^^^^^^^^^^^^^^^^^^ + +If you your build needs to check out code from a Source Control Manager +(SCM) such as git or subversion, you can use SCM objects defined in +``koji/daemon.py``. They take a specially formed URL as an argument to +the constructor. Here's an example use. The second line is important, it +makes sure the SCM is in the whitelist of SCMs allowed in +``/etc/kojid/kojid.conf``. + +:: + + scm = SCM(url) + scm.assert_allowed(self.options.allowed_scms) + directory = scm.checkout('/checkout/path', session, uploaddir, logfile) + +Checking out takes 4 arguments: where to checkout, a session object +(which is how authentication is handled), a directory to upload the log +to, and a string representing the log file name. Using this method Koji +will checkout (or clone) a remote repository and upload a log of the +standard output to the task results. + +Build Root Objects +^^^^^^^^^^^^^^^^^^ + +It is encouraged to build software in mock chroots if appropriate. That +way Koji can easily track precise details about the environment in which +the build was executed. In ``builder/kojid`` a BuildRoot class is +defined, which provides an interface to execute mock commands. Here's an +example of their use: + +:: + + broot = BuildRoot(self.session, self.options, build_tag, arch, self.id) + +A session object, task options, and a build tag should be passed in +as-is. You should also specify the architecture and the task ID. If you +ever need to pass in specialized options to mock, look in the +ImageTask.makeImgBuildRoot method to see how they are defined and passed +in to the BuildRoot constructor. + +Troubleshooting +~~~~~~~~~~~~~~~ + +The daemon writes a log file to ``/var/log/kojid.log``. Debugging output +can be turned on in ``/etc/kojid/kojid.conf``. + +Koji-Web +-------- + +koji-web is a set of scripts that run in mod\_wsgi and use the Cheetah +templating engine to provide a web interface to Koji. It acts as a +client to koji-hub providing a visual interface to perform a limited +amount of administration. koji-web exposes a lot of information and also +provides a means for certain operations, such as cancelling builds. + +The web pages are derived from Cheetah templates, the syntax of which +you can read up on +`here `__. These +templates are the ``chtml`` files sitting in ``www/kojiweb``. You'll +notice quickly that these templates are referencing variables, but where +do they come from? + +The ``www/kojiweb/index.py`` file provides them. There are several +functions named after the templates they support, and in each one a +dictionary called ``values`` is populated. This is how data is gathered +about the task, build, archive, or whatever the page is about. Take your +time with ``taskinfo.chtml`` in particular, as the conditionals there +have gotten quite long. If you are adding a new task to Koji, you will +need to extend this at a minimum. A new type of build task would require +this, and possibly another that is specific to viewing the archived +information about the build. (taskinfo vs. buildinfo) + +If your web page needs to display the contents of a list or dictionary, +use the ``$printMap`` function to help with that. It is often sensible +to define a function that easily prints options and values in a +dictionary. An example of this is in taskinfo.chtml. + +:: + + #def printOpts($opts) + #if $opts + Options:
+ $printMap($opts, '  ') + #end if + #end def + +Finally, if you need to expand the drop-down menus of "method" types +when searching for tasks in the WebUI, you will need to add them to the +``_TASKS`` list in ``www/kojiweb/index.py``. Add values where +appropriate to ``_TOPLEVEL_TASKS`` and ``_PARENT_TASKS`` as well so that +parent-child relationships show up correctly too. + +Remember whenever you update a template or index.py, you will need to +deploy and restart apache/httpd! + +Troubleshooting +~~~~~~~~~~~~~~~ + +Like the hub, this component is backed by apache, so you should follow +the same techniques for debugging Koji-Web as +`Koji-Hub <#Troubleshooting>`__. + +Kojira +------ + +kojira is a daemon that keeps the build root repodata updated. It is +responsible for removing redundant build roots and cleaning up after a +build request is completed. + +Building and Deploying Changes +============================== + +The root of the git clone for Koji code contains a ``Makefile`` that has +a few targets to make building and deployment a little easier. Among +them are: + +- tarball: create a bz2 tarball that could be consumed in an rpm build +- rpm: create Koji rpms. The NVRs will be defined by the spec file, + which is also in the same directory. The results will appear in a + ``noarch`` directory. +- test-rpm: like rpm, but append the Release field with a date and time + stamp for easy upgrade-deployment + +Plugins +======= + +This section is copied from the ``docs/Writing_a_plugin.md`` file. + +Koji supports different types of plugins, three of which are captured +here. Depending on what you are trying to do, there are different ways +to write a Koji plugin. + +Koji Builder Plugins +-------------------- + +Koji can do several things, for example build RPMs, or live CDs. Those +are types of tasks which Koji knows about. If you need to do something +which Koji does not know yet how to do, you could create a Koji Builder +plugin. Such a plugin would minimally look like this: + +:: + + from koji.tasks import BaseTaskHandler + + class MyTask(BaseTaskHandler): + Methods = ['mytask'] + _taskWeight = 2.0 + + def handler(self, arg1, arg2, kwarg1=None): + self.logger.debug("Running my task...") + # Here is where you actually do something + +A few explanations on what goes on here: + +- Your task needs to inherit from \`koji.tasks.BaseTaskHandler\` +- Your task must have a \`Methods\` attribute, which is a list of the + method names your task can handle. +- You can specify the weight of your task with the \`\_taskWeight\` + attribute. The more intensive (CPU, IO, ...) your task is, the higher + this number should be. +- The task object has a ``logger`` attribute, which is a Python logger + with the usual \`debug\`, \`info\`, \`warning\` and \`error\` + methods. The messages you send with it will end up in the Koji + Builder log. +- Your task must have a \`handler()\` method. That is the method Koji + will call to run your task. It is the method that should actually do + what you need. It can have as many positional and named arguments as + you want. + +Save your plugin as e.g ``mytask.py``, then install it in the Koji +Builder plugins folder: ``/usr/lib/koji-builder-plugins/``. Finally, +edit the Koji Builder config file, ``/etc/kojid/kojid.conf``: + +:: + + # A space-separated list of plugins to enable + plugins = mytask + +Restart the Koji Builder service, and your plugin will be enabled. You +can try running a task from your new task type with the command-line: +``$ koji make-task mytask arg1 arg2 kwarg1`` + +Hub Plugins +----------- + +Koji clients talk to the Koji Hub via an XMLRPC API. It is sometimes +desirable to add to that API, so that clients can request things Koji +does not expose right now. Such a plugin would minimally look like this: + +:: + + def mymethod(arg1, arg2, kwarg1=None): + # Here is where you actually do something + mymethod.exported = True + +What's happening? + +- Your plugin is just a method, with whatever positional and/or named + arguments you need. +- You must export your method by setting its ``exported`` attribute to + ``True`` +- The ``context.session.assertPerm()`` is how you ensure that the + correct permissions are available. + +Save your plugin as e.g \`mymethod.py\`, then install it in the Koji Hub +plugins folder, which is ``/usr/lib/koji-hub-plugins/`` + +Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``: + +:: + + # A space-separated list of plugins to enable + Plugins = mymethod + +Restart the Koji Hub service, and your plugin will be enabled. You can +try calling the new XMLRPC API with the Python client library: + +:: + + >>> import koji + >>> session = koji.ClientSession("http://koji/example.org/kojihub") + >>> session.mymethod(arg1, arg2, kwarg1='some value') + +If you want your new XMLRPC API to require specific permissions from the +user, all you need to do is add the following to your method: + +:: + + from koji.context import context + + def mymethod(arg1, arg2, kwarg1=None): + context.session.assertPerm("admin") + # Here is where you actually do something + mymethod.exported = True + +In the example above, Koji will ensure that the user is an +administrator. You could of course create your own permission, and check +for that. + +Event Plugin +------------ + +You might want to run something automatically when something else +happens in Koji. A typical example is to automatically sign a package +right after a build finished. Another would be to send a notification to +a message bus after any kind of event. + +This can be achieved with a plugin too, which would look minimally as +follows: + +:: + + from koji.plugin import callback + + @callback('preTag', 'postTag') + def mycallback(cbtype, tag, build, user, force=False): + # Here is where you actually do something + +So what is this doing? + +- The ``@callback`` decorator allows you to declare which events should + trigger your function. You can pass as many as you want. For a list + of supported events, see ``koji/plugins.py``. +- The arguments of the function depend on the event you subscribed to. + As a result, you need to know how it will be called by Koji. You + probably should use ``*kwargs`` to be safe. You can see how callbacks + are called in the ``hub/kojihub.py`` file, search for calls of the + ``run_callbacks`` function. + +Save your plugin as e.g ``mycallback.py``, then install it in the Koji +Hub plugins folder: ``/usr/lib/koji-hub-plugins`` + +Finally, edit the Koji Hub config file, ``/etc/koji-hub/hub.conf``: + +:: + + # A space-separated list of plugins to enable + Plugins = mycallback + +Restart the Koji Hub service, and your plugin will be enabled. You can +try triggering your callback plugin with the command-line. For example, +if you registered a callback for the ``postTag`` event, try tagging a +build: ``$ koji tag-build mytag mypkg-1.0-1`` + +Patch Review +============ + +If you have a patch to submit, please send it to +koji-devel@lists.fedoraproject.org. Here are some guidelines on +producing preferable patches. + +- Please do not "patch bomb". Keep them manageable, perhaps providing a + patch for each component that is getting changes. +- Adhere to `PEP8 `__ +- Provide patches a feature at a time, not a pile of several. Please + follow the same practice for bugs. +- Consider using ``git-send-email``, or just attach the patches to the + email +- Patches should apply cleanly to HEAD of the intended branch +- Please maintain backward-compatibility up to RHEL 5 (which means + Python 2.4) + +There are only 3-5 people with commit access that regularly watch the +mailing list, so responses may take a couple days if we're all occupied.