PR#595: Release 1.14.0

Merges #595
https://pagure.io/koji/pull-request/595

docs/source/conf.py

@@ -54,9 +54,9 @@ copyright = u'2017, Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McL
 # built documents.
 #
 # The short X.Y version.
-version = '1.13'
+version = '1.14'
 # The full version, including alpha/beta/rc tags.
-release = '1.13.0'
+release = '1.14.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/source/content_generator_metadata.rst

@@ -1,8 +1,8 @@
-========================================
-Koji Content Generator Metadata Proposal
-========================================
+===============================
+Koji Content Generator Metadata
+===============================
 
-This document describes the proposed Koji Content Generator Metadata
+This document describes the Koji Content Generator Metadata
 Format (version 0). This is the metadata that should be provided by a
 Content Generator in order for the content to be imported and managed by
 Koji. If you have further questions about :doc:`Content
@@ -108,7 +108,10 @@ Each map in the buildroots list contains the following entries:
    -  checksum: The checksum of the file.
    -  checksum\_type: The checksum type used.
 
+.. _metadata-kojifile:
+
 -  For maps where **type = kojifile**, the following fields will be present:
+
    -  filename: The name of the file.
    -  filesize: The size of the file.
    -  checksum: The checksum of the file.

docs/source/content_generators.rst

@@ -12,7 +12,7 @@ provide stable APIs and interfaces to other processes.
 
 Along with the content to be managed by Koji, a Content Generator will
 provide enough metadata to enable a reasonable level of auditing and
-reproduceability. The exact data provided and the format used is being
+reproducibility. The exact data provided and the format used is being
 discussed, but will include information like the upstream source URL,
 build tools used, build environment contents, and any
 container/virtualization technologies used.

docs/source/defining_hub_policies.rst

@@ -247,7 +247,7 @@ Available tests
     * for use with the channel policy
 
 ``method``
-    * matches the task method name agaist glob pattern(s)
+    * matches the task method name against glob pattern(s)
     * true if the method name matches any of the patterns
     * for use with the channel policy
 

docs/source/external_repo_server_bootstrap.rst

@@ -83,7 +83,7 @@ Examples of urls to use for external Repositories
 all these examples use mirrors.kernel.org please find the closest mirror
 to yourself. Note that the Fedora minimal buildroots download ~100Mb
 then build dependencies on top. these are downloaded each build you can
-save alot of network bandwidth by using a local mirror or running
+save a lot of network bandwidth by using a local mirror or running
 through a caching proxy.
 
 NOTE: this uses $arch **NOT** $basearch

docs/source/migrating_to_1.10.rst

@@ -67,7 +67,7 @@ And there are three new commands
 Other Configuration changes
 ---------------------------
 
-The Koji web interface can now treate ``extra-footer.html`` as a Cheetah
+The Koji web interface can now treat ``extra-footer.html`` as a Cheetah
 template. This behavior can be enabled by setting the ``LiteralFooter`` option
 to ``False`` in the kojiweb config.
 

docs/source/migrating_to_1.12.rst

@@ -71,7 +71,7 @@ Changes to calls:
     * repo related calls (e.g. ``repoInfo`` now include a boolean ``dist``
       field
     * the ``editTag2`` call can now remove tag_extra data if the
-      ``remove_extra`` keywarg argument is used
+      ``remove_extra`` keyword argument is used
     * the listTaskOutput call supports a new ``all_volumes`` keyword argument.
       If true, the results are extended to deal with files in same relative paths
       on different volumes.

docs/source/migrating_to_1.13.rst

@@ -4,7 +4,7 @@ Migrating to Koji 1.13
 ..
   reStructured Text formatted
 
-The 1.13 release of Koji includes a several changes that you should consider when
+The 1.13 release of Koji includes several changes that you should consider when
 migrating.
 
 DB Updates

docs/source/migrating_to_1.14.rst

@@ -0,0 +1,39 @@
+Migrating to Koji 1.14
+======================
+
+..
+  reStructured Text formatted
+
+You should consider the following changes when migrating to 1.14:
+
+DB Updates
+----------
+
+The schema updates this time are minor
+
+   * dropped unused ``log_messages`` table
+   * new standard entries in the ``archivetypes`` table
+
+As in previous releases, we provide a migration script that updates the
+database.
+
+::
+
+    # psql koji koji  </usr/share/doc/koji/docs/schema-upgrade-1.13-1.14.sql
+
+
+Dropped mod_python support
+--------------------------
+
+Koji's support for mod_python has been deprecated for many years. If you are
+still relying on mod_python, you will need to switch to mod_wsgi.
+
+See: :doc:`migrating_to_1.7`
+
+
+Other changes
+-------------
+
+There are numerous other changes in 1.14 that should not have a direct impact
+on migration. For details see:
+:doc:`release_notes_1.14`

docs/source/migrations.rst

@@ -5,6 +5,7 @@ Migrations
 .. toctree::
     :maxdepth: 1
 
+    migrating_to_1.14
     migrating_to_1.13
     migrating_to_1.12
     migrating_to_1.11

docs/source/release_notes.rst

@@ -5,6 +5,7 @@ Release Notes
 .. toctree::
     :maxdepth: 1
 
+    release_notes_1.14
     release_notes_1.13
 
 For releases before 1.13, see the migration guides:

docs/source/release_notes_1.13.rst

@@ -61,7 +61,7 @@ similar to the web interface.
 
 .. code-block:: text
 
-    Usage: lkoji hostinfo [options] <hostname> [<hostname> ...]
+    Usage: koji hostinfo [options] <hostname> [<hostname> ...]
     (Specify the --help global option for a list of other help options)
 
     Options:
@@ -75,7 +75,7 @@ Enhancements to restart-hosts
 The `restart-hosts` command is used by admins to safely restart the build hosts
 after a configuration change.
 
-Because multiple restarts can confict, the command will now exit with a error
+Because multiple restarts can conflict, the command will now exit with a error
 if a restart is already underway (can be overridden with --force).
 
 There are now options to limit the restart to a given channel or arch.

docs/source/release_notes_1.14.rst

@@ -0,0 +1,310 @@
+Koji 1.14 Release Notes
+=======================
+
+Migrating from Koji 1.13
+------------------------
+
+For details on migrating see :doc:`migrating_to_1.14`
+
+
+Client Changes
+--------------
+
+
+Fail fast option for builds
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/432
+
+
+When builders are configured with ``build_arch_can_fail = True`` then the
+failure of a single buildArch task does not immediately cause the build
+to fail. Instead, the remaining buildArch tasks are allowed to complete,
+at which point the build will still fail.
+
+Sometimes developers would rather a build fail immediately, so we have added
+the ``--fail-fast`` option to the build command, which overrides this setting.
+The option only has an effect if the builders are configured to fail slow.
+
+
+Custom Lorax templates
+^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/419
+
+Koji now supports custom Lorax templates for the ``spin-livemedia`` command.
+The command accepts two new options:
+
+.. code-block:: text
+
+      --lorax_url=URL       The URL to the SCM containing any custom lorax
+                            templates that are to be used to override the default
+                            templates.
+      --lorax_dir=DIR       The relative path to the lorax templates directory
+                            within the checkout of "lorax_url".
+
+
+The Lorax templates must come from an SCM, and the ``allowed_scms`` rules
+apply.
+
+When these options are used, the templates will be fetched and an appropriate
+``--lorax-templates`` option will be passed to the underlying livemedia-creator
+command.
+
+
+Allow profiles to request a specific python version
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/566
+
+On platforms with python3 available, the Koji client is built to execute
+with the python3 binary. However, there are a few client features that do not
+work under python3, notably old-style (non-gssapi) Kerberos authentication.
+
+If this issue is affecting you, you can set ``pyver=2`` in your Koji
+configuration. This can be done per profile. When Koji sees this setting
+at startup, it will re-execute itself under the requested python binary.
+
+
+New list-builds command
+^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/526
+
+The command line now has a ``list-builds`` command that has similar
+functionality to the builds tab of the web interface.
+
+.. code-block:: text
+
+    Usage: koji list-builds [options]
+    (Specify the --help global option for a list of other help options)
+
+    Options:
+      -h, --help            show this help message and exit
+      --package=PACKAGE     List builds for this package
+      --buildid=BUILDID     List specific build from ID or nvr
+      --before=BEFORE       List builds built before this time
+      --after=AFTER         List builds built after this time
+      --state=STATE         List builds in this state
+      --type=TYPE           List builds of this type.
+      --prefix=PREFIX       Only list packages starting with this prefix
+      --owner=OWNER         List builds built by this owner
+      --volume=VOLUME       List builds by volume ID
+      -k FIELD, --sort-key=FIELD
+                            Sort the list by the named field
+      -r, --reverse         Print the list in reverse order
+      --quiet               Do not print the header information
+
+
+New block-group command
+^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/509
+
+The ``block-group`` command allows admins to block package group entries
+without having to resort to the ``call`` command.
+
+.. code-block:: text
+
+    Usage: koji block-group <tag> <group>
+    (Specify the --help global option for a list of other help options)
+
+    Options:
+      -h, --help  show this help message and exit
+
+
+Exit codes for some commands
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/558
+| PR: https://pagure.io/koji/pull-request/559
+
+Several more commands will now return a non-zero exit code
+when an error occurs:
+
+    * the various image building commands
+    * the ``save-failed-tree`` command (provided by a plugin)
+
+
+Easier for scripts to use activate_session
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/493
+
+In Koji 1.13.0, it became possible for scripts to ``import koji_cli.lib`` and
+gain access to the ``activate_session`` function that the command line tool
+uses to authenticate.
+
+In this release, this function has been made easier for scripts to use:
+
+    * the options argument can now be a dictionary
+    * less options need to be specified
+
+
+Builder changes
+---------------
+
+
+Normalize paths for scms
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/591
+
+
+For many years, kojid has supported the ``allowed_scms`` option
+(see: :ref:`scm-config`) for controlling which scms can be used for building.
+In 1.13.0, Koji added the ability to explicitly block a host:path pattern.
+
+Unfortunately, 1.13.0 did not normalize the path before checking the pattern,
+making it possible for users to use equivalent paths to route around the
+block patterns.
+
+Now, Koji will normalize these paths before the ``allowed_scms`` check.
+
+
+Graceful reload
+^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/565
+
+
+For a long time kojid has handled the USR1 signal by initiating a graceful restart.
+This change exposes that in the systemd service config (and the init script
+on older platforms).
+
+Now, ``service kojid reload`` will trigger the same sort of restart that the
+``restart-hosts`` command accomplishes, but only for the build host you run it
+on. When this happens, kojid will:
+
+    * stop taking new tasks
+    * wait for current tasks to finish
+    * restart itself once all its tasks are completed
+
+
+Friendlier runroot configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/539
+| PR: https://pagure.io/koji/pull-request/528
+
+Two changes make it easier to write a configuration for the runroot plugin.
+
+The ``path_subs`` option is now more forgiving about whitespace:
+
+    * leading and trailing whitespace is ignored for each line
+    * blank lines are ignored
+
+The ``[pathNN]`` sections are no longer required to have sequential numbers.
+Previously, the plugin expected a sequence like ``[path0]``, ``[path1]``,
+``[path2]``, etc, and would stop looking for entries if the next number
+was missing. Now, any set of distinct numbers is valid and all ``[pathNN]``
+sections will be processed.
+
+
+System changes
+--------------
+
+Deprecations
+^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/554
+| PR: https://pagure.io/koji/pull-request/597
+
+The following features are deprecated and will be removed in a future release:
+
+    * the ``importBuildInPlace`` rpc call
+    * the ``use_old_ssl`` client configuration option (and the underlying
+      ``koji.compatrequests`` library)
+
+
+Removed calls
+^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/497
+| PR: https://pagure.io/koji/pull-request/507
+
+The deprecated ``buildFromCVS`` hub call has been removed. It was replaced
+by the ``buildSRPMFromCVS`` call many years ago and has been deprecated since
+version 1.6.0.
+
+The ``add_db_logger`` function has been removed from the koji library, along
+with the ``log_messages`` table in the db. This extraneous call has never been
+used in Koji.
+
+
+Dropped mod_python support
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/508
+
+
+Koji no longer supports mod_python. This option has been deprecated since
+mod_wsgi support was added in version 1.7.0.
+
+See also: :doc:`migrating_to_1.7`
+
+
+Large integer support
+^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/571
+
+
+Koji uses xmlrpc for communications with the hub, and unfortunately the
+baseline xmlrpc standard only supports 32-bit signed integers. This
+results in errors when larger integers are encountered, typically
+when a file is larger than 2 GiB.
+
+Starting with version 1.14.0, Koji will emit ``i8`` tags when encoding
+large integers for xmlrpc. Integers below the limit are still encoded
+with the standard ``int`` tag. The only time this makes a difference
+is when Koji would previously have raised an ``OverflowError``.
+
+The ``i8`` tag comes from the
+`ws-xmlrpc <https://ws.apache.org/xmlrpc/types.html>`__
+spec. Python's xmlrpc decoder has
+for many years accepted and understood this tag, even though its encoder
+would not emit it.
+
+Previous versions of Koji worked around such size issues by converting
+large integers to strings in a few targeted places. Those targeted
+workarounds have been left in place on the hub for the sake of backward
+compatibility.
+
+
+Test mode for protonmsg plugin
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/538
+
+The ``protonmsg`` plugin now accepts a boolean ``test_mode`` configuration
+option. When this option is enabled, the plugin will not actually
+send messages, but will instead log them (at the DEBUG level).
+
+This option allows testing environments to run with the plugin enabled, but
+without requiring a message bus to be set up for that environment.
+
+
+Handling of debugsource rpms
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/524
+
+Koji will now treat rpms ending in ``-debugsource`` the same way that it does
+other debuginfo rpms. Such rpms are:
+
+    * omitted from Koji's normal yum repos
+    * listed separately when displaying builds
+    * not downloaded by default in the ``download-build`` command
+
+
+Added kojifile component type for content generators
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+| PR: https://pagure.io/koji/pull-request/506
+
+Content generator imports now accept entries with type equal to ``kojifile``
+in the component lists for buildroots and images/archives. This type provides
+a more reliable way to reference archive that come from Koji.
+
+See: :ref:`Example metadata <metadata-kojifile>`.

docs/source/server_howto.rst

@@ -566,7 +566,7 @@ Set Database To Listen On All Addresses
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 The koji-hub service will attempt to connect to the database server in the
 manner you configure.  If you use the system hostname, then the database will
-need to be avalible on that address.  To configure this please perform the
+need to be available on that address.  To configure this please perform the
 following:
 
 #.  Edit /var/lib/pgsql/data/postgresql.conf
@@ -987,7 +987,7 @@ Add the host entry for the koji builder to the database
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 You will now need to add the koji builder to the database so that they can be
-utilized by kojij hub. Make sure you do this before you start kojid for the
+utilized by koji hub. Make sure you do this before you start kojid for the
 first time, or you'll need to manually remove entries from the sessions and
 users table before it can be run successfully.
 
@@ -1033,7 +1033,7 @@ the same as KojiDir)
     ; The directory root for temporary storage
     workdir=/tmp/koji
 
-The root of the koji build directory (ie, ``/mnt/koji``) must be mounted on the
+The root of the koji build directory (i.e., ``/mnt/koji``) must be mounted on the
 builder. A Read-Only NFS mount is the easiest way to handle this.
 
 ::
@@ -1134,6 +1134,9 @@ a host, but allow from it otherwise
         alt-server.example.com:/repo/dist/repos*/:no:fedpkg,sources
 
 
+The explicit block syntax was added in version 1.13.0.
+
+
 Add the host to the createrepo channel
 --------------------------------------
 

docs/source/using_the_koji_build_system.rst

@@ -66,7 +66,7 @@ by running ``fedora-cert -n`` again. You can check if it has expired
 with ``fedora-cert -v``.
 
 the following are downloaded automatically by fedora-packager-setup and
-dont need to be manually setup
+don't need to be manually setup
 
 ``~/.fedora-upload-ca.cert`` (The certificate for the Certificate
 Authority used to sign the user keys.) : It can be manually downloaded
@@ -258,7 +258,7 @@ the next group will begin building. For example:
 ``fedpkg chain-build libwidget libaselib : libgizmo :``
 
 will cause libwidget and libaselib to be built in parallel, followed by
-libgizmo and then the currect directory package. If no groups are
+libgizmo and then the correct directory package. If no groups are
 defined, packages will be built sequentially.
 
 If a build fail, following builds are cancelled but the builds that
@@ -406,7 +406,7 @@ any mock-config command to get a working mock-config from fedoras koji.
 
 .. _tuning-mock-per-tag:
 
-Tuning mock's behaviour per tag
+Tuning mock's behavior per tag
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Few options for mock can be configured per-tag. These options are stored in
@@ -423,7 +423,7 @@ environment follows:
   package manager. Typically used with `yum` or `dnf` values.
 * `mock.new_chroot` - 0/1 value. If it is set, `--new-chroot` or
   `--old-chroot` option is appended to any mock call. If it is not set,
-  mock's default behaviour is used.
+  mock's default behavior is used.
 
 Using Koji to control tasks
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

docs/source/writing_a_plugin.rst

@@ -218,7 +218,7 @@ the command.
 
 In the first case, the command will not automatically
 authenticate with the hub (though the user can still override
-this behaviour with `--force-auth` option). In the second case, the command
+this behavior with `--force-auth` option). In the second case, the command
 will perform authentication by default (this too can be overridden by the
 user with the `--noauth` option).
 

docs/source/writing_koji_code.rst

@@ -632,6 +632,7 @@ Here are some guidelines on producing preferable pull requests.
 -  Code which is imported into CLI or needed for stand-alone API calls must
    run in both 2.6+ and 3.x python versions. We use the python-six library
    for compatibility. The affected files are:
+
      - ``cli/*``
      - ``koji/__init__.py``
      - ``koji/auth.py``
@@ -639,6 +640,7 @@ Here are some guidelines on producing preferable pull requests.
      - ``koji/util.py``
      - ``tests/test_lib/*``
      - ``tests/test_cli/*``
+
 - Check, that unit tests are not broken. Simply run ``make test`` in main
   directory of your branch. For python3 compatible-code we have also ``make
   test3`` target.

koji.spec

@@ -31,7 +31,7 @@
 %define release %{baserelease}
 %endif
 Name: koji
-Version: 1.13.0
+Version: 1.14.0
 Release: %{release}%{?dist}
 License: LGPLv2 and GPLv2+
 # koji.ssl libs (from plague) are GPLv2+
@@ -480,6 +480,58 @@ fi
 %endif
 
 %changelog
+* Mon Sep 25 2017 Mike McLean <mikem at redhat.com> - 1.14.0-1
+- PR#597: use_old_ssl is deprecated
+- PR#591: Normalize paths for scms
+- PR#432: override build_arch_can_fail settings
+- PR#566: allow profiles to request a specific python version
+- PR#554: deprecate importBuildInPlace hub call
+- PR#590: support repo_include_all tag extra option
+- PR#582: Content generator metadata documentation update
+- PR#579: ignore inodes when running rpmdiff.
+- PR#493: modify activate_session to be easily used without CLI
+- PR#589: fix scratch ref for scm callback
+- PR#587: add `build_tag` argument in `postSCMCheckout` callback
+- PR#583: support rpm LONG*SIZE header fields
+- PR#526: Added list builds command to koji CLI
+- PR#581: Add a note to get_build docstring
+- PR#575: add xjb and yaml type in archivetypes table
+- PR#571: Support large ints over xmlrpc using i8 tag
+- PR#538: protonmsg plugin: test mode
+- PR#547: update version in sphinx config
+- PR#548: set task arch for indirection image builds
+- PR#568: spec: use correct macro - rhel instead redhat for RHEL version
+- PR#558: cli: Fix exit code for building images
+- PR#559: return result status in save-failed-tree
+- PR#561: rename rpm-python to python*-rpm for EOL of F24
+- PR#562: fix serverca default in kojivmd
+- PR#565: expose graceful reload in kojid service config and init script
+- PR#544: incorrect parameter for error message
+- PR#510: cli: change download-task to regular curl download
+- PR#536: fix docs links, plus minor docs cleanup
+- PR#539: runroot: friendlier parsing of path_subs config
+- PR#542: check RPMTAG_LONGSIZE is RPMTAG_SIZE is null
+- PR#419: Koji support for custom Lorax templates in LiveMedia tasks
+- PR#546: fix test_krbv_disabled unit test
+- PR#518: Error out if krbV is unavailable and gssapi did not work
+- PR#535: datetime compatibility for plugins
+- PR#524: Add support for debugsource
+- PR#528: allow some missing path sections in runroot config
+- PR#530: Spelling fixes
+- PR#506: Track artifacts coming from koji itself
+- PR#499: runroot: use /builddir/runroot.log instead of /tmp/runroot.log
+- PR#509: CLI block-group command
+- PR#514: Fix resubmit
+- PR#521: update links in README.md
+- PR#502: download-build: suppress output on quiet and add --noprogress
+- PR#511: unit tests for delete_tag() [Open]
+- PR#484: fix NoneType TypeError in deleteTag
+- PR#490: getUserPerms should throw GenericError when no user found
+- PR#497: remove deprecated buildFromCVS call
+- PR#503: Remove deprecated compat_mode from runroot plugin
+- PR#507: drop unused add_db_logger call and db table
+- PR#508: drop mod_python support
+
 * Fri Jun 30 2017 Mike McLean <mikem at redhat.com> - 1.13.0-1
 - PR#496 Makefile/spec fixes for building on el6
 - PR#491 epel-compatible macro in spec