961 lines
41 KiB
ReStructuredText
961 lines
41 KiB
ReStructuredText
=======================
|
|
Building Images in Koji
|
|
=======================
|
|
|
|
Image Building
|
|
==============
|
|
|
|
Koji is capable of building many different types of images or appliances. They
|
|
broadly fall into a few categories: LiveMedias, Disk Images, and Containers. All
|
|
types of images end up with a Name-Version-Release just like an RPM build. What
|
|
you need to provide Koji to perform a build varies by category, and the
|
|
specifics will be explained below.
|
|
|
|
For additional questions and information, ask around in ``#koji`` on FreeNode
|
|
IRC and sign up to the right mailing lists as the `Koji project website`_
|
|
dictates.
|
|
|
|
Kickstart First
|
|
===============
|
|
|
|
No matter which type of image you want to build, you need to feed Koji a
|
|
`kickstart`_ file, so we touch on how to create them and how to get them to Koji
|
|
first. Once you have a kickstart to try out, look over the kickstart-specific
|
|
caveats for each image type in the later sections.
|
|
|
|
Kickstart Reference
|
|
-------------------
|
|
|
|
If you're new to automated installations, you should read the `Anaconda
|
|
Kickstart Guide`_ first.
|
|
|
|
Getting your Kickstart to Koji
|
|
------------------------------
|
|
|
|
For scratch builds, you can get away with just using the ``--kickstart``
|
|
parameter, which accepts a path (relative or absolute) to your kickstart file
|
|
on disk. For non-scratch builds though, the kickstart file needs to live in a
|
|
remote Source Control Manager (SCM) such as git or Subversion. To access that,
|
|
you need to pass the ``--ksurl`` parameter, which accepts a wacky string in
|
|
this form:
|
|
|
|
::
|
|
|
|
git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7
|
|
|
|
Here we pointed to a repository hosted at
|
|
``git://git.fedorahosted.org/git/spin-kickstarts.git``. Note the ``?`` in
|
|
there, this delimiter separates the host URL with a directory structure to
|
|
look in for the kickstart file. In other words, if you were to clone the
|
|
repository, you should expect to see these (sub)directories laid out below the
|
|
root of the clone. The important thing to realize is the behavior of
|
|
``--kickstart`` changes if ``--ksurl`` is specified. Instead of indicating a
|
|
path to a kickstart file, you just indicate a filename. The file itself is not
|
|
specified in the fragment of the string between ``?`` and ``#`` that is passed
|
|
to ``--ksurl``, Koji still uses the ``--kickstart`` option for that. After the
|
|
directory structure comes a ``#``, which indicates the start of the commit ID.
|
|
This is the commit Koji will reset the repository to after cloning. If you
|
|
created a git repository which had just the spec file template in it, your SCM
|
|
URL would have the ``?`` and ``#`` right next to each other.
|
|
|
|
If we passed ``--ksurl`` the string above, and gave
|
|
``--kickstart "server-ec2.ks"``, then Koji would do the following:
|
|
|
|
* Consult hub SCM policy ``build_rpm``. Name of the policy could change in the
|
|
future as it has misleading name now. Policy could check if the given SCM is
|
|
allowed for this method, etc. In case it deny the build, it ends here.
|
|
* Clone the repository locally
|
|
* Call ``git reset --hard 68c40eb7``
|
|
* Search in the "fedora22" subdirectory of the clone for a file called
|
|
``server-ec2.ks``, and pass that to Anaconda
|
|
* Perform an automated install
|
|
|
|
.. _building-livemedia:
|
|
|
|
Building LiveMedias
|
|
===================
|
|
|
|
Let's describe building LiveMedias and the mechanics behind it. We also still
|
|
suport older method LiveCD which is deprecated now.
|
|
|
|
Getting Started
|
|
---------------
|
|
|
|
What you need before proceeding:
|
|
|
|
* a name for your LiveMedia, and a version number
|
|
* a Koji build target
|
|
* what architectures you want
|
|
* a flattened kickstart file
|
|
* you may also need yum repositories generated by release engineering if you
|
|
require signed packages be installed into the LiveMedia
|
|
* the permission for building these in Koji - in default installation permission
|
|
is not needed, but in production environments there will be probably some
|
|
permission or at least policy in place
|
|
|
|
The Koji command that creates a LiveMedia is called ``spin-livemedia``. It calls
|
|
out to `livemedia-creator`_ from `lorax`_ project in a mock chroot to construct
|
|
the LiveMedia. To run a LiveMedia build, use the ``spin-livemedia`` command like
|
|
so:
|
|
|
|
::
|
|
|
|
$ koji spin-livemedia --release 20200922.n.0 \
|
|
--ksurl 'git+https://pagure.io/fedora-kickstarts.git?#b4956c05028a088c641d0ce6e1a31b6d8b20176f' \
|
|
--install-tree-url 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200922.n.0/compose/Everything/$basearch/os' \
|
|
--repo 'https://kojipkgs.fedoraproject.org/compose/branched/Fedora-33-20200922.n.0/compose/Everything/$basearch/os' \
|
|
--can-fail ppc64le,aarch64 \
|
|
Fedora-Workstation-Live 33 f33 x86_64,ppc64le,aarch64 fedora-live-workstation.ks
|
|
|
|
In this example (real task is `here
|
|
<https://koji.fedoraproject.org/koji/taskinfo?taskID=52013359>`_) a LiveMedia will
|
|
be created with the N-V-R of ``Fedora-Workstation-Live-33-20200922.n.0``. If
|
|
``--release`` was not included an incrementing value would be computed by Koji
|
|
instead. spin-livemedia takes a minimum of 4 arguments:
|
|
|
|
Name
|
|
the name of the image, without versioning information. Examples could be
|
|
``Fedora-Live`` or ``Fedora-Workstation-Live``.
|
|
|
|
Version
|
|
an arbitrary version string. For Fedora images, this usually matches the
|
|
release version, such as 31, 32, or 33.
|
|
|
|
Build Target
|
|
just like RPM builds Koji must be told which target to use. This controls
|
|
what tag the image will be tagged into, and what packages are available to
|
|
install into the image.
|
|
|
|
Kickstart File
|
|
this is a recipe file that performs drives the construction of the image.
|
|
Note, that you're can either upload your kickstart or just point to some SCM
|
|
(``--ksurl``). In our example fedora repo contains
|
|
``fedora-live-workstation.ks`` kickstart file.
|
|
|
|
Use ``--help`` to discover more options to ``spin-livemedia``. You can override
|
|
the Release field with ``--release``, or ``--repo`` to override what yum repo is
|
|
used to provide packages to install to the image. Note that regardless of what
|
|
repo you use, it must contain RPMs built by Koji, otherwise you will get an
|
|
error. (unless you are building a scratch image). ``--can-fail`` options says
|
|
which architectures are optional and can fail without failing the whole build.
|
|
|
|
Mechanics
|
|
---------
|
|
|
|
The way LiveMedias are created in Koji is fairly straightforward. A chroot is
|
|
initialized and populated with the packages and their dependencies from the
|
|
``livemedia-build`` package group. Next, the kickstart file is copied into it if
|
|
it was provided from local storage. If not, it is checked out into it from an
|
|
SCM. It is then modified to use the repo associated with the build tag for the
|
|
target specified in the command unless the ``--repo`` option was given. Both the
|
|
original and the modified kickstart files are saved as part of the output for
|
|
the task for later review. A `livemedia-creator`_ command is executed using the
|
|
``mock('--chroot', ...)`` method.
|
|
|
|
.. note::
|
|
This process runs as root. This produces the desired image which is uploaded
|
|
to ``/mnt/koji/images/<image>/$imageID`` if it is not a scratch image.
|
|
|
|
.. _caveats-for-livemedias:
|
|
|
|
Caveats for LiveMedias
|
|
----------------------
|
|
|
|
There are some known caveats with using the spin-livemedia command that users
|
|
should be aware of.
|
|
|
|
%include macros in the kickstart
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
A word of caution about kickstart files and the ``%include`` macro.
|
|
`livemedia-creator`_ is smart enough to search the current directory of the
|
|
submitted kickstart file if it has ``%include`` macros. If the kickstart
|
|
specified to koji is from local storage, only that kickstart file will be copied
|
|
into the chroot, and this creates a problem if it has ``%include`` macros,
|
|
because the other kickstart files it needs will be inaccessible. This issue is
|
|
not present when the kickstart file is retrieved from a remote SCM (such as the
|
|
fedora-kickstarts git repo), because the entire repository is checked out.
|
|
Presumably it will include any other kickstart files the specified one is
|
|
including in the same directory. A workaround for the issue would be to use
|
|
``ksflatten`` (from pykickstart) on kickstart files with ``%include`` macros
|
|
that are going to be submitted to koji from the user's local disk.
|
|
|
|
Package Groups in the Kickstart File
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Package Groups in the kickstart file cause a problem if the Koji repos do not
|
|
define them, which they most likely don't since Koji's comps.xml is based on the
|
|
"groups" set up from the CLI. `livemedia-creator`_'s behavior is to ignore
|
|
package groups that are not defined in the repo it is using, so this can be
|
|
troublesome when creating the image since packages could be left out. There are
|
|
a couple possible workarounds:
|
|
|
|
* do not use package groups in the kickstart file and just specify a huge list
|
|
of packages
|
|
* use ``--repo`` and specify a repo that does have a comps.xml that defines
|
|
the groups it uses
|
|
|
|
Only Include RPMs Built in Koji
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
The image building tasks will fail if your image tries to include a package
|
|
that was not built in your build system. This is because the package does not
|
|
have any origin information stored in Koji's database. The repos defined in the
|
|
kickstart will automatically be overridden with the repo of the build tag for
|
|
the build target, unless you use the ``--repo`` option. Since only packages
|
|
you have built (or include from an external repo) should be there, you should
|
|
never have this problem unless you use ``--repo``.
|
|
|
|
No Signed or Debuginfo RPMs in Koji's Build Tags
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
If you need signed RPMs or debuginfo RPMs, you will run into trouble because
|
|
Koji does not keep those in its build tag repos. The only work around for this
|
|
is to create a repo yourself that includes these RPMs and then use ``--repo``.
|
|
This will force the image to take RPMs from that repo. Remember, the task will
|
|
fail if Koji detects RPMs were installed that were not built in the build
|
|
system.
|
|
|
|
%post Section in Kickstart
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
While `livemedia-creator`_ does support building on SELinux disabled hosts, you
|
|
can run into denials when booting if you create and use new files in the
|
|
``%post`` section of your kickstart file. If you do, you should either set the
|
|
labels appropriately at the end of the ``%post`` section, or instigate an
|
|
autorelabel at boot time.
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
If your build fails, you will be notified on the command line. In the output
|
|
will be a URL to the Koji UI, visit that and click on the red subtask link.
|
|
From that page review ``root.log`` and ``livemedia-out.log`` for errors. Often
|
|
errors are caused by packages being missing, or malformed kickstart files. The
|
|
log files are at the bottom of the page. If problem occurs later during
|
|
installation `livemedia-creator`_ will also upload a lot of other logs like
|
|
anaconda's, etc. If you're stuck, contact Release Engineering.
|
|
|
|
Build System Preparation
|
|
------------------------
|
|
|
|
This section assumes you have know-how required to install and configure a new
|
|
instance of Koji, and that you have already done so. You can learn how to do so
|
|
:doc:`here <server_howto>` if you need to. Please ensure you are using the
|
|
latest version of the software and that your database schema is updated as
|
|
well. You should also have some familiarity with how `livemedia-creator`_ works.
|
|
This section only covers preparation for LiveMedia builds.
|
|
|
|
Follow this procedure step by step to get things prepared they way they need to
|
|
be.
|
|
|
|
#. ``koji add-host-to-channel <your-host> livemedia``
|
|
add a builder to the livemedia channel
|
|
#. ``koji grant-permission livemedia <user>``
|
|
grant the permission to build an image type to a user. This step is
|
|
optional since admins have all permissions.
|
|
#. You will need a tag and target to build the images from. The yum repo
|
|
generated for the build tag of the target is what Koji will use to populate
|
|
the LiveMedias with by default. (the alternative is to use the ``--repo``
|
|
option, more on that later)
|
|
#. ``koji add-group <build-tag> livemedia-build``
|
|
add the livemedia-build group
|
|
#. ``koji add-group-pkg <build-tag> livemedia-build <pkg> ...``
|
|
add packages to the livemedia-build group. These package lists vary has
|
|
packages and dependencies change. As of September, 2020 for Fedora 33 the
|
|
needed packages for each image type are:
|
|
|
|
* bash, coreutils, glibc-all-langpacks, lorax-lmc-novirt,
|
|
selinux-policy-targeted, shadow-utils, util-linux
|
|
|
|
|
|
Building Disk Images
|
|
====================
|
|
|
|
Disk images are files that represent virtual disks. They have a partition table
|
|
and filesystems on them, and are available in a variety of formats: qcow2,
|
|
vmdk, ova, Hyper-V, raw, "base" container images, and more.
|
|
|
|
Getting Started
|
|
---------------
|
|
|
|
What you need:
|
|
|
|
* a name for your image, and a version number
|
|
* what architectures you want
|
|
* a Koji build target
|
|
* kickstart file
|
|
* installation tree
|
|
* you may also need yum repositories generated by Rel-Eng if you require signed
|
|
packages be installed into the image
|
|
|
|
The Koji command to build a disk image is called ``image-build``. The
|
|
``image-build`` command uses `ImageFactory`_ and `Oz`_ to start a VM guest and
|
|
perform an automated Anaconda installation. Here is a (lengthy) example for
|
|
building a disk image.
|
|
|
|
::
|
|
|
|
$ koji image-build --repo 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' --kickstart fedora-server.ks --scratch --distro Fedora-22 --format qcow2 fedora-server-kvm 22 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/' x86_64
|
|
|
|
This example builds a scratch qcow2 disk image using packages from an
|
|
additional yum repository. Without this option the yum repo to populate the
|
|
build root would be used instead. If this was the first image with the N-V of
|
|
fedora-server-kvm-22, then the N-V-R would be fedora-server-kvm-22-1, because
|
|
Koji uses an incrementing number for the release if you do not provide one.
|
|
Like all Koji commands, use ``--help`` to see more options that are available.
|
|
|
|
For Docker, Koji only supports Base Images right now using a kickstart file as
|
|
described above. In the future it will support layered images, but not before
|
|
some Docker requirements are met, and Koji is maintaining a Registry of its
|
|
own. This scoping effort is ongoing.
|
|
|
|
``image-build`` takes a minimum of 5 positional arguments, and 2 options must
|
|
be specified. They are reviewed in the list below, with the positional
|
|
arguments first.
|
|
|
|
Name
|
|
the name of the image, without versioning information. Examples could be
|
|
``fedora-server`` or ``fedora-workstation``.
|
|
|
|
Version
|
|
an arbitrary version string. For Fedora images, this usually matches the
|
|
release version, such as 22 or 23.
|
|
|
|
Build Target
|
|
just like RPM builds Koji must be told which target to use. This controls
|
|
what tag the image will be tagged into, and what packages are available to
|
|
install into the image.
|
|
|
|
Installation Tree URL
|
|
this is a URL to a location you can install an operating system from. It is
|
|
the same place you would direct a PXE-booted system to go. In 99% of cases
|
|
this location is provided by Release Engineering. It should have an
|
|
"isolinux" subdirectory and yum metadata somewhere within.
|
|
|
|
Architecture
|
|
only x86_64 or i386 is supported, and you can specify both on the command
|
|
line. This will cause two subtasks to be run, allowing you to build for
|
|
both arches in parallel. If either fail, the whole build will fail.
|
|
|
|
Kickstart File
|
|
this is a recipe file that performs drives the construction of the image.
|
|
Pass in the path to a kickstart file, which must be flattened.
|
|
|
|
Kickstart URL
|
|
in a non-scratch build, you'll need this too. For more details, see the
|
|
Getting your Kickstart to Koji section.
|
|
|
|
Distro
|
|
a string that indicates what OS is being built. These always follow the
|
|
convention of "Fedora-X", where X is the release number.
|
|
|
|
Since this command can get very long, a configuration file can drive the task
|
|
as well, using the ``--config`` option. It accepts a path to a configuration
|
|
file written in the Python ConfigParser format (like a Windows .ini). The
|
|
options are all named the same with one caveat, see below. Here's what one
|
|
could look like:
|
|
|
|
::
|
|
|
|
[image-build]
|
|
name = fedora-server-docker
|
|
version = 22
|
|
target = f22-candidate
|
|
install_tree = https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
|
|
arches = x86_64
|
|
|
|
format = qcow2,rhevm-ova,vsphere-ova
|
|
distro = Fedora-22
|
|
repo = https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/
|
|
disk_size = 20
|
|
|
|
ksversion = DEVEL
|
|
kickstart = fedora-22-server-docker.ks
|
|
ksurl = git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7
|
|
specfile = git://git.fedorahosted.org/git/spin-kickstarts.git?spec_templates/fedora22#68c40eb7
|
|
|
|
A few notes on the syntax:
|
|
|
|
* it allows for comments too, the lines start with a hash (#)
|
|
* options on the command line that can be used multiple times can accept
|
|
values here as comma-separated strings
|
|
* options with a hyphen need to use an underscore instead. ``--disk-size`` for
|
|
example would be ``disk_size`` in the config file
|
|
|
|
OVA Features
|
|
------------
|
|
|
|
If you're building OVAs, either for RHEVM or vSphere, you can specify OVA
|
|
options with a special section in the configuration file. It looks something
|
|
like this:
|
|
|
|
::
|
|
|
|
[ova-options]
|
|
vsphere_product_version=22
|
|
rhevm_description=Fedora Cloud 22
|
|
vsphere_product_vendor_name=Fedora Project
|
|
ovf_memory_mb=6144
|
|
rhevm_default_display_type=1
|
|
vsphere_product_name=Fedora Cloud 22
|
|
ovf_cpu_count=4
|
|
rhevm_os_descriptor=Fedora-22
|
|
|
|
or this:
|
|
|
|
::
|
|
|
|
[ova-options]
|
|
vsphere_ova_format = vagrant-virtualbox
|
|
rhevm_ova_format = vagrant-libvirt
|
|
vagrant_sync_directory = /home/vagrant/sync
|
|
|
|
The second one is actually the secret sauce for generating an image for use in
|
|
Vagrant. At this time, you would need rename the image file extension from .ova
|
|
to .box, but otherwise this should work fine.
|
|
|
|
Kickstart Preparation
|
|
---------------------
|
|
|
|
Kickstarts for the image-build command have some specific requirements which
|
|
are covered in this section.
|
|
|
|
Required Kickstart Arguments
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Anaconda of course requires many commands to be defined in the kickstart file.
|
|
If you're starting from scratch you should review the reference linked above,
|
|
or use an existing kickstart file in the spin-kickstarts git repo. It is
|
|
critically important that the installation be completely automated, if
|
|
Anaconda has to prompt for input for any reason, the build will fail because
|
|
you cannot send input to the guest. Some of the kickstart commands are optional
|
|
to Anaconda, but are required in Koji for your build to succeed. Here's the
|
|
list and the reasons why.
|
|
|
|
zerombr
|
|
You must tell Anaconda to wipe out the MBR in the virtual block device, if
|
|
you don't Anaconda will ask you.
|
|
|
|
clearpart --all --initlabel
|
|
Anaconda has to be told to wipe out all data on the virtual block device we
|
|
install on otherwise it will ask for confirmation to do so. Since it is
|
|
blank anyway this is harmless.
|
|
|
|
reboot
|
|
When the installation completes, the guest is rebooted. `ImageFactory`_ is
|
|
specifically looking for this behavior to conclude the installation
|
|
completed. Anaconda's default behavior is to wait for a key press to reboot
|
|
the system, but this is impossible from outside of Koji.
|
|
|
|
locking the root account
|
|
You have to lock the root account (rootpw --lock) or create a non-root user
|
|
(user), otherwise Anaconda will prompt for one.
|
|
|
|
Do not use the url command
|
|
The repo commands are overridden by Koji to point to internal Koji repos,
|
|
or what you specified on the command line with ``--repo``, it does not
|
|
override the url command if you provided it. Anaconda has a behavior where
|
|
it will prefer packages from the repositories given with the url command
|
|
over those with the repo command, and this is generally not what you want.
|
|
If Koji sees an RPM was installed that was not built in the system, it will
|
|
fail the build.
|
|
|
|
Recommended Kickstart Arguments
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Often you want a ``%post`` section in your kickstart to perform
|
|
post-installation configuration steps. Review that section of the reference
|
|
and note that you can specify ``--log`` and ``--interpreter``. Both of these
|
|
are recommended (but not required) to assist with the development and debugging
|
|
process. Here are some other recommendations:
|
|
|
|
* You probably want the network to use dhcp, sshd to be started, and port 22
|
|
opened in the firewall to allow access as well.
|
|
* If you're building an image that will be shipped with a product, SELinux
|
|
should be enabled.
|
|
* Images that will be used in cloud deployments like OpenStack or EC2 should
|
|
have ``cloud-init`` in the package list.
|
|
* It is discouraged to have root passwords in plaintext in your kickstart file.
|
|
* If your %post section is written in bash, consider setting -x.
|
|
* For images that have multiple partitions, use the ``--asprimary`` option for
|
|
the part command that defines the root file system. This will ensure it is
|
|
the first partition on the image, which is a requirement in some cloud
|
|
environments like EC2.
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
If your disk image build fails, follow the link in the command line output
|
|
that takes you to the task page in the Koji web UI. Click on the failed
|
|
createImage subtask in red. On that page review the screenshot.ppm file if it
|
|
was provided, or oz.log. Most failures are from Anaconda rejecting a malformed
|
|
kickstart file, which will be indicated in the screenshot. Your installation
|
|
must be completely automatic, there can be no interactivity at all, otherwise
|
|
Anaconda will sit there indefinitely until Koji (actually ImageFactory) kills
|
|
the task.
|
|
|
|
It is very easy to write a kickstart file with bugs or that results in a system
|
|
that does not boot. This section will present a series of questions to ask
|
|
yourself and examples to help diagnose where the problem lies. Once you know
|
|
that, it should be easier to understand what you can do to inspect further.
|
|
|
|
There are 4 steps in the process:
|
|
|
|
#. create a guest
|
|
#. perform an automated installation in the guest
|
|
#. boot the guest and extract the list of installed RPMs
|
|
#. upload and archive the disk image of the guest
|
|
|
|
Is it a problem with guest creation?
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
There have been unusual cases where libvirt, ImageFactory, or Oz was
|
|
misconfigured and guests could not be started properly. A misconfiguration with
|
|
Puppet or whatever Fedora Infrastructure is up to can cause this. So far the
|
|
errors have been clear in the task output, look either in the results string
|
|
or oz.log. The bad news is that in this case you really can only inform Rel-Eng
|
|
about the issue and wait for a resolution. The good news is these cases are
|
|
very rare.
|
|
|
|
Did the installation fail?
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
The Anaconda installation can fail for many reasons: missing packages, network
|
|
problems, or syntax errors in %post. Tasks will also fail if Anaconda prompts
|
|
for input for any reason. If Koji detects a lack of disk activity in the guest
|
|
for more than 5 minutes, it will fail the build and tear down the guest.
|
|
Looking in oz.log may have the answer: dracut, anaconda, and yum logs are all
|
|
printed there.
|
|
|
|
These sorts of failures often have a screenshot taken and saved with the task
|
|
output called screenshot.ppm. Viewing this will usually tell you what Anaconda
|
|
is complaining about if the installer detected an issue or prompted for input.
|
|
The string in the results output that says "No disk activity in 300 seconds,
|
|
failing." This almost always means Anaconda hit an issue and either gave up or
|
|
waited.
|
|
|
|
If Anaconda claims it is missing packages, confirm they exist in the repos you
|
|
are using with ``--repo``, if you are using that option. If you are not,
|
|
confirm the builds you expect are in the tag inheritance for the target you are
|
|
running. This is a lot like checking whether an RPM will build against the
|
|
right libraries, except we're building an image instead.
|
|
|
|
If you get the rare Anaconda dialog box that says something like "An unexpected
|
|
error occurred", try using the ``text`` command in kickstart, which will have
|
|
Anaconda boot in text mode. Sometimes the Python traceback (or whatever the
|
|
error condition is) will be printed there. I have also seen cases where
|
|
text-mode yields a black screen, but booting in graphical mode (the default)
|
|
does produce a useful dialog box. Issues like this stem from syntax errors in
|
|
the kickstart file, or bugs in pykickstart itself. If you think it is a
|
|
pykickstart bug, then someone in Rel-Eng needs to update pykickstart on the
|
|
builders.
|
|
|
|
Did the guest boot?
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
Koji waits 5 minutes for a guest to boot in this step. It unfortunately does
|
|
not give a lot of insight to why a guest may not boot, so these are a tougher
|
|
class of issues to work through. You can usually answer this question by
|
|
looking in results string. If you see "Timed out waiting for guest to boot",
|
|
then this is your problem. You can also confirm this in oz.log.
|
|
|
|
For now, the best way to investigate an issue like this is to drive a guest
|
|
installation locally using something like Gnome's Virtual Machine Manager
|
|
(VMM). The steps to perform are:
|
|
|
|
* Select a Network Install
|
|
* For the Operating System Install URL use the same one you gave to Koji. It
|
|
will be something like
|
|
https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/
|
|
* Set the Kickstart URL to where your kickstart file is. You may need to make
|
|
it available over http.
|
|
* Bump the memory to 2048M for good measure
|
|
* Launch the guest and let it complete installation
|
|
* Open a VNC session and watch what happens when the guest attempts to boot.
|
|
|
|
If the console is not providing enough information, we have to get more
|
|
creative. Anaconda supports starting an SSH daemon while the installation is
|
|
happening with the sshpw command in kickstart. Set that and comment out the
|
|
reboot command. This will let the installation complete locally and wait for a
|
|
keystroke to reboot the guest. At this point you should be able to ssh in and
|
|
inspect the environment to figure out what is going on. You should also
|
|
consider making use of the ``--log`` option to %post so that output from the
|
|
script is saved somewhere.
|
|
|
|
Another option would be to scp logs and other files off of the guest as part of
|
|
the ``%post`` script.
|
|
|
|
Other Guest Misconfigurations
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
If the guest boots but you're having problems accessing it I'd suggest
|
|
following same procedure as when the guest fails to boot. This could be a
|
|
result of firewall misconfigurations or SSH not being available for some
|
|
reason. Usually in this case the build is succeeding in Koji, but there's
|
|
something still fundamentally broken in the image. If the issue is something
|
|
you can investigate while the guest is online (you can log in), then I'd
|
|
suggest importing it locally using the libvirt.xml and the disk image provided
|
|
in Koji's task output.
|
|
|
|
You can also do investigative work in an offline mode by mounting the image
|
|
locally or using something like libguestfs to poke around without starting the
|
|
guest. The fast, dirty way to do it is by mounting it. This can often pollute
|
|
your guest environment. Here's how to do it:
|
|
|
|
* Download the image from Koji
|
|
* If the image format is not raw, you have to convert it first with qemu-img.
|
|
Something like:
|
|
::
|
|
|
|
$ qemu-img convert -O raw <image-file> <output-file></pre>
|
|
|
|
* Now mount it up using loopback devices. (as root) If your image has multiple
|
|
partitions in it, you may need to pass in a different mapped loopback device
|
|
like ``loop0p2``. Whichever one you think is the root partition or has the
|
|
issue you're trying to fix.
|
|
::
|
|
|
|
# kpartx -av <raw-image>
|
|
# mount -o loop /dev/mapper/loop0p1 /mnt/my_directory
|
|
|
|
Hopefully at this point you figure out the issue. To tear down the image you'll
|
|
run commands as root like so:
|
|
|
|
::
|
|
|
|
# umount /mnt/my_directory
|
|
# dmsetup remove loop0p1
|
|
# losetup -d /dev/loop0
|
|
|
|
Again, if you used different loopback devices, substitute those in to the
|
|
dmsetup and losetup commands.
|
|
|
|
Build System Preparation
|
|
------------------------
|
|
|
|
Follow this guide if you're a Koji admin and would like to enable image
|
|
building or want to set up some testing before enabling the integration.
|
|
|
|
When moving to ImageFactory to do image builds Koji lost the ability to easily
|
|
reproduce the build environment for images the way we do for RPMs using Mock.
|
|
This section will document how to set up an image builder for Koji. This is a
|
|
lengthy task, it will take first-timers about a week to have a useful instance,
|
|
and it is painful because it requires a bare metal system be provisioned since
|
|
`ImageFactory`_ provisions VMs to build the image. There is a significant
|
|
performance penalty for using nested virtualization.
|
|
|
|
Follow the steps below to set up your builder.
|
|
|
|
.. note::
|
|
You do not have to stand up a complete Koji instance to test the way Koji
|
|
builds images. However, if you want to test image builds with an accurate
|
|
representation of how Koji does it, or you want to test code changes
|
|
related to image builds in Koji, you should follow all the steps below.
|
|
|
|
ImageFactory/Oz Preparation
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
#. Provision a system with at least 4G of memory with the current release of
|
|
RHEL 6 or later. Sometimes builders lag behind a month or two before taking
|
|
in updates, but this will still get you pretty close to where you want to
|
|
be. For Fedora, use the latest release.
|
|
#. Install the following packages to the builder.
|
|
#. oz
|
|
#. imagefactory
|
|
#. imagefactory-plugins-TinMan
|
|
#. imagefactory-plugins-vSphere
|
|
#. imagefactory-plugins-ovfcommon
|
|
#. imagefactory-plugins-docker
|
|
#. imagefactory-plugins
|
|
#. imagefactory-plugins-OVA
|
|
#. imagefactory-plugins-RHEVM
|
|
#. python-psphere => 0.5
|
|
#. VMDKStream => 0.2
|
|
#. pykickstart
|
|
#. Edit ``/etc/kojid/kojid.conf``, and set an second value, eg: 7200 for
|
|
``oz_install_timeout``. It's a timeout waiting guest installing. Default
|
|
value is 0, that means oz will use its default value. Since ``oz-0.16.0``,
|
|
it can be configured in ``/etc/oz/oz.cfg`` as ``install`` in ``[timeouts]``
|
|
section.
|
|
#. Edit ``/etc/oz/oz.cfg``, and set the memory value in the ``[libvirt]``
|
|
section to at least 2048. Set ``safe_generation`` under ``[icicle]`` to yes.
|
|
#. Run: ``mkdir -p ~root/.psphere/templates``, and then copy the following
|
|
code into ``~root/.psphere/config.yaml``. Do not worry about the server,
|
|
username, and password credentials; they are not used anywhere.
|
|
|
|
::
|
|
|
|
general:
|
|
server: 10.16.120.224
|
|
username: Administrator
|
|
password: whatever
|
|
template_dir: ~/.psphere/templates/
|
|
logging:
|
|
destination: ~/.psphere/psphere.log
|
|
level: DEBUG # DEBUG, INFO, etc
|
|
|
|
Start up the services and ImageFactory/Oz should be ready to go. You should
|
|
read more about `how to use Oz`_ and `how to use ImageFactory`_. If you want
|
|
to try calling `ImageFactory`_ as if from a Koji Builder (but not set up a
|
|
whole Koji instance), you can use the code below to emulate that. If you want
|
|
to test the Koji integration with a full Koji instance, proceed to the next
|
|
section instead.
|
|
|
|
::
|
|
|
|
#!/usr/bin/python -tt
|
|
|
|
import logging
|
|
import os.path
|
|
import random
|
|
import sys
|
|
|
|
from imgfac.BuildDispatcher import BuildDispatcher
|
|
from imgfac.PluginManager import PluginManager
|
|
from imgfac.ReservationManager import ReservationManager
|
|
plugin_mgr = PluginManager('/etc/imagefactory/plugins.d')
|
|
plugin_mgr.load()
|
|
from imgfac.ApplicationConfiguration import ApplicationConfiguration
|
|
|
|
# logging
|
|
handler = logging.StreamHandler(sys.stdout)
|
|
tlog = logging.getLogger()
|
|
tlog.setLevel(logging.DEBUG)
|
|
tlog.addHandler(handler)
|
|
|
|
# configuration
|
|
ks = open('oztest.ks').read()
|
|
workdir = '/tmp/koji/test'
|
|
config = {
|
|
#Oz specific
|
|
'oz_data_dir': os.path.join(workdir, 'oz_data'),
|
|
'oz_screenshot_dir': os.path.join(workdir, 'oz_screenshots'),
|
|
#IF specific
|
|
'imgdir': os.path.join(workdir, 'scratch_images'),
|
|
'tmpdir': os.path.join(workdir, 'oz-tmp'),
|
|
'verbose' : True,
|
|
'timeout': 3600,
|
|
'output': 'log',
|
|
'raw': False,
|
|
'debug': True,
|
|
'image_manager': 'file',
|
|
'plugins': '/etc/imagefactory/plugins.d',
|
|
'tdl_require_root_pw': False,
|
|
'image_manager_args': {
|
|
'storage_path': os.path.join(workdir, 'output_image')},
|
|
}
|
|
random.seed() # necessary to ensure a unique mac address
|
|
rm = ReservationManager()
|
|
rm._listen_port = random.randint(rm.MIN_PORT, rm.MAX_PORT)
|
|
ApplicationConfiguration(configuration=config)
|
|
params = {'install_script': ks}
|
|
template = """<template>
|
|
<name>test-appliance</name>
|
|
<os>
|
|
<name>Fedora</name>
|
|
<version>22</version>
|
|
<arch>x86_64</arch>
|
|
<install type='url'>
|
|
<url>https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/x86_64/os/</url>
|
|
</install>
|
|
<icicle>
|
|
<extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>
|
|
</icicle>
|
|
</os>
|
|
<description>test-appliance OS</description>
|
|
<disk>
|
|
<size>16G</size>
|
|
</disk>
|
|
</template>
|
|
"""
|
|
bd = BuildDispatcher()
|
|
|
|
# build the image
|
|
base = bd.builder_for_base_image(template, parameters=params)
|
|
base.base_thread.join()
|
|
tlog.removeHandler(handler)
|
|
|
|
This script is run as root with no arguments. It uses ``oztest.ks`` in your
|
|
local directory as the kickstart you want to try to use. The URL in the XML
|
|
template is where the RPM packages will be installed from, and what the guest
|
|
will be booted with.
|
|
|
|
Koji Preparation
|
|
^^^^^^^^^^^^^^^^
|
|
|
|
#. :doc:`Install Koji <server_howto>` if you need it. This will easily be the
|
|
most time-consuming part of the process; my first time took 3 days to get
|
|
it working properly. Follow the guide closely, and go with the SSL
|
|
authentication method. SSL is a lot easier to set up locally. You will need
|
|
to install every Koji component (except koji-vmd) on the same system.
|
|
Proceed to the next step after you've had a successful Kojira repository
|
|
generated.
|
|
#. At this point, you have a system that should be ready to build images. We
|
|
just have to do some Koji configuration so that your instance is pulling
|
|
content from Koji. Replace the base tag names with whatever fits your
|
|
conventions.
|
|
|
|
#. Add tags, tag inheritance, new pkg (with pkg owner), new external repo, and regen the repo
|
|
|
|
::
|
|
|
|
koji add-tag fedora22
|
|
koji add-tag jay-fedora22
|
|
koji add-tag jay-fedora22-override --parent jay-1-fedora22
|
|
koji add-tag jay-fedora22-build --arches x86_64 --parent jay-fedora22-override
|
|
koji add-tag jay-fedora22-candidate --parent jay-fedora22
|
|
koji add-tag-inheritance --priority 40 jay-fedora22-build fedora22
|
|
koji add-pkg --owner kojiadmin jay-fedora22 fedora-server-ec2 fedora-server-kvm
|
|
koji add-external-repo -t fedora::20 fedora22 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'
|
|
koji add-target jay-fedora22-candidate jay-fedora22-build
|
|
koji regen-repo jay-fedora22-build
|
|
#. Grab a kickstart file from an image task in Koji that relates to what you want to test.
|
|
#. Finally, kick off a build!
|
|
|
|
::
|
|
|
|
koji image-build fedora-server-ec2 22 --distro Fedora-22 jay-fedora22-candidate --kickstart fedora-server-starter-ec2.ks 'https://alt.fedoraproject.org/pub/alt/releases/22/Cloud/$arch/os/'
|
|
|
|
|
|
Building Appliances
|
|
===================
|
|
|
|
This section is here for the sake of legacy. Unless you are trying to build
|
|
ARM images, you should use the image-build command described in the previous
|
|
section.
|
|
|
|
.. note::
|
|
The spin-appliance command, described herein, is deprecated.
|
|
|
|
Getting Started
|
|
---------------
|
|
|
|
Here's what you need before proceeding:
|
|
|
|
* a name for your Appliance, and a version number
|
|
* a Koji build target
|
|
* what architectures you want
|
|
* a flattened kickstart file
|
|
* you may also need yum repositories generated by release engineering if you
|
|
require signed packages be installed into the appliance
|
|
* the appliance permission in Koji
|
|
|
|
The Koji command that creates an Appliance is called ``spin-appliance``. It
|
|
calls out to appliance-creator in a mock chroot to construct the Appliance.
|
|
To run an Appliance build, use the spin-appliance command like so:
|
|
|
|
::
|
|
|
|
$ koji spin-appliance --release 4 fedora-workstation 23 f23-build fedora-workstation.ks
|
|
|
|
In this example an Appliance will be created with the N-V-R of
|
|
``fedora-workstation-23-4``. If ``--release`` was not included an incrementing
|
|
value would be computed by Koji instead. spin-appliance takes a minimum of 5
|
|
arguments:
|
|
|
|
Name
|
|
the name of the image, without versioning information. Examples could be
|
|
``fedora`` or ``fedora-workstation``.
|
|
|
|
Version
|
|
an arbitrary version string. For Fedora images, this usually matches the
|
|
release version, such as 21, 22, or 23.
|
|
|
|
Build Target
|
|
just like RPM builds Koji must be told which target to use. This controls
|
|
what tag the image will be tagged into, and what packages are available to
|
|
install into the image.
|
|
|
|
Architecture
|
|
only arm, x86_64, or i386 are supported
|
|
|
|
Kickstart File
|
|
this is a recipe file that performs drives the construction of the image.
|
|
|
|
Use ``--help`` to discover more options to spin-appliance. You can override the
|
|
Release field with ``--release``, or ``--repo`` to override what yum repo is
|
|
used to provide packages to install to the image. Note that regardless of what
|
|
repo you use, it must contain RPMs built by Koji, otherwise you will get an
|
|
error. (unless you are building a scratch image)
|
|
|
|
Mechanics
|
|
---------
|
|
|
|
The way Appliances are created in Koji is the same as :ref:`building-livemedia`.
|
|
|
|
Caveats for Appliances
|
|
----------------------
|
|
|
|
Caveats for using ``spin-appliance`` are the same as using ``spin-livemedia`` to
|
|
:ref:`caveats-for-livemedias`.
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
If your build fails, you will be notified on the command line. In the output
|
|
will be a URL to the Koji UI, visit that and click on the red subtask link.
|
|
From that page review ``root.log`` and ``appliance.log`` for errors. Often
|
|
errors are caused by packages being missing, or malformed kickstart files. The
|
|
log files are at the bottom of the page. If you're stuck, contact Release
|
|
Engineering.
|
|
|
|
Build System Preparation
|
|
------------------------
|
|
|
|
This section assumes you have know-how required to install and configure a new
|
|
instance of Koji, and that you have already done so. You can learn how to do so
|
|
:doc:`here <server_howto>` if you need to. Please ensure you are using the
|
|
latest version of the software and that your database schema is updated as
|
|
well. You should also have some familiarity with how `appliance-creator`_
|
|
works. This section only covers preparation for Appliance builds.
|
|
|
|
Follow this procedure step by step to get things prepared they way they need
|
|
to be.
|
|
|
|
#. Add a builder to the appliance channel
|
|
::
|
|
|
|
koji add-host-to-channel <your-host> appliance
|
|
|
|
#. Grant the permission to build an appliance to a user. This step is optional
|
|
since admins have all permissions.
|
|
|
|
::
|
|
|
|
koji grant-permission appliance <user>
|
|
|
|
#. You will need a tag and target to build the images from. The yum repo
|
|
generated for the build tag of the target is what Koji will use to populate
|
|
the Appliances with by default. (the alternative is to use the ``--repo``
|
|
option, more on that later)
|
|
|
|
#. Add the appliance-build group
|
|
|
|
::
|
|
|
|
koji add-group <build-tag> appliance-build``
|
|
|
|
#. Add packages to the appliance-build group. These package lists vary has
|
|
packages and dependencies change. As of October, 2015 for Fedora 24 the
|
|
needed packages for appliances:
|
|
|
|
* appliance-tools, bash, coreutils, grub, parted, perl, policycoreutils,
|
|
selinux-policy, shadow-utils, sssd-client
|
|
|
|
::
|
|
|
|
koji add-group-pkg <build-tag> appliance-build <pkg> ...
|
|
|
|
.. _Koji project website: https://fedorahosted.org/koji/wiki
|
|
.. _kickstart:
|
|
https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst
|
|
.. _Anaconda Kickstart Guide:
|
|
https://github.com/rhinstaller/pykickstart/blob/master/docs/kickstart-docs.rst
|
|
.. _lorax: https://github.com/weldr/lorax
|
|
.. _livemedia-creator: https://weldr.io/lorax/livemedia-creator.html
|
|
.. _ImageFactory: http://imgfac.org/
|
|
.. _Oz: https://github.com/clalancette/oz
|
|
.. _how to use Oz: https://github.com/clalancette/oz/wiki
|
|
.. _how to use ImageFactory: http://imgfac.org/documentation/
|
|
.. _appliance-creator: https://fedoraproject.org/wiki/Features/ApplianceTools
|