Commit graph

512 commits

Author SHA1 Message Date
Lubomír Sedlář
e02ed6c04b Merge #434 Replace mount/umount cmds with libguestfs tools 2016-10-24 07:52:29 +00:00
Dennis Gilmore
18128623b4 Merge #443 config: Don't abort on deprecated options 2016-10-20 20:26:17 +00:00
Dennis Gilmore
8f974ea109 Merge #442 metadata: Treeinfo should point to packages and repo 2016-10-20 20:25:17 +00:00
Qixiang Wan
a5273dc798 Replace mount/umount with guestfsmount and 'fusermount -u'
'mount -o loop' requires root privileges, guestmount from
libguestfs-tools-c package can work without root privileges.

Fixes: #19
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-10-20 19:38:35 +08:00
Lubomír Sedlář
c38bb4809b config: Don't abort on deprecated options
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.

The validation script still rejects configuration files with these
removed keys.

This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).

Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:51:12 +02:00
Lubomír Sedlář
1bb1c2ba28 metadata: Treeinfo should point to packages and repo
Each variant section should contain a path to a directory with packages
and to repodata. There was code to do this, but due to a typo it did not
actually work.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:05:48 +02:00
Lubomír Sedlář
74aa41f8bd Send notification when compose fails to start
This is tricky as this early in the process we don't know the compose
ID. The new message gives the full command line that was called.

Relates: #439
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 09:39:18 +02:00
Lubomír Sedlář
6e55cc6419 metadata: Stop crashing for non-bootable products
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 09:25:39 +02:00
Dennis Gilmore
a7dc4e80a7 Merge #433 createiso: Do not split bootable media 2016-10-19 16:37:25 +00:00
Dennis Gilmore
2f184f9b23 Merge #432 metadata: Correctly clone buildinstall .treeinfo 2016-10-19 16:36:45 +00:00
Lubomír Sedlář
84fcc0083f createiso: Do not split bootable media
If the media is bootable, we can not split it. Instead we will create an
ISO that is too big and issue a warning (aborting the whole compose
would be too much).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-19 18:36:19 +02:00
Dennis Gilmore
ff6c952094 Merge #430 createiso: Include layered product name in iso name 2016-10-19 16:36:07 +00:00
Dennis Gilmore
8194dd0cf0 Merge #428 buildinstall: Only transform arch for lorax 2016-10-19 16:35:03 +00:00
Dennis Gilmore
34d3be35dc Merge #426 iso-wrapper: Remove the class 2016-10-19 16:34:25 +00:00
Lubomír Sedlář
1f4f6ceb08 metadata: Correctly clone buildinstall .treeinfo
Lorax/buildinstall produce .treeinfo file that is cloned into the
compose dir. However since lorax runs separately for each arch, the
files are nested in a subdirectory. With old buildinstall method, this
causes the file to not be found and copied.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-18 09:01:30 +02:00
Lubomír Sedlář
b46af7acbb createiso: Include layered product name in iso name
When creating an ISO for a layered product, the name of the integrated
product should be included in the file name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-17 15:15:50 +02:00
Lubomír Sedlář
02ddcf7387 buildinstall: Only transform arch for lorax
The older buildinstall method fails if it sees ppc64p7 instead of ppc64.
Nothing changes for composes using Lorax.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-17 13:20:57 +02:00
Lubomír Sedlář
e58c78fd2f iso-wrapper: Remove the class
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-14 09:14:51 +02:00
Lubomír Sedlář
50e9294057 config: Validate variant regular expressions
When the mapping in configuration specifies incorrect regular expression
to match regular expressions, we should raise an error immediately and
not wait until the part of config is actually used.

This patch does not cover `live_media`, `image_build` and `osbs`
sections, as they use plain dicts and not the list of tuples format.

Fixes: #424
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-12 09:38:59 +02:00
Mark Montague
4be3c72bd5 pungi: Replace kickstart repo url
Old Pungi should replace yum baseurl/mirrorlist when doing buildinstall.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1264570
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-07 07:56:06 +02:00
Lubomír Sedlář
32059c57c0 ostree-installer: Generate correct volume ID
The volume ID should be generated in Pungi with the same format as for
other images.

Fixes: #419
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 09:17:09 +02:00
Lubomír Sedlář
cc559b12f4 ostree-installer: Use ostree as type in filename
Instead of overloading the `dvd` value, use new value `ostree` that can
be overridden by a config change.

Fixes: https://pagure.io/pungi/issue/418
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 08:41:14 +02:00
Lubomír Sedlář
8880b1ea8a ostree: Use $basearch in repo file
Fixes: #413
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-05 18:41:55 +02:00
Lubomír Sedlář
446f21d4d7 config: Accept empty branch in SCM dict
If user sets `branch` to `None`, it should behave as if the branch is
not set at all. Nagging them about removing it is not helping anything.

Fixes: #415
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-03 19:37:56 +02:00
Dennis Gilmore
4e90f81fc1 use --new-chroot when making ostree's
to complement https://pagure.io/koji/pull-request/162 we need to adjust pungi
rpm-ostree uses bublewrap that does not work in mock. --new-chroot to mock
enables the use of systemd-nspawn instead of chroot resulting in working
rpm-ostree again

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-09-28 20:58:17 -05:00
Dennis Gilmore
c0474b6cca Merge #404 Remove FSF address from file headers 2016-09-27 13:33:43 +00:00
Lubomír Sedlář
7bf12636a0 Translate paths without double slash
If the path in `translate_paths` config ends with a slash, we would
create public path with double slash.

Fixes: https://pagure.io/pungi/issue/408
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-26 15:50:49 +02:00
Lubomír Sedlář
c5f878330c Remove shebangs from non-executable files
Some modules can be executed as a sort-of test. However, the files do
not have executable bit set, so there is no need for them to have
shebangs. If someone wants to call them directly, they should do so via
python.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-23 10:26:43 +02:00
Lubomír Sedlář
d6dc269ef4 Remove FSF address from comments
The address is no longer correct. We can just as well simply point to
the web page describing the license.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-23 10:26:43 +02:00
Lubomír Sedlář
a042906717 init: Remove keep_original_comps option
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.

The config validation is updated to give a hint on how to remove the
option from the configuration.

Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-22 13:35:28 +02:00
Lubomír Sedlář
2ac45a783d ostree_installer: Add --isfinal lorax argument
If the compose is supported, we need to tell lorax to use the option.

Fixes: #399
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 13:49:35 +02:00
Dennis Gilmore
3a68a93532 Merge #385 Test and clean up pungi.linker 2016-09-19 21:27:57 +00:00
Dennis Gilmore
8d1b4dea17 Merge #390 checksums: Never skip checksumming phase 2016-09-19 21:27:10 +00:00
Lubomír Sedlář
6fede3f395 variants: Allow multiple explicit optional variants
When a variant has optional, it is possible to explictly list it in the
XML file and add extra groups. However, the original DTD did not allow
multiple variants with `id="optional"` as `id` attribute has to be
globally unique. This patch changes its type to `CDATA`, which has less
restrictions. This also means we can no longer define `<ref>` as
`IDREF` and instead check the existence of referenced variant in parser.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-19 12:21:19 +02:00
Lubomír Sedlář
f97e9258aa checksums: Never skip checksumming phase
It does not make sense for this phase to be skipped. If there are any
images, we need to generate the checksums so that writing metadata
works. If there are no images, the phase does not do anything and is
therefore very fast.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-19 12:20:34 +02:00
Lubomír Sedlář
bdf968ff3f [linker] Remove dead code
There are big parts of code that can not be reached. Other parts expose
options that are not used anywhere. To keep things simple, all of that
is removed in this patch.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-19 12:19:43 +02:00
Lubomír Sedlář
444c3a21ca [linker] Add tests
These tests are adapted from internal ones.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-19 12:19:43 +02:00
Patrick Uiterwijk
2c401a4a5c Send fedmsg message on ostree compose finishg
This sends a message when we finish composing an ostree.

Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-09-15 11:16:18 +00:00
Lubomír Sedlář
bf46048cbf createrepo: Add option to use xz compression
This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.

Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-12 11:10:43 +02:00
Pat Riehecky
10b93796bd Allow user to set a ~/.pungirc for some defaults
Merges: #64
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2016-09-08 15:47:50 +02:00
Lubomír Sedlář
1a16d94dda metadata: Improve error reporting on failed checksum
Instead of using the _doChecksum function, it now uses a function from
kobo which has nicer API. If there is an error, the message now includes
more details.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-07 13:05:11 +02:00
Jeremy Cline
ee1ee0467b extra-files: Write a metadata file enumerating extra files
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.

For example:
{
  "header": {"version": "1.0},
  "data": [
    {
      "file": "GPL",
      "checksums": {
        "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
      },
      "size": 18092
    },
    {
      "file": "release-notes/notes.html",
      "checksums": {
        "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
      },
      "size": 1120
    }
  ]
}

Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
2016-09-07 13:02:48 +02:00
Dennis Gilmore
6aeab9ee9d Merge #381 Automatically generate missing image version 2016-09-06 15:19:55 +00:00
Lubomír Sedlář
223a015898 Automatically generate missing image version
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).

This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 15:13:04 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
1881bf740d createiso: Report nice error when tag does not exist
Instead of failing with generic `AttributeError`, we can easily report
the actual problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 09:09:31 +02:00
Lubomír Sedlář
f43b8736b7 [osbs] Add NVRA of created image into main log
Currently the main log only says the phase started and finished. With
this patch each created image will be mentioned by its name, version,
release and arch.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 10:37:17 +02:00
Lubomír Sedlář
43fda1e9ef Use label to populate image release
When release for an image is specified as explicit `None`, we can
generate the value based on compose label. For example for `Alpha-1.2`
the release would be `1.2` instead of the date based one.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-25 15:24:36 +02:00
Dennis Gilmore
76f3f909ec Merge #367 Remove unused imports 2016-08-18 13:16:53 +00:00
Lubomír Sedlář
b862fc5a50 [buildinstall] Fix cleaning output dir
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.

The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-18 09:11:30 +02:00