The ostree phase now runs in parallel with a lot of other stuff. If
there's any error while the phase is running, the compose would be
aborted but the ostree threads wouldn't be stopped automatically. With
the threads left alive, the process would never finish.
This patch makes sure that whatever happens in the other code, we always
stop the ostree phases.
Fixes: https://pagure.io/pungi/issue/1799
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This reduces the legacy 'pungi' script to only its gather phase,
and removes related stuff in gather.py. The gather phase is used
in the yum path through phases/gather/methods/method_deps.py, so
it cannot be entirely removed until all users of that are gone.
But we can at least get rid of the non-Koji support for creating
install trees, ISOs and repos.
Merges: https://pagure.io/pungi/pull-request/1793
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I did a time map of a Fedora compose today, and noticed that we
spend about an hour waiting for the ostree_install phase to
complete before we start up the compose_images_phase which does
all the other image builds.
This is unnecessary. Nothing else depends on ostree_install; it
should be fine to start up the extra_phase (which contains
compose_images_phase) while the ostree stuff is still running.
This implements that by splitting the ostree phases out of the
essentials_phase which contains the real precursors to the
extra_phase. We start the essentials and ostree phases together,
but only wait for the essentials phase to complete before
kicking off extra_phase, so it can start while the ostree
phase is still running.
One tweak we have to make to accommodate this is to move
image_checksum_phase out of extra_phase, to avoid it potentially
running before all ostree installer images are built. The
checksum phase is quite fast - it takes about five minutes -
and any time benefit of running it in parallel with the osbs and
repoclosure phases seems like it must be smaller than the time
loss of waiting for ostree_install before kicking off extra.
Merges: https://pagure.io/pungi/pull-request/1790
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This was only usable in live_images phase that doesn't exist anymore,
and wasn't used much in the first place.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase was used to create live images with livecd-creator
and 32-bit ARM images with appliance-creator. We also remove
get_create_image_cmd from the Koji wrapper as it was only used
for this phase, remove associated tests, and remove related
configuration settings and documentation.
Fixes: https://pagure.io/pungi/issue/1753
Merges: https://pagure.io/pungi/pull-request/1774
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This patch connects the phase into the main script, and adds other
modifications:
* The archive is now stored in the images/ subdirectory in the compose.
* Documentation is updated to correctly mention that variant repos are
not available.
* Configuration for path and name of the final archive is dropped. There
are reasonable defaults for this and there's no point in having users
configure it.
* The extra message for the archive is no longer sent.
* The pungi-make-ostree utility is no longer required in the buildroot.
The pungi-make-ostree utility doesn't do any significant work. It
modifies configuration files (which can happen on the compose host), and
it starts other processes.
This patch changes the ostree-container phase to no longer need the
script in the buildroot. Instead, the utility is called on the compose
host to do the config manipulation and output the needed commands. Those
are then passed into the runroot task.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Pungi would by default only ever add files to the cache. That would
eventually result in essentially a mirror of the Koji volume.
This patch adds a helper cleanup script. When called, it goes through
files in the cache and deletes anything that is not hardlinked from
elsewhere and with mtime not updated recently.
Cleaning up files that hardlinked from some compose would not save any
space anyway. The mtime check should account for cases like subpackage
being downloaded but not included in any compose. This would avoid it
from being downloaded over and over again.
When a compose fails or is aborted, there can be a stale lock file left
behind in the cache. This script cleans that up too.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the --target-dir option is used, the compose can be created in CTS,
but the parent and respin information is not passed through. That leads
to data missing later on.
JIRA: RHELCMP-11411
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When nothing in configuration or the image itself changed, let's just
copy the older one.
JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When ODCS starts a compose, it will provide base composeinfo file, but
it doesn't create COMPOSE_ID. This leads to a crash when updating CTS,
since the compose id can't be read from the file. We can instead use the
value we already have in memory.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If cts_keytab is also enabled then the HTTP requests are handled with
Kerberos Authentication otherwise no authentication is used.
If cts_url is defined in the configuration, translate_paths is required.
This is needed in order to get the host and the path of the composes.
Jira: RHELCMP-6318
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
If the given directory is not a valid git directory, it raises RuntimeError.
This can be catched and raised as GitUrlResolveError, so compose can continue
to log the failure.
Jira: RHELCMP-6077
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.
This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.
JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs after image-build and osbuild and can embed an image
into a container.
JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This file can contain all Pungi specific exceptions.
It should also fix an issue encountered on Python 2.7:
AttributeError: 'module' object has no attribute 'pkgsets'
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In such case we never want to reuse the pkgset, as it risks leaking
unsigned packages. Safest option is to remove the file completely.
Fixes: https://pagure.io/pungi/issue/1480
JIRA: RHELCMP-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When Pungi receives a signal to terminate, it can sometimes get stuck
if there are threads running. It has to wait for all worker threads to
finish. They generally do finish, unless they get stuck waiting on a
subprocess.
This patch should reduce the likelihood of this happening by stopping
all subprocesses.
JIRA: RHELCMP-3056
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.
The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.
JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is useful when running pungi-patch-iso on VM with low amount of
memory but higher disk space. Without this option the operation will
fail because /tmp is tmpfs filesystem.
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
Previously, we checked only for `compose_dir`, but it makes
more sense to really check for `composeinfo-base.json` instead.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This commit also adds a --config argument allowing to
override/specify a specific fedora-messaging configuration
file to use.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
