Even if basic otel dependencies are available, this instrumentor is a
separate dependency which may be missing.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the compose is configured to include any container image, it just
followed the provided URL. This is not particularly reproducible. If the
image spec contains a tag, it may point to different images at different
time.
This commit adds a step to validating the configuration that will query
the registry and replace the tag with a digest.
This makes it more reproducible, and also fixes a problem where changing
container image would not stop ISO reuse. There's still a chance of
non-container file changing and not forcing the reuse, but that is not
very common.
JIRA: RHELCMP-14381
JIRA: RHELCMP-14465
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds support for Opentelemetry. If
OTEL_EXPORTER_OTLP_ENDPOINT env variable is defined, it will send traces
there. Otherwise there is no change.
The whole compose is wrapped in a single span. Nested under that are
spans for operations that involve a remote server.
* Talking to CTS
* Sending API requests to Koji
* Any git repo clone
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
osbuild phase needs to wait for ostree phase in some cases, this patch
makes the various image build phases waiting for ostree phase, it may
introduce some slowdown, but it's still faster than the version before
PR#1790.
JIRA: RHELCMP-14349
Fixes: https://pagure.io/pungi/issue/1816
Signed-off-by: Haibo Lin <hlin@redhat.com>
When container-images are downloaded, they would be skipped from the
extra_files.json manifest. This patch fixes that by enumerating all
files rather than relying on the getter to return a list.
JIRA: RHELCMP-14406
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`git clone` failed if the URL is specified as git+http.
git: 'remote-git+http' is not a git command. See 'git --help'.
JIRA: RHELCMP-14340
Signed-off-by: Haibo Lin <hlin@redhat.com>
The getter is already running once per architecture, it just doesn't
make the information available to the scm wrapper.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files should always be cleaned up immediately after the archive is
extracted, but we are seeing them being left behind for some reason.
With this page, even if the data is not cleaned up, it will not clog up
/tmp and be eventually deleted together with the compose.
JIRA: RHELCMP-14319
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All these are calling subprocess in 'text mode', where it will
try to decode stdout/stderr using the default encoding (utf-8
for us). If it doesn't decode, subprocess will raise an exception
and kobo doesn't handle it, it just passes it along to us, so
things blow up - see https://pagure.io/releng/issue/12474 . To
avoid this, let's set `errors="replace"`, which tells the decoder
to replace invalid data with ? characters. This way we should get
as much of the output as can be read, and no crashes.
We also replace `universal_newlines=True` with `text=True` as
the latter is shorter, clearer, and what Python 3 subprocess
wants us to use, it considers `universal_newlines` to just be
a backwards-compatibility thing - "The universal_newlines argument
is equivalent to text and is provided for backwards compatibility"
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1812
This could be useful for handling flatpak applications in the installer.
All of the specified containers are downloaded into a single oci
layout.
JIRA: RHELCMP-14302
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In https://github.com/release-engineering/productmd/pull/181 I
added new `bootable-container` and `container` types to
productmd. This makes pungi always use the bootable-container
type for ostree_container images (previously 'ociarchive'), and
default to using the container type for Kiwi-built oci.tar.xz
container images (previously 'docker').
This is a significant change for anything that relies on
productmd/fedfind conventions to 'identify' images, as these
images will now have a different identity. But I think it's a
valuable improvement in their identities. 'ociarchive' never made
any sense as an image 'type' - it's a format - and 'docker'
wasn't a very good type for images that are explicitly OCI
container images, not Docker-native ones. We also can now easily
distinguish between 'regular' container images and ones that are
intended to be bootable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The file became stale, and it likely won't work on any useful system.
Rather than trying to keep it up to date, let's admit it is not useful
and drop it completely.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no more yum anymore. This was also the only user of the
pathmatch module, which is thus also removed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The ostree phase now runs in parallel with a lot of other stuff. If
there's any error while the phase is running, the compose would be
aborted but the ostree threads wouldn't be stopped automatically. With
the threads left alive, the process would never finish.
This patch makes sure that whatever happens in the other code, we always
stop the ostree phases.
Fixes: https://pagure.io/pungi/issue/1799
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The library is imported if available, but we never build it in any
environment where the package would be installed. It was last used for
RHEL 6 builds.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This reduces the legacy 'pungi' script to only its gather phase,
and removes related stuff in gather.py. The gather phase is used
in the yum path through phases/gather/methods/method_deps.py, so
it cannot be entirely removed until all users of that are gone.
But we can at least get rid of the non-Koji support for creating
install trees, ISOs and repos.
Merges: https://pagure.io/pungi/pull-request/1793
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The pkgset phase takes around 35 minutes in current composes.
Around 20 minutes of that is spent creating these per-arch
subsets of the global package set. In a rather roundabout way
(see #1794 ), I figured out that almost all of this time is
spent in this cache check, which is broken for a subtle reason.
Python's `in` keyword works by first attempting to call the
container's magic `__contains__` method. If the container does
not implement `__contains__`, it falls back to iteration - it
tries to iterate over the container until it either hits what
it's looking for, or runs out. (If the container implements
neither, you get an error).
The FileCache instance's `file_cache` is a plain Python dict.
dicts have a very efficient `__contains__` implementation, so
doing `foo in (somedict)` is basically always very fast no matter
how huge the dict is. FileCache itself, though, implements
`__iter__` by returning an iterator over the `file_cache` dict's
keys, but it does *not* implement `__contains__`. So when we do
`foo in self.file_cache`, Python has to iterate over every key
in the dict until it hits foo or runs out. This is massively
slower than `foo in self.file_cache.file_cache`, which uses the
efficient `__contains__` method.
Because these package sets are so huge, and we're looping over
*one* huge set and checking each package from it against the cache
of another, increasingly huge, set, this effect becomes massive.
To make it even worse, I ran a few tests where I added a debug log
if we ever hit the cache, and it looks like we never actually do -
so every check has to iterate through the entire dict.
We could probably remove this entirely, but changing it to check
the dict instead of the FileCache instance makes it just about as
fast as taking it out, so I figured let's go with that in case
there's some unusual scenario in which the cache does work here.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The fedora:latest image is now based on 41, and contains dnf5. This is
causing some tests to fail due to failing imports of dnf version 4.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
I did a time map of a Fedora compose today, and noticed that we
spend about an hour waiting for the ostree_install phase to
complete before we start up the compose_images_phase which does
all the other image builds.
This is unnecessary. Nothing else depends on ostree_install; it
should be fine to start up the extra_phase (which contains
compose_images_phase) while the ostree stuff is still running.
This implements that by splitting the ostree phases out of the
essentials_phase which contains the real precursors to the
extra_phase. We start the essentials and ostree phases together,
but only wait for the essentials phase to complete before
kicking off extra_phase, so it can start while the ostree
phase is still running.
One tweak we have to make to accommodate this is to move
image_checksum_phase out of extra_phase, to avoid it potentially
running before all ostree installer images are built. The
checksum phase is quite fast - it takes about five minutes -
and any time benefit of running it in parallel with the osbs and
repoclosure phases seems like it must be smaller than the time
loss of waiting for ostree_install before kicking off extra.
Merges: https://pagure.io/pungi/pull-request/1790
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The default base name is probably fine in most cases, but there
are some where we might want to tweak it. We already allow this
for other phases (e.g. the livemedia phase).
Also, we should include the arch in the image filename. Not doing
this doesn't blow up the compose as, while they have identical
filenames, the images for different arches are in different paths,
but it's confusing for people who actually download and use the
images.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The image metadata construction code allows for subvariant to be
set in the image config dict, but checks.py doesn't expect it;
fix that. Also, when a subvariant is set, use it in the image
name template rather than the variant; otherwise you can't
build more than one subvariant in any variant (they will have
identical names, which isn't alllowed).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Koji's image-build command has not been capable of producing a
docker image with .tar.gz as its extension since 2015:
https://pagure.io/koji/c/b489f282bee7a008108534404dd2e78efb2256e7?branch=master
as that commit message implies, the files have not actually been
gzip-compressed for even longer:
https://pagure.io/koji/c/82a405c7943192e3bba3340efe7a8d07a0e26b70?branch=master
so there's no point to having this any more. It is causing the
wrong productmd 'type' to be set for GCE cloud images, which *do*
have the .tar.gz extension - because docker appears in this dict
before tar-gz, their type is being set as 'docker' not 'tar-gz'.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The image-build phase's EXTENSIONS dict is meant to exactly
mirror the 'formats' that exist in the context of the command
`koji image-build`, which is driven by this phase. That nice
association was lost, however, by adding a couple of items to it
which exist for the purposes of the osbuild phase (and in the
case of .iso, also the kiwibuild phase), which import this dict
and uses it for image identification.
To make the association 1:1 again and more clearly show what's
going on here, let's move those entries out into the osbuild and
kiwi phases. osbuild now has its own dict which starts out as a
copy of the image-build one before being extended. And let's
update the relevant comments.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The bug that caused the attribute to have wrong value has been fixed in
DNF a long time ago.
Fixes: https://pagure.io/pungi/issue/1786
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It is not possible to reliably detect what the type for an image should
be in the metadata. This commit adds an option for user to explicitly
provide it.
It can only be configured on the specific image, not globally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When Kiwi builds an ISO, it is always supposed to be bootable and should
be located in the iso/ subdirectory.
Any other kind of image should still land in images/ and be listed as
not bootable in the metadata.
Relates: https://pagure.io/pungi-fedora/issue/1342
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The version follows the same rules as versioning for live media etc.
That means it's always going to be set. The precedence goes like this:
* image specific option
* `kiwibuild_version`
* `global_version`
* `release_version` or `<release_version>_<label_milestone>`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
