Delete the `internal/upload/koji` package and replace it with
`pkg/upload/koji` package provided by `osbuild/images`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Use the version that contains the updated Koji upload code.
Also bump the version of `osbuild/blueprint` due to the new `uri` field
in file customizations.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Delete the osbuild-koji tools, which is not used by any code or test
case. This also allows to delete the koji.NewFromPlain() function. This
is to minimize the set of exported functions by the koji package, before
moving it to osbuild/images repository.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Modify the Koji image extra metadata to not depend on the upload target
data structure for the OSBuild Artifact. This is the last dependency on
the internal osbuild-composer package, allowing the move of the Koji
upload code to the osbuild/images repository.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Modify the Koji image extra metadata to not depend on the upload target
data structures for upload results. The target results are supposed to
be specific to the uploader implementation, which will eventually
change. Moreover, the definition is internal to osbuild-composer, so
this would create a problem once the Koji upload implementation is moved
to osbuild/images.
The Koji upload implementation does not really care about the exact
structure of the upload results. It is just a list of JSON objects.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The use of rpmmd.RPM is exclusive to the Koji upload implementation.
Consolidate the metadata structure for RPMs into the koji package
codebase, together with any required functions for converting osbuild
stage metadata. The respective code in osbuild/images will be eventually
removed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Increase readability by logically splitting the build metadata
structures from the actual upload API implementation.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The upload/koji package functions were creating a logger and then were
using it. This is not ideal for a library implementation.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Don't create a new transport in NewFromPlain() using
CreateRetryableTransport(), because the only place
in which NewFromPlain() is called passes the return value of
CreateRetryableTransport() when calling it.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This cleans up the linting results by adding checks for
integer underflow/overflow in several places, suppressing the error in
places where it has been checked, or fixing the types when possible.
If the Koji target result contains information about any uploaded SBOM
documents, import them to Koji as part of the finalize task.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
When uploading and distributing a commit, wait for any async tasks to
finish before returning. There are two tasks that can block this
function:
- Creating a distribution: this only happens when a new repository is
created.
- Import commit: this will always happen in this function.
Helper function that performs the whole upload and distribute procedure.
Two more helper functions are added for retrieving the href for a
repository based on its name, and for retrieving the base URL for a
repository's distribution.
Define the task state enum based on the available values defined in the
API.
Add a helper function that returns true if a task is running or waiting
and ignores errors.
Function for importing a commit artifact (that's already been uploaded)
into a given repository. Note that the "repo" argument to the
NewOstreeImportAll() function refers to the name of the repository
directory inside the archive, which for the commits we produce in
osbuild is always "repo".
Include the osbuild version used to build the image, in the image extra
metadata attached to the Koji build and image output.
Extend `koji.sh` to verify that the version is set and of the expected
value.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Expose the extra metadata attached to each manifest output, also to the
build extra metadata under `osbuild_manifest` property. The value is a
map of all manifest metadata with keys being the filename of each
manifest.
This will expose the information about osbuild-composer and
osbuild/images versions used to produce the manifest in the Koji
buildinfo, including the Web UI.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Include the osbuild-composer and its dependencies versions in the extra
metadata associated with the Manifest output when importing it to Koji.
This will make it possible to pin-point the exact version combination
which was used to generate the osbuild manifest used to built the image
imported to Koji.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Expose the osbuild information used to produce the image, in the image
extra metadata under the `osbuild_artifact` property.
This information will get included in the image / build extra metadata
and make it explicit how to reproduce the image build using osbuild and
the attached manifest.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add any non-Koji upload target results attached to an OSBuild result, to
the image extra metadata. This will make it easy to locate any image
from Koji uploaded to cloud, in the target cloud environment.
The rationale behind including only non-Koji target results is that one
can find it only in Koji, so there is no added value in including the
Koji target results at all.
Extend the `koji.sh` to check the target results in image metadata when
testing Koji scenario with cloud upload.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Import osbuild manifest and build log to the Koji build as outputs. Also
note the respective filenames in the image output extra metadata.
Note that the osbuild manifest is imported as a log file for now. Koji
has very limited set of output types defined and I still need to
determine the best way to use a custom output type in Koji instances (as
other content generators do).
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Uploads an OCI image to OCI object storage, and generates a
pre-authenticated request for the object, which can be used to import it
into custom images.
resp.IsResumable crashes if resp.MultipartUploadResponse == nil. This happens
for instance when authentication fails. Fix this by also checking the
MultipartUploadResponse field.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Also extend the Koji test case to verify that the boot mode information
is in the build extra metadata and that it contains valid value.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Expose the extra metadata information for each image output stored in
`ImageExtraInfo` also in the build extra metadata. The extra metadata
for each image is nested under key corresponding to the image filename.
Extend the Koji test case to check information in the buildinfo output
and specifically check for all expected metadata in the build Extra
metadata field.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The original Koji implementation expected that the output of a content
generator is only an image. While in reality, we will eventually upload
other types of files as outputs to Koji, such as logs and osbuild
manifest.
Rename Koji structures and their members to better map to the upstream
Koji documentation and their JSON representation. Add comments to
structures. Define type aliases and constants for string values which
are more like enums, than a free-form values.
These changes have no effect on the actual JSON representation of any of
the structures
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
UBI and the oldest support Fedora (37) now all have go 1.19, so we are
cleared to switch.
gofmt now reformats comments in certain cases, so that explains the formatting
changes in this commit.
See https://go.dev/doc/go1.19#go-doc
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
We sometimes see the following error in the logs:
Fault(1000): upload path exists: /mnt/koji/work/osbuild-cg/osbuild-composer-koji-082e1c88/Fedora-IoT-38.raw.xz.
I think this happens when we retry the upload call of the first chunk due to
random network issues. The solution is to always upload in the overwriting
mode, which ignores the already existing file.
See https://pagure.io/koji/blob/175ecb5e8f3d45a1d244b227eb889321e5dd0a29/f/kojihub/kojihub.py#_15522
This is safe because:
1) We use UUIDs in the filename, which means that there should never be a real
conflict.
2) The overwriting mode is actually the default mode in koji, see
https://pagure.io/koji/blob/175ecb5e8f3d45a1d244b227eb889321e5dd0a29/f/koji/__init__.py#_3342
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
CGImport quite often fails with the following error:
Fault(1000): File size 735051776 for Fedora-IoT-38.raw.xz (expected 738785372)
doesn't match. Corrupted upload?
When I inspect the file manually, everything seems fine, though.
I believe that this because of NFS inconsistency when multiple DNS-balanced
kojihubs are used in the setup (which is what Fedora uses). The addded
loop implements a retrying mechanism for the CGImport call to try again
whenever we see this issue.
Note that this isn't caught by other HTTP retrying mechanism because a failed
XMLRPC call returns code 200.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The size of the page blob is defined on creation and the blob is
zero-initialized. Therefore, we can just skip all the pages that contain
only zeros. This should save a lot of bandwidth if used on sparse files as
e.g. operating system images. (:
https://github.com/Azure/azure-storage-blob-go/ is deprecated, the main SDK
should be now used instead. Let's migrate the code. There should be no
functional changes.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
