We have switched how 5xx errors are being recorded
internally and we are now recording all failures
for all endpoints. As a result, a dedicated metric
only for compose failures is no longer required.
After introducing Go 1.18 to a project, it's required by law to convert at
least one method to a generic one.
Everyone hates IntToPtr, StringToPtr, BoolToPtr and Uint64ToPtr, so let's
convert them to the ultimate generic ToPtr one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The ostree options are used during the ostree resolve job, but when
generating the manifest the rhsm value comes from the image options, so
it's necessary to set it on both.
Register the custom middleware function to the cloudapi
server. This function is responsible for recording all
the status codes of all the server's endpoints.
Specifically describe that at least one of `baseurl`, `mirrorlist` or
`metalink` properties must be specified. Although this is not enforced
by the schema, it is the reality. Modifying the schema to enforce this
results in the generated data structure for `Repository` to be simply
an `interface{}`, which would just make the code more ugly, Therefore
this is covered by the API implementation logic.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Providing the `location` is no longer required for Azure Upload Options.
If it is not provided, the implementation determines the location from
the provided Resource Group. This will make the API nicer for any
client, since they won't need to provide redundant information.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Koji clean-up doesn't delete the top-level directories. As we named every
top-level directory different, they were just cumulating on kojihub, which
is obviously wrong.
This commit changes that behaviour to put all the temporary directories under
a new osbuild-cg top-level one. This way, osbuild-cg won't ve ever cleaned,
whereas osbuild-cg/osbuild-composer-koji-{UUID} will be, which is exactly
what we want.
Closes: https://github.com/osbuild/osbuild-composer/issues/3064
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
If params.Ref is an empty string, it's set to the distro's default
ref. The only difference here is that the default ref also gets
verified.
It makes splitting out resolving ostree refs to a new job easier.
In the weldr and cloud apis, ostree.ResolveParams always got executed,
also for non-ostree image types. Make it more explicit by only resolving
if the image type is actually an ostree image.
GCP Bucket to use can be now configured in the worker configuration.
Make the `Bucket` optional in the Cloud API when uploading image to GCP.
Adjust the Cloud API test case to configure GCP Bucket on the worker and
not provide it in the API request.
Instead of using the ostree.RequestParams in the OSTReeImageOptions,
define a new struct specific to ImageOptions for the ostree parameters.
This is almost identical to the new ostree.CommitSpec but the meaning of
the parameters changes based on image type and it would not be clear if
the CommitSpec was used in all cases. For example, the parameters of
the new OSTreeImageOptions do not always refer to the same commit. The
URL and Checksum may point to a parent commit to be pulled in to base
the new commit on, while the Ref refers to the new commit that will be
built (which may have a different ref from the parent).
The ostree.ResolveParams() function now returns two strings, the
resolved ref, which is replaced by the defaultRef if it's not specified
in the request, and the resolved parent checksum if a URL is specified.
The URL does not need to be returned since it's always the same as the
one specified in the request.
The function has been rewritten to make the logic more clear.
The docstring for the function has been rewritten to cover all use cases
and error conditions.
A small test is added to cover this new option. Also, a check for AWS region
in the URL was removed. The public URL doesn't actually contain it and it's
imho useless - S3 buckets are always tied to a certain region.
The elaborate grep command parsing a path from a URI was switched to being
a small python script. The python script can actually handle an URI without
a query (which was always present in the pre-signed URI, but it's not in the
ordinary one)
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Support for creating multiple amis from a single compose. It uses the
AWSEC2* jobs to push images to new regions, and share them with new
accounts.
The compose it depends upon has to have succeeded.
Add support for embedding container images via the cloud API. For
this the container resolve job was plumbed into the cloud api's
handler and the API specification updated with a new `containers`
section that mimics the blueprint section with the same name.
Dependency errors are not set by the workers, they're not set directly
in the job result. They are added by the worker server in case the job
error indicates it's a dependency error.
The ellipsis operator was used as a hack to not need to pass any details
as an argument, but it makes what the end object will actually look like
less obvious. It also makes it impossible to pass an array to details
without getting a nested array.
Fixes#2874
If the user does not pass a name, use the distribution as a name
A provided tag is used only if name is provided. It
The tag's default is a generated using UUID to avoid collisions
Worker
------
Add configuration for the default container registry.
Use the default container registry if not provided as part
of the image name.
When using the default registry use the configured values
Return the image url as part of the result.
Composer Worker API
-------------------
Add `ContainerTargetResultOptions` to return the image url
Composer API
------------
Add UploadOptions to allow setting of the image name and tag
Add UploadStatus to return the url of the uploaded image
Co-Developed-By: Christian Kellner <christian@kellner.me>
Since the `jobStatus` functions return a `JobInfo`
struct that contains the `JobStatus`, it makes sense
to rename the function names for the sake of consistency.
The number of return values from the `jobStatus`
function was growing and getting out of hand. Not
all return values were being used in all cases
and so returning a single struct with the information
and status of a job makes more sense. Then in each case
the resulting fields can be used as needed.
Add support to handle upload options in image requests for Koji
composes. The image is always uploaded to Koji, but now it can be
uploaded to the cloud environment in addition to Koji as part of the
build.
The image name used for Koji image can't be used as is for uploading to
the cloud, because each cloud provider has its own requirements for the
valid characters. For now, let the Cloud API implementation generate a
random image name. The name is always returned in the compose status's
upload status, so it should be possible to attach it to the Koji build
to allow users to find the image.
This is the first step to support embedding container images. Here
we add the `containers []container.Spec` argument to supply images
with resolved container specifications. For now all distros will
return an error in case a container is actually supplied since none
of them currently support embedding containers. NB: also no apis or
tools will actually resolve containers.
Koji API removed by the previous commit was the last user of osbuild-koji job.
Let's remove it since nothing uses it. This also removes all of the
compatibility code in Cloud API, see concerns below:
Compatibility concerns:
- the internal deployment was moved to a completely different composer
instance, thus there are no old jobs
- Fedora deployment is still unused in prod, thus we don't care about keeping
backward compatibility of the old jobs
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We no longer use it, let's remove it. If you are wondering what to use instead,
use Cloud API. It supports everything that Koji API supported and more.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This test tested two things:
1) Invalid route - this is already covered by TestUnknownRoute
2) Invalid UUID in the compose status route - this is now covered by
TestComposeStatusInvalidUUID
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The test_distro Manifest, which is used in tests across multiple
packages, was using the old structure. Updated to the v2 structure and
adapted all tests.
The package sets for an image can depend on the blueprint, and
by the same logic there is no reason it should not be able to
depend on the image options.
This is so far a non-functional change, but makes a follow-up
commit simpler (though still without actually depending on
the image options to compute the package sets).
The osbuild export is specific to the upload target and different
targets may require using a different export. While osbuild-composer
still does not support multiple exports for osbuild jobs, this prepares
the ground for such support in the future.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
message is always a super-set of the old and new way of providing the
exports to osbuild job.
The `TestKojiJobTypeValidation` unit test used the `OSBuildJob` and
Koji Target fields in the wrong way, however without any impact on the
testing itself. The reason is that no actual images were built as part
of the test nor the created jobs were ever picked up by a worker.
This change was forgotten in PR#2758.
[1] https://github.com/osbuild/osbuild-composer/pull/2758
The filename of the image as produced by osbuild for a given export is
currently set in each target options type in the `Filename` struct
member. However, the value is not really specific to any target type,
but to the specific export used for the target. For this reason move the
value form target type options to the `Target` struct inside a new
struct `OsbuildArtifact` under the name`ExportFilename`.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
object is always a super-set of the old and new way of providing the
export filename in the Target.
The `TargetErrors` is not used any more since PR#2192 [1] and there is
no need to keep the backward compatibility any more, because there are
no composer / worker instances in production, which are not running the
modified code.
In addition, delete unit tests covering this legacy error handling.
[1] https://github.com/osbuild/osbuild-composer/pull/2192
Error handling is structured in such a way that typically, a ServiceCodeError is passed
through the echo HTTP error, in reference to internally defined errors. We want to be able
to obtain and return specific external errors, for example during validation from openapi3.
Add a 'details' field to the serviceError struct, to contain extra / externally defined
information. Modify HTTPErrorHandler to anticipate either a string or a ServiceErrorCode
from echo, and respond accordingly. Edit the affected tests to expect the appropriate response.
