Commit graph

46 commits

Author SHA1 Message Date
Achilleas Koutsou
9e0af86775 ostree: rename errors and introduce ParameterComboError
Shorter and clearer error types.
- InvalidParameterError is now RefError.  It is returned when a ref (the
  new ref or the parent ref) is invalid.
- The ResolveRefError is not renamed.  It is returned when a ref can't
  be resolved (e.g., ref not found at URL).

New error type ParameterComboError.  It is returned when a parent ref is
specified but no URL, which is an invalid parameter combination.
2022-02-23 11:08:24 +01:00
Achilleas Koutsou
bfaf0c073b cloudapi/v2: support specifying parent ref for ostree requests 2022-02-23 11:08:24 +01:00
Achilleas Koutsou
24b70837c1 cloudapi/v2: validate ostree params using common validation function
Use the ostree package error types to keep the existing distinction
between Ref- and URL-related errors.
Introduce a new error condition for a general InvalidOSTreeParams
failure.
2022-02-23 11:08:24 +01:00
Achilleas Koutsou
f885de4c68 distro: remove OSTreeImageOptions type and use ostree.RequestParams 2022-02-23 11:08:24 +01:00
Sanne Raymaekers
d589317dcb internal/cloudapi: Log error in manifest job 2022-02-22 14:06:05 +01:00
Tom Gundersen
bb486bcccb cloudapi/v2/compose: don't overwrite customizations struct
The customizations struct contains different kinds of customizations,
such as users and filesystems. If both filesystems and users are
configured, the current code overwrites the user configuration.

Change this by unconditionally allocating the customizations struct
when creating the empty blueprint, and ammending it for each
customization we find.

This means that the customizations will be non-nil, also when it is
empty, which was not the case in the past. This should not change
the behavior of the code.
2022-02-19 11:13:55 +00:00
Chloe Kaubisch
0619ffa588 cloudapi: expose filesystem customizations 2022-02-19 11:13:55 +00:00
Tomas Hozza
8e6826e743 Move OSBuildMetadataToRPMs and PackageMetadataToSignature to osbuild2
Move `OSBuildMetadataToRPMs()` and `PackageMetadataToSignature()`
functions from the `rpmmd` package to `osbuild2` package to prevent
import cycles while de-duplicating `rpmStageInputs()` function from
`stage_inputs.go` of distro definitions.

Rename `PackageMetadataToSignature()` to
`RPMPackageMetadataToSignature()`, since it takes specifically
`RPMPackageMetadata` type as an argument.

Adjust affected parts of code (unit tests, cloudapi, worker).

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Achilleas Koutsou
679028bb8e cloudapi: copy all Repository fields to RepoConfig 2022-02-14 17:38:41 +01:00
Achilleas Koutsou
3320f2c57d cloudapi: separate repositories based on package_sets
- Any repository without package_sets is added to the general `Repos`
  field of the DepsolveJob, just like before.
- Repositories with package_sets are added to the `PackageSetsRepos`
  map, indexed by the package set names.
- Repositories defined in the customizations as `PayloadRepositories`
  are considered to be associated only with the `PayloadPackageSets`
  names from the image type and are added to the `PackageSetsRepos`
  under the payload sets.

The repository collection and conversion of repository structs (from
Repository to RepoConfig) has been moved to a separate function.
2022-02-14 17:38:41 +01:00
Achilleas Koutsou
82eedf5b82 DepsolveJob: rename struct field for consistency
We have two fields, `Repos` and `PackageSets`.  Renaming
`PackageSetsRepositories` to `PackageSetsRepos` for consistency.
The struct is for internal use only so the rename has no impact as long
as the serialised name is the same (json tag).

Also it's shorter.

Added docstring to the struct that explains the arguments in the same
way as they are described for the `depsolve()` function.

Changing the name of the argument in the internal `depsolve()` function
for the same reasons.
2022-02-14 17:38:41 +01:00
Achilleas Koutsou
0e5bb66516 cloudapi: small comment for readability
Comment the code that reads the user-defined repositories to explaine
what's happening.
2022-02-14 17:38:41 +01:00
Ondřej Budai
631bd21ffe cloudapi/v2: add support for aws-*-rhui image types
For the brew use-case, we also need to build AWS images containing RHUI. This
commit is thus adding them.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-10 11:26:50 +00:00
Tom Gundersen
a3f8304ae8 cloudapi/v2: support multi-build composes
This is for now only supported for koji builds.

A lot of code moved around for this, but functionally not much changed. PostCompose() now only parses the input, and queueing of all the jobs have been factored out in separate functions. PostCompose() is mostly agnostic to koji/non-koji requests.
2022-02-01 20:28:40 +00:00
Tom Gundersen
6a9ffb2086 cloudapi/v2: make ComposeStatus handle multi-build composes
This brings parity with the koji API.
2022-02-01 20:28:40 +00:00
Tom Gundersen
c81d0d08ec cloudapi/v2: support logs/manifests endpoints
For now these only work for koji builds.
2022-02-01 20:28:40 +00:00
Tom Gundersen
b32ab36e1d worker/server: typesafe Job and JobStatus
Replace Job() and JobStatus() with typesafe versions, and introduce JobType()
for the rare instances where we don't know the type up front.

Additionally, catch a few more error cases:
 - if OSBuildResult is nil, then we failed to invoke osbuild
 - make sure the same JobResult handling is done for osbuild-koji, as for osbuild
2022-02-01 20:28:40 +00:00
Tom Gundersen
9fba2dcc5d cloudapi/v2: introduce multi build composes
This only extends the API, the backend can still only deal with composes of a single build.

I aimed to keep the API practically backwards compatible, i.e., no current consumer of it should notice the change. I hope I didn't mess that up.

fixup: image statuses
2022-02-01 20:28:40 +00:00
Tom Gundersen
c1dc9d7284 cloudapi/v2: introduce compose status
In addition to individual image status, have an
overall status that captures success or failure
of the compose as a whole.

This is not as fine grained, and only distinguishes
between "pending", "failure" and "success".

This captures other jobs than the image builds, which
is relevant for the koji composes, which consists also
of koji-init and koji-finalize, in addition to the build
jobs.
2022-02-01 20:28:40 +00:00
Tom Gundersen
c892ccfde0 cloudapi/v2: make upload request optional
For now upload requests are required if and only if we are not
using koji. When using the koji integration the produced artifacts
are uploaded to koji only. In the future we may want to support
also uploading to the cloud providers.
2022-02-01 20:28:40 +00:00
Tom Gundersen
92c7fc2534 cloupapi/v2: add koji support
Extend the compose endpoints to have minimal koji support.

This is intended to replace the current koji API so that it
can be consumed through api.openshift.com.
2022-02-01 20:28:40 +00:00
Gianluca Zuccarelli
cc981b887a osbuild-worker: implement structured errors
Implement the structured errors as defined by the worker client.
Every error for each of the job types now returns a structured
error with a reason and a specific error code.  This will make
it possible to differentiate between 4xx errors and 5xx errors.

This commit refactors the way errors are implemented in the workers,
but maintains backwards compatability in composer by checking for
both kinds of errors.
2022-01-27 16:45:14 +01:00
sanne
a83cf95d5b go.mod: Update oapi-codegen and kin-openapi 2022-01-12 11:35:06 +01:00
Djebran Lezzoum
c93ea748a2 distro/depsolve/cloudapi: Add 3rd-party repository support.
Allow 3rd-party repositories to be supported and custom packages installed.
Fixes #COMPOSER-1273
2021-12-15 20:12:49 +01:00
Achilleas Koutsou
b840f452ea cloudapi: extend supported image types
Add support for:
- qcow2: guest iamge
- vmdk: vmware image
- image-installer: installer ISO

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-12-03 16:49:09 +00:00
sanne
c43ad2b22a osbuild-service-maintenance: Clean up expired images 2021-12-03 00:14:09 +00:00
sanne
c8f198166d cloudapi/v2: Add support for edge-container 2021-11-19 21:45:19 +01:00
Achilleas Koutsou
0712ed9700 cloudapi: add pipeline names to osbuild job 2021-11-19 19:37:08 +01:00
sanne
0a49dbe560 cloudapi/v2: Hard timeout for manifest job 2021-11-18 10:26:17 +01:00
sanne
028eca1b26 cloudapi/v2: Use manifest-id-only job
job dependencies:
depsolve -> manifest -> osbuild

This allows the compose handler to return the osbuild job id
immediately.
2021-11-18 10:26:17 +01:00
Sanne Raymaekers
2d1b7baa27 Revert "cloudapi/v2: Use manifest-id-only job"
This reverts commit 992e442ba7.
2021-11-16 20:23:36 +01:00
sanne
992e442ba7 cloudapi/v2: Use manifest-id-only job
job dependencies:
depsolve -> manifest -> osbuild

This allows the compose handler to return the osbuild job id
immediately.
2021-11-16 18:53:12 +00:00
Achilleas Koutsou
9aef7bfc47 osbuild-worker: attach pipeline names to jobs
Pipeline names are added to each job before adding to the queue. When a
job is finished, the names are copied to the Result object as well. This
is done for both OSBuild and Koji jobs.

The pipeline names in the result are primarily used to separate package
lists into build and payload/image packages in two cases:
1. Koji builds: for reporting the build root and image package lists to
   Koji (in Koji finalize).
2. Cloud API (v1 and v2): for reporting the payload packages in the
   metadata request.

The pipeline names are also used to print the system log output in the
order in which pipelines are executed. This still isn't used when
printing the OSBuild Result (osbuild2.Result.Write()) and we still rely
on sorting by pipeline name
(see https://github.com/osbuild/osbuild-composer/pull/1330).
2021-11-16 09:49:37 +01:00
Gianluca Zuccarelli
f8199ec41d prometheus: add middleware function
Add middleware function to track request count
and measure the latency of compose requests.
2021-10-29 20:36:18 +01:00
sanne
87cc722021 cloudapi/v2: Plural path section when querying a collection 2021-10-14 16:22:24 +02:00
Ondřej Budai
7760ca1c92 cloudapi/v2: ensure only one image per a compose in the API spec
We've never had the ability to build multiple images per a compose, this
commit thus rips out support for this on the API level:

image_requests is now image_request and it accepts only one ImageRequest
object instead of an array of them.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Ondřej Budai
b2dc90e404 cloudapi/v2: clean up targets
tbh, I don't fully understand why we wrote the code in this way.

Not a functional change.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Ondřej Budai
6b8f758c54 cloudapi/v2: move multi-image compose check to the beginning
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-14 00:14:34 +02:00
Ondřej Budai
e904397fdb cloudapi/v2: Use worker to depsolve
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-11 13:16:51 +02:00
sanne
6ce20a9ef6 cloudapi/v2: Listen on /api/image-builder-composer/v2 2021-10-11 09:52:21 +02:00
sanne
4eca54b6ed cloudapi/v2: Configurable aws bucket
Fixes #1855
2021-10-09 00:55:09 +01:00
sanne
b6c08f3056 cloudapi/v2: Replace upload types with image types
Because there's only a few combinations of upload types and image types
that make sense, enforce correct combinations by eliminating upload
types.

Fixes #1775
2021-10-09 00:55:09 +01:00
Diaa Sami
27ca55372a cloudapi: Add extra logging & improve existing
Use different logging levels depending on situation
Log HTTP 5xx errors to stdout with internal error details when possible
2021-09-30 18:34:05 +02:00
Diaa Sami
179009fec4 cloudapi: use Logrus as default logger in Echo
And remove log.Logger references
2021-09-30 18:34:05 +02:00
Diaa Sami
60e403e53e cloudapi: use Recover middleware to handle panics
recover from panics such as out-of-bounds array access & nil
pointer access, print a stack trace and return 5xx error
instead of the service crashing and relying on Execution
framework to handle crashes
2021-09-24 12:11:04 +01:00
sanne
5a9d8c792b cloudapi: V2
V2 is compliant with api.openshift.com design guidelines.

Errors are predefined, have codes, and are queryable.

All requests have an operationId set: a unique identifier which is
sortable by time. This is added to the response in case of an error.

All returned objects have the href, id, and kind field set.
2021-09-14 15:32:21 +02:00