Commit graph

172 commits

Author SHA1 Message Date
Lars Karlitski
cef7267a2d store: create compose id from PushCompose
Every caller had to make their own uuid and pass that into
PushCompose(). Make PushCompose() create the id and return it.
2020-04-01 00:29:13 +02:00
Lars Karlitski
eb10cf3124 store: remove unused arguments from PushCompose 2020-04-01 00:29:13 +02:00
Martin Sehnoutka
2f576e0964 weldr: stop using Distro.FilenameFromType
The method is available in the Distro interface, but the distro does not
have the information needed to provide this kind of information any
more. The logic is now split into Arch and ImageType interfaces. This
patch will allow us to get rid of some old code and move forward.
2020-03-28 00:21:31 +01:00
Brian C. Lane
fc205786fc weldr: Return a JSON API error response for all unknown requests
This matches the lorax-composer behavior.
2020-03-27 19:07:33 +01:00
Brian C. Lane
b5da15e2d8 client: Add /modules/ support 2020-03-25 10:01:14 +01:00
Brian C. Lane
d530d9a87e weldr: Fix projects/depsolve route handling
Return a 400 with json error message when no packages are included, with
or without trailing /
2020-03-25 10:01:14 +01:00
Brian C. Lane
271d27a41d client: Add /projects/ support 2020-03-25 10:01:14 +01:00
Brian C. Lane
5ac3cb3f46 rpmmd: Fix the format of BuildTime to match the weldr API
The format of the BuildTime returned by /projects/list and /modules/list
does not include the 'Z' at the end. This fixed the format and adjusts
the tests.
2020-03-25 10:01:14 +01:00
Lars Karlitski
ce6dc53a12 weldr: also pass custom sources to Store.PushCompose()
Also move assembly of a list of all repositories (base + custom sources)
into its own function.
2020-03-24 22:29:41 +01:00
Lars Karlitski
3544590036 store: move adding the local target to weldr
The automatic local target is only needed when accessing the API via
weldr.

In the store, the target was only added when `stateDir` was not `nil`.
This is only used for testing which doesn't exercise the branch in
weldr. Thus, the same check is not needed there.
2020-03-24 22:29:41 +01:00
Martin Sehnoutka
9d2dacbcab internal: pass architecture from the APIs to dnf-json
The following commit will introduce support for forced architecture in
dnf-json. The APIs already have this kind of information, so we can
simply pass it to the Depsolve and FetchMetadata functions.
2020-03-24 20:45:30 +01:00
Tom Gundersen
c66db42677 weldr/upload: uploadRequestToTarget() cannot fail
Drop the error return and simplify code accordingly.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-24 14:13:03 +01:00
Brian C. Lane
222d09499c weldr: Improve the error message from sourceNewHandler 2020-03-23 21:08:01 +01:00
Brian C. Lane
6e40729cd3 weldr: Update the sourceNewHandler to use the correct struct
The store supports different fields than the API, this switches to using
an API specific struct with support for proxy and gpgkey_files.
2020-03-23 21:08:01 +01:00
Brian C. Lane
3005486cb4 client: Add projects/source API support 2020-03-23 21:08:01 +01:00
Brian C. Lane
971bafbc09 Move the API struct definitions into internal/weldr
This will prevent problems with import loops. client already imports
weldr, so weldr cannot import client.
2020-03-23 21:08:01 +01:00
Tom Gundersen
d7aad58897 store/PushCompose: take ImageType and Arch as argumnet rather than strings
Resolve eagerly into real objects, and avoid having to resolve and
error-check again.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-23 15:43:55 +01:00
Tom Gundersen
8e5b79df96 weldr/uploadRequestToTarget: take ImageType as argumnet rather than a string
Resolve image type into a real object eagerly, and avoid having to
resolve and error-check again.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-23 15:43:55 +01:00
Tom Gundersen
bd86eaf6a5 weldr/composeTypesHandler: list image types per-arch
Currently all image types are supported on all arches, but in the
future we may want to restrict this. In that case, return the
image types that are valid for the arch in qusetion, rather than
all the possible ones.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-23 15:43:55 +01:00
Tom Gundersen
95203e0fff weldr/depsolveBlueprint: take ImageType as argumnet rather than strings
Resolve image type into a real object eagerly, and avoid having to
resolve and error-check again.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-23 15:43:55 +01:00
Tom Gundersen
090d01602b weldr/composeHandler: invert error handling
Check for errors and return early if they are found, rather than
check for the absence of errors.

This is not a functional change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-23 15:43:55 +01:00
Tom Gundersen
f201fc84b7 weldr: pin a real Arch object, rather than a string
weldr needs to know the host architecture. Rather than pinning
a string, pin a real Arch object, and query its name when we
need it.

This verifies the validitiy of the architecture for the given
distro before it is passed to weldr, rather than lazily on
demand.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-23 15:43:55 +01:00
Tom Gundersen
839b22026e target: pass the image filename in each target
Rather than having to assume that we only ever produce one
artifact, have each upload target contain the filename it expects
to upload from the osbuild output.

An image file is always explicitly named in the manifest, and we
leave it up to each distro to decide how this is done, but the
convention is to use the same image filename as used when
downloading the image through weldr.

Now make this policy explicit, by quering the distro for the image
name and inserting it into each upload target.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-22 22:26:09 +01:00
Tom Gundersen
61836a7079 weldr/upload: no longer reuse the structs from the target package
For serializeation, make our own private structs. The structs
in the target package are not exactly the same as the ones used by
weldr, so in order to avoid too many compromises, let's just do
an explicity translation.

As a general principle, we aim to only use private types for
serialization and rather translate than reuse for different
purposes.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-22 22:26:09 +01:00
Tom Gundersen
440a5acb41 weldr/upload: make translations between upload options explicit
The same types are used in the weldr API as internally. We want
to avoid sharing serialized types like this, as it easily leads
to layering vialotions.

For now just make the translation explicity, in a follow-up
we will introduce types dedicated to serialization in the weldr
API.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-22 22:26:09 +01:00
Lars Karlitski
069ccba894 store: pass repos to PushCompose
Repositories were not passed down into `Distro.Manifest()`. This makes
all builds fail, because these repositories are use to access gpgkeys.
2020-03-21 11:08:16 +01:00
Lars Karlitski
77556973cc distro: don't expose repositories in distro objects
Mixing the way to build a distribution with where to get the source
packages from is wrong: it breaks pre-release repos, local mirrors, and
other use cases. To accommodate those, we introduced
`/etc/osbuild-composer/repositories`.

However, that doesn't work for the RCM API, which receives repository
URLs to use from outside requests. This API has been wrongly using the
`additionalRepos` parameter to inject those repos. That's broken,
because the resulting manifests contained both the installed repos and
the repos from the request.

To fix this, stop exposing repositories from the distros, but require
passing them on every call to `Manifest()`. This makes `additionalRepos`
redundant.

Fixes #341
2020-03-18 20:46:28 +01:00
Lars Karlitski
e70e6a72a0 weldr: remove azure-specific size check
This is already done by the distros.
2020-03-18 08:33:42 +01:00
Lars Karlitski
0f1a014aca store: don't keep a "default" distro
Only the weldr API has the concept of a default distro. Pass that distro
explicitly to `PushCompose()` and fetch the distro from the compose in
all other functions that accessed Store.Distro.
2020-03-18 08:33:42 +01:00
Lars Karlitski
f68679c09f weldr: remove unused argument
819430e removed all `clean` arguments from rpmmd. This was a leftover.
2020-03-18 08:33:42 +01:00
Tom Gundersen
8400b39577 distro.Manifest: drop checksums parameter
This is no longer used to generate pipelines, but has been replaced
by the package specs.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-15 19:38:59 +01:00
Tom Gundersen
819430e659 rpmmd: no longer flush the caches on every call
When we used the dnf-based pipelines, we were relying on the fact
that the metadata was unlikely to have changed between we generated
the pipeline and called osbuild. We achieved this by always updating
to the most recent metadata on every call to rpmmd.Depsolve that
would end up in a pipelin.

Refreshing the metadata is time-consuming, and something we want
to avoid if at all possible. Now that our pipelines no longer
rely on this property, we can drop the flushing.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-15 19:38:59 +01:00
Brian C. Lane
e3d1a34ab6 weldr: Fix blueprints/depsolve response
The response always returns a 200 with a JSON response. Any errors will
be in the errors list.
2020-03-12 01:55:45 +01:00
Brian C. Lane
e13ebd46ee weldr: Fix blueprints/info response for unknown blueprints
The response is different for JSON and TOML requests. If it is JSON it
will always return a 200, but any blueprints with errors will be in the
errors list.

If TOML has an error it will return an error 400 with the error in a
standard API error response with status set to false.
2020-03-12 01:55:45 +01:00
Brian C. Lane
7bd020ca11 weldr: Return an error when an empty blueprint is received
The JSON and TOML parsers differ in how they handle an empty body so
check for a ContentLength of zero first and return a "Missing
blueprint" error to the client.

Includes updated tests for the JSON path, and new tests for empty TOML
blueprints.
2020-03-12 01:55:45 +01:00
Brian C. Lane
77fd2a0d8b store: Return an error from GetBlueprintChange
If the blueprint doesn't exist, or the commit for the selected blueprint
doesn't exist it will return an error.

This also fixes the blueprints/undo/ route to return the correct error
to the caller.
2020-03-12 01:55:45 +01:00
Brian C. Lane
b4710b52f1 store: Return an error from DeleteBlueprint and DeleteBlueprintFromWorkspace
If an unknown blueprint or workspace is deleted it will now return an
error.

Also fixes the blueprints DELETE handlers to return the correct error to
the client. Includes a new test.
2020-03-12 01:55:45 +01:00
Brian C. Lane
2675eff4eb weldr: Add tests for empty and invalid blueprints 2020-03-12 01:55:45 +01:00
Brian C. Lane
aed28ccf72 store: Add blueprint change order tracking
Previously the order that changes were made to blueprints was not being
saved. I worked around this by sorting by timestamp, but it only has 1s
resolution so it is very likely to end up with changes having the same
timestamp, especially when running tests.

This adds a new variable to the Store, it is a list of the commit hashes
for each blueprint, in the order they were made.

Since this is a change to the Store schema the first time the new code
is run with the old store state it needs to populate the commit list, as
best it can, with the existing data. To do that it sorts the changes for
each blueprint by timestamp and version and saves this ordering into the
new BlueprintsCommits list.
2020-03-10 12:41:19 +01:00
Brian C. Lane
eeef1e289c weldr: Add support for /blueprints/tag route
A POST to this route will tag the latest commit of a blueprint as a new
revision. The revision numbers start at 1 and increment on each call.
If the latest commit has already been tagged it ignores the request.
2020-03-10 12:41:19 +01:00
Jacob Kozol
4690320503 blueprint: add default version
If the user creates a new blueprint with no version specified, the
blueprint struct uses "0.0.0" as the default version. Blueprint tests
for a blueprint with an empty version now expect no error.
2020-03-04 19:20:58 +01:00
Tom Gundersen
571932db37 job: pass manifest instead of pipeline to osbuild
This is not a behavioral change, as all distros currently use
empty source objects. But when we move over to rpm-based pipelines,
this will change.

Make the same change to osbuild-pipeline, so these stay in sync.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-03 22:25:49 +01:00
Brian C. Lane
57f0892d93 weldr: Return blueprint changes in descending timestamp order 2020-03-03 12:21:40 +01:00
Ondřej Budai
d7cbc22da4 lint: fix unhandled errors 2020-03-02 14:28:55 +01:00
Ondřej Budai
b7875db046 lint: fix gosimple/S1011 errors 2020-03-02 14:28:55 +01:00
Ondřej Budai
efb100fbca lint: fix gosimple/S1005 errors 2020-03-02 14:28:55 +01:00
Lars Karlitski
740fb77d64 weldr: ensure passing valid package-specs to dnf-json
If no packages are included in a blueprint, the slice remains `nil`,
which translates to `null` in json. Always initialize the slice by
pointing it to an empty array.
2020-02-26 16:19:35 +01:00
Brian C. Lane
8e1bc2b644 Return Blueprint push errors via the API
This adds returning errors from the store PushBlueprint* functions, and
adds handling of the errors to the API code in preparation for new code
to check the blueprint before saving it.
2020-02-25 09:00:35 +01:00
Martin Sehnoutka
f1b5ee8859 rpmmd: rename fetchpackagelist to fetchmetadata
the name was misleading because the function could do more than just
download package list. In PushComposeRequest it is also used to fetch
checksums for the repositories, therefore I decided to rename it to
reflect this usage.
2020-02-20 13:04:28 +01:00
Brian C. Lane
b89475506e Add Epoch to blueprints/freeze version string
If the Epoch is > 0 the it should be added to the front of the version,
separated by a colon.

Also include a depsolve package with a non-zero Epoch and adjust the
tests accordingly.
2020-02-19 23:44:17 +01:00