Commit graph

701 commits

Author SHA1 Message Date
Ondřej Budai
3b6bc69c37 api/weldr: propagate the ComposeResult to ComposeStatus
When a job doesn't exist for a compose, just return an empty result.
In the future this will cause logs disappear from jobs created before the
store-jobqueue split but we decided we can live with that, they are still
in /var for those who really want them.
2020-05-19 22:20:20 +02:00
Ondřej Budai
78d5109c81 worker/server: add Result to ComposeStatus
Return also the OsbuildJobResult in ComposeStatus struct. This change will
allow us to remove the JobResult function, see the following commits.
2020-05-19 22:20:20 +02:00
Ondřej Budai
65c7b78d4e api/weldr: make composeToComposeEntry() take composeStatus
This makes all the calls to composeToComposeEntry() method slightly easier
to read.

Not a functional change.
2020-05-19 22:20:20 +02:00
Ondřej Budai
5806e3ea08 api/weldr: make getComposeStatus() return composeStatus struct
The reasoning is the same as in one of the previous commits. I think that
returning multiple values from one method can be confusing and potentially
error-prone. Also, this method will return ComposeResult soon too.

Also, I chose to create a new composeStatus struct instead of reusing
worker.JobStatus. I think that's a good thing in terms of consistency and
reducing the dependencies between packages.

Not a functional change.
2020-05-19 22:20:20 +02:00
Ondřej Budai
fffec5fc38 api/weldr: getComposeState() -> getComposeStatus()
State is just one of the returned values, this will make more sense when
composeStatus struct is introduced.

Not a functional change.
2020-05-19 22:20:20 +02:00
Ondřej Budai
fa5b775718 worker/server: make JobStatus() return JobStatus struct
Personally, I don't like methods returning too much values. I think it's
easy to assign the values in an incorrect order and thus interpreting them
in a wrong way. Also, the following commits will make JobStatus() return
also the JobResult, which would make the number of returned values even
higher.

Not a functional change.
2020-05-19 22:20:20 +02:00
Ondřej Budai
4ebf6ee85b upload/koji: add a simple test
This commit adds a test which uploads a random file to Koji and runs CGImport.
The result is checked using the koji cli client.
2020-05-19 13:54:53 +02:00
Ondřej Budai
87a7e90c98 upload/koji: return the CGImport result
Currently, only build id is returned, more will come when needed.
2020-05-19 13:54:53 +02:00
Ondřej Budai
e43eb4da7b upload/koji: make the hexdigest field consistent
We usually try to match the field name with its json/toml/xmlrpc tag. This
commit does exactly that.
2020-05-19 13:54:53 +02:00
Ondřej Budai
184cd24426 upload/koji: handle the unmarshalling error 2020-05-19 13:54:53 +02:00
Ondřej Budai
0c87a7f524 upload/koji: handle the xmlrpc error
This is the way to handle situations when the remote method errors.
2020-05-19 13:54:53 +02:00
Tom Gundersen
76515066a8 upload/koji: add helpers to upload to koji
This does not yet actually upload the image, and it only supports empty
images. You need to place a an empty file named <filename>, with a valid
extension (e.g., .qcow2) in /mnt/koji/work/<directory>/.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-19 13:54:53 +02:00
Major Hayden
d4b7c1d0c7 CI: Test EC2 import and boot
Build an AWS AMI image, upload it to S3, import it into EC2, and boot
it.

Signed-off-by: Major Hayden <major@redhat.com>
2020-05-18 10:27:24 -05:00
Tom Gundersen
a22cd78eb3 store/json: log on dropping composes on unmarshal
If an invalid compose is encountered, then we drop it. Make sure to log
if a logger object is provided.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
f8db2e28e1 store/json: move commits backwards compatibility
By the same logic as the previous patch.

This exposed an apparent bug, where we are sorting by the version field
of the blueprint in the change objects, but these were never
(un)marshalled, so would always be set to zero in the past.

Remove the no-op sorting for now, as it never had an effect, and
put a note whether we should instantiate the blueprint in the change
object somehow.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
15a0e78c15 store/json: move queue status backwards compatibility
Apply the backwards compatibility where the ImageBuild struct is
assembled. This should make the code simpler to reason about, by
reducing the structs we produce and later fix up.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
cdc4b1bd53 store/json: add some basic tests
In the process clarify the code a bit to make it easter to
test/understand.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
fba2af5f51 store: drop support for multi-image composes
The store only serves the weldr API, and that hard-codes the assumption
of only one image build per compose all over the place. Move this
assumption into the json serialization handler.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
e28fca493c store/json: split newComposesFromV0()
Not a functional change, just for readability.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
594858ab2c store/json: simplify ImageType handling
Serialize to a plain string, and do the translation for compatibility
from string to string.

This removes a lot of code, and if we change the store format, we should
just skip the conversion altogether.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
2c3790f20c common/ImageType: move to the store package
This is now only used for marshalling in the store, so keep the logic
where it is used.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
1fd3b9d543 weldr/ComposeEntry: marshal to strings rather than common.ImageType
This ended up being marshalled to the wrong strings, causing errors
in the queue display in cockpit-composer. This continues our effort
to keep plain strings in the JSON structs and resolve them separately
from marshal/unmarshal.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
9a6c05cfed store/fixtures: don't rename the fedoratest package
We may want to move this to the test_distro package in the future, but
calling it something else than its real name is just confusing.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
2fe4450620 store/compose/ImageType: use distro.ImageType objects
This reduces the amount of resolving and error checking we have to do.
This exposed a bug in weldr's ComposeEntry type, which will be fixed in
a follow-up commit.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
5ba7e21a72 store/compose: rename 'Id' to 'ID' to make linter happy
Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
df7a0fec22 store: merge the compose package into the store
The types exposed by the compose package are only used in the store API,
so move them there where they belong.

Making the ownership of the types clear, rather than having them live in
a package for themselves, will make it clearer how the types can be
modified in follow-up commits.

Also remove the JSON annotations, as these types are no longer used for
serialization.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
571a21c3ac store: drop pendingJobs
This is no longer used. Drop the field and the type definition.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
27e52a9755 store: use dedicated types for (un)marshaling
Distinguish between the types used at runtimes and the types used for
(un)marshaling. Also make the types private to the store package.

This should allow us to reduce the interdependencies between the
packages and more easily change things without accidentally breaking
backwards compatibility.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
8eb71ac9df mock/fixture/store: move to the store package
This way we can make more of the store fields and types private in
follow up commits.

This is not a functional change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
4c4bad5444 weldr/api_test: don't access what should be private Store fields
In a follow-up these will be made private.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
e5b3e737b6 test/helper: add comparison for ImageType objects
Two image types compare equal if they are named the same, and so are
their respective architectures and distros.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
bcf0bae2a7 distro/test: make TestArch and TestImageType public
No need to statically query these values from tests.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
eb658f5495 distro: add Distro() and Arch() accessors
These return the parent object of the Arch and ImageType, respectively.

Not a functional change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
c22797ffdb distro/test: rename to test_distro
Make the package easier to use from elsewhere, `test` is too generic.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
dc794a6d31 blueprint: DeepCopy() cannot fail
Don't return an error value, as it is always nil.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Tom Gundersen
3f2e7be9cf common/types: drop dead code
Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-18 11:50:15 +02:00
Jiri Kortus
cdcd09dd67 [tests] use assert.ElementsMatch instead of reflect.DeepEqual in distro_test
The asserts should do the same job and also make sure that the two
compared lists are the same at the same time.

Refs: #442
2020-05-14 00:34:30 +02:00
Jiri Kortus
b24957134b Improve coverage for distro packages - RHEL-8.3
Refs: #442
2020-05-13 20:32:05 +02:00
Jiri Kortus
04a0862c9a Improve coverage for distro packages - RHEL-8.2
Refs: #442
2020-05-13 20:32:05 +02:00
Brian C. Lane
369312989f blueprints: Fix handling of invalid blueprint names in the API code
Empty names are not allowed, and blueprint names should only contain
characters matching: ^[a-zA-Z0-9._-]+$

This also adds tests for the various places where the blueprint name
could potentially be wrong.
2020-05-13 20:00:52 +02:00
Brian C. Lane
56ae3d33c8 client: urlencode compose status filter parameters 2020-05-13 20:00:52 +02:00
Lars Karlitski
27e8e4b5d5 jobqueue: allow canceling jobs
This is not exposed to a worker yet. It will continue the job and get an
error when it tries to update the job's status to finished.
2020-05-13 16:45:09 +02:00
Lars Karlitski
b795ca25a2 fsjobqueue: remove unnecessary variable declaration 2020-05-13 16:45:09 +02:00
Lars Karlitski
8df143fabe fsjobqueue: pass accepted job types to New()
This makes the queue more type safe and allows to get rid of the
`pendingChannel` and `pendingChannels` helpers, which only existed to
create not-yet-existing pending channels.
2020-05-13 16:45:09 +02:00
Lars Karlitski
3240f11647 fsjobqueue: factor common functionality into maybeEnqueue() 2020-05-13 16:45:09 +02:00
Lars Karlitski
7592e38d3d jsondb: Allow passing nil as document to Read()
This makes Read() usable in situations where only the existence of a
value has to be checked.
2020-05-13 16:45:09 +02:00
Lars Karlitski
e599a95067 fsjobqueue: use a single mutex to protect all fields
In addition, it protects that all public functions operate on `db`
in transactions.
2020-05-13 16:45:09 +02:00
Lars Karlitski
e03c1fff65 fsjobqueue: factor reflect.Select out of Dequeue()
Hopefully this make it easier to read.
2020-05-13 16:45:09 +02:00
Lars Karlitski
6773c01722 jobqueue: drop JobStatus type
The enum is redundant information that can be deduced from the job's
times: queuedAt, startedAt, and finishedAt. Not having it reduces the
potential for inconsistent state.
2020-05-13 16:45:09 +02:00
Lars Karlitski
e1805d5f62 fsjobqueue: only update dependants when necessary
This is the same as the code in Enqueue(). There is no need to add a job
to q.dependants if all of its dependencies have finished.
2020-05-13 16:45:09 +02:00