If the ostree test was run on an unsupported distro, it failed but with a
very weird error message. This commit makes the test fail fast and with a
nice message.
The downloaded image may not fit inside tmpfs, especially when testing
on a constrained VM. This commit makes the test script use a different
temporary directory while handling the possibly big image.
Remove both the package osbuild-composer-koji, and the only file it
shipped: osbuild-composer-koji.socket.
It's been deprecated since 835b556, but the backwards-compatible
solution in that commit never worked, because osbuild-composer only
checks for "osbuild-composer-api.socket" when starting up.
Since this has been meant to be deprecated for a while, just remove it
outright.
Add an "Obsoletes:" for the package, so that it gets uninstalled on
existing systems.
Add an API route that returns logs for a specific compose.
For now, this contains the result of the job, in JSON. The idea is to
put more and more of this information into structured APIs. This is a
first step to make logs available at all.
Amend koji-compose.py to check that the route exist and contains as many
"image_logs" as images that were requested (currently always 1).
Based on a patch by Chloe Kaubisch <chloe.kaubisch@gmail.com>.
The integration tests are leaving the composes (which include images) in
osbuild-composer. This can lead to exhausting the disk space we have available
on our tiny testing machines. This commit adds a removal of the composes
after each integration test is finished. This issue is not present in koji.sh
and api.sh as they use different osbuild-composer APIs that doesn't use the
artifact feature.
This issue occurred when I worked on enabling the Fedora 33 tests, see:
https://osbuildci.cloud.paas.psi.redhat.com/blue/organizations/jenkins/osbuild%2Fosbuild-composer/detail/PR-1014/23/pipeline
We claim to have self-contained test cases, but the base_tests.sh script
still requires the WORKSPACE environment variable to be set outside of
the script, which is what Jenkins does.
This patch replaces WORKSPACE with a temporary directory and modifies
Jenkinsfile to use it when collecting logs.
We have several repository definitions across the tests which is quite messy.
This commit switches the Koji test to use the "central" repository configs defined in test/data/repositories/
The worker server was heavily tied to OSBuildJob(Result). Untie it so
that it can deal with different job types in the future.
This necessitates a change in the jobqueue: Dequeue() now returns the
job type, as well as job arguments as json.RawMessage. This is so that
the server can wait on multiple job types with different argument
types.
The weldr, composer, and koji APIs continue to use only "osbuild" jobs.
Introduce JobImplementation and turn the current RunJob() into
OSBuildJobImpl. Make main() select a job impl based on job type.
This is in preparation to add additional impls.
Move the fact that the worker is requesting jobs of type "osbuild" out
of the client library.
For one, require consumers to pass accepted job types to RequestJobs()
and allow querying for the job type with the new Type() function.
Also, make OSBuildArgs() and Update() generic, requiring to pass an
argument that matches the job type.
Now, main() does not deal with OSBuildJobResult anymore, and RunJob()
doesn't return it. This means we can add more job types (i.e., different
RunJob()s) now.
WatchJob() regularly checks if a job was canceled in a goroutine. It
does so by accessing composer's `/jobs/{token}` route. However, once the
main goroutine marks the job as done (by sending PATCH to that same
route), the `token` is no longer valid and thus the route not accessible
anymore.
main() does cancel the goroutine running WatchJob, but it's not
guaranteed that it gets scheduled in time to actually stop watching the
job.
Thus, don't cancel the job when fetching the `/jobs/{token}` fails. This
means that it won't cancel the job anymore when the connection to
composer goes down.
Also, we will be able to move job.Update() into RunJob().
Workers reported status via an `osbuild.Result`, which only includes
osbuild output. Make it report OSBuildJobResult instead, which was meant
to be used for this purpose and is already used as the result type in
the jobqueue.
While at it, add any errors produced by targets into this struct, as
well as an overall success flag.
Note that this breaks older workers returning the result of an osbuild
job to a new composer. I think this is fine in this case, for two
reasons:
1. We don't support running different versions of the worker and
composer in the weldr API, and remote workers aren't widely used yet.
2. Both osbuild.Result and worker.OSBuildJobResult have a top-level
`Success` boolean. Thus, logs are lost in such cases, but the overall
status of the compose is not.
This function is almost the same as the koji uploader, except that it
calls `CGFailBuild` instead of `CGImport` at the end.
Don't exit early from RunJob() when the job failed. Instead, go through
all the uploaders anyway. All the others don't do anything when the job
fails, but now we have the chance to do the necessary `CGFailBuild` call
for koji.
This moves more logic from main() into RunJob(), so that we can support
different job kinds in the future.
Add "image_name" and "stream_optimized" fields to the osbuild job as
replacement for the local target options. The former signifies the name
of the uploaded artifact and whether an artifact should be uploaded at
all (only weldr API). The latter will be deprecated at some point, when
osbuild itself can make streamoptimized vmdk images.
This change separates what have always been two distinct concepts:
artifacts that are reported back to the composer node (in practice
always running on the same machine), and upload targets to clouds and
such. Separating them makes it easier to add job types that only allow
one upload target while keeping artifacts.
Keep the local target around, so that jobs that are scheduled can still
be run after an upgrade.
The server hasn't used common.ImageBuildState to mark a job as
successful or failed for a long time. Instead, it's using the job's
return argument for that. (Jobs don't have a high-level concept of
failing).
Drop the check in the server, and always send "FINISHED" from the client
for backwards compatibility.
osbuild reports failing builds in two ways: it sets the "success" field
in its output to `false` and it returns with a non-zero exit status. The
worker used both, returning an `OSBuildError` when osbuild return
non-zero, but also forwarding the resulting object with the "success"
field.
Change this to only use the "success" field and ignore the return value.
The latter is useful for people running osbuild in a terminal or script,
but is redundant for this use-case.
This makes error reporting more consistent: `RunOSBuild` only returns an
error when *running* osbuild failed, not when the build fails.
ComposeState is only used by the weldr API.
Drop the JSON marshaller and unmarshaller, because ComposeState is not
used in an JSON-exported field anymore.
This state is specific to weldr. Previous commits removed it from the
other APIs, because they use different values.
Move the conversion into the weldr API.
This is similar to the previous commit, which did this change in
package cloudapi.
Use constants instead of string literals for compose status, and derive
the status from worker.JobStatus directly, instead of via common.State.
Don't use common.State anymore, because it has different values from
what's defined in openapi.yml. It makes sense to have these strings
defined in the same package as the spec — ideally, the code generator
would make them for us.
While at it, add a "running" status.
Fix the api.sh test to use these new statuses. Thanks to Ondřej Budai
for an additional fix there.
We don't install any packages in test cases anymore, therefore we don't need
to install EPEL there.
A slightly different explanation:
osbuild-composer-tests depends on packages from EPEL on RHEL. Therefore, you
cannot run the test cases without EPEL installed. Therefore, there's no
point in installing EPEL there.
docker.io has recently introduced a rate limiting on container pulls causing
the koji test to fail quite often.
To fix this issue, I created our own postgress mirror[1]. This commit switches
the Koji test to use it. Note that this change bumps the postgres version from
12 to 13.
[1]: 7db3c6802e
RHEL 8.3 is GA so we can run the ostree test here. This is a first step
to remove the RHEL 8.3 testing pipeline.
8.3 GA also ships a different version of composer-cli, therefore a slight
adjustment was needed.
Previous commits introduces a new way to generate all X.509 certificates
needed for testing. This commit reuses the same method for auth tests.
This has two benefits:
1) The new code generates certificates with Subject Alternative Name which
means we can use it on systems with Go 1.15 (Fedora 33, RHEL 8.4).
2) The new code generates much saner certificates.
The certificate generation is based on work by Lars Karlitski in our osbuild
CA. The server and client certs now contains Subject Alternative Name making
Python's request module and Go 1.15 happy (they deprecated certificates
without SAN).
Several reasons why we want to switch to the certificate generation:
1) The pre-generated certificates are not documented. If someone wants
to inspect them, he must know the right openssl incantation. This way,
you are able to see what's inside the certificates in a plain text.
2) The pre-generated certificates are going to expire at one point and
someone will be surprised.
3) Shipping private keys in RPMs is iffy. I know, it's just for testing but
still...
4) Auth tests are generating their own certificates. To achieve consistency,
we have two options:
a) Ship also all certificates for auth tests. That's extra 8 ones or
something like that.
b) Generate all certificates on fly. This commit does that.
5) The setup introduced by this commit is very similar to the one in our CA
making the test environment very similar to what's running in production.
tl;dr: I think this is a good step forward.
The Koji test in Github actions was always a bit quick and dirty solution.
I think it's much nicer solution to run it on Schutzbot.
Therefore, this commit moves the koji_test.go to a new osbuild-koji-tests
executable. This new test isn't run in the base test suite as one would
anticipate but inside the koji.sh test. This is needed because
osbuild-koji-tests requires a running koji instance. This might change
in the future but I think it works for now.
This was introduced in osbuild 23, so we also need to bump the dependency
in the spec file and also the submodule.
The test is also modified and a typo in its name is fixed.
We have the same thing for AWS. The AWS target also specifies under what name
should be the image available in EC2.
As requested by Brew maintainers Tomáš Kopeček and Lubomír Sedlář.
osbuild runs directly on the host, there's no intermediate container,
therefore we should set the container type to none.
As suggested by Brew maintainers Tomáš Kopeček and Lubomír Sedlář.
