By default, `qemu-img convert` creates qcow2 images usable in qemu 1.1 and
newer. RHEL 8 guest images are meant to be bootable on RHEL 6 though.
Unfortunately, RHEL 6 has qemu 0.12, therefore these images cannot be used
there.
To fix this, we need to use the new qcow2_compat option in qemu assembler
to override the default compat version and make qcow2 images that can be used
in qemu 0.10 and newer.
For this, we need osbuild 28 that isn't yet available in of any of
downstreams, therefore we need to pin it everywhere.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Change the "image-format" from a string to a dict, with a "type":
$value entry, where $value contains the previous plain string
data.
Additionally, include the qcow2 format version, if the given
image is indeed a qcow2.
Adapt all manifest test accordingly (partly done by Ondřej)
Python 3 script used for conversion of manifest tests:
import os
import json
for name in os.listdir(os.getcwd()):
if not name.endswith(".json"):
continue
print(name)
with open(name, "r") as old:
data = json.load(old)
info = data.get("image-info", {})
format = info.get("image-format")
if not format:
continue
info["image-format"] = {
"type": format
}
if format != "qcow2":
continue
info["image-format"]["compat"] = "1.1"
with open(name + ".new", "w") as new:
json.dump(data, new, indent=2)
new.write("\n")
new.flush()
os.rename(name+".new", name)
test: use the new image-info format in all test manifests
The previous commit converted only qcow2 and openstack manifests but this change
is actually needed for all manifests produced by the qemu assembler.
Co-Developed-by: Ondřej Budai <ondrej@budai.cz>
genisoimage might be removed from RHEL 9. The users are advised to switch
to mkisofs tools from the xorriso package. It should be a drop-in replacement.
The same change was recently done by libguestfs:
efb8a766ca2216ab2e32
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The template mentioned only "adequate" documentation which can be
interpreted in many ways. Mention the news/unreleased directory and the
guides explicitly and include links for convenience.
Add method to fetch Cloudbuild job log.
Add method to parse Cloudbuild job log for created resources. Parsing is
specific to the Image import Cloudbuild job and its logs format. Add
unit tests for the parsing function.
Add method to clean up all resources (instances, disks, storage objects)
after a Cloudbuild job.
Modify the worker osbuild job implementation and also the GCP upload CLI
tool to use the new cleanup method CloudbuildBuildCleanup().
Keep the StorageImageImportCleanup() method, because it is still used by
the cloud-cleaner tool. There is no way for the cloud-cleaner to figure
out the Cloudbuild job ID to be able to call CloudbuildBuildCleanup()
instead.
Add methods to delete Compute instance and disk.
Add method to get Compute instance information. This is useful for
checking if the instance has been already deleted, or whether it still
exists.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Modify all relevant methods in the internal GCP library to accept
context from the caller.
Modify all places which call the internal GCP library methods to pass
the context.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
- image_tests.sh is already being executed from the osbuild
repository by installing osbuild-composer-tests & calling the
script directly
- in this repo we've got distro_test.go::TestDistro_Manifest() and
distro_test_common.TestDistro_Manifest() which compare the static
manifests stored in this repository with the ones generated
dynamically by the code base. This is executed via `go test` and
runs against all available json files.
The above two items cover the part where we want to make sure that
the resulting content is what we expect.
Additionally the existing integration tests cover the part where
we build images, upload them to a cloud vendor and boot a new VM
from the image.
Validates the ref only when supplied through the API (i.e., doesn't
validate built-in defaults).
Regex matches ostree internal and cockpit-composer UI validation.
Added test case to compose API test.
Replacing repeated calls to u.Parse() with path.Join() on the URL's
path. This method handles certain edge cases differently:
- location not ending in / (http://example.org/repo):
- with the old method, the subsequent parsing of "refs/heads/" would
overwrite the path segment of the original URL, resulting in
http://example.org/refs/heads
- with the new method, "refs/heads" is appended to the location and
a / is added between the two parts if necessary.
- ref begins with / (location: http://example.org/repo/, ref: /ref):
- with the old method, the final parsing of ref would overwrite the
path segment of the URL, resulting in http://example.org/ref
- with the new method, the ref is appended and a / is added between
parts where necessary (same as above).
- ref is a full URL
(location: http://example.org/repo/, ref: http://example.com):
- with the old method, u.Parse(ref) would completely overwrite the
existing URL in u.
- with the new method, the ref is added as a sanitised URL path
resulting in http://example.org/refs/heads/http:/example.com.
The last one will probably result in an error in either case, but it's
probably less incorrect to coerce the ref argument into a path.
The response status code of the GET request is checked as well to
provide an appropriate error message if it is not 200 (OK).
If the data in the response is not a valid hex string, the error message
from the DecodeString() method isn't returned directly and it is
replaced by a more useful message. The original error message is
discarded.
This adds support for Packages to the store's json structures so that
they will be preserved across restarts of the osbuild-composer service.
Reading the old format will result in an empty []rpmmd.PackageSpec so
this does not require a composeV1 structure.
This adjusts current tests to account for the new struct member, and
tests osbuild-composer with empty results (eg. existing system will not
have this stored) and with the sets populated by test data.
This adds the compose's dependency list which was previously missing
from the osbuild-composer implementation of the WELDR API.
The dependencies used for the compose are saved, at compose time, in the
store. They are returned as part of the compose/info results, the 'deps'
field.
This adds a list of the depsolved packages to the store's Compose
struct. It is indexed by compose UUID and contains a list of
PackageSpecs that were used to construct the compose. This can assist in
auditing of the composes, or be used to duplicate the compose.
Using DISTRO_CODE simplifies test case selection and allows to test
different distro than the one test is running on.
This is used to run tests for RHEL 9.0 on F33 or RHEL 8.4
Previously, the Cloud API endpoint `/v1/compose/{id}` return value's
`image_status.status` for a running worker job was "running", which didn't
comply with the Cloud API specification. Equivalents allowed by the API
specification are "building", "uploading" or "registering".
As a result, the Image Builder API also does not comply in this regard
to its specification, because it currently just copies the status value
string returned by osbuild-composer.
Define the `image_status.status` as a reusable type in the Cloud API
specification. This forces openapi to generate an explicit type for it,
which can be then explicitly used in the code, instead of plain strings.
Return "building", instead of "running" for running compose.
Modify api integration test to check for all valid `image_status.status`
values for a compose.
Add News entry explaining this change.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
A result from manifest v2 contains logs from pipelines. The individual
pipelines are stored as an object, thus they have no order. Well, at least
in Go, because it doesn't guarantee one when parsing maps, see:
https://github.com/golang/go/issues/27179
Unfortunately, this makes the Result.fromV2 method return unpredictable
results because the pipeline results are processed in basically a random
order.
This caused the TestUnmarshalV2Failure test (result_test.go:124) to randomly
fail because it expects the ordering to be stable. I decided to fix this by
ordering the pipeline results by their name. When this fix is applied, the
output from Result.fromV2 is well-defined.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Detecting if a result is in the v1 format using DisallowUnknownFields was
clever. However, the format of v1 result changed in the worst way at some
point in the past: A field (id) got added. Therefore, this was an unknown
field to osbuild-composer and it fall back to decoding the input as a v2
result which led to empty logs.
This commit reimplements the result schema check: The new format always
have a top-level type field, whereas the old format doesn't have it. The
new implementation uses this fact to distinguish between v1 and v2 format
of a result.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When a stage is successful in a manifest v2, the success field is omitted from
the result. In other words, the default value of the success field is true
which is against the default value of boolean in Go. This commit implements
a workaround.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The previous version constructed multiple temporary variables and then
create job result from them. This was needed because we had multiple
upload targets but now that we have only one, this is only fragile
version of what can be done in a simplified way.
This PR removes the temporary variables and assigns errors and success
states right after the upload or build has finished.
Multiple upload targets are not supported by osbuild-composer any more.
Dropping support for this in worker therefore doesn't change anything
from the user's perspective, but it allows us to simplify the code a
bit.
Replace calls to "continue" with "return nil" because the job finished
correctly even though it failed to perform the task. But the failure was
reported to osbuild-composer for further processing so there is no need
to duplicate and report the same error in worker process
Drop support for LocalTarget, this has not been used in a long time,
and we don't really need to stay compatible across many releases
(just as long as we don't get problems with having to deploy in
lock-step), at least not yet.
Also drop support for KojiTarget, this has been replaced by the
osbuild-koji job type.
The previous implementation exited before reporting back to the worker
API in few branches. This left the compose status in RUNNING state even
though the worker did not work of the job any more. Refactoring the
API call into the `deref` part makes sure it gets called every time.
This commit only moves bits of the code around so that the status gets
back to osbuild-composer, but it still doesn't contain any useful
information in case osbuild fails etc. This will be introduced in
subsequent commits.
Run tests with user-specified ostree ref.
The ref needs to be specified in the container build as well (without a
URL) to be used for the commit that will be served to the installer.
The same variable/ref is also used to retrieve the $UPGRADE_HASH for the
final OS test.
Bug fix for changes introduced in #1244.
The new image types, rhel-edge-container and rhel-edge-installer, would
ignore the user-supplied ostree ref and use the default everywhere.
The default should only be used when a ref is not specified, which the
weldr API takes care of before calling the Manifest() method.
