After change to image-info [1], which shows the format version for qcow2
images, the `image-format` changed from string to a dictionary. However
the `open_image()` function still compares it with string. This causes
`raw` images to be converted by the script again to `raw` format. This
change fixes the issue, so that `raw` images are not converted, but used
as they are.
[1] 5937b9adca
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Update ostree-ng.sh to install and run ostree commit on UEFI VM
ostree.sh keeps on BIOS VM. Both of BIOS and UEFI are covered.
check_ostree.yaml ansible playbook has to be updated to support
both BISO and UEFI
s3cmd sync actually downloads metadata for all objects in a s3 bucket.
We have built a lot of RPMs, thus this takes 5 minutes on AWS and 25 minutes
on my laptop (!!!).
Let's use recursive put instead. This doesn't delete any files on the remote
side. As we upload RPMs only once, this also shouldn't fail on "the
object already exists". Using this method, we should be able to upload the
RPMs in seconds.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
sudo journalctl -af -n 1 -u "${WORKER_UNIT}" &
WORKER_JOURNAL_PID=$!
In this snippet, WORKER_JOURNAL_PID is set to the PID of the sudo process.
Sudo doesn't propagate any signals - therefore the child process of sudo
(journalctl in this case) isn't killed when a signal is sent to the parent.
Use pkill -P instead which kills all processes where sudo is the parent.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
GitLab CI builds its own rpms and thus it must be use a different path.
This commit modifies mockbuild.sh and deploy.sh to be able to add an
extra path segment into the path so GitLab can use a different path.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
According to Ben's suggestion, test has been updated to aligin with
customer scenario.
1. Setup ostree prod repo, building installer and edge upgrade
will be from prod repo
2. Containers for building installer and edge upgrade will be
running as stage repo
3. Before edge system update, prod repo will pull update content
from stage repo, make static-delta and summary
The `distribution` struct defined in multiple distributions contained
unused `imageTypes` field. Remove it to simplify code.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
By default, `qemu-img convert` creates qcow2 images usable in qemu 1.1 and
newer. RHEL 8 guest images are meant to be bootable on RHEL 6 though.
Unfortunately, RHEL 6 has qemu 0.12, therefore these images cannot be used
there.
To fix this, we need to use the new qcow2_compat option in qemu assembler
to override the default compat version and make qcow2 images that can be used
in qemu 0.10 and newer.
For this, we need osbuild 28 that isn't yet available in of any of
downstreams, therefore we need to pin it everywhere.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Change the "image-format" from a string to a dict, with a "type":
$value entry, where $value contains the previous plain string
data.
Additionally, include the qcow2 format version, if the given
image is indeed a qcow2.
Adapt all manifest test accordingly (partly done by Ondřej)
Python 3 script used for conversion of manifest tests:
import os
import json
for name in os.listdir(os.getcwd()):
if not name.endswith(".json"):
continue
print(name)
with open(name, "r") as old:
data = json.load(old)
info = data.get("image-info", {})
format = info.get("image-format")
if not format:
continue
info["image-format"] = {
"type": format
}
if format != "qcow2":
continue
info["image-format"]["compat"] = "1.1"
with open(name + ".new", "w") as new:
json.dump(data, new, indent=2)
new.write("\n")
new.flush()
os.rename(name+".new", name)
test: use the new image-info format in all test manifests
The previous commit converted only qcow2 and openstack manifests but this change
is actually needed for all manifests produced by the qemu assembler.
Co-Developed-by: Ondřej Budai <ondrej@budai.cz>
genisoimage might be removed from RHEL 9. The users are advised to switch
to mkisofs tools from the xorriso package. It should be a drop-in replacement.
The same change was recently done by libguestfs:
efb8a766ca2216ab2e32
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The template mentioned only "adequate" documentation which can be
interpreted in many ways. Mention the news/unreleased directory and the
guides explicitly and include links for convenience.
Add method to fetch Cloudbuild job log.
Add method to parse Cloudbuild job log for created resources. Parsing is
specific to the Image import Cloudbuild job and its logs format. Add
unit tests for the parsing function.
Add method to clean up all resources (instances, disks, storage objects)
after a Cloudbuild job.
Modify the worker osbuild job implementation and also the GCP upload CLI
tool to use the new cleanup method CloudbuildBuildCleanup().
Keep the StorageImageImportCleanup() method, because it is still used by
the cloud-cleaner tool. There is no way for the cloud-cleaner to figure
out the Cloudbuild job ID to be able to call CloudbuildBuildCleanup()
instead.
Add methods to delete Compute instance and disk.
Add method to get Compute instance information. This is useful for
checking if the instance has been already deleted, or whether it still
exists.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Modify all relevant methods in the internal GCP library to accept
context from the caller.
Modify all places which call the internal GCP library methods to pass
the context.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
- image_tests.sh is already being executed from the osbuild
repository by installing osbuild-composer-tests & calling the
script directly
- in this repo we've got distro_test.go::TestDistro_Manifest() and
distro_test_common.TestDistro_Manifest() which compare the static
manifests stored in this repository with the ones generated
dynamically by the code base. This is executed via `go test` and
runs against all available json files.
The above two items cover the part where we want to make sure that
the resulting content is what we expect.
Additionally the existing integration tests cover the part where
we build images, upload them to a cloud vendor and boot a new VM
from the image.
Validates the ref only when supplied through the API (i.e., doesn't
validate built-in defaults).
Regex matches ostree internal and cockpit-composer UI validation.
Added test case to compose API test.
Replacing repeated calls to u.Parse() with path.Join() on the URL's
path. This method handles certain edge cases differently:
- location not ending in / (http://example.org/repo):
- with the old method, the subsequent parsing of "refs/heads/" would
overwrite the path segment of the original URL, resulting in
http://example.org/refs/heads
- with the new method, "refs/heads" is appended to the location and
a / is added between the two parts if necessary.
- ref begins with / (location: http://example.org/repo/, ref: /ref):
- with the old method, the final parsing of ref would overwrite the
path segment of the URL, resulting in http://example.org/ref
- with the new method, the ref is appended and a / is added between
parts where necessary (same as above).
- ref is a full URL
(location: http://example.org/repo/, ref: http://example.com):
- with the old method, u.Parse(ref) would completely overwrite the
existing URL in u.
- with the new method, the ref is added as a sanitised URL path
resulting in http://example.org/refs/heads/http:/example.com.
The last one will probably result in an error in either case, but it's
probably less incorrect to coerce the ref argument into a path.
The response status code of the GET request is checked as well to
provide an appropriate error message if it is not 200 (OK).
If the data in the response is not a valid hex string, the error message
from the DecodeString() method isn't returned directly and it is
replaced by a more useful message. The original error message is
discarded.
This adds support for Packages to the store's json structures so that
they will be preserved across restarts of the osbuild-composer service.
Reading the old format will result in an empty []rpmmd.PackageSpec so
this does not require a composeV1 structure.
This adjusts current tests to account for the new struct member, and
tests osbuild-composer with empty results (eg. existing system will not
have this stored) and with the sets populated by test data.
This adds the compose's dependency list which was previously missing
from the osbuild-composer implementation of the WELDR API.
The dependencies used for the compose are saved, at compose time, in the
store. They are returned as part of the compose/info results, the 'deps'
field.
This adds a list of the depsolved packages to the store's Compose
struct. It is indexed by compose UUID and contains a list of
PackageSpecs that were used to construct the compose. This can assist in
auditing of the composes, or be used to duplicate the compose.
Using DISTRO_CODE simplifies test case selection and allows to test
different distro than the one test is running on.
This is used to run tests for RHEL 9.0 on F33 or RHEL 8.4
