This replaces Packages() and BuildPackages() by returning a map of
package sets, the semantics of which is up to the distro to define.
They are meant to be depsolved and the result returned back as a
map to Manifest(), with the same keys.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use en_US.UTF-8 as default for LANG, which is what previously was
used and is also needed to properly work on non-us/latin setups[1].
In the customization tests, use a different value than the default
one to check that the customization does in fact work.
[1] http://git.app.eng.bos.redhat.com/git/spin-kickstarts.git/tree/rhel8/rhel-8.2-kvm-x86_64.ks#n4
Co-authored-by: Achilleas Koutsou <achilleas@koutsou.net>
Note that this doesn't actually test for the ostree fields, I'm not sure
if that's possible with this test framework. But it does make sure that
a test compose won't try to fetch the url.
This commit adds support for uploading images directly to Azure using the
cloud API.
The UploadStatus part is currently not implemented and will be added in a
follow-up PR.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit adds and implements org.osbuild.azure.image target.
Let's talk about the already implemented org.osbuild.azure target firstly:
The purpose of this target is to authenticate using the Azure Storage
credentials and upload the image file as a Page Blob. Page Blob is basically
an object in storage and it cannot be directly used to launch a VM. To achieve
that, you need to define an actual Azure Image with the Page Blob attached.
For the cloud API, we would like to create an actual Azure Image that is
immediately available for new VMs. The new target accomplishes it.
To achieve this, it must use a different authentication method: Azure OAuth.
The other important difference is that currently, the credentials are stored
on the worker and not in target options. This should lead to better security
because we don't send the credentials over network. In the future, we would
like to have credential-less setup using workers in Azure with the right
IAM policies applied but this requires more investigation and is not
implemented in this commit.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This file contains a client for Azure Storage API. As we soon introduce the
client for Azure API, we need a distinction here.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The UploadImage method doesn't actually create an image. It creates a Page
Blob. Blob is something like S3 object but in the Azure terminology. Page
Blob means that's optimized for random access and it's the only blob type
that can be used to create images.
This commit cleans up the terminology so it's less confusing.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
If the image size isn't aligned to 512 bytes, the Azure API returns very hard
to understand error message. Let's do this check ourselves early so we can
return a sane error.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Return GCP-specific target results form the worker, similar as it is
done for AWS.
Extend Cloud API to allow GCP-specific upload Options.
Modify Cloud API to return UploadOptions as part of the UploadStatus.
Modify Cloud API integration test to check returned upload Options and
upload Type.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add the TargetResult struct to OSBuildJobResult. Include the 'options'
interface on TargetResult to contain target-specific information,
for example amiID and region from AWS. Expose 'options' on a status
call as an UploadStatus field. Add logic to support AWS within this
format, which can be used as a template for other targets.
Upload target type is currently not returned form the worker, but
hardcoded in the cloudapi code to always return "aws". This make testing
of the cloudapi for other cloud providers quite complicated.
Since extending the target status information returned from the
worker is currently in progress, work around the situation for now
by returning an empty string as the upload type.
This will allows other types of upload targets to be tested as part of
cloudapi test case.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for GCP as an upload target to the internal API.
Extend the cloudapi to allow GCP as an upload target in the compose
request. Regenerate the cloudapi go code. Added GCP-specific upload
result component in the API definition, similar to AWS. It is not yet
used, but it will be once returning a target-specific result from
worker is supported.
Add support for GCP upload target to the worker job implementation.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new internal upload target for Google Cloud Platform and
osbuild-upload-gcp CLI tool which uses the API.
Supported features are:
- Authenticate with GCP using explicitly provided JSON credentials
file or let the authentication be handled automatically by the
Google cloud client library. The later is useful e.g. when the worker
is running in GCP VM instance, which has associated permissions with
it.
- Upload an existing image file into existing Storage bucket.
- Verify MD5 checksum of the uploaded image file against the local
file's checksum.
- Import the uploaded image file into Compute Node as an Image.
- Delete the uploaded image file after a successful image import.
- Delete all cache files from storage created as part of the image
import build job.
- Share the imported image with a list of specified accounts.
GCP-specific image type is not yet added, since GCP supports importing
VMDK and VHD images, which the osbuild-composer already supports.
Update go.mod, vendor/ content and SPEC file with new dependencies.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The openstack boot test often ruins our days with:
Waiting for instance 63ac19be-2e19-44e2-8bef-9770d68a190c to become Active
failed: A timeout occurred
I decided to investigate. It turns out the first boot of an image can take
up to 18 minutes. The subsequent ones are usually much faster (but don't rely
on this fact, I saw 15 minutes there).
This commit bumps the timeout to 30 minutes. This should be plenty of time
for the instance to spin up and get into the ACTIVE state.
Honestly, I'm not very happy with the solution but it should help with the
failing Schutzbot. As a follow up, I will reach to the PSI OpenStack team
and ask them if we could somehow speed up the process (maybe by using another
flavor, ci.m1.medium.ephemeral just might be slow for some reason, I don't
know).
Anyway, this should help us in the short term because I strongly believe that
a slow test is still better than a failing one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Explicitly set the kernel to boot into.
Also change the blueprint/kernenl handling:
Rather than only falling back to the default kernel name for
getting the package list, let GetKernel() always return the
correct result so we can rely on this being consistent.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For now this is simply used to resolve the parent commit, in case
one is not provided. In the future it will be used by new image
types to actually pull content from.
This extends the weldr API, so that future work does not have to
modify that.
The logic we now implement for the ostree commit image types is:
If the URL is provided, but the parent commit is not. The parent
commit is taken to be the current HEAD of the ostree repo at the
given url, with the given (or default) ref.
This only provides a small optional convenience, but we will
soon introduce image types where the URL of the repository is
required.
This commit still needs testing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than setting this automagically, expose it to the caller. For
now the only caller we have simply passes it back in, so this is a
noop.
In follow-up commits this will be used to resolve the parent commit.
This is tested by verifying that the generated manifests do not
change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We aim at shrinking our deps eventually but we need subman for the time
being. This patch basically un-exclude subman which was introduced by
https://github.com/osbuild/osbuild-composer/pull/893
Signed-off-by: Antonio Murdaca <runcom@linux.com>
New upload target for VMWare, similar to the ones for AWS and Azure,
allowing users to set credentials for their vSphere instance.
Commit also includes function that performs the actual upload.
The kernel now comes from the blueprint packages even when it's not
specified. Removing from the base packages of the image types avoids
duplication and allows for alternative kernels to be specified without
also including the default.
The latter is necessary for RHEL for Edge and Fedora IoT images (ostree
commits) that fail to build when multiple kernels are installed.
ImageType tests modified to fix expected package order.
Blueprints can now be used to specify a kernel as part of the kernel
customizations. Specifying a kernel adds it to the package list.
If no known kernel is specified (neither in the customizations nor the
package list), the default "kernel" is included automatically.
If kernels are specified in both the package list and the
customizations, both are added (even if they're duplicates).
s390x isn't supported on Centos.
rhel-commit-edge sounds just wrong for Centos. We can revert this change any
time. The thing is that I wasn't able to find something like CentOS IoT and
we don't want to be in a position of defining a new distribution spin.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I'm sorry, I need to extend this condition and my brain isn't powerful enough
to reason about complex negative conditions.
Not a functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The image definition is shared with the latest RHEL 8.y one (8.4 currently).
I expect that we the introduction of 8.5 support, we point the centos 8
distro at it.
The test repositories and manifests use the official CentOS composes. From
what I can tell, they are persistent. This is not guaranteed though, so we
might need to switch to RPMRepo at some point.
The "classic" CentOS 8 should also be buildable but due to the chicken and egg
issue (this commit will get into Centos "8.4" but Centos "8.4" isn't a thing
yet), we cannot test it and therefore it might be broken.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Test for each distro that runs through all architecture - image type
combinations and calls the Manifest() method with a kernel boot option
customization and checks if the ostree image types produce the expected
error.
Kernel boot parameters have no effect on ostree type images (Fedora IoT
and RHEL for Edge). Catch this and fail early in the pipeline creation
and communicate the issue to the user.
Optionally allow a pacakge set to be included in the compose request.
The specified packages are added to the base packages before
depsolving. As the base packages differ between the image types
the package customizations may have different results on the different
images part of the compose request.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Modify RHEL 8.3 and 8.4 KVM guest images definition to produce osbuild
manifest with `org.osbuild.rhsm` stage to disable both RHSM DNF plugins
(`product-id` and `subscription-manager`).
Update `/docs/news/unreleased/osbuild-rhsm-stage.md` to note that RHEL
8.3 and 8.4 qcow2 image definitions are updated to disable RHSM DNF
plugins by default.
Enhance `tools/image-info` tool to add RHSM-specific section to its
output in case RHSM DNF plugins configurations exist in the tree.
Regenerate all RHEL image test cases affected by the patch set.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support to configure `org.osbuild.rhsm` osbuild stage. This stage
allows the configuration of Red Hat Subscription Manager (RHSM) related
components. Currently it is possible to configure only the enablement
status of RHSM DNF plugins.
Add `/docs/news/unreleased/osbuild-rhsm-stage.md` with information about
the added support for `org.osbuild.rhsm` osbuild stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The org.osbuild.sysconfig stage is now supported. Config updates can be
made to the kernel and network files. Currently, the same values are
used for all image types in rhel84. The image-info script is updated to
allow testing the sysconfig info.
Expose a more detailed job status result - specifically, include upload status
alongside image status. Expand openapi.yml accordingly and add an UploadStatus
field to the OSBuildJobResult struct. At the moment, only represent the
"success" and "failure" states of UploadStatus - to differentiate between
"pending" and "running" would involve significant design decisions and should be
addressed in a separate commit.
There is some confusion surrounding the format of the source TOML that
can be sent to the server. The format it accepts doesn't match the
output from composer-cli which includes the source id in [] eg.
[k8s]
name = "kubernetes packages"
...
This patch changes the parsing to allow the id to be set as 'id = "k8s"'
or passed as a map in [k8s]. If the id is passed in the body it takes
precedence over the map name.
