Port all of the pipeline refactoring done to RHEL-90 to RHEL-86. Both
distros now use the same approach.
Regenerate all RHEL-8.6 and CentOS 8 image test cases.
[1] https://git.centos.org/centos/kickstarts/tree/master
Signed-off-by: Tomas Hozza <thozza@redhat.com>
RHSM configuration is now applied conditionally only on RHEL. The same
applies to the customization to subscribe the system on first boot.
The reason is that the CentOS `@core` package group does not contain
`subscription-manager`. Thus it is not installed on CentOS Stream by
default and also CentOS 8 image definitions don't apply any changes
to the RHSM configuration [1].
In addition, make sure to not install any subscription-manager
packages on CentOS Stream images.
Regenerate all CentOS 8 image test cases.
[1] https://git.centos.org/centos/kickstarts/tree/master
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Enhance the `osPipeline` to add necessary stages to the returned
pipeline, in case the image is RPM OSTree based. As a result, delete the
`ostreeTreePipeline` and replace its uses by `osPipeline`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Make the `osPipeline` self-contained in the sense, that no stages are
added to the returned pipeline outside of the function and the returned
pipeline is usable as returned.
Modify the `osPipeline` to add Kernel Cmdline, FSTab and bootloader
config stages to the pipeline if a valid partition table was passed to
the function. As the last one, the SELinux stage is appended to the
returned pipeline.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the EC2 SAP image specific configuration from `ec2SapPipelines`
to the EC2 SAP default image configurations data structure. As a
result, remove the `ec2SapPipelines` and `rhelEc2SapPipelines` entirely
and use `rhelEc2Pipelines` for all RHEL EC2 images.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the x86_64 specific configuration from `ec2X86_64BaseTreePipeline`
to x86_64-specific image configurations for EC2 / AMI images. As a
result, remove the `ec2X86_64BaseTreePipeline` entirely and replace it
with `osPipeline`.
Regenerate image test cases. While there are changed in the manifests,
the actual image configuration didn't change at all and thus the
`image-info` report was not changed.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move all hard-coded image configuration from the `ec2BaseTreePipeline`
function to the `ImageConfig` structure and update the respective EC2
images default configuration structure.
Update `osPipeline` and `ostreeTreePipeline` to handle all of the new
configuration values from `ImageConfig`.
Completely remove the `ec2BaseTreePipeline` and replace it with
`osPipeline`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the RHSM configuration settings to `ImageConfig` structure and use
when handling subscriptions in `osPipeline`, `ec2BaseTreePipeline` and
`ostreeTreePipeline` functions.
Regenerate image test cases. While there are changed in the manifests,
the actual image configuration didn't change at all and thus the
`image-info` report was not changed.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Introduce a new data structure `ImageConfig` holding the default OS
configuration applied when building an image. The structure can be used
to hold the default image configuration on the distribution level with
possible overrides defined on the image-type level.
As a starting point, move hard-coded default values and configuration
common for `osPipeline`, `ec2BaseTreePipeline` and `ostreeTreePipeline`
to the distribution and image-type default image configuration. This is
preparing the ground for merging all of these three pipeline functions
into `osPipeline`, which will produce the appropriate OS pipeline based
on the image-type configuration and the fact if it is rpmOstree or not.
Regenerate affected EC2 and AMI manifests. There is however no change in
the resulting image configuration and image-info report.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Because of the way the gcp secrets are stored for the workers, and how
the mapping from vault to openshift works (unable to map a multiple key
secret into a single json file), there's a bit of juggling required to
get the gcp credentials in the right format.
This test provisions a RHEL-8.6 VM locally, installs osbuild-composer on
it, upgrades the system to RHEL-9.0 and verifies osbuild-composer still
works by building a simple qcow image.
This directory is not used on worker instances. It was a left-over from the
times when this AMI was also used for running composer.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The worker needs quite a lot of configuration involving secrets. Baking them
in the AMI is just awful so we need to fetch them during the instance startup.
Previously, this was all done using cloud-init. This makes the cloud-init
config huge and it is also very hard to test.
This commit moves all the configuration scripts into the image itself.
Cloud-init still needs to be used to push the secret variables into the
instance. The configuration scripts are run after cloud-init. They pick up
yhe secrets and initialize the worker correctly.
These scripts were adopted from
75b752a1c0
(private repository).
During the adoption, some changes has to be applied to make shellcheck happy.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Detect folders that are not used since some timeout and delete them.
The cache folder must be empty when dnf-json is started in order to
avoid the situation where some folders can never be cleaned up (dnf-json
does not look at the cache directory content but uses information from
the requests to deduce which folders to keep and to delete).
Solves #2020
Bring the Tar stage implementation on par with the current osbuild
schema. Specifically add the 'format' and 'root-node' options to the
stage options structure.
Add stage options validation along with appropriate unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The existing test cases for `image-installer` image type were not
regenerated as part of PR #2135 which introduced 'image_type_tags'
because regenerating them using `generate-all-test-cases` failed on them
due to trying to analyze them using 'image-info'. These test cases were
most probably originally generated manually with --keep-image-info
option passed to `generate-test-cases`. And I missed that their
regeneration failed at that time.
Add the `image-installer` image type to all distros in
`distro-arch-imagetype-map.json` and (re)generate the test cases for it.
The image test case for CentOS Stream 9 is not generated, because there
are no repos defined yet for the distro. This is part of the CentOS 9
PR #2142.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new option `no-image-info`, which can be specified for an image
type in `format-request-map.json` in case the image can not be analyzed
using `image-info` and thus it should not be run as part of generating
the image test case.
This is the case for e.g. the `image-installer` image type in RHEL-8.6,
which effectively breaks generating all image test cases for RHEL-8.6
due to `image-info` failing to analyze the resulting image.
This change will allow us to consistently generate image test cases
(manifests only) for all image types, which were so far not covered by
the image testing.
Modify the `generate-test-cases` script to recognize the `no-image-info`
option in the `format-request-map.json`
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use `make scratch` for building RPMs without tests. This fixes the case,
when the RPM build with test fails to build due to changes in image
manifests. The whole reason of running the script is to regenerate image
test cases when the manifest changed, so this was a chicken and egg
problem.
No CI testing is needed, as this is a development tool.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
These tests can be run only in our CI environment, and not as pure unit tests
launched by go test ./... Let's make sure that all files that belong to these
tests are properly marked with build constraints.
This should fix the following error in Coverity Scan runs:
# github.com/osbuild/osbuild-composer/cmd/osbuild-auth-tests
runtime.main_main·f: function main is undeclared in the main package
Warning: Build command go build -o bin/ ./... exited with code 2. Please verify that the build completed successfully.
See https://github.com/osbuild/osbuild-composer/runs/4427232580?check_suite_focus=true
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
During manual cleanup of unused resources, the storage account can get
removed. The current storage account is not possible to remove
mannually, but adding this check to make it more resielient in future
scenarios.
The `Kernel` and `Network` members of the sysconfig stage options
structure were previously not declared as pointers. As a result, they
always appeared in the resulting JSON object, even though they were
empty. Use pointers to ensure that the members are omitted from the
resulting JSON object, if they were not defined.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
