Introduce a new data structure `ImageConfig` holding the default OS
configuration applied when building an image. The structure can be used
to hold the default image configuration on the distribution level with
possible overrides defined on the image-type level.
As a starting point, move hard-coded default values and configuration
common for `osPipeline`, `ec2BaseTreePipeline` and `ostreeTreePipeline`
to the distribution and image-type default image configuration. This is
preparing the ground for merging all of these three pipeline functions
into `osPipeline`, which will produce the appropriate OS pipeline based
on the image-type configuration and the fact if it is rpmOstree or not.
Regenerate affected EC2 and AMI manifests. There is however no change in
the resulting image configuration and image-info report.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Because of the way the gcp secrets are stored for the workers, and how
the mapping from vault to openshift works (unable to map a multiple key
secret into a single json file), there's a bit of juggling required to
get the gcp credentials in the right format.
This test provisions a RHEL-8.6 VM locally, installs osbuild-composer on
it, upgrades the system to RHEL-9.0 and verifies osbuild-composer still
works by building a simple qcow image.
This directory is not used on worker instances. It was a left-over from the
times when this AMI was also used for running composer.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The worker needs quite a lot of configuration involving secrets. Baking them
in the AMI is just awful so we need to fetch them during the instance startup.
Previously, this was all done using cloud-init. This makes the cloud-init
config huge and it is also very hard to test.
This commit moves all the configuration scripts into the image itself.
Cloud-init still needs to be used to push the secret variables into the
instance. The configuration scripts are run after cloud-init. They pick up
yhe secrets and initialize the worker correctly.
These scripts were adopted from
75b752a1c0
(private repository).
During the adoption, some changes has to be applied to make shellcheck happy.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Detect folders that are not used since some timeout and delete them.
The cache folder must be empty when dnf-json is started in order to
avoid the situation where some folders can never be cleaned up (dnf-json
does not look at the cache directory content but uses information from
the requests to deduce which folders to keep and to delete).
Solves #2020
Bring the Tar stage implementation on par with the current osbuild
schema. Specifically add the 'format' and 'root-node' options to the
stage options structure.
Add stage options validation along with appropriate unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The existing test cases for `image-installer` image type were not
regenerated as part of PR #2135 which introduced 'image_type_tags'
because regenerating them using `generate-all-test-cases` failed on them
due to trying to analyze them using 'image-info'. These test cases were
most probably originally generated manually with --keep-image-info
option passed to `generate-test-cases`. And I missed that their
regeneration failed at that time.
Add the `image-installer` image type to all distros in
`distro-arch-imagetype-map.json` and (re)generate the test cases for it.
The image test case for CentOS Stream 9 is not generated, because there
are no repos defined yet for the distro. This is part of the CentOS 9
PR #2142.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new option `no-image-info`, which can be specified for an image
type in `format-request-map.json` in case the image can not be analyzed
using `image-info` and thus it should not be run as part of generating
the image test case.
This is the case for e.g. the `image-installer` image type in RHEL-8.6,
which effectively breaks generating all image test cases for RHEL-8.6
due to `image-info` failing to analyze the resulting image.
This change will allow us to consistently generate image test cases
(manifests only) for all image types, which were so far not covered by
the image testing.
Modify the `generate-test-cases` script to recognize the `no-image-info`
option in the `format-request-map.json`
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use `make scratch` for building RPMs without tests. This fixes the case,
when the RPM build with test fails to build due to changes in image
manifests. The whole reason of running the script is to regenerate image
test cases when the manifest changed, so this was a chicken and egg
problem.
No CI testing is needed, as this is a development tool.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
These tests can be run only in our CI environment, and not as pure unit tests
launched by go test ./... Let's make sure that all files that belong to these
tests are properly marked with build constraints.
This should fix the following error in Coverity Scan runs:
# github.com/osbuild/osbuild-composer/cmd/osbuild-auth-tests
runtime.main_main·f: function main is undeclared in the main package
Warning: Build command go build -o bin/ ./... exited with code 2. Please verify that the build completed successfully.
See https://github.com/osbuild/osbuild-composer/runs/4427232580?check_suite_focus=true
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
During manual cleanup of unused resources, the storage account can get
removed. The current storage account is not possible to remove
mannually, but adding this check to make it more resielient in future
scenarios.
The `Kernel` and `Network` members of the sysconfig stage options
structure were previously not declared as pointers. As a result, they
always appeared in the resulting JSON object, even though they were
empty. Use pointers to ensure that the members are omitted from the
resulting JSON object, if they were not defined.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for the new `org.osbuild.dnf-automatic.config` stage for
configuring DNF Automatic.
Add appropriate new unit tests for the stage implementation and modify
necessary existing unit tests.
Related to https://github.com/osbuild/osbuild/pull/936
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for the new `org.osbuild.yum.repos` stage for creating DNF /
YUM repository configuration files.
Add appropriate new unit tests for the stage implementation and modify
necessary existing unit tests.
Related to https://github.com/osbuild/osbuild/pull/932
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for the `PermitRootLogin` option in the `sshd.config` stage.
Valid values can be of type `bool` or `string`. Due to this reason, a
custom interface type is defined and a custom `UnmarshalJSON()` method is
defined for the `SshdConfigConfig` structure.
Modify unit tests to test the newly added option and test
(un)marhsalling of valid values of both types.
Related to https://github.com/osbuild/osbuild/pull/917
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use image_type_tags in repo definitions used for generating image test
cases. Enhance the `generate-test-cases` script to take the
image_type_tags into account when creating a compose request for a
specific image test case.
The motivation for the change is to keep the list of repositories in the
compose request of a image test case as small as possible. Previously,
all of the defined repositories were part of each image test cases for a
specific architecture, even those that were not needed.
Regenerate affected image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When backed by a DB, composer has no need of a queue directory.
This also addresses "Error moving artifacts for job" logging noise.
Signed-off-by: sanne <sanne.raymaekers@gmail.com>
The change between the 32s bucket and the 64s bucket is too drastic
for measuring the duration of depsolve jobs. At present, 90% of the
depsolve jobs have a duration inbetween 32s and 64s, making the 32s
bucket too sensitive and the 64s bucket not sensitive enough.
