The bug wasn't caught because the PackageSets field of the repository
wasn't being copied after parsing the compose request for the test
manifest.
This should now catch future occurrences of this bug.
The value comes from the command line args of the test binary
(osbuild-composer-manifest-tests). It works in our tests because we use
the default value, but if it was set differently it would have been
ignored.
Pass through all repos to the initalizeManifest() function. Each
pipeline will then select which repositories it needs based on the
PackageSets field of each repository.
Before, we only passed global repos down to the manifest generators and
pipeline-specific repositories would only be used if they were attached
to package sets and were handled explicitly by a pipeline generator.
The repositories of the "blueprint" package set are explicitly added to
the workload and returned by the "os" pipeline.
The repositories of the "installer" package set are explicitly added to
the "anaconda-tree" pipeline.
If a repository was specified for any other pipeline, for example
"build", the repositories for the that package set were never added to
the pipeline.
Fixes#3290
Function that filters a list of repositories to return only the global
repos and any that define a given pipeline or package set name in their
list of PackageSets.
Match the key for the OS packages to the name of the OS pipeline ("os").
We will use this key to identify package sets that are returned from the
OS pipeline since the manifest returns package sets indexed by each
pipeline's name.
New test that runs through a few different scenarios of assigning
repositories to specific package sets. Each scenario defines a set of
global repositories (or none) and a set of pipeline-specific
repositories (or none) and an expected result. It then calls the
distro's PackageSets() method. The test checks that the package sets
that are returned are assigned the correct repositories based on the
test case's expected result.
A basic string-set struct is implemented to help with merging and
comparing string slices.
This test should currently fail due to #3290.
Regenerate image manifests which use the files customization and are
affected by the recent change to `osbuild/fsnode.go`, specifically to
always setting the `RemoveDestination` to `true`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Default to always remove the destination before copying when generating
the copy stage options for custom files in the image. This will ensure
that if the destination is an existing symlink to another file, it won't
be followed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Set the dependency to osbuild version 80, which contains the
`remove_destination` option in the copy stage.
https://github.com/osbuild/osbuild/releases/tag/v80
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add support for newly added `remove_destination` option in the copy
stage.
Related to https://github.com/osbuild/osbuild/pull/1241
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Test the directories and files customization with the ostree container
image type deployed using the (Anaconda-based) installer image.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Reformat calls of `ansible-playbook` by splitting them into multiple
lines. This makes it easier to read.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the `check_ostree.yaml` to check the result of directories
and files customizations on a running ostree-based system. The test
also checks if a custom systemd unit `custom.service` got enabled
and started on boot.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Use directories and files customization in the compose request for image
types that support this customization (only ostree installer and raw
image do not support it).
Extend the instance verification to check for the custom directories and
files.
Extend the ostree commit verification to check for the custom
directories and files.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the `--format=json` was always set in the `$GCP_CMD`. This
was working OK in general, but it is creating issues when using
`$GCP_CMD compute ssh` to execute commands on the provisioned VM. In
such case, an extra `[]` is added to the output, which then causes
various checks to fail, since the captured stdout is different than
expected.
Instead remove the `--format=json` by default from `$GCP_CMD` and use it
explicitly only in cases when `jq` is used to parse the command's
output. In all other cases, it should be OK to not set the output format
to JSON and it is very much desirable to do so when SSH-ing to the VM.
Also by default pass the `--quiet` option to `gcloud` when using it from
the container. This was previously done only when using the tool
installed on the system.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add `iot-container` test manifests with customizations. Specifically with
files, directories and services.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add `edge-container` test manifests with customizations. Specifically with
files, directories and services.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add `edge-commit` test manifests with customizations. Specifically with
files, directories and services.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Drop unused `rhel-edge-*` requests, which are not used any more. They
have been replaced by `edge-*` requests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add the new directories and files customization to the customized qcow2
image manifests used for testing.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Expose the Directory and File customizations in the Cloud API. Also
validate the provided customizations while processing the compose
request by trying to convert them to internal representations
`fsnode.File` and `fsnode.Directory`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add new error `ErrorInvalidCustomization` used in situations when the
user-provided customization values don't pass validation. This will be
used by the Directory / File customization.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a default policy for custom directories and files to constrain what
users can do. The intention is to ensure that directories and files can
be created only in `/etc` and also that none of the important
configuration files can be overwritten by this customization.
Add the policy validation to all distro implementation.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add helper functions for checking directory and file blueprint
customizations against the policy of allowed paths.
These functions are not yet used in the distro definitions.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Hook up the custom BP directories and files implementation with OS
pipeline implementation. The user-provided values are now set in the OS
customizations structure and will be used by the OS pipeline generator
when adding stages to the pipeline.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a helper function for validating the user-provided directory and
file customizations. This is necessary to fail early on invalid input,
instead of when building the image.
The function ensures that:
- No file path is a prefix of another file or directory path
- There are no duplicate file or directory paths in the
customizations
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add helper functions for converting slices of directory and file
customizations structures from the `blueprint` package to a slice of
structures from the `fsnode` package, which are used in image type
definitions.
These will be used to convert BP customizations to the os pipeline
customization then used by the pipeline generator.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the Blueprint customizations with the representation for custom
Directories and Files specified by the user.
Implement custom Unmarshalers for TOML and JSON. These ensure that all
user-provided values are validated before use and also handle the fact
that user and group ownership for directories and files can be
specifies as a string or as an integer.
Implement helper functions for converting the Blueprint-specific types
for these customizations to their internal representation from `fsnode`
package.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Move the `CheckMountpoints()` implementation to `blueprint` package,
since it does not operate on any data structures from the `disk`.
Move the default mountpoint allow list policy definition to the
`pathpolicy` package.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Move the `FilesystemCustomization` structure and its custom
unmarshallers to a dedicated file. This makes `customizations.go` easier
to read.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `PathPolicies` implements a generic concept that can be fit on more
use cases than just mountpoints. Another one would be a policy for
creating custom files and directories in the image. Having the
implementation in the `disk` package and using data structures from the
`disk` and `blueprint` packages makes it impossible to use it for any
additional BP customization due to a circular dependencies that always
occurs.
Split out the implementation into a separate package `pathpolicy` as the
first step.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extend the `OSCustomizations` with a list of custom files and
directories, that should be created in the image. If any `Files` or
`Directories` are specified in the `OSCustomizations`, the appropriate
osbuild stages will be added to the `os` pipeline. In addition to that,
any custom files data will be returned by the `getInline()` method of
the `os` pipeline.
This customization can't be yet used by users, because the translation
from BP customization to the `OSCustomizations` is missing.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This will allow to conveniently add multiple stages to the pipeline at
once, which is useful if a generator function wrapping some
functionality generates more than one `Stage`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Move the handling of the list of enabled and disabled systemd services
more to the end of the `os` pipeline, just before the SELinux stage.
This has no functional effect on produced images, but it will make it
nicer once the handling of the custom files and directories will be
added to the pipeline. Specifically it should be added right before the
services stage to allow enabling custom service files, but after all
other configurations that are applied to the image.
Regenerate all manifests.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Implement helper functions for generating osbuild stages for a slices of
`fsnode` types, such as Files and Directories. The generated stages will
ensure that the provided FS nodes will be created in the FS tree and
will have their respective properties set (such as ownership, mode,
etc).
These functions are not yet used by any pipeline code, but the idea is
that they will be used in pipeline generator functions to create custom
directories and files based on the pipeline-specific customizations.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
