The `org.osbuild.chrony` stage currently supports only a single option
'timeservers' which does not allow specifying additional options for the
configured timeservers. The option can not be easily extended to allow
specifying additional options and at the same time keep the backward
compatibility with old manifests.
The need for a lower-level stage option allowing to configure additional
options of the 'server' directive is required by RHEL AMI images, which
use 'maxpoll', 'minpoll' and 'prefered' options.
Extend the `org.osbuild.chrony` stage with two additional options. The
'servers' option accepts a list of dictionaries specifying timeservers
to be configured using the 'server' directive, including a subset of the
directive options. The 'servers' option can not be used at the same time
in the stage options as the 'timeservers' option.
The second added option is 'leapsectz' corresponding with a directive of
the same name. The value of the option is a string. If the provided
string is empty, then all occurrences of the 'leapsectz' directive are
removed from the `chrony.conf`. Otherwise, the 'leapsectz' directive is
added to the `chrony.conf` with the provided value, while all original
occurrences of the option are removed.
Add a new stage test case under `test/data/stages/chrony-servers` to
test the new variant of the stage options. The reason is that the
'timeservers' option conflicts with 'servers' option, which makes it
impossible to test both of them in the same test case.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new `org.osbuild.cloud-init` stage, which currently allows to create
configuration files for cloud-init under `/etc/cloud/cloud.cfg.d`. The
stage supports only a very limited subset of cloud-init configuration
options, which is covering needs of RHEL AMI images.
The schema mandates that if the 'configuration_files' option is
specified, then at least one configuration file must be defined. In
addition each section of the configuration must contain at least one
property (section or configuration option).
Add `python3-pyyaml` package to the `F34-build` testing manifest,
because it is required for running and testing the new stage.
Regenerate all affected manifests.
Add test for the new stage.
Update the `osbuild-ci` container image used for testing to a new tag,
which includes python3-pyyaml, the dependency of the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new `org.osbuild.dracut.conf` stage. The option
'configuration_files' allows to create dracut configuration files under
`/usr/lib/dracut/dracut.conf.d` and thus make the
configuration persistent. The stage supports only a subset of all
configuration options allowed in dracut configuration. The intention is
to provide almost functional parity with the options supported by
`org.osbuild.dracut` stage.
The schema mandates that at least one configuration file must be defined
in the stage options. In addition, each configuration file must contain
at least one configuration option.
Add test for the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The logic to sort urls was added globally in `mpp-import-pipeline`
but only the in the v1 code path was the `state.manifest_urls`
variable set and thus for v2 the actual sorting did not happen.
Fix this and set the `manifest_urls` to the `org.osbuild.curl`
items, which makes sense because we only know how to sort those.
For `setfiles` in the `org.osbuild.selinux` to work properly it
needs itself have the correct labels. This is true for setfiles
on the host, but also needs to be true for the `setfiles` that
is located in the build root. Therefore we need to label the
build root via `org.osbuild.selinux`. Additionally re-label
the `cp` and `tar` binary to have the `install_exec_t` which is
needed to read and write unknown labels (to the host). Since
`cp` and `tar` are used to read (and write) files inside stages
and assemblers they need to have the special label.
The commit that added the ostree tarball to manifest version 2
went in after the PR to sort the urls and thus the source urls
for that manifest were not sorted. This of course no breaks ci
which makes sure that the test data up to date. I blame the
ci model used by github but I am also sorry.
Convert the test to use `pytest` and split out the individual
tests. The temp-directory fixture has the session scope so
that checkpoints can be shared between the individual tests.
Remove the dependency on unittest for the `OSBuild` class which
used the `unittest` instance only for `assertEqual`, which can
easily also be done via a plain `assert`.
In both mpp-depsolve and mpp-import-pipeline, sort the packages to
url dictionary before writing the JSON. This makes it easier to
look for packages but more importantly ensures that the resulting
set of packages has the same ordering in the sources section
independently of how it was assembled.
Instead of operating directly on a file, which was previously specified
by `filename`, operate on a device. This is more flexible since a file
can be accessed via a loop back device; but the inverse is obviously
not true, like other devices can not be accessed via a plain file.
Therefore, re-factor the stage to use a device and adapt the existing
test (`fedora-ostree-image`).
Add new stage `org.osbuild.systemd-logind` allowing to create
systemd-logind configuration drop-ins in `/usr/lib/systemd/logind.conf.d`.
Currently only the `NAutoVTs` option in the `Login` section can be
configured.
The schema mandates that:
- There must be at least one configuration file defined.
- The 'Login' section is required, as it is the only one in the
systemd-logind configuration.
- At least one option must be configured in the 'Login' section.
Add test for the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new stage `org.osbuild.modprobe` allowing to create modprobe
configuration files in `/usr/lib/modprobe.d`. Currently only the
`blacklist` command can be used in the configuration files.
The schema mandates, that at least one configuration file must be
defined.
Add test for the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `org.osbuild.sysconfig` stage to create `ifcfg-*` files
under `network-scripts` subdirectory. It is possible to set only values
currently set in RHEL AMI images, specifically:
- BOOTPROTO
- DEVICE
- IPV6INIT
- ONBOOT
- PEERDNS
- TYPE
- USERCTL
Change all `configure_*` functions to raise ValueError exception,
instead of returning values. As a follow up change, remove all checks of
the returned value from these functions.
Update the `org.osbuild.sysconfig` stage test case to create ifcfg
configuration files for two interfaces.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `org.osbuild.systemd` stage to create drop-in configuration
files for Systemd `.service` units under `/usr/lib/systemd/system`.
Currently only the `Environment` option in the `Service` section can be
configured.
Update the `org.osbuild.systemd` stage test case to create drop-in
configuration `10-rh-enable-for-ec2.conf` for `nm-cloud-setup.service`
unit, as used in RHEL AMI images.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `org.osbuild.rhsm` stage to configure selected options in the
subscription-manager configuration (in `/etc/rhsm/rhsm.conf`). It is
possible to set only values currently set in RHEL AMI images,
specifically:
- `manage_repos` option in `rhsm` section
- `auto_registration` option in `rhsmcertd` section
Ensure that the stage does not "touch" any configuration files, unless
it actually changes them. This prevents changing the file modification
time.
Update the `org.osbuild.rhsm` stage test case to set the additional
configuration options.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new manifest that creates an ostree commit, deploys that,
creates a raw image and copies the deployment into it. The
resulting artefact is a bootlabel qcow2 image.
Add the ability to only read a sub-tree of a tree via `Object.read_at`.
Expose the functionality via the `Store{Server,Client}.read_tree_at`.
Extend the tests to check this new functionality.
Often, a message is being sent and followed by a call to `recv`
to wait for a reply. Create a simple helper `send_and_recv` that
does both in one method.
Add a simple check for that helper to the tests.
Add a new constructor method that allows creating a `Socket` from
an existing file-descriptor of a socket. This might be need when
the socket was passed to a child process.
Add a simple test for the new constructor method.
Add a new constructor method, `Socket.new_pair`, to create a pair
of connected sockets (via `socketpair`) and wrap both sides via
`jsoncomm.Socket`.
Add a simple test to check it.
Move from using 'zram' to 'zram-generator-defaults' in the ostree bootiso
testing manifest. More information is available in Fedora 33 Change
document [1].
Add org.osbuild.kernel-cmdline stage to fedora-boot.json manifest
because of change in how grub handles the kernel command line arguments
[2].
GRUB2 Stage 2 checksums in assemblers test are updated. The change have
been verified by building the fedora-boot.json manifest with each checked
filesystem and booting the image in QEMU with legacy mode.
[1] https://fedoraproject.org/wiki/Changes/SwapOnZRAM
[2] https://github.com/osbuild/osbuild-composer/pull/982#issuecomment-697356929
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The previous version covered too few use cases, more specifically a
single subscription. That is of course not the case for many hosts, so
osbuild needs to understand subscriptions.
When running org.osbuild.curl source, read the
/etc/yum.repos.d/redhat.repo file and load the system subscriptions from
there. While processing each url, guess which subscription is tied to
the url and use the CA certificate, client certificate, and client key
associated with this subscription. It must be done this way because the
depsolving and fetching of RPMs may be performed on different hosts and
the subscription credentials are different in such case.
More detailed description of why this approach was chosen is available
in osbuild-composer git: https://github.com/osbuild/osbuild-composer/pull/1405
Test that `checksum.verify_file` works correctly, which internally
uses the only other utility function `checksum.hexdigest_file`.
Check all algorithms currently supported by the `org.osbuild.curl`
source.
Add basic checks for the ostree source, which includes a successful
pull of a commit, an empty source entry and one where the specified
commit is non-existant. For this create a simple commit in a ostree
repo is checked in. The commit was created via:
mkdir "/tmp/data"
echo "Hello World" > /tmp/data/hello.txt
ostree init --repo test/data/sources/org.osbuild.ostree/data/repo \
--mode=archive
ostree commit --repo test/data/sources/org.osbuild.ostree/data/ \
--branch "test/ostree" /tmp/data \
--timestamp="1995-05-13 12:34:56 +0000"
This should give an commit with the following commit id:
d6243b0d0ca3dc2aaef2e0eb3e9f1f4836512c2921007f124b285f7c466464d8
Parameterize `test_sources` via `pytest.mark.parameterize`, so that
now the product of source and test cases for that source is visible
to pytest and thus also the caller.
Instead of testing the legacy `SourcesServer` and `sources.get`,
test the `Source.download` method, which is the new and exclusive
way how sources are used in osbuild. For this, the complete info
for a specific source is now included the specific test case.
For the `org.osbuild.curl` source this means that the respective
information is moved from `sources.json`, which is then unused
and thus removed. The test case that checks for an unknown
checksum is also removed because `Source.download` just fetches
everything instead of a subset.
Use `pytest`, which is a dependency of osbuild since a few versions,
instead of `unittest` for the source tests. The main reason is to be
able to use `pytest.mark.parameterize` in the near future to easily
generate a test matrix.
In order to have a more stable package metadata representation,
sort the generated metadata by name. Adapt the tests' metadata
file to reflect that change.
When the expected metadata does not match the produced metadata,
write the latter to `/tmp` for closer inspection; additionally
this should help update the metadata file in case the changes
are expected.
This is a partial revert of commit d584a1e, which converted the
dynamically generate stage tests to sub tests. The problem with
the latter is that they can't be individually run, which makes
testing changes to specific stages cumbersome to develop.
Therefore switch back to a model where the stage tests are
dynamically generated via a class decorator.
In the output test, check that for a successful pipeline run all
the elements are present: main result, assembler result, stage
result.
NB: Build result is hard to test because we would need to actually
build a valid build root.
Extend the current ostree container test case to use the new
bootiso manifest to build a bootable iso. This is done in the
existing container test case in order to share the build root
and ostree commit results and speed up testing. In the future,
the test infrastructure should be extended so that the cache
can be optionally shared between test cases.
This new manifests creates an boot iso which contains an ostree
commit that is build in the same manifest. The boot iso can then
be installed via the ISO.
This is to test the functionality of the new `saved_entry` grub2
config file. Ideally an integration test would install a new non-
default kernel and check that it does not get selected. Something
for the future.
In addition to the required base layer, provided via the the input
of the same name, the oci-archive stage now accepts up to nine
additional layers that get added on top of each other, sorted in
ascending order, i.e. `layer.1` to `layer.9`.
Adapt the `fedora-ostree-container` example manifest so that the
ostree commit is now in a separate layer, which makes it possible
to share the base layer between different commits container.
