Use the new partiton layout support in MPP. NB: start data as
well as the size of the last partition have been omitted now,
since `sfdisk` will figure those out for us.
Previous versions of mpp would already set the arch and basearch
substitution, which would work for dep-solving itself, but not
properly re-write the resulting URLs which means that the manifest
was broken. Fix this by properly replacing the substitutions in
the URL. Also support official 'releasever' substitution.
Instead of passing dictionaries around that are inconvenient to
use in code and even more in the `mpp-format-*` directives, use
a simple class to represent package information. Use that in
the `pkginfo` dict that can be accessed via `mpp-format-*`. Use
the `evra` property instead of string manipulation in the
`fedora-boot.mpp.json` and `-ostree-bootiso.mpp.json` manifest.
This uses size computations to simplify the partition size/offsets
and the depsolv results to get the kernel version.
This makes no changes to the resulting json files.
Add a new `org.osbuild.authselect` stage for configuring system identity
and authentication sources using `authselect`. The stage runs
`authselect select` command from the image in a chroot to set the
desired 'profile_id' profile. Optionally, a list of specific profile
features to enable can be passed using the 'features' option.
Add a test case for the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new option 'x11-keymap' for configuring the X11 keyboard settings.
The value of the option is a dictionary with keys representing settings
which can be configured. Currently, only the 'layouts' setting can be
configured. Its value is a list of strings, representing the specific
layouts, which should be configured for the X11 keyboard.
Update the stage test case with the new option 'x11-keymap'.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The `org.osbuild.chrony` stage currently supports only a single option
'timeservers' which does not allow specifying additional options for the
configured timeservers. The option can not be easily extended to allow
specifying additional options and at the same time keep the backward
compatibility with old manifests.
The need for a lower-level stage option allowing to configure additional
options of the 'server' directive is required by RHEL AMI images, which
use 'maxpoll', 'minpoll' and 'prefered' options.
Extend the `org.osbuild.chrony` stage with two additional options. The
'servers' option accepts a list of dictionaries specifying timeservers
to be configured using the 'server' directive, including a subset of the
directive options. The 'servers' option can not be used at the same time
in the stage options as the 'timeservers' option.
The second added option is 'leapsectz' corresponding with a directive of
the same name. The value of the option is a string. If the provided
string is empty, then all occurrences of the 'leapsectz' directive are
removed from the `chrony.conf`. Otherwise, the 'leapsectz' directive is
added to the `chrony.conf` with the provided value, while all original
occurrences of the option are removed.
Add a new stage test case under `test/data/stages/chrony-servers` to
test the new variant of the stage options. The reason is that the
'timeservers' option conflicts with 'servers' option, which makes it
impossible to test both of them in the same test case.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new `org.osbuild.cloud-init` stage, which currently allows to create
configuration files for cloud-init under `/etc/cloud/cloud.cfg.d`. The
stage supports only a very limited subset of cloud-init configuration
options, which is covering needs of RHEL AMI images.
The schema mandates that if the 'configuration_files' option is
specified, then at least one configuration file must be defined. In
addition each section of the configuration must contain at least one
property (section or configuration option).
Add `python3-pyyaml` package to the `F34-build` testing manifest,
because it is required for running and testing the new stage.
Regenerate all affected manifests.
Add test for the new stage.
Update the `osbuild-ci` container image used for testing to a new tag,
which includes python3-pyyaml, the dependency of the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new `org.osbuild.dracut.conf` stage. The option
'configuration_files' allows to create dracut configuration files under
`/usr/lib/dracut/dracut.conf.d` and thus make the
configuration persistent. The stage supports only a subset of all
configuration options allowed in dracut configuration. The intention is
to provide almost functional parity with the options supported by
`org.osbuild.dracut` stage.
The schema mandates that at least one configuration file must be defined
in the stage options. In addition, each configuration file must contain
at least one configuration option.
Add test for the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The logic to sort urls was added globally in `mpp-import-pipeline`
but only the in the v1 code path was the `state.manifest_urls`
variable set and thus for v2 the actual sorting did not happen.
Fix this and set the `manifest_urls` to the `org.osbuild.curl`
items, which makes sense because we only know how to sort those.
For `setfiles` in the `org.osbuild.selinux` to work properly it
needs itself have the correct labels. This is true for setfiles
on the host, but also needs to be true for the `setfiles` that
is located in the build root. Therefore we need to label the
build root via `org.osbuild.selinux`. Additionally re-label
the `cp` and `tar` binary to have the `install_exec_t` which is
needed to read and write unknown labels (to the host). Since
`cp` and `tar` are used to read (and write) files inside stages
and assemblers they need to have the special label.
The commit that added the ostree tarball to manifest version 2
went in after the PR to sort the urls and thus the source urls
for that manifest were not sorted. This of course no breaks ci
which makes sure that the test data up to date. I blame the
ci model used by github but I am also sorry.
In both mpp-depsolve and mpp-import-pipeline, sort the packages to
url dictionary before writing the JSON. This makes it easier to
look for packages but more importantly ensures that the resulting
set of packages has the same ordering in the sources section
independently of how it was assembled.
Instead of operating directly on a file, which was previously specified
by `filename`, operate on a device. This is more flexible since a file
can be accessed via a loop back device; but the inverse is obviously
not true, like other devices can not be accessed via a plain file.
Therefore, re-factor the stage to use a device and adapt the existing
test (`fedora-ostree-image`).
Add new stage `org.osbuild.systemd-logind` allowing to create
systemd-logind configuration drop-ins in `/usr/lib/systemd/logind.conf.d`.
Currently only the `NAutoVTs` option in the `Login` section can be
configured.
The schema mandates that:
- There must be at least one configuration file defined.
- The 'Login' section is required, as it is the only one in the
systemd-logind configuration.
- At least one option must be configured in the 'Login' section.
Add test for the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add new stage `org.osbuild.modprobe` allowing to create modprobe
configuration files in `/usr/lib/modprobe.d`. Currently only the
`blacklist` command can be used in the configuration files.
The schema mandates, that at least one configuration file must be
defined.
Add test for the new stage.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `org.osbuild.sysconfig` stage to create `ifcfg-*` files
under `network-scripts` subdirectory. It is possible to set only values
currently set in RHEL AMI images, specifically:
- BOOTPROTO
- DEVICE
- IPV6INIT
- ONBOOT
- PEERDNS
- TYPE
- USERCTL
Change all `configure_*` functions to raise ValueError exception,
instead of returning values. As a follow up change, remove all checks of
the returned value from these functions.
Update the `org.osbuild.sysconfig` stage test case to create ifcfg
configuration files for two interfaces.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `org.osbuild.systemd` stage to create drop-in configuration
files for Systemd `.service` units under `/usr/lib/systemd/system`.
Currently only the `Environment` option in the `Service` section can be
configured.
Update the `org.osbuild.systemd` stage test case to create drop-in
configuration `10-rh-enable-for-ec2.conf` for `nm-cloud-setup.service`
unit, as used in RHEL AMI images.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the `org.osbuild.rhsm` stage to configure selected options in the
subscription-manager configuration (in `/etc/rhsm/rhsm.conf`). It is
possible to set only values currently set in RHEL AMI images,
specifically:
- `manage_repos` option in `rhsm` section
- `auto_registration` option in `rhsmcertd` section
Ensure that the stage does not "touch" any configuration files, unless
it actually changes them. This prevents changing the file modification
time.
Update the `org.osbuild.rhsm` stage test case to set the additional
configuration options.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add a new manifest that creates an ostree commit, deploys that,
creates a raw image and copies the deployment into it. The
resulting artefact is a bootlabel qcow2 image.
Move from using 'zram' to 'zram-generator-defaults' in the ostree bootiso
testing manifest. More information is available in Fedora 33 Change
document [1].
Add org.osbuild.kernel-cmdline stage to fedora-boot.json manifest
because of change in how grub handles the kernel command line arguments
[2].
GRUB2 Stage 2 checksums in assemblers test are updated. The change have
been verified by building the fedora-boot.json manifest with each checked
filesystem and booting the image in QEMU with legacy mode.
[1] https://fedoraproject.org/wiki/Changes/SwapOnZRAM
[2] https://github.com/osbuild/osbuild-composer/pull/982#issuecomment-697356929
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add basic checks for the ostree source, which includes a successful
pull of a commit, an empty source entry and one where the specified
commit is non-existant. For this create a simple commit in a ostree
repo is checked in. The commit was created via:
mkdir "/tmp/data"
echo "Hello World" > /tmp/data/hello.txt
ostree init --repo test/data/sources/org.osbuild.ostree/data/repo \
--mode=archive
ostree commit --repo test/data/sources/org.osbuild.ostree/data/ \
--branch "test/ostree" /tmp/data \
--timestamp="1995-05-13 12:34:56 +0000"
This should give an commit with the following commit id:
d6243b0d0ca3dc2aaef2e0eb3e9f1f4836512c2921007f124b285f7c466464d8
Instead of testing the legacy `SourcesServer` and `sources.get`,
test the `Source.download` method, which is the new and exclusive
way how sources are used in osbuild. For this, the complete info
for a specific source is now included the specific test case.
For the `org.osbuild.curl` source this means that the respective
information is moved from `sources.json`, which is then unused
and thus removed. The test case that checks for an unknown
checksum is also removed because `Source.download` just fetches
everything instead of a subset.
In order to have a more stable package metadata representation,
sort the generated metadata by name. Adapt the tests' metadata
file to reflect that change.
This new manifests creates an boot iso which contains an ostree
commit that is build in the same manifest. The boot iso can then
be installed via the ISO.
This is to test the functionality of the new `saved_entry` grub2
config file. Ideally an integration test would install a new non-
default kernel and check that it does not get selected. Something
for the future.
In addition to the required base layer, provided via the the input
of the same name, the oci-archive stage now accepts up to nine
additional layers that get added on top of each other, sorted in
ascending order, i.e. `layer.1` to `layer.9`.
Adapt the `fedora-ostree-container` example manifest so that the
ostree commit is now in a separate layer, which makes it possible
to share the base layer between different commits container.
Add a new basic test for the dracut stage. It uses a osbuild
pipeline to build an initrd and inspects it via the initrd.py
module. The content is compared to a reference located in the
same directory as the pipeline (test/data/stages/dracut/).
Change the test manifests that use containers to not include
docs when installing. Also don't install docs in the build root
for those manifests. Since the fedora-ostree-container.mpp is
being built in CI, this also tests that the new option.
The `org.osbuild.files` source provides files, but might in the
future not be the only one that does. Therefore rename it to
match the internal tool that is being used to fetch the files.
This is done for most other osbuild modules that target tools.
The format v1 loader is adapted to make this change transparent
for users of the v1 format, so we are backwards compatible.
Change the MPP depsolve preprocessor so that for format v2 based
manifest `org.osbuild.curl` source is used. Also rename the
corresponding source test. Adapt the format v2 mod test to use
the curl source.
Add a v2 manifest to builds an ostree commit, then builds a small
container with a webserver and puts that commit into it.
Start a new test suit where this manifest is built and then checked
that the artifact exists. In the future the artifact should also be
inspected and run by a container engine and the commit be pulled
via ostree.
Add an example that builds a small container, based on the minimal
container fedora kickstart (see [1, 2]). There is a deviation
because osbuild currently does not support removing
random files like it is done in [2].
The build pipeline "build" in f32-build-v2.json) is a port of the
existing build pipeline in f32-build.json.
This manifest is currently not used in any automated tests (yet),
but is there as a showcase for the new format. It does however
test the format version 2 support for mpp, since that is called
in CI.
[1] https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-common.ks
[2] https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base-minimal.ks
The stage was structured in a way so that it supports many different
inputs, like, `archive` and in the future maybe trees, git checkouts.
This does not fit very well into the new input model, where instead
seperate `copy.tree` and `copy.files` stages might make more sense.
Since the stage is not been used anywhere, and mainly got added
as a way to get the ignition dracut module via an archive of a git
repo, remove the stage for now. A replacement will be brought back
in the future.
