This commit implements the `reboot` option for kickstart files.
Note that there are two ways this can be enabled via the json.
Either via a boolean or by passing a dict with options.
```
{"reboot": true}
{"reboot": {"eject": true, "kexec": true}
```
Passing the empty dict
```
{"reboot": {}}
```
is not allowed by the schema.
Add functional/regression around the schema validation for the
kickstart stage. The goal is to ensure that the regexp matching
in the schema allows the expected uses and rejects clearly
forbidden ones.
Run the `test_stages` test in parallel in the github runner. This
test currently takes about 1:30h to 2:30h and running it in parallel
will give us big wins in terms of test time. The time is observed
to go down to 0:30h to 1h.
Note that the other tests are not run in parallel. The reason is
that they fail randomly, it looks like insufficient isolation
between them. Some are easy to fix, e.g.:
721521220b
but it's probably not worth it as the other tests run a lot faster.
Include the new `stages/test` test category to the CI runs as well.
Note that because `stages/__init__.py` and `stages/test/__init__.py`
are missing it is not possible to use the existing style of just
doing `stages.test`. Adding `stages/__init__.py` feels wrong and
the desire is to have the stages tests close to the stages so this
seems the least invasive way.
This commit adds `osbuild.testutil.imports.import_module_from_path`
that can be used to import arbitrary python source files. This
allows importing files from the stages directory that have a
non python friendly filename like `org.osbuild.kickstart`.
Add support for the kickstart options:
- zerombr
- clearpart
Note that for clearpart the `drives` and `list` options have a
regexp pattern to limit the valid inputs. In theory we could only
exclude the `,` here as this is used in the kickstart config as
the list delimiter. Similarly `disklabel` also needs to exclude
` ` or one could write:
```
{"disklabel": "foo --unknown-option-that-confuses-kickstart"}
```
This script reads the RUNNER entries from .gitlab-ci.yml and the
snapshot repo names from Schutzfile. It then prints which runners don't
have entries in Schutzfile and which Schutzfile entries don't have
runners.
This is slightly complicated by the fact that centos-stream reports
itself as centos. So when checking for missing entries they are
considered to be the same thing. eg. If centos is a RUNNER and
centos-stream is in Schutzfile it doesn't report anything missing.
Functions for parsing mounts and inputs raise an exception if the
referenced entity can't be found in the stage options. However, the
exception message always included the value of the `root` variable,
which is `None` when an exception is risen. Instead of the `root`
value, the `name` variable with the entity name should be used.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The items of an array are defined under the `items` key, not under `paths`.
Let's fix this.
Btw, this is possible because JSON Schema itself doesn't use
additionalProperties = false. This allows extending the schemas easily, but
is sadly a bit error-prone.
Sadly, since this issue effectively disabled validation of the stage options,
we also need to relax the schema a bit:
We found out that there are manifests in the wild, that use relative paths,
instead of absolute ones. Thus, this commit changes the validation regex to
allow relative paths. However, this now emits a warning and it's strongly
discouraged. The associated stage test was modified to accommodate for this.
Co-authored-by: Tomáš Hozza <thozza@redhat.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This is a feature that was added in rpm-ostree 2023.10 and is needed
for the new transient /etc feature to work. What it does is change the
labeling of /usr/etc to match those of /etc, so that /usr/etc can be used
directly as a bind-mount or an overlay mount when mounted on /etc.
See https://github.com/coreos/rpm-ostree/pull/4640 for details.
The import of `dnf-json` came from `osbuild-composer`, as we sadly
have/had multiple copies this wasn't the *right* version as it was
lacking previous work to disable weak dependencies.
In the case we are not using a buildroot (i.e. we are using
the host as the buildroot) let's also mount in /etc/containers
into the environment. There are sometimes where software running
from /usr can't operate without configuration in /etc and this
will allow it to work.
An example of software hitting this problem is skopeo. With a
simple config like:
```
version: '2'
mpp-vars:
release: 38
pipelines:
- name: skopeo-tree
# build: name:build
source-epoch: 1659397331
stages:
- type: org.osbuild.skopeo
inputs:
images:
type: org.osbuild.containers
origin: org.osbuild.source
mpp-resolve-images:
images:
- source: quay.io/fedora/fedora-coreos
tag: stable
name: localhost/fcos
options:
destination:
type: containers-storage
storage-path: /usr/share/containers/storage
```
We end up hitting an error like this:
```
time="2023-10-24T18:27:14Z" level=fatal msg="Error loading trust policy: open /etc/containers/policy.json: no such file or directory"
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.skopeo", line 90, in <module>
r = main(args["inputs"], args["tree"], args["options"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/osbuild/bin/org.osbuild.skopeo", line 73, in main
subprocess.run(["skopeo", "copy", image_source, dest], check=True)
File "/usr/lib64/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['skopeo', 'copy', 'dir:/tmp/tmp5_qcng99/image', 'containers-storage:[overlay@/run/osbuild/tree/usr/share/containers/storage+/run/containers/storage]localhost/fcos']' returned non-zero exit status 1.
```
This PR adds in a mount for /etc/containers from the host so that
/etc/containers/policy.json can be accessed.
This example shows how to build a qcow2 very similar to the one
delivered by Fedora CoreOS. It uses an input container that has
the Fedora CoreOS OSTree already baked into it, which means that
the OSTree isn't built using this manifest, but taken as an input
from a remote registry.
This drains some of the logic out of `main()` into a
`ostree_commit_deploy()` function. Doing this will make it easier
to diff this stage with the recently added `ostree.deploy.container`
stage.
This commit also changes the `ref` in the schema to be optional,
which is a fixup for 3cc733d. We need to make the ref optional because
the ref could come from the user in the toplevel schema or it could
come from input commit in the schema.
This stage is similar to ostree.deploy, but deploys from a container
image rather than from an OSTree commit by using the `ostree container
image deploy` command. An example stage definition could look like:
```
- type: org.osbuild.ostree.deploy.container
options:
osname: fedora-coreos
target_imgref: ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:stable
mounts:
- /boot
- /boot/efi
kernel_opts:
- rw
- console=tty0
- console=ttyS0
- ignition.platform.id=qemu
- '$ignition_firstboot'
inputs:
images:
type: org.osbuild.containers
origin: org.osbuild.source
mpp-resolve-images:
images:
- source: quay.io/fedora/fedora-coreos
tag: stable
```
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
The test for this stage is failing because etc/shadow changes content
depending on the date that it runs on (due to the "date of last password
change" field). This causes the checksums to not be constant for our
tests and depends on the date.
This commit removes the checksums for etc/shadow from the test so that
they are not checked as part of the test. This worksaround the test
failure issue for now until a solution to the dynamic contents is
determined.
Signed-off-by: Michael Ho <michael.ho@ieee.org>
Import the `dnf-json` Python script that is included in the "images" and
"osbuild-composer" repository into "osbuild".
`dnf-json` provides dependency solving capabilities by taking in some
JSON and then outputting the package set that JSON resolves to. Since it
is a Python script it moves into a subpackage of `osbuild`.
The script is renamed to `osbuild-depsolve-dnf`.
This adds an early return to the `_process_ostree_commits` function to
prevent an empty ostree commit object from being created in the deploy
stage which causes violations to the deploy stage input schema.
Added three users to the test to test all combinations of ssh key
definitions:
- a user called `onekey` that has a single key defined in the `key`
option.
- a user called `multikey` that has a multiple keys defined in the
`keys` array.
- a user called `bothy` that defines both a single key in the `key`
option and two other keys in the `keys` array.
Add a new attribute, "keys", to allow specifying multiple public SSH
keys to install to a users authorized_keys file.
This maintains backwards compatibility with the existing "key" attribute
that can only specify a single SSH key to install into the file (without
using some newline hacks).
Signed-off-by: Michael Ho <michael.ho@ieee.org>
This will allow a user to specify an input directly to the deploy
stage rather than requiring a ostree.pull stage to be called first.
Adding inputs will also be useful when we soon add support for
deploying from a container rather than just deploying from an existing
ostree commit in a repo.
