This commit migrates the test to a brand new V2, F38-based manifest. It's
actually based on osbuild-composer interpretation of the Fedora Cloud Base
image.
This form of signatures has been (build-time-optionally) supported
since ostree 2020.4 as an alternative to the old gpg signatures. With
the current work on composefs[1] they are becomming more important, as
they will allow verification of the commit (and thus the composefs
image) during boot, giving us a full trusted boot chain all the way
into the ostree userspace.
Note: `ostree sign` used to require libsodium and was thus disabled in
e.g. the Fedora build of ostree. However, recently[2] it is also supported
with openssl, which will let it be more widely used.
[1] https://github.com/ostreedev/ostree/pull/2921
[2] https://github.com/ostreedev/ostree/pull/2922
This stage allows you to create new (random) ed25519 keys as used by
`ostree sign`.
The primary usecase for this is composefs. You can generate a
transient key-pair during the build (unique to the build) that binds
the initrd to the userspace tree.
You put the public key in the initrd, sign the resulting commit with
the private key and then throw away the private key. During boot of a
(secureboot trusted) initrd, we use this public key to validate that
we're booting the right commit.
This is similar to how the transient kernel module signatures work.
It similarly generates a keypair during the kernel rpm build, sign the
modules, throw away the private key and embed the public key in the
kernel binary.
Of course, this stage can also be used to generate keys used for
persistant signatures.
The `./tools` dir was not part of the LINTABLES in the `tox.ini`
which meant that pep8/pylint etc checks were not run on the tools
there.
This commit adds it and fixes the issues that `make lint` found.
Now that inputs can be relatively easily validated against
the schema this should also be used for all the "good" test
inputs to ensure that all tests test against valid inputs.
This commit implements the `reboot` option for kickstart files.
Note that there are two ways this can be enabled via the json.
Either via a boolean or by passing a dict with options.
```
{"reboot": true}
{"reboot": {"eject": true, "kexec": true}
```
Passing the empty dict
```
{"reboot": {}}
```
is not allowed by the schema.
Add functional/regression around the schema validation for the
kickstart stage. The goal is to ensure that the regexp matching
in the schema allows the expected uses and rejects clearly
forbidden ones.
Run the `test_stages` test in parallel in the github runner. This
test currently takes about 1:30h to 2:30h and running it in parallel
will give us big wins in terms of test time. The time is observed
to go down to 0:30h to 1h.
Note that the other tests are not run in parallel. The reason is
that they fail randomly, it looks like insufficient isolation
between them. Some are easy to fix, e.g.:
721521220b
but it's probably not worth it as the other tests run a lot faster.
Include the new `stages/test` test category to the CI runs as well.
Note that because `stages/__init__.py` and `stages/test/__init__.py`
are missing it is not possible to use the existing style of just
doing `stages.test`. Adding `stages/__init__.py` feels wrong and
the desire is to have the stages tests close to the stages so this
seems the least invasive way.
This commit adds `osbuild.testutil.imports.import_module_from_path`
that can be used to import arbitrary python source files. This
allows importing files from the stages directory that have a
non python friendly filename like `org.osbuild.kickstart`.
Add support for the kickstart options:
- zerombr
- clearpart
Note that for clearpart the `drives` and `list` options have a
regexp pattern to limit the valid inputs. In theory we could only
exclude the `,` here as this is used in the kickstart config as
the list delimiter. Similarly `disklabel` also needs to exclude
` ` or one could write:
```
{"disklabel": "foo --unknown-option-that-confuses-kickstart"}
```
This script reads the RUNNER entries from .gitlab-ci.yml and the
snapshot repo names from Schutzfile. It then prints which runners don't
have entries in Schutzfile and which Schutzfile entries don't have
runners.
This is slightly complicated by the fact that centos-stream reports
itself as centos. So when checking for missing entries they are
considered to be the same thing. eg. If centos is a RUNNER and
centos-stream is in Schutzfile it doesn't report anything missing.
Functions for parsing mounts and inputs raise an exception if the
referenced entity can't be found in the stage options. However, the
exception message always included the value of the `root` variable,
which is `None` when an exception is risen. Instead of the `root`
value, the `name` variable with the entity name should be used.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The items of an array are defined under the `items` key, not under `paths`.
Let's fix this.
Btw, this is possible because JSON Schema itself doesn't use
additionalProperties = false. This allows extending the schemas easily, but
is sadly a bit error-prone.
Sadly, since this issue effectively disabled validation of the stage options,
we also need to relax the schema a bit:
We found out that there are manifests in the wild, that use relative paths,
instead of absolute ones. Thus, this commit changes the validation regex to
allow relative paths. However, this now emits a warning and it's strongly
discouraged. The associated stage test was modified to accommodate for this.
Co-authored-by: Tomáš Hozza <thozza@redhat.com>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This is a feature that was added in rpm-ostree 2023.10 and is needed
for the new transient /etc feature to work. What it does is change the
labeling of /usr/etc to match those of /etc, so that /usr/etc can be used
directly as a bind-mount or an overlay mount when mounted on /etc.
See https://github.com/coreos/rpm-ostree/pull/4640 for details.
The import of `dnf-json` came from `osbuild-composer`, as we sadly
have/had multiple copies this wasn't the *right* version as it was
lacking previous work to disable weak dependencies.
In the case we are not using a buildroot (i.e. we are using
the host as the buildroot) let's also mount in /etc/containers
into the environment. There are sometimes where software running
from /usr can't operate without configuration in /etc and this
will allow it to work.
An example of software hitting this problem is skopeo. With a
simple config like:
```
version: '2'
mpp-vars:
release: 38
pipelines:
- name: skopeo-tree
# build: name:build
source-epoch: 1659397331
stages:
- type: org.osbuild.skopeo
inputs:
images:
type: org.osbuild.containers
origin: org.osbuild.source
mpp-resolve-images:
images:
- source: quay.io/fedora/fedora-coreos
tag: stable
name: localhost/fcos
options:
destination:
type: containers-storage
storage-path: /usr/share/containers/storage
```
We end up hitting an error like this:
```
time="2023-10-24T18:27:14Z" level=fatal msg="Error loading trust policy: open /etc/containers/policy.json: no such file or directory"
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.skopeo", line 90, in <module>
r = main(args["inputs"], args["tree"], args["options"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/osbuild/bin/org.osbuild.skopeo", line 73, in main
subprocess.run(["skopeo", "copy", image_source, dest], check=True)
File "/usr/lib64/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['skopeo', 'copy', 'dir:/tmp/tmp5_qcng99/image', 'containers-storage:[overlay@/run/osbuild/tree/usr/share/containers/storage+/run/containers/storage]localhost/fcos']' returned non-zero exit status 1.
```
This PR adds in a mount for /etc/containers from the host so that
/etc/containers/policy.json can be accessed.
This example shows how to build a qcow2 very similar to the one
delivered by Fedora CoreOS. It uses an input container that has
the Fedora CoreOS OSTree already baked into it, which means that
the OSTree isn't built using this manifest, but taken as an input
from a remote registry.
This drains some of the logic out of `main()` into a
`ostree_commit_deploy()` function. Doing this will make it easier
to diff this stage with the recently added `ostree.deploy.container`
stage.
This commit also changes the `ref` in the schema to be optional,
which is a fixup for 3cc733d. We need to make the ref optional because
the ref could come from the user in the toplevel schema or it could
come from input commit in the schema.
This stage is similar to ostree.deploy, but deploys from a container
image rather than from an OSTree commit by using the `ostree container
image deploy` command. An example stage definition could look like:
```
- type: org.osbuild.ostree.deploy.container
options:
osname: fedora-coreos
target_imgref: ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:stable
mounts:
- /boot
- /boot/efi
kernel_opts:
- rw
- console=tty0
- console=ttyS0
- ignition.platform.id=qemu
- '$ignition_firstboot'
inputs:
images:
type: org.osbuild.containers
origin: org.osbuild.source
mpp-resolve-images:
images:
- source: quay.io/fedora/fedora-coreos
tag: stable
```
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
The test for this stage is failing because etc/shadow changes content
depending on the date that it runs on (due to the "date of last password
change" field). This causes the checksums to not be constant for our
tests and depends on the date.
This commit removes the checksums for etc/shadow from the test so that
they are not checked as part of the test. This worksaround the test
failure issue for now until a solution to the dynamic contents is
determined.
Signed-off-by: Michael Ho <michael.ho@ieee.org>
Import the `dnf-json` Python script that is included in the "images" and
"osbuild-composer" repository into "osbuild".
`dnf-json` provides dependency solving capabilities by taking in some
JSON and then outputting the package set that JSON resolves to. Since it
is a Python script it moves into a subpackage of `osbuild`.
The script is renamed to `osbuild-depsolve-dnf`.
This adds an early return to the `_process_ostree_commits` function to
prevent an empty ostree commit object from being created in the deploy
stage which causes violations to the deploy stage input schema.
Added three users to the test to test all combinations of ssh key
definitions:
- a user called `onekey` that has a single key defined in the `key`
option.
- a user called `multikey` that has a multiple keys defined in the
`keys` array.
- a user called `bothy` that defines both a single key in the `key`
option and two other keys in the `keys` array.
