tools: add deploy-qemu and gen-user-data

tools/gen-user-data generates a cloud-init user-data file from a
configuration directory. It is mostly useful to embed files in the
user-data.

tools/deploy-qemu uses above tool to make a user-data file and spins up
a virtual machine with it. This is useful to locally run, test, and
debug osbuild-composer.

A simple user-data directory for running tests locally is included in
tests/deploy-user-data. It expects a repository with osbuild-composer
rpms to be served on the host's port 8000.

HACKING.md

@@ -0,0 +1,35 @@
+# Hacking on osbuild-composer
+
+*osbuild-composer* cannot be run from the source tree, but has to be installed
+onto a system. We recommend doing this by building rpms, with:
+
+    make rpm
+
+This will build rpms from the latest git HEAD (remember to commit changes), for
+the current operating system, with a version that contains the commit hash. The
+packages end up in `./rpmbuild/RPMS/$arch`.
+
+RPMS are easiest to deal with when they're in a dnf repository. To turn this
+directory into a dnf repository and serve it on localhost:8000, run:
+
+    createrepo_c ./rpmbuild/RPMS/x86_64
+    python3 -m http.server --directory ./rpmbuild/RPMS/x86_64 8000
+
+To start a ephemeral virtual machine using this repository, run:
+
+    tools/deploy-qemu IMAGE tools/deploy/test
+
+`IMAGE` has to be a path to an cloud-init-enabled image matching the host
+operating system, because that's what the packages where built for above.
+
+The second argument points to a directory from which cloud-init user-data is
+generated (see `tools/gen-user-data` for details). The one given above tries to
+mimick what is run on *osbuild-composer*'s continuous integration
+infrastructure, i.e., installing `osbuild-composer-tests` and starting the
+service.
+
+You can log into the running machine as user `admin`, with the
+password `foobar`. Stopping the machine loses all data.
+
+For a quick compile and debug cycle, we recommend iterating code using thorough
+unit tests before going through the full workflow described above.

tools/deploy-qemu

@@ -0,0 +1,59 @@
+#!/usr/bin/bash
+
+#
+# deploy-qemu IMAGE USERDATA
+#
+# Starts an ephemeral virtual machine in qemu, injecting configuration via
+# cloud-init. Stopping this script stops the VM and discards all data.
+#
+# IMAGE    -- An os image that can be booted by qemu and has cloud-init
+#             installed and enabled. No changes are made to this file.
+#
+# USERDATA -- A cloud-init user-data config file, or a directory of
+#             configuration as accepted by the `gen-user-data` tool.
+#
+
+set -euo pipefail
+
+if [[ -z "$1" || -z "$2" ]]; then
+  echo "usage: $0 IMAGE USERDATA"
+  exit 1
+fi
+
+scriptdir=$(dirname "$0")
+image=$1
+userdata=$2
+workdir=$(mktemp -d "$scriptdir/qemu-tmp-XXXXXX")
+function cleanup() {
+  rm -rf "$workdir"
+}
+trap cleanup EXIT
+
+if [ -d "$userdata" ]; then
+  "$scriptdir/gen-user-data" "$userdata" > "$workdir/user-data"
+else
+  cp "$userdata" "$workdir/user-data"
+fi
+
+echo -e "instance-id: nocloud\nlocal-hostname: vm\n" > "$workdir/meta-data"
+
+genisoimage \
+  -input-charset utf-8 \
+  -output "$workdir/cloudinit.iso" \
+  -volid cidata \
+  -joliet \
+  -rock \
+  -quiet \
+  -graft-points \
+  "$workdir/user-data" \
+  "$workdir/meta-data"
+
+qemu-system-x86_64 \
+  -enable-kvm \
+  -m 1024 \
+  -snapshot \
+  -cpu host \
+  -net nic,model=virtio \
+  -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::4430-:443 \
+  -cdrom "$workdir/cloudinit.iso" \
+  "$image"

tools/deploy/test/files/run/provision-scripts/deploy.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+set -euxo pipefail
+
+dnf -y install osbuild-composer-tests

tools/deploy/test/files/run/provision-scripts/provision.sh

@@ -0,0 +1 @@
+../../../../../../schutzbot/provision.sh

tools/deploy/test/user-data.yml

@@ -0,0 +1,16 @@
+#cloud-config
+yum_repos:
+  osbuild:
+    name: osbuild
+    baseurl: "http://10.0.2.2:8000"
+    enabled: true
+    gpgcheck: false
+user: admin
+password: foobar
+ssh_pwauth: True
+chpasswd:
+  expire: False
+sudo: 'ALL=(ALL) NOPASSWD:ALL'
+runcmd:
+  - /run/provision-scripts/deploy.sh
+  - /run/provision-scripts/provision.sh

tools/gen-user-data

@@ -0,0 +1,75 @@
+#!/usr/bin/python3
+
+"""
+gen-user-data
+
+This tool generates a cloud-config user-data file from a directory containing
+configuration. Its main purpose is to make it easy to include files in the
+user-data, which need to be encoded in base64.
+
+It writes the assembled user-data to standard out.
+
+The configuration directory may contain:
+
+* user-data.yml -- a base user-data. Anything that exists in this file will be
+                   transferred as-is. Any additional configuration is appended
+                   to already existing configuration.
+
+* files/        -- a directory containing additional files to include. The
+                   file's path on the target system mirrors its path relative
+                   to this directore (`files/etc/hosts` → `/etc/hosts`). Its
+                   permissions are copied over, but the owner will always be
+                   root:root. Empty directories are ignored.
+
+The `python3-pyyaml` package is required to run this tool.
+"""
+
+
+import argparse
+import base64
+import os
+import stat
+import sys
+import yaml
+
+
+def octal_mode_string(mode):
+    """Convert stat.st_mode to the format cloud-init expects.
+
+    cloud-init's write_files plugin expects file permissions in the format
+    returned by python2's oct() function, for example '0644'. In python3, oct()
+    returns a string in the new octal notation, '0o644'.
+    """
+    return "0" + oct(stat.S_IMODE(mode))[2:]
+
+
+def main():
+    p = argparse.ArgumentParser(description="Generate cloud-config user-data")
+    p.add_argument("configdir", metavar="CONFIGDIR", help="input directory")
+    args = p.parse_args()
+
+    try:
+        with open(f"{args.configdir}/user-data.yml") as f:
+            userdata = yaml.load(f, Loader=yaml.SafeLoader)
+    except FileNotFoundError:
+        userdata = {}
+
+    filesdir = f"{args.configdir}/files"
+    for directory, dirs, files in os.walk(filesdir):
+        for name in files:
+            path = f"{directory}/{name}"
+            with open(path, "rb") as f:
+                content = base64.b64encode(f.read()).decode("utf-8")
+            userdata.setdefault("write_files", []).append({
+                "path": "/" + os.path.relpath(path, filesdir),
+                "encoding": "b64",
+                "content": content,
+                "permissions": octal_mode_string(os.lstat(path).st_mode)
+            })
+
+    sys.stdout.write("#cloud-config\n")
+    yaml.dump(userdata, sys.stdout, Dumper=yaml.SafeDumper)
+
+
+if __name__ == "__main__":
+    sys.exit(main())