schutzbot: run openstack/* jobs on a dedicated runner

We have limited resources in openstack. We can only run about 40 concurrent VMs.
Previously, the rate limiting was kinda stupid:

All (aws and openstack) jobs were run using the same runner. This runner was
globally limited to 60 concurrent jobs. For openstack, the individual
images were also limited to a certain number of concurrent jobs in
the gitlab-ci-terraform repository so we don't hit the quota. This limit
was applied at runtime - the first thing that an openstack job did was to
wait for a slot. This job counted towards the global limit of jobs (60)
and thus was blocking one slot without doing any useful work.

Applying local limits to please global quota is stupid though. We have much
more demand for rhel-8.5 runner than for e.g. Fedora. It would be much
better to just use global limit that would map much nicely to global
quota.

Today, I've introduced a new runner with tag terraform/openstack. It's
currently limited to 20 concurrent jobs. All jobs running on openstack should
run on the new runner. This runner has the local rate limiting for openstack
disabled. This means that we can run 20 concurrent openstack jobs and it
doesn't matter which distribution they run.

To sum it up, this has two benefits:
- no local limits, we can just use the full quota
- no idling jobs waiting for an openstack slot

Note that the openstack global limit is currently set to 20, I will raise
it once all PRs are rebased on top of this change.

Side effect: I moved all libvirt test to openstack. I think this is overall
better because testing guest images on KVM makes more sense than testing them
on TCG.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>

.gitlab-ci.yml

@@ -6,7 +6,7 @@ stages:
   - cleanup
   - finish
 
-.terraform:
+.base:
   after_script:
     - schutzbot/update_github_status.sh update
     - schutzbot/save_journal.sh
@@ -19,6 +19,17 @@ stages:
       - ci-artifacts
     when: always
 
+.terraform:
+  extends: .base
+  tags:
+    - terraform
+
+.terraform/openstack:
+  extends: .base
+  tags:
+    - terraform/openstack
+
+
 init:
   stage: init
   tags:
@@ -150,7 +161,7 @@ Regression:
 
 OSTree:
   stage: test
-  extends: .terraform
+  extends: .terraform/openstack
   rules:
     - if: '$CI_PIPELINE_SOURCE != "schedule"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[8-9]\.[\S]+/ && $NIGHTLY == "true"'
@@ -234,18 +245,6 @@ Integration:
         RUNNER:
           - aws/rhel-8.5-x86_64
         INTERNAL_NETWORK: ["true"]
-      - SCRIPT:
-          - libvirt.sh
-        RUNNER:
-          - aws/fedora-33-x86_64
-          - openstack/centos-stream-8-x86_64
-      - SCRIPT:
-          - libvirt.sh
-        RUNNER:
-          - aws/rhel-8-x86_64
-          - openstack/rhel-8.5-x86_64
-          - openstack/rhel-9.0-beta-nightly-x86_64
-        INTERNAL_NETWORK: ["true"]
       - SCRIPT:
           - aws.sh
           - azure.sh
@@ -284,6 +283,26 @@ API:
           - aws/rhel-9.0-beta-nightly-x86_64
         INTERNAL_NETWORK: ["true"]
 
+libvirt:
+  stage: test
+  extends: .terraform/openstack
+  rules:
+    - if: '$CI_PIPELINE_SOURCE != "schedule"'
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[8-9]\.[\S]+/ && $NIGHTLY == "true"'
+  script:
+    - schutzbot/deploy.sh
+    - /usr/libexec/tests/osbuild-composer/libvirt.sh
+  parallel:
+    matrix:
+      - RUNNER:
+          - openstack/fedora-33-x86_64
+          - openstack/centos-stream-8-x86_64
+      - RUNNER:
+          - openstack/rhel-8-x86_64
+          - openstack/rhel-8.5-x86_64
+          - openstack/rhel-9.0-beta-nightly-x86_64
+        INTERNAL_NETWORK: ["true"]
+
 RHEL 9 on 8:
   stage: test
   extends: .terraform
@@ -318,7 +337,7 @@ NIGHTLY_SUCCESS:
 
 Installer:
   stage: test
-  extends: .terraform
+  extends: .terraform/openstack
   rules:
     - if: '$CI_PIPELINE_SOURCE != "schedule"'
     - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[8-9]\.[\S]+/ && $NIGHTLY == "true"'