974c258382
We have limited resources in openstack. We can only run about 40 concurrent VMs. Previously, the rate limiting was kinda stupid: All (aws and openstack) jobs were run using the same runner. This runner was globally limited to 60 concurrent jobs. For openstack, the individual images were also limited to a certain number of concurrent jobs in the gitlab-ci-terraform repository so we don't hit the quota. This limit was applied at runtime - the first thing that an openstack job did was to wait for a slot. This job counted towards the global limit of jobs (60) and thus was blocking one slot without doing any useful work. Applying local limits to please global quota is stupid though. We have much more demand for rhel-8.5 runner than for e.g. Fedora. It would be much better to just use global limit that would map much nicely to global quota. Today, I've introduced a new runner with tag terraform/openstack. It's currently limited to 20 concurrent jobs. All jobs running on openstack should run on the new runner. This runner has the local rate limiting for openstack disabled. This means that we can run 20 concurrent openstack jobs and it doesn't matter which distribution they run. To sum it up, this has two benefits: - no local limits, we can just use the full quota - no idling jobs waiting for an openstack slot Note that the openstack global limit is currently set to 20, I will raise it once all PRs are rebased on top of this change. Side effect: I moved all libvirt test to openstack. I think this is overall better because testing guest images on KVM makes more sense than testing them on TCG. Signed-off-by: Ondřej Budai <ondrej@budai.cz> |
||
|---|---|---|
| .devcontainer | ||
| .github | ||
| cmd | ||
| containers/osbuild-composer | ||
| distribution | ||
| docs | ||
| image-types | ||
| internal | ||
| repositories | ||
| schutzbot | ||
| test | ||
| tools | ||
| vendor | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| .golangci.yml | ||
| .packit.yaml | ||
| codecov.yml | ||
| CONTRIBUTING.md | ||
| DEPLOYING.md | ||
| dnf-json | ||
| go.mod | ||
| go.sum | ||
| HACKING.md | ||
| krb5.conf | ||
| LICENSE | ||
| Makefile | ||
| NEWS.md | ||
| osbuild-composer.spec | ||
| README.md | ||
| RELEASING.md | ||
| Schutzfile | ||
OSBuild Composer
Operating System Image Composition Services
The composer project is a set of HTTP services for composing operating system images. It builds on the pipeline execution engine of osbuild and defines its own class of images that it supports building.
Multiple APIs are available to access a composer service. This includes support for the lorax-composer API, and as such can serve as drop-in replacement for lorax-composer.
You can control a composer instance either directly via the provided APIs, or through higher-level user-interfaces from external projects. This, for instance, includes a Cockpit Module or using the composer-cli command-line tool.
Project
- Website: https://www.osbuild.org
- Bug Tracker: https://github.com/osbuild/osbuild-composer/issues
- IRC: #osbuild on Libera.Chat
About
Composer is a middleman between the workhorses from osbuild and the user-interfaces like cockpit-composer, composer-cli, or others. It defines a set of high-level image compositions that it supports building. Builds of these compositions can be requested via the different APIs of Composer, which will then translate the requests into pipeline-descriptions for osbuild. The pipeline output is then either provided back to the user, or uploaded to a user specified target.
The following image visualizes the overall architecture of the OSBuild infrastructure and the place that Composer takes:
Consult the osbuild-composer(7) man-page for an introduction into composer,
information on running your own composer instance, as well as details on the
provided infrastructure and services.
Requirements
The requirements for this project are:
osbuild >= 26systemd >= 244
At build-time, the following software is required:
go >= 1.15python-docutils >= 0.13
Build
The standard go package system is used. Consult upstream documentation for detailed help. In most situations the following commands are sufficient to build and install from source:
mkdir build
go build -o build ./...
The man-pages require python-docutils and can be built via:
make man
Repository:
- web: https://github.com/osbuild/osbuild-composer
- https:
https://github.com/osbuild/osbuild-composer.git - ssh:
git@github.com:osbuild/osbuild-composer.git
Pull request gating
Each pull request against osbuild-composer starts a series of automated
tests. Tests run via GitHub Actions and Jenkins. Each push to the pull request
will launch theses tests automatically.
Jenkins only tests pull requests from members of the osbuild organization in
GitHub. A member of the osbuild organization must say ok to test in a pull
request comment to approve testing. Anyone can ask for testing to run by
saying the bot's favorite word, schutzbot, in a pull request comment.
Testing will begin shortly after the comment is posted.
Test results in Jenkins are available by clicking the Details link on the right side of the Schutzbot check in the pull request page.
License:
- Apache-2.0
- See LICENSE file for details.