Commit graph

76 commits

Author SHA1 Message Date
Diaa Sami
962ad9e582 containers/osbuild-composer: add prometheus port parameter 2023-06-29 14:34:15 +02:00
Diaa Sami
8398f27742 internal/cloudapi: additional prometheus listener
Listening on another port, while keeping the existing endpoint until
transition is complete
2023-06-07 17:05:32 +02:00
Ondřej Budai
128f56169e Dockerfile*: chown the copied files
When `go install` is called, go tries to get the git commit hash and embed it
into the built binary. Internally, go just calls the git executable.

The newer go-toolset seems to be based on RHEL 9.2 that ships a newer version
of git (2.39.1). This version contains the safe directory patch that
disallows git from operating on repositories owned by different users.

Thus, we need to chown the files when copying.

See

https://git-scm.com/docs/git-config/2.35.2#Documentation/git-config.txt-safedirectory
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-05-17 13:08:29 +02:00
Ondřej Budai
7ce51e0ab5 distribution: update to ubi9
This commit also adds `-y` to `microdnf install`. Microdnf shipped in RHEL 9
apparently requires it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-05-17 13:08:29 +02:00
Ondřej Budai
4bafe1ead9 remove Fedora 35 support
Fedora 35 is going EOL on Tue 2022-12-13. At the time of writing this commit
message, that's the next day. As we do releases on Wednesdays, the next
release will never find its way to F35 and thus, there's no point in keeping
support for it.

Let's delete everything that relates to Fedora 35. If there's something that
cannot be deleted (e.g. CI containers based on F35), let's upgrade it to F37.

TestCrossArchDepsolve now uses CentOS Stream 8 because RHEL 8.4 cannot read
F37 repository metadata. This is a similar issue to
https://bugzilla.redhat.com/show_bug.cgi?id=2004853 . Basically, newer
repositories can be only read by libmodulemd >= 2.11.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-01-06 11:34:50 +01:00
Brian C. Lane
d73171fdde Use go install instead of deprecated go get
In go 1.18 'go install' needs to be used to install an executable. See
https://go.dev/doc/go-get-install-deprecation and
https://go.dev/doc/manage-install
2022-11-09 12:08:43 +01:00
Tomas Hozza
036303694c test: suffix the currently used test configurations with -tls
This is a preparation to support a different authentication method in
`provision.sh`, specifically JWT.
2022-08-04 11:55:43 +02:00
Christian Kellner
4c55063a55 Dockerfile-ubi: install gpgme devel package in builder
The go package `proglottis/gpgme` a dependency of `containers/image/v5`
package uses `libgpgme`. In the near future `internal/container`, which
depends on `containers/image/v5`, will be used directly in composer and
thus we need to install the `gpgme` devel package and its build deps.
2022-07-21 13:32:07 +02:00
Gianluca Zuccarelli
00f4f1690f Dockerfile: update worker builder
Additional packages are required to build the
docker worker. This fix updates the builder
container to install the required libraries
and then create the worker binary.
2022-07-15 03:06:43 +02:00
Ondřej Budai
c0df932fc6 Dockerfile: bump the shutdown period to 15 seconds
5 seconds didn't fully help with the issue, let's try 15 and see what happens.

See https://github.com/osbuild/osbuild-composer/pull/2797#issuecomment-1172862359
for more details.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-07 15:33:32 +02:00
Sanne Raymaekers
e52bd38ccf distribution/Dockerfile-ubi: Graceful shutdown of 5 seconds 2022-07-05 11:04:20 +02:00
Sanne Raymaekers
03b57f002c jobqueue: Move jobqueue out of internal 2022-07-04 15:37:28 +02:00
zwtop
5c79d4e98c docker-compose: fix osbuild-worker unable load libcrypt.so.1
Signed-off-by: zwtop <wang.zhan@smartx.com>
2022-06-18 11:39:10 +01:00
Achilleas Koutsou
03e86cc31a spec: remove dnf-json service and socket
dnf-json is a standalone script again.  The service and socket are
removed.
2022-06-01 11:36:52 +01:00
Ygal Blum
fc87b1740e entrypoint - add parameters for socket bind address and port
entrypoint.py
-------------
Allow the user to set the port number also for the remote worker
Allow the user to set the binding address for the composer api or remote worker api
Set the default port of the composer API to 8080

Dockerfile-ubi
--------------
Remove setting the port for the composer API since the default is
already 8080
2022-05-04 09:13:40 +02:00
Feng Huang
c64eb98011 use app-sre packer image
Signed-off-by: Feng Huang <fehuang@redhat.com>
2022-03-11 09:24:26 +01:00
Tom Gundersen
9b89df57b6 docker-compose: integrate dev container
Integrate the dev container into docker compose with the aim
of making test-case generation work as well as composer-cli.

This also makes docker-compose self contained, and no setup or configuration is required beyond running `docker compose up --build`.
2022-02-27 20:55:03 +00:00
sanne
a6f40f807d tools/appsre-build-worker-packer: Run entire workflow in container
Avoids the issue with artifacts that jenkins can't clean up.
2022-01-13 15:05:52 +01:00
sanne
71da979c81 tools: AppSRE packer build 2022-01-05 22:13:55 +01:00
Thomas Lavocat
ca126e9747 dnf-json: Change dnf-json to be a daemon
The service is started via systemd activation sockets.
The service serves http POST requests, the same json as before is
expected as the body of the request, and the same json as before is sent
as the response of the request.
2021-12-15 09:41:32 +01:00
sanne
c43ad2b22a osbuild-service-maintenance: Clean up expired images 2021-12-03 00:14:09 +00:00
Gianluca Zuccarelli
44017890ca containers: mock oauth container
Add a mock oauth container to simulate
the openshift SSO offline_token
2021-11-12 14:07:13 +01:00
Gianluca Zuccarelli
cfa8d7a636 distribution: worker dnf-json & cache dir
Since the depsolving has been moved to the worker,
the Dockerfile for the worker needed to have the
dnf-json executable. Additionally there was a missing
cache directory.
2021-11-05 13:34:43 +01:00
sanne
4e56f04dd7 templates: Composer OSD template 2021-10-05 16:45:55 +02:00
sanne
2f328b0e97 workers: Backwards compatible api.openshift.com spec compliance
The main changes are:
- Kind, Href, Id fields for every object returned
- Attach operationIds to each request, return it for errors
- Errors are predefined and queryable
2021-09-27 13:10:05 +01:00
Martin Sehnoutka
e310d4e4f0 distribution: Use After=network.target instead of multi-user
There is no need for osbuild-composer to wait until the whole multi-user
target is completed. It can be started earlier as it doesn't have any
dependencies in the target.

This can be a problem if there is a unit in the target that is not starting
and still is unrelated to osbuild-composer. There was a bug like this
with Plymouth where the service didn't finish and it was hanging. That
prevented osbuild-composer from starting and the user was left with
working SSH connection and shell, but composer-cli and systemctl start
osbuild-composer.service were both unresponsive.

Replace After=multi-user.target with After=network.target to start
osbuild-composer earlier.
2021-09-24 14:15:10 +02:00
Gianluca Zuccarelli
b09040efd5 containers: worker client base url protocol
The base url for the osbuild-worker container was
missing the `https` protocol in the docker-compose file.
This commit adds this to prevent the worker from exiting
2021-09-23 13:28:27 +01:00
Lars Karlitski
9c2c92f729 jobqueue: Introduce jobqueue backed by a postgres database
Co-authored-by: sanne <sanne.raymaekers@gmail.com>
2021-07-28 21:52:31 +01:00
sanne
234c51e562 distribution: Spell osbuild-composer correctly 2021-07-16 12:14:46 +02:00
sanne
c53283f5e9 distribution: Change clouddot template to allow for unavailable pods
Because the state directory, which is still currently used, only allows
for a single pod to mount it, allow for 0 available pods when updating
the deployment. Otherwise it will block updating until the state
directory is available for mounting, which will be never.
2021-07-14 22:19:20 +02:00
sanne
d0b5aaae95 distribution: Switch to ConfigMap for configuration
This will allow us to drop vault secrets, and make it easier to
configure composer.
2021-07-14 18:43:34 +02:00
sanne
6fac4dcb70 distribution: Add cache dir to composer clouddot deployment 2021-06-23 10:33:22 +02:00
sanne
95cd5b782e distribution: The composer-api listens on 9196
The openshift service is listening on 443, but composer is running on
9196.
2021-06-21 17:31:15 +02:00
Achilleas Koutsou
d701d237d0 Fix cache path creation in worker Dockerfile 2021-06-18 14:02:09 +01:00
sanne
2a42d05a10 distribution: Add PVC to clouddot template
This will make the job queue persistent over redeployments.
2021-06-04 20:21:16 +02:00
sanne
65b7ce10ba distribution: Add second service to composer clouddot deployment 2021-05-18 12:07:30 +02:00
sanne
0518933676 distribution: Add state directory
This should become a PersistentVolumeClaim in future, but let's get the
deployment running first.
2021-05-17 21:43:55 +02:00
sanne
a825b87c35 distribution: Add OCP template for clouddot 2021-04-28 14:07:23 +02:00
Achilleas Koutsou
14183ba10a containers: Make config path configurable
Container configuration directory can be set in the .env file which is
automatically used by docker-compose.  The default value is the one used
previously (containers/config).
The HACKING.md guide is updated to describe how to set this and how it
affects other commands.
2021-01-30 13:20:11 +00:00
Achilleas Koutsou
7c3a3c785f containers: docker-compose for multi-container setup
Defines how to build and run two containers for osbuild-composer and
worker in the same subnet for testing.
2021-01-30 13:20:11 +00:00
Achilleas Koutsou
172cd4f816 dockerfile: Second image for worker
Worker container image. Follows the same general structure as the
composer image
2021-01-30 13:20:11 +00:00
Achilleas Koutsou
937911feac dockerfile: dnf-json requires python3-dnf 2021-01-30 13:20:11 +00:00
Sanne Raymaekers
946a0b425a containers: Specify port for the composer-api as argument 2020-12-23 17:31:29 +01:00
Tom Gundersen
c3185466ae containers: simplify Dockerfile and container build
This changes the following:
 - Only produce one container. There is no particular benefit to
   supporting several different base containers, so unify on ubi
   as that is what we need ourselves.
 - Build directly from git. Now that the RPM we include in our
   container does not have any dependencies and only contains a
   couple of executables, the indirection via RPM has less value.
   Eventually the value will be reduced even further as we merge
   the entrypoint into the main binary and move dnf-json into the
   worker, leaving us with only a go binary. The only potential
   benefit might be that the build environment of RPMs is more
   clearly defined, but there is no real reason to believe that
   our mockbuild is any better than using the UBI golang build
   container.

This simplifies the container builds, and brings us more in line
with what is done in image-builder, and what is needed to deploy
to openshift.
2020-12-22 15:27:39 +01:00
Ondřej Budai
71428d3131 composer: do not require the weldr socket
It should be totally possible to run osbuild-composer without the Weldr API
and local worker socket. This commit accomplishes this task!

In cloud environments where we don't want the Weldr API, it's now totally
possible to run osbuild-composer without it. If you also want to get rid of
the local worker, this command still works:

systemctl mask osbuild-worker@1.service

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2020-11-17 17:01:18 +00:00
Ondřej Budai
dc1b84fcfe composer: split out the local worker socket
Everybody hates the local workers. The first step of getting rid of them
is to split their socket out of osbuild-composer.socket - we need to keep
this one to support the Weldr API but the local worker socket can live in
its own file.

The behaviour should be the same for now: osbuild-composer.service always
starts the local worker socket.

However, this split allows the osbuild-composer executable to be run without
the Weldr API activated. The following commit explores this option more
in depth.

Note that the new socket can be used by root only because workers are always
run as root.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2020-11-17 17:01:18 +00:00
Ondřej Budai
0ac554c139 sockets: clarify their descriptions
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2020-11-17 17:01:18 +00:00
Lars Karlitski
41a797547a distribution: remove osbuild-composer-koji
Remove both the package osbuild-composer-koji, and the only file it
shipped: osbuild-composer-koji.socket.

It's been deprecated since 835b556, but the backwards-compatible
solution in that commit never worked, because osbuild-composer only
checks for "osbuild-composer-api.socket" when starting up.

Since this has been meant to be deprecated for a while, just remove it
outright.

Add an "Obsoletes:" for the package, so that it gets uninstalled on
existing systems.
2020-11-10 16:26:03 +01:00
Lars Karlitski
b25a350502 osbuild-composer: merge cloud API into main binary
This removes the osbuild-composer-cloud package, binary, systemd units,
the (unused) test binary, and the (only-run-on-RHEL) test in aws.sh.

Instead, move the cloud API into the main package, using the same
socket as the koji API, osbuild-composer-api.socket. Expose it next to
the koji API on route `/api/composer/v1`.

This is a backwards incompatible change, but only of the -cloud parts,
which have been marked as subject to change.
2020-10-16 09:37:04 +02:00
Lars Karlitski
835b556db7 distribution: deprecate osbuild-composer-koji.socket
Instead, call it osbuild-composer-api.socket, but provide a symlink for
backwards compatibility. Change `schutzbot/provision.sh` to only enable
osbuild-composer-api.socket.

In the future, this new socket is the only API socket, which provides
both the "cloud" API and the one for koji.

This means that the koji API is always enabled.
2020-10-16 09:37:04 +02:00