particle-os/debian-forge-composer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

dbjobqueue: reimplement the jobqueue to use only one listening connection

Browse source 
Previously, all dequeuers (goroutines waiting for a job to be dequeued) were
listening for new messages on postgres channel jobs (LISTEN jobs). This didn't
scale well as each dequeuer required to have its own DB connection and the
number of DB connections is hard-limited in the pool's config.

I changed the logic to work somewhat differently: dbjobqueue.New() now spawns
a goroutine that listens on the postgres channel. If there's a new message,
the goroutine just wakes up all dequeuers using a standard go channel.
Go channels are cheap so this should scale much better.

A test was added that confirms that 100 dequeuers are not a big deal now. This
test failed when I tried to run on it on the previous commit. I tried even 1000
locally and it was still fine.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
This commit is contained in:
Ondřej Budai 2022-03-07 15:22:28 +01:00 • committed by Tom Gundersen
parent c8dbe0de74
commit c4c7f44fcb
4 changed files with 169 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

1
go.mod
View file

@ -31,7 +31,6 @@ require (
github.com/google/uuid v1.3.0
github.com/gophercloud/gophercloud v0.22.0
github.com/hashicorp/go-retryablehttp v0.7.0
github.com/jackc/pgconn v1.10.0
github.com/jackc/pgtype v1.8.1
github.com/jackc/pgx/v4 v4.13.0
github.com/julienschmidt/httprouter v1.3.0