Async Job Processing based on PostgreSQL

Question

I search an async job processing system like celery or python-rq but based on PostgreSQL.

Needed features:

• Async processing of jobs
• Uses PostgreSQL
• API for Python
• Store message in language independent format (json, yaml, protcol buffers, ...) NOT python pickle.
• open source

Why PostgreSQL:

• We already use postgreSQL
• Jobs most not get lost. This not the default of Redis and RabbitMQ.
• We only have few jobs per hour. Simplicity is more important than performance.
• Celery is too big

Related article about SELECT FOR UPDATE .. SKIP LOCKED which was introduced in PostgreSQL 9.5: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/

Answer 1

This one does not solve all wanted features, but most of them:

https://www.npmjs.com/package/pg-boss

It is based on node.js, not Python.

It uses PostgresSQL with the new SKIP LOCKED feature.