I search an async job processing system like celery or python-rq but based on PostgreSQL.
Needed features:
- Async processing of jobs
- Uses PostgreSQL
- API for Python
- Store message in language independent format (json, yaml, protcol buffers, ...) NOT python pickle.
- open source
Why PostgreSQL:
- We already use postgreSQL
- Jobs most not get lost. This not the default of Redis and RabbitMQ.
- We only have few jobs per hour. Simplicity is more important than performance.
- Celery is too big
Related article about SELECT FOR UPDATE .. SKIP LOCKED
which was introduced in PostgreSQL 9.5: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/