0

I search an async job processing system like celery or python-rq but based on PostgreSQL.

Needed features:

  • Async processing of jobs
  • Uses PostgreSQL
  • API for Python
  • Store message in language independent format (json, yaml, protcol buffers, ...) NOT python pickle.
  • open source

Why PostgreSQL:

  • We already use postgreSQL
  • Jobs most not get lost. This not the default of Redis and RabbitMQ.
  • We only have few jobs per hour. Simplicity is more important than performance.
  • Celery is too big

Related article about SELECT FOR UPDATE .. SKIP LOCKED which was introduced in PostgreSQL 9.5: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/

0

This one does not solve all wanted features, but most of them:

https://www.npmjs.com/package/pg-boss

It is based on node.js, not Python.

It uses PostgresSQL with the new SKIP LOCKED feature.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.