I am looking for a Python 3.7+ library for a key-value store that:

  • Can take tuples of primitive types as keys & values
  • Stores its data on disk so that it persists between processes
  • Does not require a server process
  • Is thread-safe and multiprocessing-safe
  • Works on NFS
  • Works on Linux, macOS, and Windows
  • Can handle a collection of data up to about 1 GB in size
  • BONUS: Can clear out old entries to reduce the on-disk storage to a given size

The best I've found so far is diskcache, but that uses SQLite databases, which are not reliable on NFS.

  • I doubt you can find anything like this. As shown in the SQLite FAQ, SQLite claims that NFS implementations have a lot of problems with the file locking needs by concurrent writes, so I don't think any other program or library could make them reliable. Your best bet would be to use a server based database. – Alejandro Mar 12 at 18:05
  • @Alejandro: Things like flufl.lock provide NFS-safe locking, so something built on that could work. And a server is 100% out of the question for my needs. – jwodder Mar 12 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.