I have a Discord Bot that's on a many a servers, I'm revamping most all of it and came to a personal crossroads. While developing I used the schema that I saw Red Discord Bot use which under the hood is dicts in memory saved on update to a json file for each cog. (cog being a category that the bot deals in).

This uses

stuff = dataIO.load_json(...)
dataIO.save_json(... , stuff)

upon init / close of each cog. This performs well, although I feel as though using a database may be a better approach considering it can encompass all cogs as a whole rather than dealing with each cog as a file individually, thus duplicating data.

I've seen the json file method as a standard among many developers and another being mongoDB to store their data. I have used both in my original application but I'd like to see what I could use that would eliminate using both.

So, I know that there will be opinion based biases involved in this, but dealing with python strictly, what is the most efficient method of storing data for each server and user on those servers. (my point being made on using some sort of database relational or nonSQL/un-relational)

The reason I ask is sometimes using pure python is (in some cases) "better" than using an API to a database written in another language.

I have taken a dive (even tested) quite a few other databases including TinyDB, ZODB, and sqlite (eww) and seems I just need it laid out for me. I'm looking for something that is more efficient than using the json approach I touched on above but not heavy that will make coding/debugging a hassle. Plus, I really want to steer clear from writing one more blasted SQL statement

Thanks all!


I know I should have included this in the above but I want to make sure it's read. I'm looking for something that is not in-memory as I'd like to conserve my memory usage. I have no concerns as to the disk space used but memory could pose a concern depending on the usage it imposes. Memory mapped is ok, although I found MongoDB a hassle to configure to not use ALL my memory as my current setup is standalone but plan to move to using 2 servers very soon.

  • " really want to steer clear from writing one more blasted SQL statement" - what a sad thing to hear :-( MySQL or SQLite would be what I would recommend, especially as the amount of data increases (which answers your "most efficient"). Whatever you choose (and there are lots of pro/con articles, so I won't repeat), be sure that you have an automatic mechanism to backup your data. – Mawg says reinstate Monica Nov 21 '18 at 10:12
  • 1
    Honestly, in the end if it's more efficient / the best option then is there a less verbose or more pythonic library/approach than using SQL statements to interact with the database? – Jab Nov 21 '18 at 10:44
  • @SteveBarnes is the acknowledged Python guru hereabouts. Maybe he can recommend a package; but, I can't see how you can get too far away from SQL. It's not a particularly difficult "language" & there are tool sot generate/validate SQL. How much data do you expect - in total and per second? And will you be threading? – Mawg says reinstate Monica Nov 21 '18 at 10:47
  • In specific cases I do expect to be threading, but on a regular basis no the majority will be async calls. I'd just like one database for the whole application. possibly 2 as with the threading it will have heavy reads and writes – Jab Nov 21 '18 at 11:46
  • "heavy reads and writes" sounds like a job for an RBMS – Mawg says reinstate Monica Nov 21 '18 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.