Right now, I have a system that generates about 20 million objects per day. I want to retain a year's worth of data. I need to store these so a data team can analyze them and make predictions based off of them. Since the volume is so high, I tolerate the additional loss of objects as it isn't really a big deal. In fact, the data team regularly undersamples the data anyways to make it workable.

The objects are streamed from servers to a single machine using TLS connections. The client is written in python but could be rewritten at any point

My first attempt at storing these objects was writing JSON into a file on the disk. The files are rotated every 30 minutes so we don't wind up with huge files or a large number of small files.

This became very large very quickly. I switched to writing compressed files using the bz2 module. This saves greater than 90% of the disk space since it is compressible.

Unfortunately, this makes reading anything in the files require decompression first. So my third attempt writes an additional uncompressed file with summary data. This is basic stuff like the first and last time of the messages in the file.

The data team is using the summary files to determine which files to process. This obviously speeds things up a great deal since they can often eliminate 90% of the dataset before they even start decompressing files. After that point, answering any question still involved scanning the whole file. Obviously, I can add more summary data to allow more questions to be answered without decompression.

At this point, I'm damn close to writing a pseudo-database. I don't really want to go down that path yet as I'm hoping it isn't necessary.

So far I think my requirements are the following

  1. I need to persist objects to disk. Essentially, each file would just be a massive array with no real ordering about the objects. An objects is a key-value mapping from a set of keys to a set of values (1 to 1). The only types I care about are strings, lists, dictionaries, integers, and floating point numbers.
  2. I need to be able to index the file somehow, preferably when the file is closed.
  3. I need to be able to query the file using some sort of language and get an answer using those indices.
  4. I don't need mutability, but it doesn't matter if a particularly solution allows for it.

Is there anything that can do something like this?

  • 1
    You should look at datawarehouse storage solutions and the systems used by e.g. banks and credit card companies, that have to analyze enormous amounts of data to detect patterns. One name that comes to mind is Teradata. And I think that avoiding databases is a bad choice, these are developed especially for handling large amounts of data. But a lot depend on the structure of your 'objects'. Maybe your question is more appropriate on Programmers.SE
    – user416
    Oct 6, 2014 at 9:29
  • I guess I should have added: the budget for this is $0.
    – Eric Urban
    Oct 6, 2014 at 22:56
  • 1
    edit this into your question and maybe delete the comments (that goes for your comment under Nicolas' answer as well). And you still have not answered what 'objects'
    – user416
    Oct 7, 2014 at 6:22

1 Answer 1


You could use a compressed table with MySQL/InnoDB:


  • Persisted to disk
  • Compressed
  • Indexes
  • SQL queries

It is mutable though. Performance could probably be better with an nnon-mutable database engine.

  • We tried throwing the data at an RDBMS a while back. Writes were fast but queries took forever. Part of the appeal of not using a real database in my opinion is the lack of any ongoing maintenance. We can throw files on network attached storage we already have.
    – Eric Urban
    Oct 6, 2014 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.