1

Around 150K of records daily, expect to grow 100% in a year. Average record size is below 200B. One day of records is around 30MB.
(That would be 11 GB of data at the end of the year, or 22 GB with the 100% growth). Max QPS is around 6.

Records will be added by either Go or PHP written app.

The first approach was to write simple text files. The second, is to write binary encoded ProtoBuf or Avro messages to files. Both of this are simple length delimited binary files.

I'm concerned with data corruption.

What I would like:

  • validation checksum
  • or correction codes

I was considering HBase but it's like taking a sledgehammer to crack a nut.

I'm looking either for a file format which allows to skip/recover corrupted data or a minimalistic storage solution.

4
  • The records never ever change, right? You never modify or delete past data?
    – Nicolas Raoul
    Dec 15 '16 at 8:31
  • That's true. I will process them, but the processed data is a new set of logs and will be stored in a different place. Dec 15 '16 at 9:04
  • 1
    This use case is called en.wikipedia.org/wiki/Write_once_read_many. The HBase website says "Use Apache HBase™ when you need random, realtime read/write access". Since you don't need write access, HBase is probably not the most efficient for you. Still not WORM, but by the way would it be too basic to just append to files on a journalized filesystem like ext4?
    – Nicolas Raoul
    Dec 15 '16 at 10:29
  • Thanks for a link! At this point, I'm going to append to files ProtoBuf binary encoded messages. Seems like the simplest solution. Dec 15 '16 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.