I have a system with millions of active clients sending logs to my backend. My current log management system uses Kafka-HDFS-Drill infrastructure, but i'm experiencing large troubles with Drill-HDFS interaction. The functionality i'm trying to implement is saving logs and requesting logs for one client by some filters. Assume that client base is growing and the system should be reasonably scalable. So, what are best practices, tools or frameworks to store client logs?


Raw data income is approximately 20-30 Gb per day, and log records should also be available for 3 months. Max space should be not more than 3 Tb, and the system should have replication factor 2 or more. So, we must compress data somehow to fit these conditions.

Logs have the following structure:

clientID       int
level          byte
logTimestamp   long
serverTime     long
message        String

So, the system should be able to find all logs by clientID, possibly with level, ordered by logTimestamp, and response time should be not more than 10 seconds. Message is up to 10 Kb size.

What we have now: six middle-class instances in AWS, running on Ubuntu 16.04, and we shouldn't be running out of this.

  • Please read meta.softwarerecs.stackexchange.com/questions/336/… and give the additional info mentioned in there. Also: what format are the logs in, actual and expected files sizes and numbers, and if you can elaborate on requesting logs for one client by some filters that would be great. – user416 Oct 26 '16 at 12:55
  • @JanDoggen updated my question, hope that it will be helpful now. – Everv0id Oct 26 '16 at 13:48
  • It looks like you overlooked and give the additional info mentioned in there. OS, price, etc? – user416 Oct 26 '16 at 14:19

You didn't mention about managed service or not, then you can try AWS Redshift (OLAP). Starting with Dense Storage type (ds2.xlarge), with configuration : 1 Leader Node, and 2 Compute Node.

To send data to Redshift, use Kinesis Firehose, it is the easiest. It will use S3 as intermediate storage in CSV format. Beside, csv format can be use everywhere. From S3 you can replicate into another redshift cluster, but you need to create the job for this.

Redshift Table structure :

CREATE TABLE "public"."log" (
    "logTimestamp" int4 NOT NULL encode delta distkey,
    "clientID" int NOT NULL encode delta,
    "level" smallint NOT NULL encode delta,
    "serverTime" int4 NOT NULL encode delta,
    "message" varchar(max) NOT NULL encode lzo --untested
compound sortkey (logTimestamp, clientID, level);

Using right sortkey and distkey will increase the performance.

In firehose part, use smaller s3 buffer & smaller s3 interval to make the load operation faster, remember, to do this, it need more than 1 compute node.

Another worth to try solution (I haven't tested it yet) :


So, I think I can finally resolve my question. We built a system with ClickHouse DB as a storage, and Apache Kafka as an event receiver. We receive 200Gb of client logs per day (and this is current size of input data per day for our Kafka cluster), and then our kafka consumers (written on java) insert large batches of events from Kafka to Clickhouse cluster.

Some specific info: each event contains approximately 1Kb of data, and we receive 200kk of such events per day, which is exactly 200Gb of pure data per day. Clickhouse cluster compresses 200Gb of these events into 3Gb. Queries like 'find some substring' or 'find all events by ClientID' in Clickhouse table of such size take 1 or 2 seconds. So, I think that Clickhouse fits perfectly for solving the problem of storing large amount of events with text payload. We expect that we could handle 10 times more of input data with current hardware configuration (3 Kafka instances, 3 Clickhouse instances, and some ZooKeeper cluster outside for storing configurations, all of those are middle-class AWS servers) without nesessity to scale.

Both Kafka and Clickhouse are open-source and free to use, so if you are reading this you should definitely take a look.

UPDATE After two months of production use of ClickHouse we finally removed it from our stack. The reason was that ClickHouse performs big amounts of disk reads when we select something by key. For each 1MB of useful data ClickHouse reads 2GB of data from disk. It leaded to slow performance, each by-key request took 2 minutes which is not acceptable.

As an alternative we used plain file system, where we store our data in JSON format. File system backend is known as not fault-tolerant and not write-scalable, but it gives us the ability to get data by key in 1-5 seconds which is acceptable time for web requests.

By the way, we are looking for a storage that is scalable and shows performance close to disk i/o.

  • FYI: Now a days you should be able to fix any slow i/o issues with either SSD(550mb/s) or M.2 SSD.(2-3gb/s). If that is not enough RAID a couple together. You can get 4 m.2 on a pci-e x16 card. – cybernard Sep 25 '18 at 0:56

Try graylog. It is remote syslog collection software, working on elasticsearch database backend. It can process thousands of messages per second. You can store any text logs you have there and even use regular expressions to gather some structured fields from raw data. It is possible to feed data via rsyslog protocol or by agents on legacy ( eg. Microsoft Windows) systems

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.