I have a system with millions of active clients sending logs to my backend. My current log management system uses Kafka-HDFS-Drill infrastructure, but i'm experiencing large troubles with Drill-HDFS interaction. The functionality i'm trying to implement is saving logs and requesting logs for one client by some filters. Assume that client base is growing and the system should be reasonably scalable. So, what are best practices, tools or frameworks to store client logs?
Raw data income is approximately 20-30 Gb per day, and log records should also be available for 3 months. Max space should be not more than 3 Tb, and the system should have replication factor 2 or more. So, we must compress data somehow to fit these conditions.
Logs have the following structure:
clientID int level byte logTimestamp long serverTime long message String
So, the system should be able to find all logs by
clientID, possibly with
level, ordered by
logTimestamp, and response time should be not more than 10 seconds. Message is up to 10 Kb size.
What we have now: six middle-class instances in AWS, running on Ubuntu 16.04, and we shouldn't be running out of this.