Ours is a enterprise based SaaS application which is hosted on AWS. We use a lot of AWS resources including RDS, Dynamo, Kinesis, Memcache, SQS.
Our application has accounts and each account can have many users who use the service. Number of accounts and users in a single AWS region as of now is 61k accounts and 1.9 million users. We have 3 such hosted regions.
We have a monolith server and have adopted miscroservices architecture thereafter for newer services. Total number of distributed services would be around 50.
My goals are:
- to identify the top 3 accounts that are using the maximum number of resources across all services.
- Identify top 3 requests consuming maximum resources across all services
- Identify unfinished requests. NOTE: Not longest requests but the requests that have run for too long and are not yet completed.
Roughly I will need to track every request based on account_id, user_id and request_uuid along with a timestamp of start and completed for every service to achieve the above goals.
I have explored Prometheus but the fact that I have to track by request_uuid, account_id and user_id eliminates this choice as using these as labels will lead to exploding time series. Or is it still fine for the numbers I have quoted?
I explored distributed tracing tools like Zipkin but then again it wont give me the granularity of the request specific details like the ids.
This leaves me with log aggregation as an option. We already use splunk but we have seen a lot of data loss during very high traffic because of their rate limiting.
ELK does not seem like a good option as we already are facing the re indexing latencies in another service.
Should I just use a TSDB and dump only necessary data in a format that can be retrieved in a way I want to? This sounds like re inventing the wheel. I would like to know how do others achieve such goals? Surely this must be a common problem that is already solved.
Also, we use NewRelic and Zabbix for metrics and alerting. The above solution is to bridge the gaps between these tools.