I am trying to develop endpoint detection and response (EDR) as a security solution. I am currently researching how to actually develop it. I googled quite a few websites, but they only described just the concept, benefit in terms of security management and business, and/or difference between EDR and Endpoint Protection Platform (EPP) while I want to know more technical aspect of EDR: what tools, frameworks, programming languages, services are used and what the architecture of EDR system is like on integrating these factors.
So far, I found Elastic Stack (combination of Beats, Logstash, Elasticsearch and Kibana: https://www.elastic.co/products/) seems to be one of the best solutions, which collects various logs from endpoints (done by Beats like packetbeat and filebeat), sends logs to my DB (by Logstash or Beats), stores them as Json format (Elasticsearch), and visualize the accumulated logs (Kibana).
As a first step of my EDR, I supposed only windows PC (not MAC or Linux, or smartphones, but will be expanded to them later) as endpoints. I have successfully collected the logs from plural endpoints and stored the data to a server. However, I have also found some problems (below).
- Logstash occupies pretty big communication band (about 250 Mbps)
- Logstash occupies pretty big memory resource on endpoints (about 800 MB)
- Logstash can send data by HTTPS only to Elasticsearch directly (direct access to DB from endpoint is not secure). Beats cannot send log data by HTTPS.
I am searching the ways to solve these problems as well, but, if there is any alternatives of Elastic Stack (or Logstash) which can solve the above issues, I want to try them. I found Apache Solr and Splunk are usually compared with Elastic Stack as a scalable semi-structured DB (suitable for search engine DB), but Apache Solr was a quite older tool as far as I searched, and Splunk requires quite expensive paid service. So I don't wanna use it.
Looking for any suggestion, advice and comments. Thank you!!