I am looking for a solution to to process and analyze large amounts of network traffic without writing to disk.
Because I'm looking to see if there is an open source project or tool out there in general, I'm not going to specify the amount of data coming in. I think it is suffice to say that the amount of data coming in over the network is large enough that the writing of pcaps via python libpcap cannot keep up with the traffic.
Here is the general solution workflow I'm trying to work out -- the following steps are an active in-memory processing stream as network data comes in:
- Process all network traffic coming into one chokepoint -- some Linux server.
- Filter traffic that is irrelevant (likely based on protocol/port and maybe ip)
- Analysis engine will look at incoming packet data and determine if it is of interest (this could be as simple as matching a string in the packet payload)
- If an interesting hit is triggered by analysis engine, write a pcap to disk. (this would imply a buffer of some sort surrounding the interesting event, perhaps 1 day)
This solution would be a service on the Linux server (assuming the server has sufficient specs).
I feel like I am touching on a very common problem here that must have been solved before, but I could not easily find a packet streaming framework or tool that can scale to large amounts of data. Perhaps the idea is to use Python libpcap with something like Apache Spark Streaming?