I am looking for a solution to to process and analyze large amounts of network traffic without writing to disk.

Because I'm looking to see if there is an open source project or tool out there in general, I'm not going to specify the amount of data coming in. I think it is suffice to say that the amount of data coming in over the network is large enough that the writing of pcaps via python libpcap cannot keep up with the traffic.

Here is the general solution workflow I'm trying to work out -- the following steps are an active in-memory processing stream as network data comes in:

  1. Process all network traffic coming into one chokepoint -- some Linux server.
  2. Filter traffic that is irrelevant (likely based on protocol/port and maybe ip)
  3. Analysis engine will look at incoming packet data and determine if it is of interest (this could be as simple as matching a string in the packet payload)
  4. If an interesting hit is triggered by analysis engine, write a pcap to disk. (this would imply a buffer of some sort surrounding the interesting event, perhaps 1 day)

This solution would be a service on the Linux server (assuming the server has sufficient specs).

I feel like I am touching on a very common problem here that must have been solved before, but I could not easily find a packet streaming framework or tool that can scale to large amounts of data. Perhaps the idea is to use Python libpcap with something like Apache Spark Streaming?

  • If the volume of data is so big then buffering it is likely to exceed the available RAM on just about any machine, especially a days worth. If you really need to capture that much then you need to capture to dedicated, very fast, drive. Jun 19 '16 at 4:31
  • Try ntop, which uses libpcap, but doesn't to long-term analysis and doesn't avoid disk altogether (but is open-source).
    – Jedi
    Jun 19 '16 at 5:19
  • Obviously you could mount a ram drive to tmpfs and get RAM speed writing. M2 ssd and pci-e ssd offer 2gb/s+ write speeds which can be increased using a multi drive raid.
    – cybernard
    Jun 20 '16 at 21:53
  • You might want to read this: ics.forth.gr/dcs/Activities/papers/gnort.raid08.pdf Someone made a GPU accelerated version of snort called gnort. I have no idea if this is a viable product or a lab experiment, but worth a look.
    – cybernard
    Jun 20 '16 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.