Software/framework in Linux for filtering HTTP traffic?

Question

I have been using mitmproxy for filtering HTTP traffic. mitmproxy is simple and written in Python. It will start a new thread to handle every new HTTP request.

Here are some things I need to get from those request flows:

1. URL
2. parameters
3. IP of the request
4. MAC of the request
5. parse the content and do some logistics

mitmdump is shipped from project mitmproxy and it works in command line. Extensively I used it in this way:

mitmdump -T -s ./my_written_request_handler.py (it listens on port 8000, and I used iptables to forward all HTTP traffic to this port)

I handled all the requests in this handler ./my_written_request_handler.py, but I found it to be less efficient. Maybe because it's in Python!

So I want a new easy-efficient tool or framework to do this job.

Will a firewall do this job without proxy?

Answer 1

I think HAProxy is one of your best bets if you want an efficient way to filter traffic with a lot of rules under Linux. Over the years, they say Haproxy has become "de-facto standard opensource load balancer, is now shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms". It's been around for quite a while and can be tricky but worth it.

From their documentation about "What HAProxy is and is not", I can see the following that would fit your most important requirement:

HAProxy is [also] an HTTP fixing tool: it can modify / fix / add / remove / rewrite the URL or any request or response header. [...] HAProxy is [also] a content-based switch: it can consider any element from the request to decide what server to pass the request or connection to. [...]