Tool for statistical performance measurement in production environments

Question

I search a tool for statistical performance measurement in production environment.

I want to know what the Python interpreter is doing most of the time. With other words I want to detect hot spots.

I want to observe the production environment for several hours.

Later I want to aggregate where the interpreter spends the most time.

Required Features:

• Open source and no viral license like GPL
• Only very little performance impacts since I want to check my production environment
• Suited for Python and web environments (we use Django).

Answer 1

This tool might help: https://github.com/guettli/live-trace

live-trace

live-trace is a Python library to log stacktraces of a running application in a daemon thread N times per second. The log file can be analyzed to see where the interpreter spends most of the time. It is called "live-trace" since it can be used on production systems without noticeable performance impact.

If you want to see why a particular request is slow, then use a profiler, not live-trace.

Use live-trace if you want to see the bird's-eye view. If you ask yourself the question:

What is the interpreter doing all day long?

Then use live-trace in your production environment. Let it collect a lot of snaphosts of the interpreter state. The current implementation uses stacktraces for freezing the state of the interpreter. After running for some hours you can aggregate the collected stacktraces and identify hot spots.

Background to this question and answer at "Software Recommendation"

I wrote this tool "live-trace" and asked here if there is a tool like this.

I did this, since I am not very happy with the current implementation of live-trace. If the interpreter executes a lot of code in c extensions, then the result is not reliable. AFAIK the daemon thread can't record stacktraces if the main thread is inside a c extensions (except the GIL get's released, like explained here thread state and GIL).

A work-around this would be a second process which collects stacktraces via gdb. Or even a better solution which I have not found up to now...

Answer 2

PyFlame looks good: https://github.com/uber/pyflame

From the docs:

Pyflame: A Ptracing Profiler For Python

Pyflame is a high performance profiling tool that generates flame graphs for Python. Pyflame is implemented in C++, and uses the Linux ptrace(2) system call to collect profiling information. It can take snapshots of the Python call stack without explicit instrumentation, meaning you can profile a program without modifying its source code. Pyflame is capable of profiling embedded Python interpreters like uWSGI. It fully supports profiling multi-threaded Python programs.

Pyflame usually introduces significantly less overhead than the builtin profile (or cProfile) modules, and emits richer profiling data. The profiling overhead is low enough that you can use it to profile live processes in production.