Tool to determine what the Python worker processes are doing?

Question

I have a django wsgi application running on Apache on Linux.

I would like to get an overview of what is going on during an average day.

Some months ago I wrote a tool to see what the python interpreter is doing all day long: https://github.com/guettli/live-trace

It has a major draw-back: If the python interpreter is inside the GIL (Global Interpreter Lock), then no logging of stacktraces happen. This means the result is wrong.

Is there a different tool which helps me to see in which python source code lines the interpreter is most of the time?

Answer 1

https://github.com/uber/pyflame

from the README

A Ptracing Profiler For Python

Pyflame is a high performance profiling tool that generates flame graphs for Python. Pyflame is implemented in C++, and uses the Linux ptrace(2) system call to collect profiling information. It can take snapshots of the Python call stack without explicit instrumentation, meaning you can profile a program without modifying its source code. Pyflame is capable of profiling embedded Python interpreters like uWSGI. It fully supports profiling multi-threaded Python programs.

Pyflame usually introduces significantly less overhead than the builtin profile (or cProfile) modules, and emits richer profiling data. The profiling overhead is low enough that you can use it to profile live processes in production.