I'm looking for a simple wall-clock profiler for Linux. I just want to see the list of symbols/methods, the count of calls to that method, and the average time spent in that method per call to it in microseconds. I tried the built-in perf tools; they seem to report active CPU time only. I'm also open to suggestions for how to write a tool on Linux that knows when methods are called & returned in some other process (using the perf API or whatever).
After wasting a whole day researching this and testing various tools (after an entire career of depending on gprof, which never really worked), I realized that such a tool just doesn't exist. There are lots of nice tools (like perf, gperftools, etc.), but none of them give you a true wall-clock sample of where your program is really spending its time, including time that it spends waiting on I/O. Valgrind callgrind is the only one that can do it, but the overhead (50x slowdown) is unacceptable in a production environment.
So I wrote my own, and it actually works. The only dependency is GDB, which you probably have already if you're doing this stuff.
One way to do this is with Google's CPU Profiler. A link, along with instructions is on the github: https://github.com/gperftools/gperftools.
To make it use walltime, set
CPUPROFILE_REALTIME=1 (Warning, this will break sleeps)