I'm looking for a simple wall-clock profiler for Linux. I just want to see the list of symbols/methods, the count of calls to that method, and the average time spent in that method per call to it in microseconds. I tried the built-in perf tools; they seem to report active CPU time only. I'm also open to suggestions for how to write a tool on Linux that knows when methods are called & returned in some other process (using the perf API or whatever).

  • Please clarify what strace appears to lack. – agc Aug 29 '18 at 1:19
  • The strace summary seems to be close to what I want; I just need the top 5 stack traces for each of the top 5 slow system calls in the summary. – Brannon Sep 5 '18 at 23:34

After wasting a whole day researching this and testing various tools (after an entire career of depending on gprof, which never really worked), I realized that such a tool just doesn't exist. There are lots of nice tools (like perf, gperftools, etc.), but none of them give you a true wall-clock sample of where your program is really spending its time, including time that it spends waiting on I/O. Valgrind callgrind is the only one that can do it, but the overhead (50x slowdown) is unacceptable in a production environment.

So I wrote my own, and it actually works. The only dependency is GDB, which you probably have already if you're doing this stuff.


  • Thanks for posting this. I will try it in the near future. – Brannon Jan 22 '19 at 15:01
  • I'm still working on it, but I think it's pretty close to done.... spent a lot of time improving it yesterday through use in a production environment. GDB/MI (the interface that it uses to control GDB) is a little flaky, and I also got bitten by SIGPIPE (which my target program was ignoring for networking reasons, but GDB was still catching). But now it can attach to a live running server process (my game server, which currently has 100+ players connected to it at the same time), grab samples for however many seconds, and then detach without disturbing the server process at all. Good stuff. – Jason Rohrer Jan 23 '19 at 18:23
  • I tried it out and worked great for me, very useful. – Zitrax Apr 11 '19 at 20:32
  • Following up: I did try this some time ago. However, as it did not support multiple threads, it did not work for my situation. – Brannon Oct 26 '20 at 12:49

One way to do this is with Google's CPU Profiler. A link, along with instructions is on the github: https://github.com/gperftools/gperftools.

To make it use walltime, set CPUPROFILE_REALTIME=1 (Warning, this will break sleeps)

  • Breaking sleeps sounds pretty severe. I'm certain my application uses a number of those as well as various timers. Part of the point of this was to see if I was hitting any of those inadvertently. – Brannon Aug 28 '18 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.