I'm looking for a simple wall-clock profiler for Linux. I just want to see the list of symbols/methods, the count of calls to that method, and the average time spent in that method per call to it in microseconds. I tried the built-in perf tools; they seem to report active CPU time only. I'm also open to suggestions for how to write a tool on Linux that knows when methods are called & returned in some other process (using the perf API or whatever).
After wasting a whole day researching this and testing various tools (after an entire career of depending on gprof, which never really worked), I realized that such a tool just doesn't exist. There are lots of nice tools (like perf, gperftools, etc.), but none of them give you a true wall-clock sample of where your program is really spending its time, including time that it spends waiting on I/O. Valgrind callgrind is the only one that can do it, but the overhead (50x slowdown) is unacceptable in a production environment.
So I wrote my own, and it actually works. The only dependency is GDB, which you probably have already if you're doing this stuff.
-
-
I'm still working on it, but I think it's pretty close to done.... spent a lot of time improving it yesterday through use in a production environment. GDB/MI (the interface that it uses to control GDB) is a little flaky, and I also got bitten by SIGPIPE (which my target program was ignoring for networking reasons, but GDB was still catching). But now it can attach to a live running server process (my game server, which currently has 100+ players connected to it at the same time), grab samples for however many seconds, and then detach without disturbing the server process at all. Good stuff. – Jason Rohrer Jan 23 '19 at 18:23
-
-
Following up: I did try this some time ago. However, as it did not support multiple threads, it did not work for my situation. – Brannon Oct 26 '20 at 12:49
One way to do this is with Google's CPU Profiler. A link, along with instructions is on the github: https://github.com/gperftools/gperftools.
To make it use walltime, set CPUPROFILE_REALTIME=1 (Warning, this will break sleeps)
-
Breaking sleeps sounds pretty severe. I'm certain my application uses a number of those as well as various timers. Part of the point of this was to see if I was hitting any of those inadvertently. – Brannon Aug 28 '18 at 20:37
straceappears to lack. – agc Aug 29 '18 at 1:19stracesummary seems to be close to what I want; I just need the top 5 stack traces for each of the top 5 slow system calls in the summary. – Brannon Sep 5 '18 at 23:34