Log highest disk-reading process, every second, even on almost-freezed Linux

Question

My computer sometimes freezes because some process starts to read huge amounts of data from the hard disk. I want to find out what process does this.

The tool must:

• Add one line to a given text file every second
• That line must contain:
  • The name (for instance "virtualbox" or "/usr/lib/firefox/firefox -new-window" or something similar) of the program that has read the most data from the hard disk (I only have one hard disk) during that second
  • A timestamp, without which I can not know whether the reads happened before or during the freeze.
  • If possible, the quantity of data read by that program during that second (for instance "17899 blocks" or "2 MB/s" or something similar)
• It must run even if the computer is running super slowly:
  • Not start any new process, because in these near-freeze conditions starting a new process (even a very tiny one) takes a very long time. That means no bash script calling ps every second.
  • Be efficient. If possible, not compute statistics for all processes and then throw all but the first one.
  • Write directly to disk, because when these freezes happen the only thing I can do is wait an hour, reboot the computer, then read that text file to investigate on the cause of the crash.

I will run the program as root with high priority.

Answer 1

iotop -bot > ~/iotop.log is a not-so-bad solution. Other solutions are very welcome!

• It runs as a single process.
• The -b argument makes it run in batch mode, and output information to the standard output rather than in an interactive console.
• The -o argument makes it only list processes that actually performed IO operations, which is more efficient than the default mode of listing all processes.
• The -t argument makes it print a timestamp at the beginning of each line.

Unfortunately, it also computes disk writes and other statistics, which probably wastes resources. The information can at least be removed from the output by using the following command:

iotop -bot | grep -v "^.........Total\|^.........Actual\|^ TIME" > ~/iotop.log