How to find per-process I/O statistics on Linux

Newer Linux kernels have per-process I/O accounting and you can use the iotop tool to find out what’s performing I/O, but in many cases I’m trying to find the source of an I/O problem in an older kernel. I found sort of a hack-ish way to do that today, while trying to figure out why a system was basically unresponsive.

I found a post on Stack Overflow that showed a way you can get per process I/O statistics from the kernel even in older kernels. I adapted this to my needs, and wrote a little script.

Here’s how you use it. First, get it:

wget http://aspersa.googlecode.com/svn/trunk/iodump

Then turn on kernel messages about I/O:

echo 1 > /proc/sys/vm/block_dump

This makes the kernel start writing messages about every I/O operation that takes place. Now all you have to do is get those messages and feed them into my script:

while true; do sleep 1; dmesg -c; done | perl iodump

Wait a little while, then cancel the script. The results should look something like the following:

root@kanga:~# while true; do sleep 1; dmesg -c; done | perl iodump

^C# Caught SIGINT.

TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES

firefox               4450       4538        251       4287          0 sda4, sda3

kjournald             2100        551          0        551          0 sda4

firefox              28452        185        185          0          0 sda4

kjournald              782         59          0         59          0 sda3

pdflush                 31         30          0         30          0 sda4, sda3

syslogd               2485          2          0          2          0 sda3

firefox              28414          2          2          0          0 sda4, sda3

firefox              28413          1          1          0          0 sda4

firefox              28410          1          1          0          0 sda4

firefox              28307          1          1          0          0 sda4

firefox              28451          1          1          0          0 sda4

I deliberately generated a bunch of I/O by deleting my Firefox history and cache.

Got any better ideas, warnings, etc? Post them in the comments.

Written by Baron Schwartz

August 23rd, 2009 at 1:26 am

Posted in GNU/Linux,PostgreSQL,SQL,Sys Admin,Tools

«  A Review of Beginning Database Design by Clare Churcher

OpenSQL Camp events in 2009 »

22 Responses to 'How to find per-process I/O statistics on Linux'

Subscribe to comments with RSS

1.  

   iotop (http://guichaz.free.fr/iotop/) is not too dissimilar from this, only giving real-time usage statistics, without the need to sample for a while before reviewing.

   __

   23 Aug 09 at 4:42 am

    
2.  

   is there similar way to see I/O dump or mac osx? i use osx86 on my laptop and week ago , I badly needed to see I/O write activity which was in MB/s constantly with no real activity carried by me.

   anshuman gholap.

   anshuman

   23 Aug 09 at 6:47 am

    
3.  

   Good tip, thanks!

   It’s minor, but I think “while true; do sleep 1″ can be replaced by “watch 1″

   Ludwig

   23 Aug 09 at 6:58 am

    
4.  

   Another good tool is iotop. Debian users can simply “sudo aptitude install iotop”. iotop provides real-time disk i/o stats. Even without cumulative totals, it’s still fairly useful!

   Sohail Mirza

   23 Aug 09 at 7:02 am

    
5.  

   sysstat 8.1.7-1 from ubuntu contains pidstat tool
   It’s useful for per-proc io statistic with -d key
   This is an example.
   pidstat -d 1
   Linux 2.6.31-rc5-generic (vega) 08/23/2009 _i686_ (1 CPU)

   04:20:05 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
   04:20:06 PM 6452 41009.71 0.00 0.00 dd

   04:20:06 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
   04:20:07 PM 6452 39920.79 0.00 0.00 dd

   Nickolay Ihalainen

   23 Aug 09 at 8:20 am

    
6.  

   Right, pidstat, iotop, iopp — all are very useful, but not on an old CentOS server that doesn’t have per-process IO accounting :-)

   Xaprb

   23 Aug 09 at 1:41 pm

    
7.  

   I/O accounting should be part of RHEL/CentOS 5.4.

   http://dag.wieers.com/blog/red-hat-backported-io-accounting-to-rhel5

   Bruce Locke

   23 Aug 09 at 3:09 pm

    
8.  

   Have a look at blktrace (the rpm or deb should include blktrace, blkparse, btrace and btt). Just point it at a block device and run for a while. Very useful. Lots of good howto’s turn up on google if you care to look.

   John McNulty

   23 Aug 09 at 6:16 pm

    
9.  

   [...] How to find per-process I/O statistics on Linux at Xaprb (tags: linux sysadmin performance monitoring tools process kernel tips) [...]

   links for 2009-08-23 at DeStructUred Blog

   23 Aug 09 at 10:03 pm

    
10.  

    Why not write a single script to do everything required, something like:

    #!/bin/sh

    dmesg -c
    /etc/init.d/klogd stop
    echo 1 > /proc/sys/vm/block_dump

    # allow 30 seconds of stats to be logged
    sleep 30

    dmesg -c | perl iodump

    echo 0 > /proc/sys/vm/block_dump
    /etc/init.d/klogd start

    Tom Collins

    24 Aug 09 at 12:31 am

     
11.  

    John, I’ll look into that, thanks for the tip.

    Tom, I wrote it as a separate script because some systems actually don’t have a klogd running as a daemon (and/or you might not want to disable it), and I also want to be able to log on one system and analyze on another system.

    Xaprb

    24 Aug 09 at 8:27 am

     
12.  

    [...] How to find per-process I/O statistics on Linux at Xaprb (tags: linux sysadmin performance io monitoring) [...]

    andy.edmonds.be › links for 2009-08-25

    25 Aug 09 at 8:35 pm

     
13.  

    [...] Schwartz shared How to find per-process I/O statistics on Linux. He writes, “Newer Linux kernels have per-process I/O accounting and you can use the iotop [...]

    Log Buffer #159: a Carnival of the Vanities for DBAs | Pythian Group Blog

    28 Aug 09 at 1:05 pm

     
14.  

    Could you include cpu percent, memory usage (which working set in windows XP) for all processess in this?
    Also i would like to know about the unit of i/o information. Is it KB or Bytes? Thank you in advance.

    Sanjeev

    25 Jun 10 at 3:38 am

     
15.  

    Last post got no answer, I’d very much like to know as well – what units is this measuring? B, kB, MB?

    sahumani

    19 Oct 10 at 4:21 am

     
16.  

    I don’t remember. If you find out, post the answer in a new comment.

    Xaprb

    19 Oct 10 at 6:47 am

     
17.  

    Looks like the unit is simply “times”, like when you ask How many times has this process done READ’s (or WRITE’s)? Every time, a READ (or WRITE) reads (writes) one block. Not sure how vm/block_dump records scatter/gather I/O (multiple blocks per read or write). I guess it breaks it up into multiple READ/WRITE lines but I’m not sure. The next question is How big is the block? I’m pretty sure it’s the file system block size as when you created the file system. With that info, you can convert the iodump result into KB or MB.

    Yong Huang

    17 Nov 10 at 11:59 am

     
18.  

    Here is the script :

    #!/bin/bash
    dmesg -c
    echo 1 > /proc/sys/vm/block_dump
    sleep 30
    go=0
    while [ $go = 0 ]; do
    sleep 1
    clear
    dmesg | perl iodump
    trap “go=1″ SIGINT
    done
    echo 0 > /proc/sys/vm/block_dump
    dmesg -c
    clear
    echo Caught SIGINT.

    terrabit

    18 Dec 10 at 3:41 am

     
19.  

    The unit for block_dump data is blocks, and the easiest way to determine that size in a way that works for any filesystem type is:

    /sbin/blockdev –getbsz /dev/sda1

    Note that even though I/O accounting showed up in RHEL 5.4, you still can’t get iotop to work because of other dependencies. As of iotop 0.4, it will compile and run against the Python 2.4 included with RHEL, so long as you’ve installed the python-ctypes package. But iotop still doesn’t work. Even though the main per-process I/O patch was backported in RHEL 5.4, the current kernel isn’t configured correctly for it, and there are other issues with the backport being incomplete. If you build iotop and run it, against the most recent RHEL 5.5 kernel I have here (2.6.18-194.26) it gives the error “CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %” and the detail level is accordingly low.

    There is an open enhancement request to sort this out so iotop works:https://bugzilla.redhat.com/show_bug.cgi?id=557062 but it keeps missing being included in RHEL releases. It was hoped for in 5.6 but missed that deadline. The kernel patches needed are available (along with RHEL engineer packaged iotop related RPMs) are available athttp://people.redhat.com/jolsa/iotop/ ; while the patches there aren’t large, it isn’t as easy as just turning on the kernel config option.

    Greg Smith

    3 Jan 11 at 3:02 pm

     
20.  

    For those trying to get Guillaume Chazarain’s sophisticated iotop to work, try my humble little topio program as a workaround:
    http://yong321.freeshell.org/freeware/pio.html#linux

    Over the past 7 years, I ported my program to multiple OS’es. The Linux version was done only a few months ago.

    Yong Huang

    3 Jan 11 at 3:48 pm

     
21.  

    iodump appears to have a bug in it, as it always reports a 0 for “DIRTY”. Unfortunately, I am not a perl or regex expert and after spending several hours with this script cannot seem to get a valid value for DIRTY count. The reason I believe DIRTY count is important is the system I’m testing has an ext3 file system with kjournald doing reads and writes on behalf of processes. So if a process “dirty” requires a kjournald write, that won’t show up without the DIRTY value. Plus, the key measure I’m looking at right now is per process startup resource usage, and a command line linux tool such as iostat that can’t be run until well after startup has completed won’t help. Was hoping to use iodump to process the kernel log to crunch the startup data. Thanks!

    Chris Rizzo

    10 Feb 12 at 5:12 pm

     
22.  

    Ahhh… I believe I’ve found the bugs in the script. The dirtied regex should be:
    m/(\S+)\((\d+)\): (dirtied) inode (\d+) \(.*?\) on (\S+)/
    And line 69 should be $task->{‘dirtied’}, not $task->{‘dirty’} (given the following input):
    Feb 9 20:20:23 XRX_0000AA9E8ABA kernel: [ 14.974373] klogd(242): dirtied inode 47718 (klogd.pid) on sda5

    Chris Rizzo