system frequently IO writes

First get this script and name it: iodump

#!/usr/bin/env perl

=pod

=head1 NAME

iodump - Compute per-PID I/O stats for Linux when iotop/pidstat/iopp are not available.

=head1 SYNOPSIS

Prepare the system:

  dmesg -c
  /etc/init.d/klogd stop
  echo 1 > /proc/sys/vm/block_dump

Start the reporting:

  while true; do sleep 1; dmesg -c; done | perl iodump
  CTRL-C

Stop the system from dumping these messages:

  echo 0 > /proc/sys/vm/block_dump
  /etc/init.d/klogd start

=head1 LICENSE

This software is released to the public domain, with no guarantees whatsoever.

=cut

use strict;
use warnings FATAL => 'all';
use English qw(-no_match_vars);
use sigtrap qw(handler finish untrapped normal-signals);

my %tasks;

my $oktorun = 1;
my $line;
while ( $oktorun && (defined ($line = <>)) ) {
   my ( $task, $pid, $activity, $where, $device );
   ( $task, $pid, $activity, $where, $device )
      = $line =~ m/(\S+)\((\d+)\): (READ|WRITE) block (\d+) on (\S+)/;
   if ( !$task ) {
      ( $task, $pid, $activity, $where, $device )
         = $line =~ m/(\S+)\((\d+)\): (dirtied) inode \(.*?\) (\d+) on (\S+)/;
   }
   if ( $task ) {
      my $s = $tasks{$pid} ||= { pid => $pid, task => $task };
      ++$s->{lc $activity};
      ++$s->{activity};
      ++$s->{devices}->{$device};
   }
}

printf("%-15s %10s %10s %10s %10s %10s %s\n",
   qw(TASK PID TOTAL READ WRITE DIRTY DEVICES));
foreach my $task (
   reverse sort { $a->{activity} <=> $b->{activity} } values %tasks
) {
   printf("%-15s %10d %10d %10d %10d %10d %s\n",
      $task->{task}, $task->{pid},
      ($task->{'activity'}  || 0),
      ($task->{'read'}      || 0),
      ($task->{'write'}     || 0),
      ($task->{'dirty'}     || 0),
      join(', ', keys %{$task->{devices}}));
}

sub finish {
   my ( $signal ) = @_;
   if ( $oktorun ) {
      print STDERR "# Caught SIG$signal.\n";
      $oktorun = 0;
   }
   else {
      print STDERR "# Exiting on SIG$signal.\n";
      exit(1);
   }
}

Then turn on kernel messages about I/O:

echo 1 > /proc/sys/vm/block_dump

This makes the kernel start writing messages about every I/O operation that takes place. Now all you have to do is get those messages and feed them into the iodump script:

while true; do sleep 1; dmesg -c; done | perl iodump

Wait a little while, then cancel the script. The results should look something like the following:

# while true; do sleep 1; dmesg -c; done | perl iodump
^C# Caught SIGINT.
TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
firefox               4450       4538        251       4287          0 sda4, sda3
kjournald             2100        551          0        551          0 sda4
firefox              28452        185        185          0          0 sda4
kjournald              782         59          0         59          0 sda3
pdflush                 31         30          0         30          0 sda4, sda3
syslogd               2485          2          0          2          0 sda3
firefox              28414          2          2          0          0 sda4, sda3
firefox              28413          1          1          0          0 sda4
firefox              28410          1          1          0          0 sda4
firefox              28307          1          1          0          0 sda4
firefox              28451          1          1          0          0 sda4

When finished, turn off kernel messages about I/O:

echo 0 > /proc/sys/vm/block_dump

主要是因为有太多系统日志记录，而系统日志记录会引发kjournald内核文件系统日志记录程序大量频繁读写。

[root@no02 ~]# while true; do sleep 1; dmesg -c; done | perl iodump
# Caught SIGINT.
TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
kjournald             1873       2082          0       2082          0 dm-0
kjournald              520        258          0        258          0 sda1
pdflush              18732         78          0         78          0 sda1
bash                 15430         18         18          0          0 sda1
named                13285         11         11          0          0 dm-0
mysqld                6910          6          6          0          0 dm-0
syslogd              27718          3          0          3          0 sda1
sendmail               373          2          0          2          0 sda1
sendmail               466          2          0          2          0 sda1
procmail               670          1          0          1          0 sda1
sendmail               605          1          0          1          0 sda1

关闭syslog相关日志：

cat /etc/syslog.conf
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                            /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;named.none        /var/log/messages

# The authpriv file has restricted access.
authpriv.*                        /var/log/secure

# Log all the mail messages in one place.
mail.*                            -/var/log/maillog


# Log cron stuff
cron.*                            /var/log/cron

# Everybody gets emergency messages
*.emerg                            *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                        /var/log/spooler

# Save boot messages also to boot.log
local7.*                        /var/log/boot.log

重启syslog日志，发现io写频繁降下来。

service syslog restart