记一次性你调优 IO瓶颈
http://blog.csdn.net/hualusiyu/article/details/10023281
top - 18:38:56 up 5 days, 14:48, 7 users, load average: 53.43, 188.00, 422.23
Tasks: 504 total, 1 running, 494 sleeping, 9 stopped, 0 zombie
Cpu(s): 1.6%us, 0.4%sy, 0.0%ni, 97.8%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 65923236k total, 16464576k used, 49458660k free, 194628k buffers
Swap: 4095992k total, 0k used, 4095992k free, 6482424k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15435 root 20 0 19.6g 4.9g 10m S 157.6 7.8 92:22.20 java
22890 root 20 0 15288 1428 828 R 3.8 0.0 0:00.04 top
1 root 20 0 19352 1624 1312 S 0.0 0.0 0:02.53 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.22 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.30 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 1:34.54 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.03 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/0
发现load巨高，，，，【最好看看单个核的wa参数】
iostat -x -d -m 1 10
Linux 2.6.32-279.el6.x86_64 (localhost.localdomain) 03/03/2014 _x86_64_ (24 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.01 9.00 0.05 1.65 0.00 0.04 51.01 0.00 0.45 0.23 0.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 24.00 0.00 4.00 0.00 0.11 56.00 0.00 0.25 0.25 0.10
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 23.00 0.00 2.00 0.00 0.10 100.00 0.01 3.00 1.50 0.30
[root@localhost classes]# vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 0 49571696 194440 6387936 0 0 0 2 0 3 2 1 98 0 0
2 1 0 49557080 194440 6388180 0 0 0 66 16366 19851 5 2 92 2 0
2 1 0 49556576 194440 6388364 0 0 0 0 15859 19556 4 1 92 3 0
从iostat跟vmstat命令来看，，，，，貌似 io的负载不是很高，，，，，，
这只能说明本地磁盘不是瓶颈，，，，但是网络硬盘呢？？？发单牵扯到语言单的存储等
iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
16816 be/4 root 0.00 B/s 5.93 K/s 0.00 % 0.04 % java -Djava.util.logging.config.fil~che.catalina.startup.Bootstrap start
15811 be/4 root 0.00 B/s 2.97 K/s 0.00 % 0.03 % java -Djava.util.logging.config.fil~che.catalina.startup.Bootstrap start
16462 be/4 root 0.00 B/s 2.97 K/s 0.00 % 0.00 % java -Djava.util.logging.config.fil~che.catalina.startup.Bootstrap start
15968 be/4 root 0.00 B/s 2.97 K/s 0.00 % 0.00 % java -Djava.util.logging.config.fil~che.catalina.startup.Bootstrap start
16289 be/4 root 0.00 B/s 2.97 K/s 0.00 % 0.00 % java -Djava.util.logging.config.fil~che.catalina.startup.Bootstrap start
其中TID为线程id，可以通过jstack -l 进程号 把线程信息dump出来，，，，然后就知道是哪个线程占用io比较高
"http-10.132.23.74-8090-exec-54" daemon prio=10 tid=0x00002aab36ecec00
nid=0x3175 waiting on condition [0x00000000498c0000..0x00000000498c0e20]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaaba3cf768> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
at java.lang.Thread.run(Thread.java:619)
不停的比对（一两次搞不出来，，多搞几次就可以看出来了），，，最终可以定位到 写语音文件的线程，，，，写的是网络磁盘