本文转载:http://nolinux.blog.51cto.com/4824967/1662942
公司在测试环境加了一批新的服务器,因为还没有急着上架就简单的使用FIO工具测试了下服务器磁盘在不同RAID级别下的IOPS的值,然后比对一下差距,增加一下经验。
这里简单描述下服务器的状态:
品牌:DELL 型号:PowerEdge R720 CPU:2 * Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz 内存:4 * 8G 阵列卡:PERC H310 Mini (Embedded) 硬盘:6 * 300G 15k SAS
本次主要使用FIO分别对RAID 0,RAID 5,RAID 10,RAID 50和单盘做了IOPS的测试,
下面首先介绍FIO的安装和使用,后面即为实验的输出和总结信息。
FIO工具
一、FIO 简述
FIO是测试IOPS的非常好的工具,用来对硬件进行压力测试和验证,支持13种不同的I/O引擎,包括:sync,mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio等。
fio 官网地址:http://freecode.com/projects/fio/
二、FIO 安装
[root@kvm-phy27-jz ~]# yum install -y gcc gcc-c++ libaio-devel [root@kvm-phy27-jz ~]# cd /usr/local/src [root@kvm-phy27-jz src]# wget http://brick.kernel.dk/snaps/fio-2.1.10.tar.gz [root@kvm-phy27-jz ~]# tar -zxf fio-2.1.10.tar.gz [root@kvm-phy27-jz ~]# cd fio-2.1.10 [root@kvm-phy27-jz fio-2.1.10]# make [root@kvm-phy27-jz fio-2.1.10]# make install [root@kvm-phy27-jz fio-2.1.10]# which fio /usr/local/bin/fio
三、FIO参数
filename=/test.xxx # 测试文件名称,通常选择需要测试的盘的data目录 direct=1 # 测试过程绕过机器自带的buffer,使测试结果更真实 rw=randwrite # 测试随机写的I/O rw=randrw # 测试随机写和读的I/O bs=16k # 单次io的块文件大小为16k bsrange=512-2048 # 同上,提定数据块的大小范围 size=200g # 本次的测试文件大小为200g,以每次4k的io进行测试 numjobs=30 # 本次的测试线程为30 runtime=1000 # 测试时间为1000秒,如果不写则一直将5g文件分4k每次写完为止 ioengine=psync # io引擎使用pync方式 rwmixwrite=30 # 在混合读写的模式下,写占30% group_reporting # 关于显示结果的,汇总每个进程的信息 lockmem=1g # 只使用1g内存进行测试 zero_buffers # 用0初始化系统buffer nrfiles=8 # 每个进程生成文件的数量
四、有关fio命令输出含义
read # 顺序读 write # 顺序写 rw,readwrite # 顺序混合读写 randwrite # 随机写 randread # 随机读 randrw # 随机混合读写 bw # 带宽,单位KB/s iops # 每秒钟的IO数 runt # 总运行时间 lat (msec) # 延迟(毫秒),msec为毫秒,usec为微秒
五、常用FIO命令
顺序读: fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=nolinux_test 随机写: fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=nolinux_test 顺序写: fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=nolinux_test 混合随机读写: fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test -ioscheduler=noop
OK,下面就是不同级别的RAID的IOPS情况:
这一系列的测试都是基于上面的服务器配置,系统使用CentOS6.6,经过简单优化,磁盘的文件系统为EXT4。
一、单盘情况
[root@kvm-phy27-jz fio-2.1.10]# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 ... nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 fio 2.0.7 Starting 30 threads nolinux_test: Laying out IO file(s) (1 file(s) / 204800MB) Jobs: 22 (f=22): [mmmmmmmmmmm_mmmm__mm_m_m_EmmmE] [14.4% done] [4521K/2129K /s] [276 /130 iops] [eta 09m:59s] nolinux_test: (groupid=0, jobs=30): err= 0: pid=2413 read : io=444528KB, bw=4437.8KB/s, iops=277 , runt=100170msec clat (msec): min=3 , max=1070 , avg=101.45, stdev=89.86 lat (msec): min=3 , max=1070 , avg=101.45, stdev=89.86 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 17], 10.00th=[ 23], 20.00th=[ 34], | 30.00th=[ 45], 40.00th=[ 59], 50.00th=[ 76], 60.00th=[ 96], | 70.00th=[ 121], 80.00th=[ 153], 90.00th=[ 212], 95.00th=[ 273], | 99.00th=[ 429], 99.50th=[ 515], 99.90th=[ 709], 99.95th=[ 791], | 99.99th=[ 922] bw (KB/s) : min= 18, max= 375, per=3.40%, avg=150.67, stdev=51.94 write: io=194288KB, bw=1939.6KB/s, iops=121 , runt=100170msec clat (usec): min=221 , max=246125 , avg=15053.58, stdev=14396.12 lat (usec): min=223 , max=246127 , avg=15056.39, stdev=14396.24 clat percentiles (usec): | 1.00th=[ 247], 5.00th=[ 1480], 10.00th=[ 2384], 20.00th=[ 5728], | 30.00th=[ 9664], 40.00th=[11840], 50.00th=[13760], 60.00th=[15680], | 70.00th=[17792], 80.00th=[20864], 90.00th=[25984], 95.00th=[31104], | 99.00th=[45824], 99.50th=[95744], 99.90th=[205824], 99.95th=[220160], | 99.99th=[238592] bw (KB/s) : min= 10, max= 335, per=4.05%, avg=78.50, stdev=51.26 lat (usec) : 250=0.41%, 500=0.43%, 750=0.33%, 1000=0.10% lat (msec) : 2=0.95%, 4=2.48%, 10=5.93%, 20=18.04%, 50=24.99% lat (msec) : 100=19.59%, 250=22.31%, 500=4.04%, 750=0.35%, 1000=0.05% lat (msec) : 2000=0.01% cpu : usr=0.42%, sys=2.22%, ctx=2520168, majf=5, minf=166643 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=27783/w=12143/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=444528KB, aggrb=4437KB/s, minb=4437KB/s, maxb=4437KB/s, mint=100170msec, maxt=100170msec WRITE: io=194288KB, aggrb=1939KB/s, minb=1939KB/s, maxb=1939KB/s, mint=100170msec, maxt=100170msec Disk stats (read/write): sda: ios=27782/12215, merge=0/25, ticks=2591396/91815, in_queue=2684776, util=100.00%
二、6块盘做RAID 0
[root@kvm-phy27-jz fio-2.1.10]# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 ... nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 fio 2.0.7 Starting 30 threads nolinux_test: Laying out IO file(s) (1 file(s) / 204800MB) Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [3.0% done] [22685K/9444K /s] [1384 /576 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [4.0% done] [22783K/10197K /s] [1390 /622 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [5.0% done] [22489K/9755K /s] [1372 /595 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [6.0% done] [22063K/9444K /s] [1346 /576 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [7.0% done] [22734K/9771K /s] [1387 /596 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [8.0% done] [22047K/8740K /s] [1345 /533 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [9.0% done] [21376K/9706K /s] [1304 /592 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [10.0% done] [23487K/9951K /s] [1433 /607 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [11.0% done] [22325K/8609K /s] [1362 /525 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [12.0% done] [23078K/10164K /s] [1408 /620 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [13.9% done] [23716K/10229K /s] [1447 /624 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [14.9% done] [23978K/9886K /s] [1463 /603 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [15.8% done] [23471K/9836K /s] [1432 /600 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [16.8% done] [23782K/10638K /s] [1451 /649 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [17.8% done] [23160K/10082K /s] [1413 /615 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [18.8% done] [22243K/9345K /s] [1357 /570 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [19.8% done] [21392K/9133K /s] [1305 /557 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [20.8% done] [22505K/9706K /s] [1373 /592 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [21.8% done] [23045K/10458K /s] [1406 /638 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [22.8% done] [22800K/10000K /s] [1391 /610 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [23.8% done] [23127K/10049K /s] [1411 /613 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [24.8% done] [22767K/9656K /s] [1389 /589 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [25.7% done] [23372K/10295K /s] [1426 /628 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [26.7% done] [23176K/9820K /s] [1414 /599 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [27.7% done] [22636K/9345K /s] [1381 /570 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [28.7% done] [22570K/9886K /s] [1377 /603 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [29.7% done] [20885K/9886K /s] [1274 /603 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [30.7% done] [22276K/9673K /s] [1359 /590 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [31.7% done] [21916K/9411K /s] [1337 /574 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [32.7% done] [24420K/10098K /s] [1490 /616 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [33.7% done] [22865K/9444K /s] [1395 /576 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [34.7% done] [21850K/9869K /s] [1333 /602 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [35.6% done] [22783K/9869K /s] [1390 /602 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [36.6% done] [22030K/8953K /s] [1344 /546 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [37.6% done] [21948K/9460K /s] [1339 /577 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [38.6% done] [20181K/8920K /s] [1231 /544 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [39.6% done] [23471K/9722K /s] [1432 /593 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [40.6% done] [23814K/10458K /s] [1453 /638 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [41.6% done] [24764K/10704K /s] [1511 /653 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [42.6% done] [22963K/10016K /s] [1401 /611 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [43.6% done] [21376K/10360K /s] [1304 /632 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [44.6% done] [21163K/8854K /s] [1291 /540 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [45.5% done] [21703K/9460K /s] [1324 /577 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [46.5% done] [23602K/9525K /s] [1440 /581 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [47.5% done] [22129K/10688K /s] [1350 /652 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [48.5% done] [23602K/10327K /s] [1440 /630 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [49.5% done] [22603K/10229K /s] [1379 /624 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [50.5% done] [23045K/10229K /s] [1406 /624 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [51.5% done] [21327K/9706K /s] [1301 /592 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [52.5% done] [22560K/10158K /s] [1377 /620 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [53.5% done] [24977K/10393K /s] [1524 /634 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [54.5% done] [21998K/9116K /s] [1342 /556 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [55.4% done] [22701K/9951K /s] [1385 /607 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [56.4% done] [22849K/10377K /s] [1394 /633 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [57.4% done] [21916K/9509K /s] [1337 /580 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [58.4% done] [22063K/9869K /s] [1346 /602 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [59.4% done] [20950K/9525K /s] [1278 /581 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [60.4% done] [23945K/10409K /s] [1461 /635 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [61.4% done] [24166K/10321K /s] [1475 /630 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [62.4% done] [22446K/9584K /s] [1370 /585 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [63.4% done] [25247K/10190K /s] [1541 /622 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [64.4% done] [22921K/9469K /s] [1399 /578 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [65.3% done] [23494K/9863K /s] [1434 /602 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [66.3% done] [21381K/9551K /s] [1305 /583 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [67.3% done] [21086K/8224K /s] [1287 /502 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [68.3% done] [23658K/9912K /s] [1444 /605 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [69.3% done] [21823K/9011K /s] [1332 /550 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [70.3% done] [23183K/9240K /s] [1415 /564 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [71.3% done] [21266K/8454K /s] [1298 /516 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [72.3% done] [22577K/10027K /s] [1378 /612 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [73.3% done] [22374K/9869K /s] [1365 /602 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [74.3% done] [21588K/8642K /s] [1317 /527 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [75.2% done] [22200K/9945K /s] [1355 /607 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [76.2% done] [21954K/10256K /s] [1340 /626 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [77.2% done] [21676K/9469K /s] [1323 /578 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [78.2% done] [21823K/10190K /s] [1332 /622 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [79.2% done] [21381K/9748K /s] [1305 /595 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [80.2% done] [20873K/9240K /s] [1274 /564 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [81.2% done] [23576K/10698K /s] [1439 /653 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [82.2% done] [23003K/9945K /s] [1404 /607 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [83.2% done] [21397K/10403K /s] [1306 /635 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [84.2% done] [23396K/9469K /s] [1428 /578 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [85.1% done] [21801K/9280K /s] [1330 /566 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [86.1% done] [20737K/9280K /s] [1265 /566 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [87.1% done] [22554K/9444K /s] [1376 /576 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [88.1% done] [24158K/10311K /s] [1474 /629 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [89.1% done] [21736K/9345K /s] [1326 /570 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [90.1% done] [22669K/9362K /s] [1383 /571 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [91.1% done] [20541K/9296K /s] [1253 /567 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [92.1% done] [23838K/10240K /s] [1455 /625 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [93.1% done] [22085K/9388K /s] [1348 /573 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [94.1% done] [22773K/9109K /s] [1390 /556 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [95.0% done] [21561K/9306K /s] [1316 /568 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [96.0% done] [20840K/8519K /s] [1272 /520 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [97.0% done] [22822K/9797K /s] [1393 /598 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [98.0% done] [22069K/9797K /s] [1347 /598 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [99.0% done] [22167K/10534K /s] [1353 /643 iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [23248K/10633K /s] [1419 /649 iops] [eta 00m:00s] nolinux_test: (groupid=0, jobs=30): err= 0: pid=2480 read : io=2146.2MB, bw=21948KB/s, iops=1371 , runt=100132msec clat (usec): min=296 , max=588194 , avg=20452.68, stdev=30073.79 lat (usec): min=296 , max=588195 , avg=20453.01, stdev=30073.79 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 40], 95.00th=[ 63], | 99.00th=[ 165], 99.50th=[ 212], 99.90th=[ 326], 99.95th=[ 371], | 99.99th=[ 453] bw (KB/s) : min= 72, max= 1346, per=3.37%, avg=738.97, stdev=201.50 write: io=950496KB, bw=9492.5KB/s, iops=593 , runt=100132msec clat (usec): min=176 , max=131922 , avg=3208.13, stdev=3135.80 lat (usec): min=177 , max=131927 , avg=3210.90, stdev=3135.89 clat percentiles (usec): | 1.00th=[ 223], 5.00th=[ 243], 10.00th=[ 338], 20.00th=[ 1336], | 30.00th=[ 2352], 40.00th=[ 2864], 50.00th=[ 3248], 60.00th=[ 3632], | 70.00th=[ 4048], 80.00th=[ 4512], 90.00th=[ 5280], 95.00th=[ 5984], | 99.00th=[ 7776], 99.50th=[ 8768], 99.90th=[26496], 99.95th=[86528], | 99.99th=[129536] bw (KB/s) : min= 21, max= 960, per=3.37%, avg=320.15, stdev=144.88 lat (usec) : 250=1.88%, 500=1.96%, 750=0.78%, 1000=0.58% lat (msec) : 2=2.50%, 4=15.39%, 10=35.36%, 20=23.54%, 50=13.16% lat (msec) : 100=3.06%, 250=1.58%, 500=0.20%, 750=0.01% cpu : usr=1.43%, sys=10.07%, ctx=11331053, majf=0, minf=166561 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=137355/w=59406/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=2146.2MB, aggrb=21947KB/s, minb=21947KB/s, maxb=21947KB/s, mint=100132msec, maxt=100132msec WRITE: io=950496KB, aggrb=9492KB/s, minb=9492KB/s, maxb=9492KB/s, mint=100132msec, maxt=100132msec Disk stats (read/write): sda: ios=137228/59403, merge=0/19, ticks=2562622/89096, in_queue=2652072, util=99.99%
三、6块盘做RAID 5
[root@kvm-phy27-jz fio-2.1.10]# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 ... nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 fio 2.0.7 Starting 30 threads nolinux_test: Laying out IO file(s) (1 file(s) / 204800MB) Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [3077K/1456K /s] [187 /88 iops] [eta 00m:00s] nolinux_test: (groupid=0, jobs=30): err= 0: pid=3013 read : io=324576KB, bw=3242.7KB/s, iops=202 , runt=100097msec clat (msec): min=1 , max=345 , avg=101.77, stdev=64.98 lat (msec): min=1 , max=345 , avg=101.77, stdev=64.98 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 10], | 30.00th=[ 79], 40.00th=[ 100], 50.00th=[ 115], 60.00th=[ 128], | 70.00th=[ 141], 80.00th=[ 157], 90.00th=[ 178], 95.00th=[ 196], | 99.00th=[ 235], 99.50th=[ 249], 99.90th=[ 277], 99.95th=[ 285], | 99.99th=[ 314] bw (KB/s) : min= 24, max= 212, per=3.34%, avg=108.17, stdev=28.20 write: io=140704KB, bw=1405.7KB/s, iops=87 , runt=100097msec clat (msec): min=3 , max=307 , avg=106.48, stdev=65.12 lat (msec): min=3 , max=307 , avg=106.48, stdev=65.13 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 84], 40.00th=[ 105], 50.00th=[ 120], 60.00th=[ 133], | 70.00th=[ 147], 80.00th=[ 161], 90.00th=[ 182], 95.00th=[ 202], | 99.00th=[ 239], 99.50th=[ 251], 99.90th=[ 289], 99.95th=[ 289], | 99.99th=[ 310] bw (KB/s) : min= 21, max= 278, per=4.17%, avg=58.56, stdev=35.44 lat (msec) : 2=0.30%, 4=4.24%, 10=13.41%, 20=6.98%, 50=0.69% lat (msec) : 100=13.61%, 250=60.28%, 500=0.48% cpu : usr=0.45%, sys=2.42%, ctx=1953443, majf=0, minf=166561 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=20286/w=8794/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=324576KB, aggrb=3242KB/s, minb=3242KB/s, maxb=3242KB/s, mint=100097msec, maxt=100097msec WRITE: io=140704KB, aggrb=1405KB/s, minb=1405KB/s, maxb=1405KB/s, mint=100097msec, maxt=100097msec Disk stats (read/write): sda: ios=20258/8866, merge=0/32, ticks=153891/101181, in_queue=255029, util=99.54%
四、6块盘做RAID 10
[root@kvm-phy27-jz fio-2.1.10]# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 - runtime=100 -group_reporting -name=nolinux_test nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 ... nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 fio 2.0.7 Starting 30 threads nolinux_test: Laying out IO file(s) (1 file(s) / 204800MB) Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [3.0% done] [15613K/7438K /s] [953 /454 iops] [eta 01m:Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [4.0% done] [16596K/7323K /s] [1013 /447 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [5.0% done] [16482K/6897K /s] [1006 /421 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [5.9% done] [17235K/7389K /s] [1052 /451 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [6.9% done] [16646K/7208K /s] [1016 /440 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [7.9% done] [17547K/6799K /s] [1071 /415 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [8.9% done] [17711K/7995K /s] [1081 /488 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [9.9% done] [17448K/6684K /s] [1065 /408 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [10.9% done] [17252K/6799K /s] [1053 /415 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [11.9% done] [17104K/7323K /s] [1044 /447 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [12.9% done] [17039K/7061K /s] [1040 /431 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [13.9% done] [17219K/7127K /s] [1051 /435 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [14.9% done] [17874K/6930K /s] [1091 /423 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [15.8% done] [17170K/6848K /s] [1048 /418 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [16.8% done] [14614K/5750K /s] [892 /351 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [17.8% done] [16629K/7536K /s] [1015 /460 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [18.8% done] [17776K/7634K /s] [1085 /466 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [19.8% done] [16596K/7176K /s] [1013 /438 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [20.8% done] [16252K/7389K /s] [992 /451 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [21.8% done] [14696K/6619K /s] [897 /404 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [22.8% done] [15745K/6619K /s] [961 /404 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [23.8% done] [13008K/5554K /s] [794 /339 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [24.8% done] [17809K/6864K /s] [1087 /419 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [25.7% done] [17252K/7634K /s] [1053 /466 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [26.7% done] [17006K/6799K /s] [1038 /415 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [27.7% done] [18415K/7340K /s] [1124 /448 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [28.7% done] [17022K/7553K /s] [1039 /461 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [29.7% done] [16596K/7045K /s] [1013 /430 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [30.7% done] [15171K/7192K /s] [926 /439 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [31.7% done] [15777K/7061K /s] [963 /431 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [32.7% done] [16564K/7749K /s] [1011 /473 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [33.7% done] [15843K/7454K /s] [967 /455 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [34.7% done] [16613K/7798K /s] [1014 /476 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [35.6% done] [17563K/7880K /s] [1072 /481 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [36.6% done] [15974K/6946K /s] [975 /424 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [37.6% done] [16269K/7454K /s] [993 /455 iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [38.6% done] [17088K/7176K /s] [1043 /438 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [39.6% done] [16859K/7471K /s] [1029 /456 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [40.6% done] [16662K/6733K /s] [1017 /411 iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [41.6% done] [17039K/7487K /s] [1040 /457 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [42.6% done] [15810K/7716K /s] [965 /471 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [43.6% done] [17858K/7241K /s] [1090 /442 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [44.6% done] [16252K/7225K /s] [992 /441 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [45.5% done] [17416K/7569K /s] [1063 /462 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [46.5% done] [16482K/7127K /s] [1006 /435 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [47.5% done] [17301K/7569K /s] [1056 /462 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [48.5% done] [16941K/7438K /s] [1034 /454 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [49.5% done] [16039K/7536K /s] [979 /460 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [50.5% done] [17399K/7290K /s] [1062 /445 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [51.5% done] [17383K/7487K /s] [1061 /457 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [52.5% done] [17039K/7127K /s] [1040 /435 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [53.5% done] [16662K/7602K /s] [1017 /464 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [54.5% done] [17104K/7847K /s] [1044 /479 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [55.4% done] [16531K/7127K /s] [1009 /435 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [56.4% done] [16072K/7012K /s] [981 /428 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [57.4% done] [15925K/7454K /s] [972 /455 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [58.4% done] [16711K/7667K /s] [1020 /468 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [59.4% done] [17498K/7602K /s] [1068 /464 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [60.4% done] [16859K/7143K /s] [1029 /436 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [61.4% done] [17088K/6602K /s] [1043 /403 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [62.4% done] [17104K/6799K /s] [1044 /415 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [63.4% done] [17301K/7208K /s] [1056 /440 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [64.4% done] [17596K/7061K /s] [1074 /431 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [65.3% done] [17661K/7127K /s] [1078 /435 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [66.3% done] [16924K/6930K /s] [1033 /423 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [67.3% done] [17629K/6815K /s] [1076 /416 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [68.3% done] [17514K/7258K /s] [1069 /443 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [69.3% done] [17432K/7585K /s] [1064 /463 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [70.3% done] [16662K/7585K /s] [1017 /463 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [71.3% done] [16596K/7028K /s] [1013 /429 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [72.3% done] [17334K/7585K /s] [1058 /463 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [73.3% done] [17825K/7258K /s] [1088 /443 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [74.3% done] [17891K/7536K /s] [1092 /460 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [75.2% done] [15925K/7405K /s] [972 /452 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [76.2% done] [17186K/7340K /s] [1049 /448 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [77.2% done] [16269K/7520K /s] [993 /459 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [78.2% done] [16318K/7389K /s] [996 /451 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [79.2% done] [17154K/7110K /s] [1047 /434 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [80.2% done] [16351K/7667K /s] [998 /468 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [81.2% done] [17006K/7897K /s] [1038 /482 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [82.2% done] [18022K/7258K /s] [1100 /443 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [83.2% done] [16236K/7323K /s] [991 /447 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [84.2% done] [16465K/7602K /s] [1005 /464 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [85.1% done] [17039K/7159K /s] [1040 /437 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [86.1% done] [17350K/7258K /s] [1059 /443 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [87.1% done] [17432K/7454K /s] [1064 /455 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [88.1% done] [16777K/7421K /s] [1024 /453 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [89.1% done] [17121K/7323K /s] [1045 /447 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [90.1% done] [16531K/7356K /s] [1009 /449 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [91.1% done] [17137K/7585K /s] [1046 /463 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [92.1% done] [17612K/7143K /s] [1075 /436 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [93.1% done] [16596K/7274K /s] [1013 /444 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [94.1% done] [17694K/7454K /s] [1080 /455 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [95.0% done] [16793K/6979K /s] [1025 /426 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [96.0% done] [18169K/7634K /s] [1109 /466 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [97.0% done] [17399K/6619K /s] [1062 /404 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [98.0% done] [17842K/6897K /s] [1089 /421 iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [99.0% done] [16334K/6897K /s] [997 /421 iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [16875K/7356K /s] [1030 /449 iops] [eta 00m:00s] nolinux_test: (groupid=0, jobs=30): err= 0: pid=28011 read : io=1606.9MB, bw=16445KB/s, iops=1027 , runt=100055msec clat (usec): min=207 , max=623956 , avg=27266.40, stdev=26421.57 lat (usec): min=207 , max=623957 , avg=27266.74, stdev=26421.57 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 19], 60.00th=[ 24], | 70.00th=[ 31], 80.00th=[ 42], 90.00th=[ 60], 95.00th=[ 77], | 99.00th=[ 124], 99.50th=[ 153], 99.90th=[ 229], 99.95th=[ 285], | 99.99th=[ 392] bw (KB/s) : min= 30, max= 1021, per=3.35%, avg=550.46, stdev=118.04 write: io=706368KB, bw=7059.9KB/s, iops=441 , runt=100055msec clat (usec): min=206 , max=380138 , avg=4432.19, stdev=6592.84 lat (usec): min=208 , max=380139 , avg=4434.78, stdev=6592.91 clat percentiles (usec): | 1.00th=[ 235], 5.00th=[ 294], 10.00th=[ 502], 20.00th=[ 1816], | 30.00th=[ 3024], 40.00th=[ 3632], 50.00th=[ 4192], 60.00th=[ 4768], | 70.00th=[ 5344], 80.00th=[ 6112], 90.00th=[ 7392], 95.00th=[ 8768], | 99.00th=[12224], 99.50th=[14400], 99.90th=[69120], 99.95th=[146432], | 99.99th=[366592] bw (KB/s) : min= 22, max= 731, per=3.35%, avg=236.33, stdev=107.84 lat (usec) : 250=0.87%, 500=2.13%, 750=0.87%, 1000=0.52% lat (msec) : 2=1.95%, 4=8.84%, 10=29.78%, 20=22.78%, 50=22.25% lat (msec) : 100=8.56%, 250=1.37%, 500=0.06%, 750=0.01% cpu : usr=1.15%, sys=7.45%, ctx=8559338, majf=0, minf=166568 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=102836/w=44148/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=1606.9MB, aggrb=16444KB/s, minb=16444KB/s, maxb=16444KB/s, mint=100055msec, maxt=100055msec WRITE: io=706368KB, aggrb=7059KB/s, minb=7059KB/s, maxb=7059KB/s, mint=100055msec, maxt=100055msec Disk stats (read/write): sda: ios=102742/44197, merge=0/22, ticks=2546701/103319, in_queue=2650955, util=99.98%
五、6块盘做RAID 50
[root@kvm-phy27-jz fio-2.1.10]# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 ... nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1 fio 2.0.7 Starting 30 threads nolinux_test: Laying out IO file(s) (1 file(s) / 204800MB) Jobs: 8 (f=8): [_m___mm__mmm____m__m__________] [4.4% done] [4435K/2127K /s] [270 /129 iops] [eta 36m:39s] s] nolinux_test: (groupid=0, jobs=30): err= 0: pid=2660 read : io=468304KB, bw=4678.9KB/s, iops=292 , runt=100091msec clat (usec): min=521 , max=212024 , avg=71598.16, stdev=42935.63 lat (usec): min=521 , max=212025 , avg=71598.60, stdev=42935.59 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 6], 20.00th=[ 10], | 30.00th=[ 57], 40.00th=[ 71], 50.00th=[ 81], 60.00th=[ 89], | 70.00th=[ 98], 80.00th=[ 109], 90.00th=[ 122], 95.00th=[ 135], | 99.00th=[ 157], 99.50th=[ 165], 99.90th=[ 180], 99.95th=[ 190], | 99.99th=[ 206] bw (KB/s) : min= 26, max= 283, per=3.33%, avg=155.83, stdev=32.69 write: io=200320KB, bw=2001.4KB/s, iops=125 , runt=100091msec clat (msec): min=2 , max=212 , avg=72.30, stdev=43.07 lat (msec): min=2 , max=212 , avg=72.31, stdev=43.07 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 11], | 30.00th=[ 58], 40.00th=[ 71], 50.00th=[ 81], 60.00th=[ 90], | 70.00th=[ 99], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 135], | 99.00th=[ 157], 99.50th=[ 167], 99.90th=[ 184], 99.95th=[ 192], | 99.99th=[ 208] bw (KB/s) : min= 24, max= 351, per=3.75%, avg=75.13, stdev=43.18 lat (usec) : 750=0.01% lat (msec) : 2=0.23%, 4=4.20%, 10=15.54%, 20=3.47%, 50=3.88% lat (msec) : 100=44.91%, 250=27.75% cpu : usr=0.53%, sys=2.99%, ctx=2748999, majf=5, minf=166631 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=29269/w=12520/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=468304KB, aggrb=4678KB/s, minb=4678KB/s, maxb=4678KB/s, mint=100091msec, maxt=100091msec WRITE: io=200320KB, aggrb=2001KB/s, minb=2001KB/s, maxb=2001KB/s, mint=100091msec, maxt=100091msec Disk stats (read/write): sda: ios=29273/12617, merge=0/23, ticks=212137/99979, in_queue=312063, util=99.54%
各种级别RAID的总结如下表:
RAID级别 | read(iops) | write(iops) |
单盘,无RAID | 277 | 121 |
0 | 1371 | 593 |
5 | 202 | 87 |
10 |
1027 | 441 |
50 | 292 | 125 |
OK,以上就是本次针对相同的服务器进行不同磁盘阵列下的IOPS测试情况。
虽然实验之前没有对系统做过多的深度优化,但是由于本次的所有测试都是在相同环境下进行,因此干扰也不会很大。由于RAID 5的原理,测试出来的效果和磁盘阵列卡的缓存关系更密切一点,不过暂时只有这种阵列卡可以测试了。从不同RAID级别的IOPS中可以看出,如果对空间需求不是很高的情况下,使用RAID 10还是不错的选择。虽然RAID 10更倾向于数据库服务器去使用,不过做虚拟化的母机使用效果还会很不错的。