（9）ceph pool 性能测试

Ceph包含在rados bench RADOS存储集群上执行性能基准测试的命令。该命令将执行写测试和两种类型的读测试。--no-cleanup在测试读写性能时，该选项很重要。默认情况下，该rados bench命令将删除已写入存储池的对象。留在这些对象后面允许两个读取测试来测量顺序和随机读取性能。

（1）准备工作，清除所有的文件系统缓存
[root@node141 mnt]# echo 3 > /proc/sys/vm/drop_caches
[root@node141 mnt]# sync

（2）创建一个新的存储池
[root@node141 mnt]# ceph osd pool create rbd 128 128

（3）测试一下rbd的写性能，添加 --no-cleanup参数

[root@node141 mnt]# rados bench -p rbd 10 write  --no-cleanup 
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_node141_28363
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        81        65   259.999       260    0.107901    0.200157
    2      16       161       145   289.976       320    0.190659    0.215092
    3      16       252       236   314.632       364   0.0807696    0.193549
    4      16       351       335   334.958       396    0.191582    0.185652
    5      16       445       429   343.157       376    0.158618    0.178393
    6      16       548       532   354.621       412    0.162274    0.178178
    7      16       652       636   363.378       416    0.190742    0.173784
    8      16       753       737   368.448       404     0.15392    0.170866
    9      16       863       847   376.391       440    0.103038    0.169129
   10      16       964       948   379.144       404    0.138741    0.167306
Total time run:         10.1023
Total writes made:      965
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     382.09
Stddev Bandwidth:       53.3267
Max bandwidth (MB/sec): 440
Min bandwidth (MB/sec): 260
Average IOPS:           95
Stddev IOPS:            13.3317
Max IOPS:               110
Min IOPS:               65
Average Latency(s):     0.167422
Stddev Latency(s):      0.0765119
Max latency(s):         0.668499
Min latency(s):         0.0569634

（4）测试一下，10秒的顺序读取：

[root@node141 mnt]# rados bench -p rbd 10 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       257       241   963.679       964   0.0607106   0.0623843
    2      16       549       533   1065.73      1168   0.0450988   0.0583824
    3      16       835       819   1091.76      1144    0.044447    0.057111
Total time run:       3.47417
Total reads made:     965
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1111.06
Average IOPS:         277
Stddev IOPS:          27.8747
Max IOPS:             292
Min IOPS:             241
Average Latency(s):   0.056581
Max latency(s):       0.388605
Min latency(s):       0.015351

（5）测试10秒随机读写性能

[root@node141 mnt]# rados bench -p rbd 10 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       408       392   1566.56      1568   0.0715088    0.038812
    2      16       794       778   1554.08      1544   0.0219942   0.0396005
    3      16      1211      1195   1591.93      1668    0.125987   0.0389289
    4      15      1604      1589    1587.9      1576   0.0497586   0.0392412
    5      16      2013      1997   1596.67      1632  0.00512184   0.0388821
    6      16      2415      2399   1598.52      1608    0.146983   0.0389842
    7      16      2806      2790   1593.56      1564    0.110503   0.0391283
    8      16      3186      3170   1584.34      1520   0.0226104   0.0393014
    9      15      3598      3583   1591.83      1652   0.0117805   0.0392624
   10      16      3965      3949   1579.03      1464   0.0752829   0.0395739
Total time run:       10.0536
Total reads made:     3965
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1577.54
Average IOPS:         394
Stddev IOPS:          15.6308
Max IOPS:             417
Min IOPS:             366
Average Latency(s):   0.0397174
Max latency(s):       0.353027
Min latency(s):       0.00414538

（6）要增加并发读取和写入的数量，请使用-t 默认为16个现场的选项，-b参数可以调整正在写入的对象的大小，默认对象大小为4MB。安全的最大对象为16MB.
添加 --run-name 选项以控制在基准测试期间写入的对象的名称。多个rados bench命令可以同时通过改变跑出--run-name标签每个运行命令的实例。这可以防止在多个客户端尝试访问同一对象并允许不同客户端访问不同对象时可能发生的潜在I / O错误。--run-name在尝试模拟真实世界的工作负载时，该选项也很有用。例如：

[root@node141 mnt]# rados bench -p rbd 10 write  -t 4 --run-name zabbix71
hints = 1
Maintaining 4 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_node141_28450
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1       4        40        36   143.992       144    0.065704    0.103059
    2       4        86        82   163.981       184   0.0763531   0.0955013
    3       4       126       122   162.641       160    0.106746   0.0953542
    4       4       173       169   168.972       188   0.0872597   0.0934176
    5       4       218       214   171.171       180    0.170851   0.0927783
    6       4       255       251   167.305       148   0.0943133    0.093864
    7       4       294       290   165.686       156   0.0976048   0.0959294
    8       4       340       336   167.971       184   0.0882485   0.0946923
    9       4       386       382   169.748       184   0.0703278   0.0933611
   10       4       430       426    170.37       176   0.0623777   0.0933391
Total time run:         10.1109
Total writes made:      431
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     170.51
Stddev Bandwidth:       16.6747
Max bandwidth (MB/sec): 188
Min bandwidth (MB/sec): 144
Average IOPS:           42
Stddev IOPS:            4.16867
Max IOPS:               47
Min IOPS:               36
Average Latency(s):     0.0935851
Stddev Latency(s):      0.0433054
Max latency(s):         0.579434
Min latency(s):         0.0533891
Cleaning up (deleting benchmark objects)
Removed 431 objects
Clean up completed and total clean up time :1.62203

(7) 删除rados bench 命令创建的数据
[root@node141 mnt]# rados -p rbd cleanup
Removed 965 objects