ceph存储性能测试
环境信息
主机：3台
osd：每台6个
ceph 版本：L
测试工具：fio
存储网卡：万兆
单块磁盘性能
随机读写：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=randread -bs=4k -size=2G -numjobs=64 -runt
ime=100 -group_reporting -name=test-rand-read
read: IOPS=1517, BW=6070KiB/s (6215kB/s)(524MiB/88369msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=2G -numjobs=64 -run
time=30 -group_reporting -name=test-rand-read
write: IOPS=642, BW=2571KiB/s (2633kB/s)(76.4MiB/30448msec)
顺序读：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=5G -numjobs=1 -run
time=30 -group_reporting -name=test-rand-read
read: IOPS=54, BW=220MiB/s (230MB/s)(5120MiB/23292msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=read -bs=4M -size=5G -numjobs=64 -run
time=30 -group_reporting -name=test-rand-read
read: IOPS=117, BW=470MiB/s (493MB/s)(13.0GiB/30508msec)
顺序写
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=16M -size=5G -numjobs=1 -runtime
=100 -group_reporting -name=test-rand-read
write: IOPS=57, BW=229MiB/s (240MB/s)(5120MiB/22360msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=4M -size=5G -numjobs=64 -runtime
=30 -group_reporting -name=test-rand-read
write: IOPS=831, BW=3326MiB/s (3488MB/s)(97.8GiB/30120msec)
两块raid1性能
随机读写：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=randread -bs=4k -size=2G -numjobs=64 -runtim
e=100 -group_reporting -name=test-rand-read
read: IOPS=2875, BW=11.2MiB/s (11.8MB/s)(353MiB/31385msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=2G -numjobs=64 -runti
me=100 -group_reporting -name=test-rand-read
write: IOPS=639, BW=2559KiB/s (2620kB/s)(76.0MiB/30431msec)
顺序读：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=read -bs=4M -size=5G -numjobs=1 -run
time=30 -group_reporting -name=test-rand-read
read: IOPS=83, BW=334MiB/s (351MB/s)(5120MiB/15310msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=read -bs=4M -size=5G -numjobs=64 -ru
ntime=100 -group_reporting -name=test-rand-read
read: IOPS=239, BW=959MiB/s (1005MB/s)(28.3GiB/30214msec)
顺序写：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=4M -size=5G -numjobs=1 -ru
ntime=30 -group_reporting -name=test-rand-read
write: IOPS=53, BW=213MiB/s (223MB/s)(5120MiB/24077msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=4M -size=5G -numjobs=64 -r
untime=30 -group_reporting -name=test-rand-read
write: IOPS=829, BW=3319MiB/s (3480MB/s)(97.6GiB/30118msec)
结果对比
| 单块磁盘 | 两块raid1 | 结论 | |
|---|---|---|---|
| 随机读 | iops=1517 | IOPS=2875 | raid1对于单块磁盘随机读有近一倍提升 |
| 随机写 | IOPS=624 | IOPS=642 | 两者无太大差别 |
| 顺序读 | BW=230MB/s numjobs=1 | bw=351MB/s numjobs=1 | raid1对于单块磁盘顺序读有50%左右的提升 |
| 顺序读 | BW=493MB/s numjobs=64 | bw=1005MB/s numjobs=64 | numjobs=64时raid1对于单块硬盘顺序读有近1倍提升 |
| 顺序写 | bw=240MB/s numjobs=1 | bw=223MB/s numjobs=1 | raid1对于单块硬盘顺序写无性能提升 |
| 顺序写 | bw=3488MB/s numjobs=64 | bw=3480MB/s numjobs=64 | numjobs=64时raid1对于单块硬盘顺序写无性能提升 |
ceph性能测试
随机读写：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=randread -bs=4k -size=2G -numjobs=64 -runtim
e=100 -group_reporting -name=test-rand-read
read: IOPS=24.1k, BW=94.3MiB/s (98.9MB/s)(9433MiB/100009msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=2G -numjobs=64 -runti
me=100 -group_reporting -name=test-rand-read
write: IOPS=3924, BW=15.3MiB/s (16.1MB/s)(1536MiB/100194msec)
顺序读：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=read -bs=4M -size=20G -numjobs=1 -runtime=
30 -group_reporting -name=test-rand-read
read: IOPS=250, BW=1002MiB/s (1051MB/s)(20.0GiB/20430msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=read -bs=4M -size=20G -numjobs=64 -ru
ntime=100 -group_reporting -name=test-rand-read
read: IOPS=174, BW=696MiB/s (730MB/s)(41.9GiB/61550msec)
顺序写：
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=4M -size=5G -numjobs=1 -ru
ntime=30 -group_reporting -name=test-rand-read
write: IOPS=210, BW=841MiB/s (882MB/s)(20.0GiB/24348msec)
fio --filename=./test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=4M -size=5G -numjobs=64 -r
untime=30 -group_reporting -name=test-rand-read
write: IOPS=210, BW=844MiB/s (885MB/s)(25.0GiB/31521msec)
结果对比
| 单块磁盘 | ceph集群 | 结论 | |
|---|---|---|---|
| 随机读 | iops=1517 | IOPS=24.1k | 提升约16倍，比较理想，性能线性增加 |
| 随机写 | IOPS=624 | IOPS=3924 | 提升约6倍，数据写三份，性能没有读高 |
| 顺序读 | BW=230MB/s numjobs=1 | bw=1051MB/s numjobs=1 | 已被网络带宽限制 |
| 顺序读 | BW=493MB/s numjobs=64 | bw=730MB/s numjobs=64 | 已被网络带宽限制 |
| 顺序写 | bw=240MB/s numjobs=1 | bw=882MB/s numjobs=1 | 已被网络带宽限制 |
| 顺序写 | bw=3488MB/s numjobs=64 | bw=885MB/s numjobs=64 | 已被网络带宽限制 |