MooseFS VS NFS

MooseFS(MFS)是一个分布式集群文件系统，介绍看 这里, 还有官方的 中文文档, 这里主要是进行 NFS和MFS的benchmark, 测试方法为使用 ab进行单个文件访问测试，后端server使用 ngnix, write性能测试使用 dd.

环境:

server: 192.168.1.8, Ubuntu, NFS, MFS, 8G内存, 双核Intel(R) Core(TM) i3 CPU 2.93GHz, MFS挂载目录为/mnt/mfsdir1, 是独立的磁盘块(dd建立)

client: 192.168.1.106, Mac OS X, Nginx, Nginx访问目录中分别挂载192.168.1.8的/mnt/mfsdir1(MFS分区)和NFS共享目录/mnt/nfs(属于/分区)

NFS和MFS的配置均为默认的, 这里要注意的是因为测试平台是Mac OS, 所以在NFS共享文件配置中需要加入参数insecure, 否则Mac OS会报opeartion not permite

小文件read测试

测试的文件为389K的图片文件

NFS测试结果:

#ab -c 100 -n 1000 http://192.168.1.106/filesystem_benckmark/nfs/big.jpg

Server Software:        nginx/0.7.65
Server Hostname:        192.168.1.106
Server Port:            80

Document Path:          /filesystem_benckmark/nfs/big.jpg
Document Length:        398102 bytes

Concurrency Level:      100
Time taken for tests:   1.510 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0

Total transferred:      398317000 bytes
HTML transferred:       398102000 bytes
Requests per second:    662.46 [#/sec] (mean)
Time per request:       150.952 [ms] (mean)
Time per request:       1.510 [ms] (mean, across all concurrent requests)
Transfer rate:          257684.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.1      0      10
Processing:    12  147  43.8    148     326
Waiting:        1   62  29.6     59     159
Total:         12  147  44.1    148     329

Percentage of the requests served within a certain time (ms)
  50%    148
  66%    164
  75%    172
  80%    177
  90%    200
  95%    228
  98%    249
  99%    264
 100%    329 (longest request)

MFS测试结果:

#ab -c 100 -n 1000 http://192.168.1.106/filesystem_benckmark/mfs/big.jpg

Server Software:        nginx/0.7.65
Server Hostname:        192.168.1.106
Server Port:            80

Document Path:          /filesystem_benckmark/mfs/big.jpg
Document Length:        398102 bytes

Concurrency Level:      100
Time taken for tests:   0.721 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0

Total transferred:      401711927 bytes
HTML transferred:       401494992 bytes
Requests per second:    1386.30 [#/sec] (mean)
Time per request:       72.135 [ms] (mean)
Time per request:       0.721 [ms] (mean, across all concurrent requests)
Transfer rate:          543840.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   1.4      2      13
Processing:    31   68  11.8     68      96
Waiting:        1   11   8.5      9      33
Total:         33   70  11.8     70      97

Percentage of the requests served within a certain time (ms)
  50%     70
  66%     73
  75%     75
  80%     76
  90%     87
  95%     91
  98%     95
  99%     96
 100%     97 (longest request)

 

从以上测试可以看出MooseFS在读性能上还是完胜NFS的, 为NFS的200%

大文件read测试

测试的文件为5.6M的二进制文件

NFS测试结果:

#ab -c 10 -n 100 http://192.168.1.106/filesystem_benckmark/nfs/production-20100326.log.tar.gz

Concurrency Level:      10
Time taken for tests:   2.156 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      591628100 bytes
HTML transferred:       591605100 bytes
Requests per second:    46.38 [#/sec] (mean)
Time per request:       215.603 [ms] (mean)
Time per request:       21.560 [ms] (mean, across all concurrent requests)
Transfer rate:          267974.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0       5
Processing:    17  211 110.5    215     480
Waiting:        1   12  10.4      9      64
Total:         17  211 110.6    215     481

Percentage of the requests served within a certain time (ms)
  50%    215
  66%    261
  75%    287
  80%    305
  90%    340
  95%    387
  98%    474
  99%    481
 100%    481 (longest request)

MFS测试结果:

#ab -c 10 -n 100 http://192.168.1.106/filesystem_benckmark/mfs/production-20100326.log.tar.gz

Concurrency Level:      10
Time taken for tests:   0.671 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      591628100 bytes
HTML transferred:       591605100 bytes
Requests per second:    149.06 [#/sec] (mean)
Time per request:       67.089 [ms] (mean)
Time per request:       6.709 [ms] (mean, across all concurrent requests)
Transfer rate:          861190.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    50   65   9.8     63      93
Waiting:        0    2   1.0      1       4
Total:         50   65   9.8     63      93
WARNING: The median and mean for the waiting time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     63
  66%     68
  75%     71
  80%     73
  90%     76
  95%     88
  98%     91
  99%     93
 100%     93 (longest request)

在大文件的读取上MFS更是表现明显, 性能为NFS的300%

write测试

写测试这里使用了dd方法, 写入数据大小为100M

NFS write

#time dd if=/dev/zero of=sometestfile bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 8.829279 secs (11597776 bytes/sec)
        8.91 real         0.10 user         3.39 sys

MFS write

#time dd if=/dev/zero of=sometestfile bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 6.531960 secs (15676765 bytes/sec)
        8.81 real         0.12 user         2.56 sys

可以看出MFS的写性能达到了百兆网卡的全速(14Mb/s), NFS为11Mb/s, 在时间上也可以看出MFS的优势, 再加上MooseFS的分布式存储和容错功能, 的确非常不错

对于单目录下存在大量文件时单个文件查找和操作仍需要测试，待续