cosbench通过aws-S3协议测试对象存储服务

COSBench是Intel团队基于java开发,对云存储的测试工具,全称是Cloud object Storage Bench,本文主要针对的是支持aws-s3协议的存储服务进行测试,包括seaweedfs、以及华为云存储。
--配置信息基于xml配置,也可以在cosbench的
github地址:https://github.com/intel-cloud/cosbench
本文下载的是最新的0.4.2.c4版本,可以根据自己的需求去下载github上的相应的release版本, 本文所说的cosbench的下载地址: https://github.com/intel-cloud/cosbench/releases/download/v0.4.2.c4/0.4.2.c4.zip
也可以直接在girhub的release目录下下载。下载后解压如下所示:

0.4.2.c4/
0.4.2.c4.zip
total 4132
drwxr-xr-x 15 root root    4096 May 15 11:34 ./
drwxr-xr-x  3 root root    4096 May 13 11:07 ../
-rw-r--r--  1 root root  348912 Jul  9  2014 3rd-party-licenses.pdf
drwxr-xr-x 34 root root    4096 May 16 12:01 archive/
-rw-r--r--  1 root root    5266 Apr 27  2016 BUILD.md
-rw-r--r--  1 root root      10 Jun 15  2016 BUILD.no
-rw-r--r--  1 root root    7178 Apr 27  2016 CHANGELOG
-rw-r--r--  1 root root    2639 Jul  9  2014 cli.sh
drwxr-xr-x  4 root root    4096 May 16 14:40 conf/
-rw-r--r--  1 root root  985318 Jul  9  2014 COSBenchAdaptorDevGuide.pdf
-rw-r--r--  1 root root    2974 May 13 15:08 cosbench-start.sh
-rw-r--r--  1 root root    1423 Dec 30  2014 cosbench-stop.sh
-rw-r--r--  1 root root 2516640 Apr 27  2016 COSBenchUserGuide.pdf
drwxr-xr-x  4 root root    4096 Jun 17  2016 ext/
-rw-r--r--  1 root root      16 May 15 11:34 ip-port.list
drwxr-xr-x  5 root root    4096 Jun 17  2016 javadoc/
drwxr-xr-x  2 root root    4096 Jun 17  2016 lib-src/
-rw-r--r--  1 root root     578 Jul  9  2014 LICENSE
drwxr-xr-x  2 root root    4096 Jun 17  2016 licenses/
drwxr-xr-x  3 root root    4096 May 15 11:34 log/
drwxr-xr-x  2 root root    4096 May 15 11:34 main/
-rw-r--r--  1 root root     478 Jul  9  2014 NOTICE
drwxr-xr-x  4 root root    4096 Jun 17  2016 osgi/
-rw-r--r--  1 root root  201244 Jul  9  2014 pkg.lst
-rw-r--r--  1 root root    2182 Apr 27  2016 README.md
drwxr-xr-x  2 root root    4096 Jun 17  2016 scripts/
-rw-r--r--  1 root root     131 May 16 11:54 start
-rw-r--r--  1 root root     716 Jul  9  2014 start-all.bat
-rw-r--r--  1 root root     727 Apr 27  2016 start-all.sh
-rw-r--r--  1 root root     788 Jul  9  2014 start-controller.bat
-rw-r--r--  1 root root    1062 Jul  9  2014 start-controller.sh
-rw-r--r--  1 root root     783 Jul  9  2014 start-driver.bat
-rw-r--r--  1 root root    1910 Apr 27  2016 start-driver.sh
-rw-r--r--  1 root root     127 May 16 12:01 stop
-rw-r--r--  1 root root     724 Apr 27  2016 stop-all.sh
-rw-r--r--  1 root root     809 Jul  9  2014 stop-controller.sh
-rw-r--r--  1 root root    1490 Apr 27  2016 stop-driver.sh
-rw-r--r--  1 root root     278 Apr 27  2016 TODO.md
-rw-r--r--  1 root root       5 Apr 27  2016 VERSION
-rw-r--r--  1 root root     677 Jul  9  2014 web.bat
drwxr-xr-x  3 root root    4096 Jun 17  2016 work/
drwxr-xr-x  5 root root    4096 May 13 11:09 workloads/
drwxr-xr-x  3 root root    4096 Jun 17  2016 workspace/

基本概念介绍,可以参看上图中的COSBenchUserGuide.pdf 文件:
下面我们开始测试之旅吧:
进入conf目录下:

-rw-r--r--  1 root root 2000 Jul  9  2014 ampli-config-sample.xml
-rw-r--r--  1 root root 1189 Jul  9  2014 cdmi-base-config-sample.xml
-rw-r--r--  1 root root 1537 Jul  9  2014 cdmi-swift-config-sample.xml
-rw-r--r--  1 root root  914 May 16 11:54 config.xml
drwxr-xr-x  3 root root 4096 May 15 11:34 .controller/
-rw-r--r--  1 root root  163 Apr  8  2015 controller.conf
-rw-r--r--  1 root root  439 Jul  9  2014 controller-tomcat-server.xml
-rw-r--r--  1 root root  168 Jul  9  2014 cosbench-users.xml
-rw-r--r--  1 root root 2341 Jul  9  2014 delay-stage-config-sample.xml
drwxr-xr-x  3 root root 4096 May 15 11:34 .driver/
-rw-r--r--  1 root root   64 May 15 11:34 driver_1.conf
lrwxrwxrwx  1 root root   13 May 15 11:34 driver.conf -> driver_1.conf
-rw-r--r--  1 root root   55 Apr 27  2016 driver_template.conf
-rw-r--r--  1 root root  439 May 15 11:34 driver-tomcat-server_1.xml
-rw-r--r--  1 root root  439 Apr  7  2015 driver-tomcat-server2.xml
-rw-r--r--  1 root root  439 Apr 27  2016 driver-tomcat-server_template.xml
lrwxrwxrwx  1 root root   26 May 15 11:34 driver-tomcat-server.xml -> driver-tomcat-server_1.xml
-rw-r--r--  1 root root 1337 Jul  9  2014 filewriter-config-explanation.txt
-rw-r--r--  1 root root 1339 Jan 27  2016 gcs-config-sample.xml
-rw-r--r--  1 root root  846 Jan 27  2016 gcs-service-account-sample.json
-rw-r--r--  1 root root 1555 Jul  9  2014 hashcheck.xml
-rw-r--r--  1 root root 1078 Jul  9  2014 librados-config-sample.xml
-rw-r--r--  1 root root 1510 Jul  9  2014 librados-sample-annotated.xml
-rw-r--r--  1 root root 1648 Jul  9  2014 noop-config.xml
-rw-r--r--  1 root root 1543 Jul  9  2014 noop-read-config.xml
-rw-r--r--  1 root root 1571 Jul  9  2014 noop-write-config.xml
-rw-r--r--  1 root root 4647 Jun  8  2016 openio-config-sample.xml
-rw-r--r--  1 root root 2174 Jul  9  2014 reusedata.xml
-rw-r--r--  1 root root 1253 May 14 14:37 s3-config-sample.xml
-rw-r--r--  1 root root 1855 Jul  9  2014 splitrw.xml
-rw-r--r--  1 root root 1127 Jul  9  2014 sproxyd-config-sample.xml
-rw-r--r--  1 root root 1716 Apr 27  2016 swift-config-sample.xml
-rw-r--r--  1 root root 1630 Jul  9  2014 workload-config.xml
-rw-r--r--  1 root root 1038 May 13 16:01 workload-s3-config.xml

如上图所示,我们可以看到cosbench已经给我们提供了和你丰富的测试样例,本文主要是对S3协议的测试就看到了s3-config-sample.xml文件,我们就编写一个S3的测试脚本:




  

  
    //初始化创建一个桶信息 cprefix 代表桶名的前缀,conttainers和前缀拼接成桶的唯一id。多个桶时r(1,n) n代表桶的个数理论上是100以内 s3的要求
    
      
    
//向桶里放大小为sizes的对象信息,oprefix同样代表对象名的前缀,objects和oprefix拼成对象的唯一标识
    
      
        
      
    

  



: 对于putobject的config编写要注意,s3的bucketname 和objectid都是唯一的,当对同一个对象(bucketname和objectid一样)进行反复put操作代表的是更新操作,测试的内容就发生了根本变化,
故每次测试时,要不就是重新填写桶的名称,要不就修改对象的前缀保证唯一性,不会影响我们的测试结果此处我们修改对象的前缀比如
objects-50k-1-
若是下次测试是100k的就把oprefix=objects-100k-1-
若是要重新测试100k的就把oprefix=objects-100k-2-
目的是保证测试的时候不受objectid+bucketname一样导致数据从新建操作变成了更新操作。
最后执行测试:
1.启动cosbench,进入上述的0.4.2.c4/ 目录下,此处为ubuntu系统故使用了一下命令:

0.4.2.c4# sh start-all.sh
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench driver ...
Starting    cosbench-log_0.4.2    [OK]
Starting    cosbench-tomcat_0.4.2    [OK]
Starting    cosbench-config_0.4.2    [OK]
Starting    cosbench-http_0.4.2    [OK]
Starting    cosbench-cdmi-util_0.4.2    [OK]
Starting    cosbench-core_0.4.2    [OK]
Starting    cosbench-core-web_0.4.2    [OK]
Starting    cosbench-api_0.4.2    [OK]
Starting    cosbench-mock_0.4.2    [OK]
Starting    cosbench-ampli_0.4.2    [OK]
Starting    cosbench-swift_0.4.2    [OK]
Starting    cosbench-keystone_0.4.2    [OK]
Starting    cosbench-httpauth_0.4.2    [OK]
Starting    cosbench-s3_0.4.2    [OK]
Starting    cosbench-librados_0.4.2    [OK]
Starting    cosbench-scality_0.4.2    [OK]
Starting    cosbench-cdmi-swift_0.4.2    [OK]
Starting    cosbench-cdmi-base_0.4.2    [OK]
Starting    cosbench-driver_0.4.2    [OK]
Starting    cosbench-driver-web_0.4.2    [OK]
Successfully started cosbench driver!
Listening on port 0.0.0.0/0.0.0.0:18089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 18088 !!!
----------------------------------------------

======================================================

Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench controller ...
Starting    cosbench-log_0.4.2    [OK]
Starting    cosbench-tomcat_0.4.2    [OK]
Starting    cosbench-config_0.4.2    [OK]
Starting    cosbench-core_0.4.2    [OK]
Starting    cosbench-core-web_0.4.2    [OK]
Starting    cosbench-controller_0.4.2    [OK]
Starting    cosbench-controller-web_0.4.2    [OK]
Successfully started cosbench controller!
Listening on port 0.0.0.0/0.0.0.0:19089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 19088 !!!
----------------------------------------------

2.将修改的测试用例加入到测试计划中

sh cli.sh submit conf/config.xml
#出现下列内容代表成功
Accepted with ID: w54

3.查看测试内容
可以在浏览器中输入:http://ip:19088/controller/index.html
ip: 为启动cosbench的ip地址
查看测试信息和测试结果,以及在运行中的测试。
在这个界面可以上传测试计划,就是将编写好的xml文件上传到测试服务器上并执行,如下图所示:

cosbench通过aws-S3协议测试对象存储服务_第1张图片
红色框内

测试结果都会保存在服务端的archive/ 目录下,供你随时翻阅。

4.停止服务

0.4.2.c4# sh stop-all.sh
Stopping cosbench controller ...
Successfully stopped cosbench controller.

======================================================

127.0.0.1:18088
18088
Stopping cosbench driver ...
Successfully stopped cosbench driver.

你可能感兴趣的:(cosbench通过aws-S3协议测试对象存储服务)