COSBench是Intel团队基于java开发,对云存储的测试工具,全称是Cloud object Storage Bench,本文主要针对的是支持aws-s3协议的存储服务进行测试,包括seaweedfs、以及华为云存储。
--配置信息基于xml配置,也可以在cosbench的
github地址:https://github.com/intel-cloud/cosbench
本文下载的是最新的0.4.2.c4版本,可以根据自己的需求去下载github上的相应的release版本, 本文所说的cosbench的下载地址: https://github.com/intel-cloud/cosbench/releases/download/v0.4.2.c4/0.4.2.c4.zip
也可以直接在girhub的release目录下下载。下载后解压如下所示:
0.4.2.c4/
0.4.2.c4.zip
total 4132
drwxr-xr-x 15 root root 4096 May 15 11:34 ./
drwxr-xr-x 3 root root 4096 May 13 11:07 ../
-rw-r--r-- 1 root root 348912 Jul 9 2014 3rd-party-licenses.pdf
drwxr-xr-x 34 root root 4096 May 16 12:01 archive/
-rw-r--r-- 1 root root 5266 Apr 27 2016 BUILD.md
-rw-r--r-- 1 root root 10 Jun 15 2016 BUILD.no
-rw-r--r-- 1 root root 7178 Apr 27 2016 CHANGELOG
-rw-r--r-- 1 root root 2639 Jul 9 2014 cli.sh
drwxr-xr-x 4 root root 4096 May 16 14:40 conf/
-rw-r--r-- 1 root root 985318 Jul 9 2014 COSBenchAdaptorDevGuide.pdf
-rw-r--r-- 1 root root 2974 May 13 15:08 cosbench-start.sh
-rw-r--r-- 1 root root 1423 Dec 30 2014 cosbench-stop.sh
-rw-r--r-- 1 root root 2516640 Apr 27 2016 COSBenchUserGuide.pdf
drwxr-xr-x 4 root root 4096 Jun 17 2016 ext/
-rw-r--r-- 1 root root 16 May 15 11:34 ip-port.list
drwxr-xr-x 5 root root 4096 Jun 17 2016 javadoc/
drwxr-xr-x 2 root root 4096 Jun 17 2016 lib-src/
-rw-r--r-- 1 root root 578 Jul 9 2014 LICENSE
drwxr-xr-x 2 root root 4096 Jun 17 2016 licenses/
drwxr-xr-x 3 root root 4096 May 15 11:34 log/
drwxr-xr-x 2 root root 4096 May 15 11:34 main/
-rw-r--r-- 1 root root 478 Jul 9 2014 NOTICE
drwxr-xr-x 4 root root 4096 Jun 17 2016 osgi/
-rw-r--r-- 1 root root 201244 Jul 9 2014 pkg.lst
-rw-r--r-- 1 root root 2182 Apr 27 2016 README.md
drwxr-xr-x 2 root root 4096 Jun 17 2016 scripts/
-rw-r--r-- 1 root root 131 May 16 11:54 start
-rw-r--r-- 1 root root 716 Jul 9 2014 start-all.bat
-rw-r--r-- 1 root root 727 Apr 27 2016 start-all.sh
-rw-r--r-- 1 root root 788 Jul 9 2014 start-controller.bat
-rw-r--r-- 1 root root 1062 Jul 9 2014 start-controller.sh
-rw-r--r-- 1 root root 783 Jul 9 2014 start-driver.bat
-rw-r--r-- 1 root root 1910 Apr 27 2016 start-driver.sh
-rw-r--r-- 1 root root 127 May 16 12:01 stop
-rw-r--r-- 1 root root 724 Apr 27 2016 stop-all.sh
-rw-r--r-- 1 root root 809 Jul 9 2014 stop-controller.sh
-rw-r--r-- 1 root root 1490 Apr 27 2016 stop-driver.sh
-rw-r--r-- 1 root root 278 Apr 27 2016 TODO.md
-rw-r--r-- 1 root root 5 Apr 27 2016 VERSION
-rw-r--r-- 1 root root 677 Jul 9 2014 web.bat
drwxr-xr-x 3 root root 4096 Jun 17 2016 work/
drwxr-xr-x 5 root root 4096 May 13 11:09 workloads/
drwxr-xr-x 3 root root 4096 Jun 17 2016 workspace/
基本概念介绍,可以参看上图中的COSBenchUserGuide.pdf 文件:
下面我们开始测试之旅吧:
进入conf目录下:
-rw-r--r-- 1 root root 2000 Jul 9 2014 ampli-config-sample.xml
-rw-r--r-- 1 root root 1189 Jul 9 2014 cdmi-base-config-sample.xml
-rw-r--r-- 1 root root 1537 Jul 9 2014 cdmi-swift-config-sample.xml
-rw-r--r-- 1 root root 914 May 16 11:54 config.xml
drwxr-xr-x 3 root root 4096 May 15 11:34 .controller/
-rw-r--r-- 1 root root 163 Apr 8 2015 controller.conf
-rw-r--r-- 1 root root 439 Jul 9 2014 controller-tomcat-server.xml
-rw-r--r-- 1 root root 168 Jul 9 2014 cosbench-users.xml
-rw-r--r-- 1 root root 2341 Jul 9 2014 delay-stage-config-sample.xml
drwxr-xr-x 3 root root 4096 May 15 11:34 .driver/
-rw-r--r-- 1 root root 64 May 15 11:34 driver_1.conf
lrwxrwxrwx 1 root root 13 May 15 11:34 driver.conf -> driver_1.conf
-rw-r--r-- 1 root root 55 Apr 27 2016 driver_template.conf
-rw-r--r-- 1 root root 439 May 15 11:34 driver-tomcat-server_1.xml
-rw-r--r-- 1 root root 439 Apr 7 2015 driver-tomcat-server2.xml
-rw-r--r-- 1 root root 439 Apr 27 2016 driver-tomcat-server_template.xml
lrwxrwxrwx 1 root root 26 May 15 11:34 driver-tomcat-server.xml -> driver-tomcat-server_1.xml
-rw-r--r-- 1 root root 1337 Jul 9 2014 filewriter-config-explanation.txt
-rw-r--r-- 1 root root 1339 Jan 27 2016 gcs-config-sample.xml
-rw-r--r-- 1 root root 846 Jan 27 2016 gcs-service-account-sample.json
-rw-r--r-- 1 root root 1555 Jul 9 2014 hashcheck.xml
-rw-r--r-- 1 root root 1078 Jul 9 2014 librados-config-sample.xml
-rw-r--r-- 1 root root 1510 Jul 9 2014 librados-sample-annotated.xml
-rw-r--r-- 1 root root 1648 Jul 9 2014 noop-config.xml
-rw-r--r-- 1 root root 1543 Jul 9 2014 noop-read-config.xml
-rw-r--r-- 1 root root 1571 Jul 9 2014 noop-write-config.xml
-rw-r--r-- 1 root root 4647 Jun 8 2016 openio-config-sample.xml
-rw-r--r-- 1 root root 2174 Jul 9 2014 reusedata.xml
-rw-r--r-- 1 root root 1253 May 14 14:37 s3-config-sample.xml
-rw-r--r-- 1 root root 1855 Jul 9 2014 splitrw.xml
-rw-r--r-- 1 root root 1127 Jul 9 2014 sproxyd-config-sample.xml
-rw-r--r-- 1 root root 1716 Apr 27 2016 swift-config-sample.xml
-rw-r--r-- 1 root root 1630 Jul 9 2014 workload-config.xml
-rw-r--r-- 1 root root 1038 May 13 16:01 workload-s3-config.xml
如上图所示,我们可以看到cosbench已经给我们提供了和你丰富的测试样例,本文主要是对S3协议的测试就看到了s3-config-sample.xml文件,我们就编写一个S3的测试脚本:
//初始化创建一个桶信息 cprefix 代表桶名的前缀,conttainers和前缀拼接成桶的唯一id。多个桶时r(1,n) n代表桶的个数理论上是100以内 s3的要求
//向桶里放大小为sizes的对象信息,oprefix同样代表对象名的前缀,objects和oprefix拼成对象的唯一标识
注: 对于putobject的config编写要注意,s3的bucketname 和objectid都是唯一的,当对同一个对象(bucketname和objectid一样)进行反复put操作代表的是更新操作,测试的内容就发生了根本变化,
故每次测试时,要不就是重新填写桶的名称,要不就修改对象的前缀保证唯一性,不会影响我们的测试结果此处我们修改对象的前缀比如
objects-50k-1-
若是下次测试是100k的就把oprefix=objects-100k-1-
若是要重新测试100k的就把oprefix=objects-100k-2-
目的是保证测试的时候不受objectid+bucketname一样导致数据从新建操作变成了更新操作。
最后执行测试:
1.启动cosbench,进入上述的0.4.2.c4/ 目录下,此处为ubuntu系统故使用了一下命令:
0.4.2.c4# sh start-all.sh
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench driver ...
Starting cosbench-log_0.4.2 [OK]
Starting cosbench-tomcat_0.4.2 [OK]
Starting cosbench-config_0.4.2 [OK]
Starting cosbench-http_0.4.2 [OK]
Starting cosbench-cdmi-util_0.4.2 [OK]
Starting cosbench-core_0.4.2 [OK]
Starting cosbench-core-web_0.4.2 [OK]
Starting cosbench-api_0.4.2 [OK]
Starting cosbench-mock_0.4.2 [OK]
Starting cosbench-ampli_0.4.2 [OK]
Starting cosbench-swift_0.4.2 [OK]
Starting cosbench-keystone_0.4.2 [OK]
Starting cosbench-httpauth_0.4.2 [OK]
Starting cosbench-s3_0.4.2 [OK]
Starting cosbench-librados_0.4.2 [OK]
Starting cosbench-scality_0.4.2 [OK]
Starting cosbench-cdmi-swift_0.4.2 [OK]
Starting cosbench-cdmi-base_0.4.2 [OK]
Starting cosbench-driver_0.4.2 [OK]
Starting cosbench-driver-web_0.4.2 [OK]
Successfully started cosbench driver!
Listening on port 0.0.0.0/0.0.0.0:18089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 18088 !!!
----------------------------------------------
======================================================
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench controller ...
Starting cosbench-log_0.4.2 [OK]
Starting cosbench-tomcat_0.4.2 [OK]
Starting cosbench-config_0.4.2 [OK]
Starting cosbench-core_0.4.2 [OK]
Starting cosbench-core-web_0.4.2 [OK]
Starting cosbench-controller_0.4.2 [OK]
Starting cosbench-controller-web_0.4.2 [OK]
Successfully started cosbench controller!
Listening on port 0.0.0.0/0.0.0.0:19089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 19088 !!!
----------------------------------------------
2.将修改的测试用例加入到测试计划中
sh cli.sh submit conf/config.xml
#出现下列内容代表成功
Accepted with ID: w54
3.查看测试内容
可以在浏览器中输入:http://ip:19088/controller/index.html
ip: 为启动cosbench的ip地址
查看测试信息和测试结果,以及在运行中的测试。
在这个界面可以上传测试计划,就是将编写好的xml文件上传到测试服务器上并执行,如下图所示:
测试结果都会保存在服务端的archive/ 目录下,供你随时翻阅。
4.停止服务
0.4.2.c4# sh stop-all.sh
Stopping cosbench controller ...
Successfully stopped cosbench controller.
======================================================
127.0.0.1:18088
18088
Stopping cosbench driver ...
Successfully stopped cosbench driver.