YCSB测试hbase-0.96.0

1.YCSB简介
YCSB的全称是Yahoo! Cloud Serving Benchmark，这是一个用于测试cloud serving/NoSQL/Key-Value Store的benchmark。 论文发表在SoCC’10， 代码开源。有很多出色的工作（包括SILT@SOSP’11，HyperDex@SIGCOMM’12）使用YCSB进行测试。为了了解hbase-0.96.0的性能，因此学习一下如何使用YCSB。本文主要论述利用ycsb测试hbase-0.96.0。
2、YCSB测试hbase-0.96.0
1）下载ycsb源代码

git clone  https://github.com/brianfrankcooper/YCSB.git

删除原来hbase/pom.xml中的hbase和hadoop依赖项，改为如下：

<dependency>
          <groupId>org.apache.hbase</groupId>
          <artifactId>hbase-it</artifactId>
          <version>0.96.0-hadoop2</version>
</dependency>
<dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.2.0</version>
</dependency>

切换到YCSB目录，并使用mvn编译打包源代码（需要安装maven），如下：

cd YCSB
mvn -DskipTests package

编译结束后，将集群的配置信息写入hbase/src/main/conf/hbase-site.xml文件。例如：

<configuration>
  <property>
      <name>hbase.zookeeper.quorum</name>
      <value>l-hbase1.data.cn8,l-hbase2.data.cn8,l-hbase3.data.cn8</value>
  </property>
</configuration>

务必将hbase.zookeeper.quorum属性配置好，以便 YCSB使用它找到集群的入口。
在运行工作负载之前，必须创建YCSB写入的HBase表。可以在shell中创建表（ycsb默认测试的表名就是usertable）：

hbase(main):002:0> create 'usertable ','cf' 
0 row(s) in 1.1940 seconds

创建好之后，准备开始测试集群。首先是加载数据，执行如下命令：

bin/ycsb load hbase -P workloads/workloada -p columnfamily=cf 

执行结果如下：
YCSB Client 0.1
Command line: -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=cf -load
2014-01-14 14:43:28,214 WARN [Thread-2] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-01-14 14:43:28,396 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:host.name=ruifengshan-ubuntu
2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.version=1.6.0_37
2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.vendor=Sun Microsystems Inc.
2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.home=/usr/lib/jvm/java/jdk1.6.0_37/jre
2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.class.path=/home/ruifengshan/work/YCSB-master/jdbc/src/main/conf:/home/ruifengshan/work/YCSB-master/gemfire/src/main/conf:/home/ruifengshan/work/YCSB-master/core/target/core-0.1.4.jar:/home/ruifengshan/work/YCSB-master/hbase/target/hbase-binding-0.1.4.jar:/home/ruifengshan/work/YCSB-master/hbase/target/archive-tmp/hbase-binding-0.1.4.jar:/home/ruifengshan/work/YCSB-master/hbase/src/main/conf:/home/ruifengshan/work/YCSB-master/nosqldb/src/main/conf:/home/ruifengshan/work/YCSB-master/dynamodb/conf:/home/ruifengshan/work/YCSB-master/voldemort/src/main/conf:/home/ruifengshan/work/YCSB-master/infinispan/src/main/conf
2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.library.path=/usr/lib/jvm/java/jdk1.6.0_37/jre/lib/amd64/server:/usr/lib/jvm/java/jdk1.6.0_37/jre/lib/amd64:/usr/lib/jvm/java/jdk1.6.0_37/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=/tmp
2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.compiler=<NA>
2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.name=Linux
2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.arch=amd64
2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.version=3.5.0-45-generic
2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=ruifengshan
2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=/home/ruifengshan
2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=/home/ruifengshan/work/YCSB-master
2014-01-14 14:43:28,402 INFO [Thread-2] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=l-hbase2.data.cn8:2181,l-hbase1.data.cn8:2181,l-hbase3.data.cn8:2181 sessionTimeout=90000 watcher=hconnection-0x3bc0f2e5
2014-01-14 14:43:28,429 INFO [Thread-2] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x3bc0f2e5 connecting to ZooKeeper ensemble=l-hbase2.data.cn8:2181,l-hbase1.data.cn8:2181,l-hbase3.data.cn8:2181
2014-01-14 14:43:28,434 INFO [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(966)) - Opening socket connection to server l-hbase3.data.cn8/192.168.44.7:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
2014-01-14 14:43:28,440 INFO [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(849)) - Socket connection established to l-hbase3.data.cn8/192.168.44.7:2181, initiating session
2014-01-14 14:43:28,447 WARN [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxnSocket (ClientCnxnSocket.java:readConnectResult(139)) - Connected to an old server; r-o mode will be unavailable
2014-01-14 14:43:28,448 INFO [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1207)) - Session establishment complete on server l-hbase3.data.cn8/192.168.44.7:2181, sessionid = 0x34351eb5ea30c49, negotiated timeout = 40000
[OVERALL], RunTime(ms), 1777.0
[OVERALL], Throughput(ops/sec), 562.7462014631401
[UPDATE], Operations, 1
[UPDATE], AverageLatency(us), 520191.0
[UPDATE], MinLatency(us), 520191
[UPDATE], MaxLatency(us), 520191
[UPDATE], 95thPercentileLatency(ms), 520
[UPDATE], 99thPercentileLatency(ms), 520
[UPDATE], 0, 0
[UPDATE], 1, 0
[UPDATE], 2, 0
[UPDATE], 3, 0
[UPDATE], 4, 0
[UPDATE], 5, 0
[UPDATE], 6, 0
[UPDATE], 7, 0
[UPDATE], 8, 0
[UPDATE], 9, 0
[UPDATE], 10, 0
[UPDATE], 11, 0
[UPDATE], 12, 0
[UPDATE], 13, 0
....
[CLEANUP], 998, 0
[CLEANUP], 999, 0
[CLEANUP], >1000, 0
java -cp /home/ruifengshan/work/YCSB-master/jdbc/src/main/conf:/home/ruifengshan/work/YCSB-master/gemfire/src/main/conf:/home/ruifengshan/work/YCSB-master/core/target/core-0.1.4.jar:/home/ruifengshan/work/YCSB-master/hbase/target/hbase-binding-0.1.4.jar:/home/ruifengshan/work/YCSB-master/hbase/target/archive-tmp/hbase-binding-0.1.4.jar:/home/ruifengshan/work/YCSB-master/hbase/src/main/conf:/home/ruifengshan/work/YCSB-master/nosqldb/src/main/conf:/home/ruifengshan/work/YCSB-master/dynamodb/conf:/home/ruifengshan/work/YCSB-master/voldemort/src/main/conf:/home/ruifengshan/work/YCSB-master/infinispan/src/main/conf com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=cf -load

其中，workloada 为测试的配置文件，详细说明：https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties

然后是执行事务，主要有read和update，执行命令如下：

bin/ycsb run hbase -P workloads/workloada -p columnfamily=cf

执行结果如下：

[OVERALL], RunTime(ms), 3060.0
[OVERALL], Throughput(ops/sec), 326.797385620915
[UPDATE], Operations, 475
[UPDATE], AverageLatency(us), 464.33894736842103
[UPDATE], MinLatency(us), 70
[UPDATE], MaxLatency(us), 116412
[UPDATE], 95thPercentileLatency(ms), 0
[UPDATE], 99thPercentileLatency(ms), 0
[UPDATE], Return=0, 474
[UPDATE], 0, 471
[UPDATE], 1, 0
[UPDATE], 2, 2
[UPDATE], 3, 0
[UPDATE], 4, 1
...
[UPDATE], 998, 0
[UPDATE], 999, 0
[UPDATE], >1000, 0
[READ], Operations, 526
[READ], AverageLatency(us), 5223.148288973384
[READ], MinLatency(us), 1439
[READ], MaxLatency(us), 1099990
[READ], 95thPercentileLatency(ms), 4
[READ], 99thPercentileLatency(ms), 5
[READ], Return=0, 526
[READ], 0, 0
[READ], 1, 14
[READ], 2, 232
[READ], 3, 223
[READ], 4, 47
[READ], 5, 6
[READ], 6, 1
[READ], 7, 0
[READ], 8, 1
...

转载请注明出处：http://blog.csdn.net/iAm333