大数据压力测试工具HiBench

需求描述:需要测试cdh集群的组件的性能和稳定性。

压力测试工具选型:HiBench

HiBench测试工具说明:

HiBench是一套基准测试套件,用于帮助我们评估不同的大数据框架性能指标(包括处理速度、吞吐等)的负载指标,可以评估Hadoop、Spark和流式负载等

源码下载:wget https://github.com/intel-hadoop/HiBench/archive/HiBench-7.0.zip

源码编译:命令:mvn -Dspark=2.1 -Dscala=2.11 clean package

官方文档:https://github.com/intel-hadoop/HiBench/blob/master/docs/build-hibench.md


前提条件:

1)、需要python2.x(>=2.6)

2)、需要linux bc命令,用于生成Hibench 报告,若没有,则安装:yum install bc


修改配置:

vim conf/hadoop.conf

hibench.hadoop.home:hadoop安装目录

hibench.hadoop.executable:你的bin/hadoop所在目录,一般是{HADOOP_HOME}/bin/hadoop

hibench.hadoop.configure.dir:hadoop配置文件所在目录,一般位于HADOOP_HOME}/etc/hadoop

hibench.hdfs.master:hdfs上存储Hibench数据的目录,如:hdfs://localhost:8020/user/hibench

hibench.hadoop.release:hadoop发行版提供者,支持value:apache, cdh5, hdp


 /root/apps/HiBench-master/conf

          -rw-r--r-- 1 root root  245 Dec 12 21:30 benchmarks.lst

-rw-r--r-- 1 root root  332 Aug 13 02:34 flink.conf.template

-rw-r--r-- 1 root root   13 Dec 12 21:30 frameworks.lst

-rw-r--r-- 1 root root  246 Aug 13 02:34 gearpump.conf.template

-rw-r--r-- 1 root root  471 Dec 13 20:30 hadoop.conf

-rw-r--r-- 1 root root  448 Aug 13 02:34 hadoop.conf.template

-rw-r--r-- 1 root root 6600 Dec 18 04:57 hibench.conf

-rw-r--r-- 1 root root 1640 Dec 18 22:42 spark.conf

-rw-r--r-- 1 root root 1655 Aug 13 02:34 spark.conf.template

-rw-r--r-- 1 root root  942 Aug 13 02:34 storm.conf.template

drwxr-xr-x 8 root root  109 Dec 14 02:49 workloads #里面有对应配置产生的数据量配置

举例:micro

[root@gsafety1 conf]# ll /root/apps/HiBench-master/conf/workloads/micro

total 20

-rw-r--r-- 1 root root 1920 Aug 13 02:34 dfsioe.conf

-rwxr-xr-x 1 root root  805 Aug 13 02:34 sleep.conf

-rw-r--r-- 1 root root  657 Aug 13 02:34 sort.conf

-rw-r--r-- 1 root root  571 Dec 18 04:40 terasort.conf

-rwxr-xr-x 1 root root  658 Dec 14 02:38 wordcount.conf

目录结构:

  /root/apps/HiBench-master/bin/workloads下面有:

                  graph

                  micro#选这个!!测试

                  ml

                  sql

                  streaming

                  websearch


测试算法包:

  /root/apps/HiBench-master/bin/workloads/micro/dfsioe# hdfsio测试

  /root/apps/HiBench-master/bin/workloads/micro/sleep

  /root/apps/HiBench-master/bin/workloads/micro/sort

  /root/apps/HiBench-master/bin/workloads/micro/wordcount


   cd /root/apps/HiBench-master/bin/workloads/micro/terasort#排序

   准备数据启动:

   /root/apps/HiBench-master/bin/workloads/micro/terasort/prepare/prepare.sh

   启动mr任务:

   /root/apps/HiBench-master/bin/workloads/micro/terasort/hadoop/run.sh

   启动spark任务:

   /root/apps/HiBench-master/bin/workloads/micro/terasort/spark/run.sh


测试报告:

 /root/apps/HiBench-master/report

               drwxr-xr-x 4 root root   44 Dec 14 03:58 bayes

                -rw-r--r-- 1 root root 6651 Dec 19 01:39 hibench.report

                drwxr-xr-x 3 root root   28 Dec 13 23:56 sort

                drwxr-xr-x 4 root root   44 Dec 13 03:45 terasort

                drwxr-xr-x 5 root root   61 Dec 12 04:11 wordcount

你可能感兴趣的:(大数据压力测试工具HiBench)