hibench适配3.1.1

参数说明

/*
 * 
read or write test
 * 
date and time the test finished
 * 
number of files
 * 
total number of bytes processed
 * 
throughput in mb/sec (total number of bytes / sum of processing times)
 * 
average i/o rate in mb/sec per file
 * 
standard deviation of i/o rate 
 * 
 */

• Number of files （处理的文件个数，每个都对应一个map任务数）
• throughput(计算公式如下)

$$Throughput(N)=\frac{{\sum }_{i=0}^{N}filesiz{e}_i}{{\sum }_{i=0}^{N}tim{e}_i}$$

• average i/o rate(计算公式如下)可以看到这个是单个文件

$$AverageIOrate(N)=\frac{{\sum }_{i=0}^{N}rat{e}_i}{N}=\frac{{\sum }_{i=0}^N\frac{filesiz{e}_i}{tim{e}_i}}{N}$$

• concurrent throughput(并发平均吞吐)

$$ConcurrentThroughput=Throughput(N)\ast N=\frac{{\sum }_{i=0}^{N}filesiz{e}_i}{{\sum }_{i=0}^{N}tim{e}_i}\ast N$$

• concurrent average IO rate(并发平均IO)

$$ConcurrentAverageIOrate=AverageIOrate(N)\ast N=\sum _{i=0}^N\frac{filesiz{e}_i}{tim{e}_i}$$

上述2个公式N表示的是集群mapslot数目，本次测试的时候，为6

• N计算公式（mapred.tasktracker.map.tasks.maximum缺省为2,mapred.tasktracker.reduce.tasks.maximum缺省为2,maxreduces为你mapreduce集群中集群的机器的数量)

$$N=mapSlots=mapred.tasktracker.map.tasks.maximum\ast \frac{maxreduces}{mapred.tasktracker.reduce.tasks.maximum}$$

环境信息

• hdp 3.1.4
• os CentOS 7.6
• hibench

hadoop-2.7.3到hadoop-3.1.1跨度较大，部分接口有变动，因此需要修改hibench来支持测试

// pom.xml 修改hadoop.mr2.version为3.1.1
<hadoop.mr2.version>3.1.1</hadoop.mr2.version>
// TestDFSIOEnh.java移除copyMerge接口

-			 FileUtil.copyMerge(fs, DfsioeConfig.getInstance().getReportDir(fsConfig), fs, DfsioeConfig.getInstance().getReportTmp(fsConfig), false, fsConfig, null);
-			 FileUtil.
-			 LOG.info("remote report file " + DfsioeConfig.getInstance().getReportTmp(fsConfig) + " merged.");
+			 BufferedReader lines = new BufferedReader(new InputStreamReader(fs.open(new Path(DfsioeConfig.getInstance().getReportDir(fsConfig),"part-r-00000"))));
//TestDFSIO.java FileInputFormat log接口依赖在2.7中为org.apache.commons.logging.Log，3.1.1中变更为org.slf4j.Logger，修改log处理方式
- private static final Log LOG = FileInputFormat.LOG;
+ private static final Log LOG = (Log) FileInputFormat.LOG;

编译：

 ~/apache-maven-3.5.4/bin/mvn -Phadoopbench -Dspark=2.2 -Dscala=2.11 clean package -X