[置顶] Hadoop 实战之运行AveragingWithCombiner

环境：Vmware 8.0 和ubuntu11.04

Hadoop 实战之运行AveragingWithCombiner---使用combiner提升性能

第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：

sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input 
hadoop dfsadmin -safemode leave

第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:

chmod 777 /home/tanglg1987/ start.sh
./start.sh 

执行过程如下：

12/10/15 23:05:38 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = tanglg1987/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
12/10/15 23:05:39 INFO namenode.FSNamesystem: fsOwner=tanglg1987,tanglg1987,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
12/10/15 23:05:39 INFO namenode.FSNamesystem: supergroup=supergroup
12/10/15 23:05:39 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/10/15 23:05:39 INFO common.Storage: Image file of size 100 saved in 0 seconds.
12/10/15 23:05:39 INFO common.Storage: Storage directory /tmp/hadoop-tanglg1987/dfs/name has been successfully formatted.
12/10/15 23:05:39 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at tanglg1987/127.0.1.1
************************************************************/
12/10/15 23:05:40 INFO datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = tanglg1987/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
Usage: java DataNode
[-rollback]
12/10/15 23:05:40 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at tanglg1987/127.0.1.1
************************************************************/
starting namenode, logging to /home/tanglg1987/hadoop-0.20.2/bin/../logs/hadoop-tanglg1987-namenode-tanglg1987.out
localhost: starting datanode, logging to /home/tanglg1987/hadoop-0.20.2/bin/../logs/hadoop-tanglg1987-datanode-tanglg1987.out
localhost: starting secondarynamenode, logging to /home/tanglg1987/hadoop-0.20.2/bin/../logs/hadoop-tanglg1987-secondarynamenode-tanglg1987.out
starting jobtracker, logging to /home/tanglg1987/hadoop-0.20.2/bin/../logs/hadoop-tanglg1987-jobtracker-tanglg1987.out
localhost: starting tasktracker, logging to /home/tanglg1987/hadoop-0.20.2/bin/../logs/hadoop-tanglg1987-tasktracker-tanglg1987.out
Safe mode is OFF

第四步：上传本地文件到hdfs

在专利局http://data.nber.org/patents/网站下载专利数据

http://data.nber.org/patents/apat63_99.zip

hadoop fs -put /home/tanglg1987/apat63_99.txt input

五步：新建一个AveragingWithCombiner.java，代码如下：

package com.baison.action;
import java.io.IOException;
import java.util.regex.PatternSyntaxException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class AveragingWithCombiner extends Configured implements Tool {
public static class MapClass extends MapReduceBase
implements Mapper<LongWritable, Text, Text, Text> { 
public void map(LongWritable key, Text value,
OutputCollector<Text, Text> output,
Reporter reporter) throws IOException { 
String fields[] = value.toString().split(",", -20);
String country = fields[4];
String numClaims = fields[8];

if (numClaims.length() > 0 && !numClaims.startsWith("\"")) {
output.collect(new Text(country), new Text(numClaims + ",1"));
}
}
} 
public static class Combine extends MapReduceBase
implements Reducer<Text, Text, Text, Text> { 
public void reduce(Text key, Iterator<Text> values,
OutputCollector<Text, Text> output,
Reporter reporter) throws IOException { 
double sum = 0;
int count = 0;
while (values.hasNext()) {
String fields[] = values.next().toString().split(",");
sum += Double.parseDouble(fields[0]);
count += Integer.parseInt(fields[1]);
}
output.collect(key, new Text(sum + "," + count));
}
} 
public static class Reduce extends MapReduceBase
implements Reducer<Text, Text, Text, DoubleWritable> { 
public void reduce(Text key, Iterator<Text> values,
OutputCollector<Text, DoubleWritable> output,
Reporter reporter) throws IOException { 
double sum = 0;
int count = 0;
while (values.hasNext()) {
String fields[] = values.next().toString().split(",");
sum += Double.parseDouble(fields[0]);
count += Integer.parseInt(fields[1]);
}
output.collect(key, new DoubleWritable(sum/count));
}
}
public int run(String[] args) throws Exception {
// Configuration processed by ToolRunner
Configuration conf = getConf(); 
// Create a JobConf using the processed conf
JobConf job = new JobConf(conf, AveragingWithCombiner.class); 
// Process custom command-line options
Path in = new Path(args[0]);
Path out = new Path(args[1]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out); 
// Specify various job-specific parameters 
job.setJobName("AveragingWithCombiner");
job.setMapperClass(MapClass.class);
job.setCombinerClass(Combine.class);
job.setReducerClass(Reduce.class); 
job.setInputFormat(TextInputFormat.class);
job.setOutputFormat(TextOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class); 
// Submit the job, then poll for progress until the job is complete
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
// Let ToolRunner handle generic command-line options 
String [] arg={"hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt","hdfs://localhost:9100/user/tanglg1987/output3"};
int res = ToolRunner.run(new Configuration(), new AveragingWithCombiner(), arg); 
System.exit(res);
}
}

第六步：Run On Hadoop，运行过程如下：

12/10/16 21:12:07 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/16 21:12:07 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/16 21:12:07 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/16 21:12:08 INFO mapred.JobClient: Running job: job_local_0001
12/10/16 21:12:08 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/16 21:12:08 INFO mapred.MapTask: numReduceTasks: 1
12/10/16 21:12:08 INFO mapred.MapTask: io.sort.mb = 100
12/10/16 21:12:08 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/16 21:12:08 INFO mapred.MapTask: record buffer = 262144/327680
12/10/16 21:12:09 INFO mapred.JobClient: map 0% reduce 0%
12/10/16 21:12:11 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/16 21:12:12 INFO mapred.JobClient: map 63% reduce 0%
12/10/16 21:12:12 INFO mapred.MapTask: Starting flush of map output
12/10/16 21:12:12 INFO mapred.MapTask: Finished spill 0
12/10/16 21:12:12 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/16 21:12:12 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/16 21:12:12 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/16 21:12:12 INFO mapred.MapTask: numReduceTasks: 1
12/10/16 21:12:12 INFO mapred.MapTask: io.sort.mb = 100
12/10/16 21:12:13 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/16 21:12:13 INFO mapred.MapTask: record buffer = 262144/327680
12/10/16 21:12:13 INFO mapred.JobClient: map 100% reduce 0%
12/10/16 21:12:14 INFO mapred.MapTask: Spilling map output: record full = true
12/10/16 21:12:14 INFO mapred.MapTask: bufstart = 0; bufend = 2467890; bufvoid = 99614720
12/10/16 21:12:14 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/16 21:12:15 INFO mapred.MapTask: Finished spill 0
12/10/16 21:12:15 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/16 21:12:16 INFO mapred.JobClient: map 73% reduce 0%
12/10/16 21:12:16 INFO mapred.MapTask: Spilling map output: record full = true
12/10/16 21:12:16 INFO mapred.MapTask: bufstart = 2467890; bufend = 4942611; bufvoid = 99614720
12/10/16 21:12:16 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/16 21:12:17 INFO mapred.MapTask: Finished spill 1
12/10/16 21:12:18 INFO mapred.MapTask: Starting flush of map output
12/10/16 21:12:18 INFO mapred.MapTask: Finished spill 2
12/10/16 21:12:18 INFO mapred.Merger: Merging 3 sorted segments
12/10/16 21:12:18 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 4411 bytes
12/10/16 21:12:18 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
12/10/16 21:12:18 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/16 21:12:18 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
12/10/16 21:12:18 INFO mapred.MapTask: numReduceTasks: 1
12/10/16 21:12:18 INFO mapred.MapTask: io.sort.mb = 100
12/10/16 21:12:18 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/16 21:12:18 INFO mapred.MapTask: record buffer = 262144/327680
12/10/16 21:12:19 INFO mapred.JobClient: map 100% reduce 0%
12/10/16 21:12:20 INFO mapred.MapTask: Spilling map output: record full = true
12/10/16 21:12:20 INFO mapred.MapTask: bufstart = 0; bufend = 2496693; bufvoid = 99614720
12/10/16 21:12:20 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/16 21:12:21 INFO mapred.MapTask: Finished spill 0
12/10/16 21:12:21 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/16 21:12:22 INFO mapred.JobClient: map 84% reduce 0%
12/10/16 21:12:22 INFO mapred.MapTask: Spilling map output: record full = true
12/10/16 21:12:22 INFO mapred.MapTask: bufstart = 2496693; bufend = 5000178; bufvoid = 99614720
12/10/16 21:12:22 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/16 21:12:23 INFO mapred.MapTask: Finished spill 1
12/10/16 21:12:24 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/16 21:12:24 INFO mapred.MapTask: Starting flush of map output
12/10/16 21:12:25 INFO mapred.JobClient: map 99% reduce 0%
12/10/16 21:12:25 INFO mapred.MapTask: Finished spill 2
12/10/16 21:12:25 INFO mapred.Merger: Merging 3 sorted segments
12/10/16 21:12:25 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 4656 bytes
12/10/16 21:12:25 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
12/10/16 21:12:25 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/16 21:12:25 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000002_0' done.
12/10/16 21:12:25 INFO mapred.MapTask: numReduceTasks: 1
12/10/16 21:12:25 INFO mapred.MapTask: io.sort.mb = 100
12/10/16 21:12:25 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/16 21:12:25 INFO mapred.MapTask: record buffer = 262144/327680
12/10/16 21:12:26 INFO mapred.JobClient: map 100% reduce 0%
12/10/16 21:12:26 INFO mapred.MapTask: Spilling map output: record full = true
12/10/16 21:12:26 INFO mapred.MapTask: bufstart = 0; bufend = 2526823; bufvoid = 99614720
12/10/16 21:12:26 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/16 21:12:27 INFO mapred.MapTask: Finished spill 0
12/10/16 21:12:28 INFO mapred.MapTask: Starting flush of map output
12/10/16 21:12:28 INFO mapred.MapTask: Finished spill 1
12/10/16 21:12:28 INFO mapred.Merger: Merging 2 sorted segments
12/10/16 21:12:28 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2613 bytes
12/10/16 21:12:28 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
12/10/16 21:12:28 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:201326592+35576587
12/10/16 21:12:28 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000003_0' done.
12/10/16 21:12:28 INFO mapred.LocalJobRunner:
12/10/16 21:12:28 INFO mapred.Merger: Merging 4 sorted segments
12/10/16 21:12:28 INFO mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 7858 bytes
12/10/16 21:12:28 INFO mapred.LocalJobRunner:
12/10/16 21:12:28 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/16 21:12:28 INFO mapred.LocalJobRunner:
12/10/16 21:12:28 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/16 21:12:28 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output
12/10/16 21:12:28 INFO mapred.LocalJobRunner: reduce > reduce
12/10/16 21:12:28 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/16 21:12:29 INFO mapred.JobClient: map 100% reduce 100%
12/10/16 21:12:29 INFO mapred.JobClient: Job complete: job_local_0001
12/10/16 21:12:29 INFO mapred.JobClient: Counters: 15
12/10/16 21:12:29 INFO mapred.JobClient: FileSystemCounters
12/10/16 21:12:29 INFO mapred.JobClient: FILE_BYTES_READ=129472
12/10/16 21:12:29 INFO mapred.JobClient: HDFS_BYTES_READ=876508703
12/10/16 21:12:29 INFO mapred.JobClient: FILE_BYTES_WRITTEN=236890
12/10/16 21:12:29 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2658
12/10/16 21:12:29 INFO mapred.JobClient: Map-Reduce Framework
12/10/16 21:12:29 INFO mapred.JobClient: Reduce input groups=151
12/10/16 21:12:29 INFO mapred.JobClient: Combine output records=1063
12/10/16 21:12:29 INFO mapred.JobClient: Map input records=2923923
12/10/16 21:12:29 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/16 21:12:29 INFO mapred.JobClient: Reduce output records=151
12/10/16 21:12:29 INFO mapred.JobClient: Spilled Records=1721
12/10/16 21:12:29 INFO mapred.JobClient: Map output bytes=18862764
12/10/16 21:12:29 INFO mapred.JobClient: Map input bytes=236903179
12/10/16 21:12:29 INFO mapred.JobClient: Combine input records=1984625
12/10/16 21:12:29 INFO mapred.JobClient: Map output records=1984055
12/10/16 21:12:29 INFO mapred.JobClient: Reduce input records=49
第七步：查看结果集，运行结果如下：