[置顶] Hadoop 实战之运行CitationHistogram

大家好，今天我们在Hadoop 实战之分析专利引用数据集（一）和Hadoop 实战之分析专利引用数据集（三）的基础上来实现计算不同引用次数专利的数目

环境：Vmware 8.0 和Ubuntu11.04

Hadoop 实战之运行CitationHistogram---计算不同引用次数专利的数目

第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：

sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input 
hadoop dfsadmin -safemode leave

第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:

chmod 777 /home/tanglg1987/start.sh
./start.sh 

执行过程如下：

第四步：把我上一篇博客：linux下运行MyJob3的运行结果result当成这里的输入input

博客地址：http://blog.csdn.net/xiaotom5/article/details/8088304

五步：新建一个CitationHistogram.java，代码如下：

package com.baison.action;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.KeyValueTextInputFormat;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class CitationHistogram extends Configured implements Tool {   
    public static class MapClass extends MapReduceBase
        implements Mapper<Text, Text, IntWritable, IntWritable> {       
        private final static IntWritable uno = new IntWritable(1);
        private IntWritable citationCount = new IntWritable();     
        public void map(Text key, Text value,
                        OutputCollector<IntWritable, IntWritable> output,
                        Reporter reporter) throws IOException {                      
            citationCount.set(Integer.parseInt(value.toString()));
            output.collect(citationCount, uno);
        }
    }
    public static class Reduce extends MapReduceBase
        implements Reducer<IntWritable,IntWritable,IntWritable,IntWritable>
    {      
        public void reduce(IntWritable key, Iterator<IntWritable> values,
                           OutputCollector<IntWritable, IntWritable>output,
                           Reporter reporter) throws IOException {                       
            int count = 0;
            while (values.hasNext()) {
                count += values.next().get();
            }
            output.collect(key, new IntWritable(count));
        }
    } 
    public int run(String[] args) throws Exception {
        Configuration conf = getConf();    
        JobConf job = new JobConf(conf, CitationHistogram.class);  
        Path in = new Path(args[0]);
        Path out = new Path(args[1]);
        FileInputFormat.setInputPaths(job, in);
        FileOutputFormat.setOutputPath(job, out);   
        job.setJobName("CitationHistogram");
        job.setMapperClass(MapClass.class);
        job.setReducerClass(Reduce.class); 
        job.setInputFormat(KeyValueTextInputFormat.class);
        job.setOutputFormat(TextOutputFormat.class);
        job.setOutputKeyClass(IntWritable.class);
        job.setOutputValueClass(IntWritable.class); 
        JobClient.runJob(job); 
        return 0;
    }
    public static void main(String[] args) throws Exception { 
    	String [] arg={"hdfs://localhost:9100/user/tanglg1987/output"
	,"hdfs://localhost:9100/user/tanglg1987/output/CitationHistogram"};     
        int res = ToolRunner.run(new Configuration(), 
                                 new CitationHistogram(), 
                                 arg);    
        System.exit(res);
    }
}

第六步：Run On Hadoop，运行过程如下：

12/10/18 23:26:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 23:26:49 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 23:26:49 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 23:26:50 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 23:26:50 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 23:26:50 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 23:26:50 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 23:26:50 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 23:26:50 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 23:26:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:50 INFO mapred.MapTask: bufstart = 0; bufend = 2097152; bufvoid = 99614720
12/10/18 23:26:50 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 23:26:51 INFO mapred.JobClient: map 0% reduce 0%
12/10/18 23:26:51 INFO mapred.MapTask: Finished spill 0
12/10/18 23:26:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:51 INFO mapred.MapTask: bufstart = 2097152; bufend = 4194296; bufvoid = 99614720
12/10/18 23:26:51 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 23:26:52 INFO mapred.MapTask: Finished spill 1
12/10/18 23:26:52 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:52 INFO mapred.MapTask: bufstart = 4194296; bufend = 6291440; bufvoid = 99614720
12/10/18 23:26:52 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 23:26:52 INFO mapred.MapTask: Finished spill 2
12/10/18 23:26:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:53 INFO mapred.MapTask: bufstart = 6291440; bufend = 8388584; bufvoid = 99614720
12/10/18 23:26:53 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 23:26:53 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:53 INFO mapred.MapTask: Finished spill 3
12/10/18 23:26:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:53 INFO mapred.MapTask: bufstart = 8388584; bufend = 10485736; bufvoid = 99614720
12/10/18 23:26:53 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 23:26:54 INFO mapred.JobClient: map 31% reduce 0%
12/10/18 23:26:54 INFO mapred.MapTask: Finished spill 4
12/10/18 23:26:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:54 INFO mapred.MapTask: bufstart = 10485736; bufend = 12582880; bufvoid = 99614720
12/10/18 23:26:54 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 23:26:55 INFO mapred.MapTask: Finished spill 5
12/10/18 23:26:55 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:55 INFO mapred.MapTask: bufstart = 12582880; bufend = 14680024; bufvoid = 99614720
12/10/18 23:26:55 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 23:26:56 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:56 INFO mapred.MapTask: Finished spill 6
12/10/18 23:26:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:56 INFO mapred.MapTask: bufstart = 14680024; bufend = 16777168; bufvoid = 99614720
12/10/18 23:26:56 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 23:26:57 INFO mapred.JobClient: map 58% reduce 0%
12/10/18 23:26:57 INFO mapred.MapTask: Finished spill 7
12/10/18 23:26:57 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:57 INFO mapred.MapTask: bufstart = 16777168; bufend = 18874312; bufvoid = 99614720
12/10/18 23:26:57 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 23:26:58 INFO mapred.MapTask: Finished spill 8
12/10/18 23:26:58 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:58 INFO mapred.MapTask: bufstart = 18874312; bufend = 20971464; bufvoid = 99614720
12/10/18 23:26:58 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 23:26:58 INFO mapred.MapTask: Finished spill 9
12/10/18 23:26:59 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:59 INFO mapred.MapTask: bufstart = 20971464; bufend = 23068608; bufvoid = 99614720
12/10/18 23:26:59 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 23:26:59 INFO mapred.MapTask: Finished spill 10
12/10/18 23:26:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:59 INFO mapred.MapTask: bufstart = 23068608; bufend = 25165752; bufvoid = 99614720
12/10/18 23:26:59 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 23:27:00 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 23:27:00 INFO mapred.MapTask: Finished spill 11
12/10/18 23:27:00 INFO mapred.MapTask: Starting flush of map output
12/10/18 23:27:00 INFO mapred.MapTask: Finished spill 12
12/10/18 23:27:00 INFO mapred.Merger: Merging 13 sorted segments
12/10/18 23:27:00 INFO mapred.Merger: Merging 4 intermediate segments out of a total of 13
12/10/18 23:27:01 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 32589860 bytes
12/10/18 23:27:02 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:03 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 23:27:05 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:05 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 23:27:05 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:05 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 23:27:05 INFO mapred.LocalJobRunner:
12/10/18 23:27:05 INFO mapred.Merger: Merging 1 sorted segments
12/10/18 23:27:05 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 32589842 bytes
12/10/18 23:27:05 INFO mapred.LocalJobRunner:
12/10/18 23:27:06 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/18 23:27:06 INFO mapred.LocalJobRunner:
12/10/18 23:27:06 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/18 23:27:06 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output/CitationHistogram
12/10/18 23:27:06 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 23:27:06 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/18 23:27:07 INFO mapred.JobClient: map 100% reduce 100%
12/10/18 23:27:07 INFO mapred.JobClient: Job complete: job_local_0001
12/10/18 23:27:07 INFO mapred.JobClient: Counters: 15
12/10/18 23:27:07 INFO mapred.JobClient: FileSystemCounters
12/10/18 23:27:07 INFO mapred.JobClient: FILE_BYTES_READ=118774812
12/10/18 23:27:07 INFO mapred.JobClient: HDFS_BYTES_READ=65717226
12/10/18 23:27:07 INFO mapred.JobClient: FILE_BYTES_WRITTEN=151399190
12/10/18 23:27:07 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1721
12/10/18 23:27:07 INFO mapred.JobClient: Map-Reduce Framework
12/10/18 23:27:07 INFO mapred.JobClient: Reduce input groups=258
12/10/18 23:27:07 INFO mapred.JobClient: Combine output records=0
12/10/18 23:27:07 INFO mapred.JobClient: Map input records=3258984
12/10/18 23:27:07 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/18 23:27:07 INFO mapred.JobClient: Reduce output records=258
12/10/18 23:27:07 INFO mapred.JobClient: Spilled Records=10825525
12/10/18 23:27:07 INFO mapred.JobClient: Map output bytes=26071872
12/10/18 23:27:07 INFO mapred.JobClient: Map input bytes=32858613
12/10/18 23:27:07 INFO mapred.JobClient: Combine input records=0
12/10/18 23:27:07 INFO mapred.JobClient: Map output records=3258984
12/10/18 23:27:07 INFO mapred.JobClient: Reduce input records=3258984

第七步：查看结果集，运行结果如下：