[置顶] Hadoop 实战之使用RegexMapper

环境：Vmware 8.0 和Ubuntu11.04

Hadoop 实战之RegexMapper

第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：

sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input 
hadoop dfsadmin -safemode leave

第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:

chmod 777 /home/tanglg1987/ start.sh
./start.sh 

执行过程如下：

第四步：上传本地文件到hdfs

在/home/tanglg1987/a目录下新建config.properties内容如下：

#次数
TIME=100000
#间隔,单位秒
#SLEEP=60000
SLEEP=10
DELAY=5
TIME=700000
SLEEP=100
DELAY=24

第五步：新建一个RegexMapperTest.java，代码如下：

package com.baison.action;
import java.io.IOException;
import java.util.Iterator;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
public class RegexMapperTest {
    public static class RegexMapper implements Mapper<WritableComparable,Text,Text,LongWritable> {
        private Pattern pattern;
        private int group;
        public void configure(JobConf job) {
            pattern = Pattern.compile(job.get("mapred.mapper.regex"));
            group = job.getInt("mapred.mapper.regex.group", 0);
            System.out.println(group);
        }
        @Override
        public void map(WritableComparable key, Text value, OutputCollector<Text,LongWritable> output, Reporter reporter) throws IOException {
            String text = value.toString();
                Matcher matcher = pattern.matcher(text);
                while (matcher.find()) {
                    output.collect(value, new LongWritable(1));
                }
        }
        @Override
        public void close() throws IOException {
            // TODO Auto-generated method stub
        }
    }
    public static class LongSumReducer implements Reducer<Text,LongWritable,Text,LongWritable> {
        @Override
        public void reduce(Text key, Iterator<LongWritable> values, OutputCollector<Text,LongWritable> output, Reporter reporter) throws IOException {
            long sum = 0;
            while (values.hasNext()) {
                LongWritable next = (LongWritable)values.next();
                sum += next.get();
            }
            output.collect(key, new LongWritable(sum));
        }
        @Override
        public void configure(JobConf arg0) {
            // TODO Auto-generated method stub
            
        }
        @Override
        public void close() throws IOException {
            // TODO Auto-generated method stub
            
        }
    }
    public static void main(String[] args) throws IOException {
        JobConf job = new JobConf(RegexMapperTest.class);
        FileInputFormat.addInputPath(job, new Path("/home/tanglg1987/a"));
        job.setMapperClass(RegexMapper.class);
        job.set("mapred.mapper.regex", "=");
        FileOutputFormat.setOutputPath(job, new Path("/home/tanglg1987/b"));
        job.setReducerClass(LongSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);
        JobClient.runJob(job);
    }
}

第六步：Run On Hadoop，运行过程如下：

12/10/18 22:41:19 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 22:41:19 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/10/18 22:41:19 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 22:41:19 INFO mapred.FileInputFormat: Total input paths to process : 2
12/10/18 22:41:19 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 22:41:19 INFO mapred.FileInputFormat: Total input paths to process : 2
12/10/18 22:41:19 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:41:19 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:41:20 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:41:20 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:41:20 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:41:20 INFO mapred.MapTask: Finished spill 0
12/10/18 22:41:20 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 22:41:20 INFO mapred.LocalJobRunner: file:/home/tanglg1987/a/config.properties:0+108
12/10/18 22:41:20 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 22:41:20 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:41:20 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:41:20 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:41:20 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:41:20 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:41:20 INFO mapred.MapTask: Finished spill 0
12/10/18 22:41:20 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
12/10/18 22:41:20 INFO mapred.LocalJobRunner: file:/home/tanglg1987/a/config.properties~:0+110
12/10/18 22:41:20 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
12/10/18 22:41:20 INFO mapred.LocalJobRunner:
12/10/18 22:41:20 INFO mapred.Merger: Merging 2 sorted segments
12/10/18 22:41:20 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 290 bytes
12/10/18 22:41:20 INFO mapred.LocalJobRunner:
12/10/18 22:41:20 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/18 22:41:20 INFO mapred.LocalJobRunner:
12/10/18 22:41:20 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/18 22:41:20 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/tanglg1987/b
12/10/18 22:41:20 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:41:20 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/18 22:41:20 INFO mapred.JobClient: map 100% reduce 100%
12/10/18 22:41:20 INFO mapred.JobClient: Job complete: job_local_0001
12/10/18 22:41:20 INFO mapred.JobClient: Counters: 13
12/10/18 22:41:20 INFO mapred.JobClient: FileSystemCounters
12/10/18 22:41:20 INFO mapred.JobClient: FILE_BYTES_READ=39665
12/10/18 22:41:20 INFO mapred.JobClient: FILE_BYTES_WRITTEN=79969
12/10/18 22:41:20 INFO mapred.JobClient: Map-Reduce Framework
12/10/18 22:41:20 INFO mapred.JobClient: Reduce input groups=8
12/10/18 22:41:20 INFO mapred.JobClient: Combine output records=0
12/10/18 22:41:20 INFO mapred.JobClient: Map input records=19
12/10/18 22:41:20 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/18 22:41:20 INFO mapred.JobClient: Reduce output records=8
12/10/18 22:41:20 INFO mapred.JobClient: Spilled Records=28
12/10/18 22:41:20 INFO mapred.JobClient: Map output bytes=258
12/10/18 22:41:20 INFO mapred.JobClient: Map input bytes=218
12/10/18 22:41:20 INFO mapred.JobClient: Combine input records=0
12/10/18 22:41:20 INFO mapred.JobClient: Map output records=14
12/10/18 22:41:20 INFO mapred.JobClient: Reduce input records=14

第七步：查看结果集，运行结果如下：