mapreduce: InputFormat详解 -- RecordReader篇

InputFormat是MapReduce中一个很常用的概念，它在程序的运行中到底起到了什么作用呢？

InputFormat其实是一个接口，包含了两个方法：

public interface InputFormat<K, V> {
  InputSplit[] getSplits(JobConf job, int numSplits) throws IOException;
  RecordReader<K, V> getRecordReader(InputSplit split,
                                     JobConf job,
                                     Reporter reporter) throws IOException;
}

这两个方法有分别完成着以下工作：

      方法getSplits将输入数据切分成splits，splits的个数即为map tasks的个数，splits的大小默认为块大小，即64M

     方法 getSplits将每个s plit解析成records, 再依次将record解析成<K,V>对

也就是说 InputFormat完成以下工作：

 InputFile -->  splits  -->  <K,V>

 

系统常用的  InputFormat 又有哪些呢？

                      

其中Text InputFormat便是最常用的，它的 <K,V>就代表 <行偏移,该行内容>

 

然而系统所提供的这几种固定的将  InputFile转换为 <K,V>的方式有时候并不能满足我们的需求：

此时需要我们自定义   InputFormat ，从而使Hadoop框架按照我们预设的方式来将

InputFile解析为<K,V>

在领会自定义   InputFormat 之前，需要弄懂一下几个抽象类、接口及其之间的关系：

 

InputFormat(interface), FileInputFormat(abstract class), TextInputFormat(class),

RecordReader (interface), Line RecordReader(class)的关系

      FileInputFormat implements  InputFormat

      TextInputFormat extends  FileInputFormat

      TextInputFormat.get RecordReader calls  Line RecordReader

      Line RecordReader  implements  RecordReader

 

对于InputFormat接口，上面已经有详细的描述

再看看 FileInputFormat，它实现了 InputFormat接口中的 getSplits方法，而将 getRecordReader与isSplitable留给具体类(如 TextInputFormat )实现， isSplitable方法通常不用修改，所以只需要在自定义的 InputFormat中实现

getRecordReader方法即可，而该方法的核心是调用 Line RecordReader(即由LineRecorderReader类来实现 " 将每个s plit解析成records, 再依次将record解析成<K,V>对" )，该方法实现了接口RecordReader

 

  public interface RecordReader<K, V> {

  boolean next(K key, V value) throws IOException;
  K createKey();
  V createValue();
  long getPos() throws IOException;
  public void close() throws IOException;
  float getProgress() throws IOException;
}

 

     因此自定义InputFormat的核心是自定义一个实现接口RecordReader类似于LineRecordReader的类，该类的核心也正是重写接口RecordReader中的几大方法，

     定义一个InputFormat的核心是定义一个类似于LineRecordReader的，自己的RecordReader

     示例，数据每一行为 “物体，x坐标，y坐标，z坐标”

    ball 3.5,12.7,9.0

   car 15,23.76,42.23

   device 0.0,12.4,-67.1

每一行将要被解析为<Text, Point3D>（Point3D是我们在上一篇日志中自定义的数据类型）

 

方式一，自定义的RecordReader使用中LineRecordReader，

 

 

 1  public class ObjectPositionInputFormat extends
 2     FileInputFormat<Text, Point3D> {
 3  
 4   public RecordReader<Text, Point3D> getRecordReader(
 5       InputSplit input, JobConf job, Reporter reporter)
 6       throws IOException {
 7  
 8     reporter.setStatus(input.toString());
 9     return new ObjPosRecordReader(job, (FileSplit)input);
10   }
11 }
12 class ObjPosRecordReader implements RecordReader<Text, Point3D> {
13   private LineRecordReader lineReader;
14   private LongWritable lineKey;
15   private Text lineValue;
16  
17   public ObjPosRecordReader(JobConf job, FileSplit split) throws IOException {
18     lineReader = new LineRecordReader(job, split);
19  
20     lineKey = lineReader.createKey();
21     lineValue = lineReader.createValue();
22   }
23  
24   public boolean next(Text key, Point3D value) throws IOException {
25     // get the next line
26     if (!lineReader.next(lineKey, lineValue)) {
27       return false;
28     }
29  
30     // parse the lineValue which is in the format:
31     // objName, x, y, z
32     String [] pieces = lineValue.toString().split(",");
33     if (pieces.length != 4) {
34       throw new IOException("Invalid record received");
35     }
36  
37     // try to parse floating point components of value
38     float fx, fy, fz;
39     try {
40       fx = Float.parseFloat(pieces[1].trim());
41       fy = Float.parseFloat(pieces[2].trim());
42       fz = Float.parseFloat(pieces[3].trim());
43     } catch (NumberFormatException nfe) {
44       throw new IOException("Error parsing floating point value in record");
45     }
46    // now that we know we'll succeed, overwrite the output objects
47     key.set(pieces[0].trim()); // objName is the output key.
48     value.x = fx;
49     value.y = fy;
50     value.z = fz;
51     return true;
52   }
53   public Text createKey() {
54     return new Text("");
55   }
56   public Point3D createValue() {
57     return new Point3D();
58   }
59   public long getPos() throws IOException {
60     return lineReader.getPos();
61   }
62   public void close() throws IOException {
63     lineReader.close();
64   }
65 public float getProgress() throws IOException {
66     return lineReader.getProgress();
67   }
68 }

 

 

方式二：自定义的RecordReader中使用LineReader，

 1 public class ObjectPositionInputFormat extends FileInputFormat<Text, Point3D> {   
 2     @Override
 3     protected boolean isSplitable(JobContext context, Path filename) {
 4         // TODO Auto-generated method stub
 5         return false;
 6     }
 7     @Override
 8     public RecordReader<Text, Point3D> createRecordReader(InputSplit inputsplit,
 9             TaskAttemptContext context) throws IOException, InterruptedException {
10         // TODO Auto-generated method stub
11         return new objPosRecordReader();
12     }
13     public static class objPosRecordReader extends RecordReader<Text,Point3D>{
14  
15         public LineReader in;
16         public Text lineKey;
17         public Point3D lineValue;
18         public StringTokenizer token=null;       
19         public Text line;      
20         @Override
21         public void close() throws IOException {
22             // TODO Auto-generated method stub           
23         }
24      @Override
25         public Text getCurrentKey() throws IOException, InterruptedException {
26             // TODO Auto-generated method stub
27             System.out.println("key");
28             //lineKey.set(token.nextToken());
29             System.out.println("hello");
30             return lineKey;
31         }
32         @Override
33         public Point3D getCurrentValue() throws IOException,
34                 InterruptedException {
35             // TODO Auto-generated method stub           
36             return lineValue;
37         }
38         @Override
39         public float getProgress() throws IOException, InterruptedException {
40             // TODO Auto-generated method stub
41             return 0;
42         }
43         @Override
44         public void initialize(InputSplit input, TaskAttemptContext context)
45                 throws IOException, InterruptedException {
46             // TODO Auto-generated method stub
47             FileSplit split=(FileSplit)input;
48             Configuration job=context.getConfiguration();
49             Path file=split.getPath();
50             FileSystem fs=file.getFileSystem(job);
51            
52             FSDataInputStream filein=fs.open(file);
53             in=new LineReader(filein,job);
54            
55             line=new Text();
56             lineKey=new Text();
57             lineValue=new Point3D();                     
58         }
59         @Override
60         public boolean nextKeyValue() throws IOException, InterruptedException {
61             // TODO Auto-generated method stub
62             int linesize=in.readLine(line);
63             if(linesize==0)
64                 return false;
65                token=new StringTokenizer(line.toString());
66             String []temp=new String[2];
67             if(token.hasMoreElements()){
68                 temp[0]=token.nextToken();
69                 if(token.hasMoreElements()){
70                     temp[1]=token.nextToken();
71                 }
72             }
73             System.out.println(temp[0]);
74             System.out.println(temp[1]);
75             String []points=temp[1].split(",");
76             System.out.println(points[0]);
77             System.out.println(points[1]);
78             System.out.println(points[2]);
79             lineKey.set(temp[0]);
80             lineValue.set(Float.parseFloat(points[0]),Float.parseFloat(points[1]), Float.parseFloat(points[2]));
81             System.out.println("pp");
82             return true;
83         }       
84     }
85 }

 

 

     从以上可以看出，自定义一个InputFormat的核心是定义一个类似于LineRecordReader的，自己的RecordReader，而在其中可能会到LineReader/LineRecordReader/KeyValueLineRecordReader类

     因此，要自定义InputFormat，这三个类的源码就必须很熟悉~