InputFormat到key-value生成流程

public abstract class InputFormat<K, V> {

  public abstract
    List<InputSplit> getSplits(JobContext context
                               ) throws IOException, InterruptedException;
 
  public abstract
    RecordReader<K,V> createRecordReader(InputSplit split,
                                         TaskAttemptContext context
                                        ) throws IOException,
                                                 InterruptedException;

}

public abstract class FileInputFormat<K, V> extends InputFormat<K, V> {

public class TextInputFormat extends FileInputFormat<LongWritable, Text> {

getSplits方法,获得对输入文件的切分数量,每一个split对应一个map。
创建RecordReader,该RecordReader接收切分好的split,实现nextKeyValue、getCurrentKey、getCurrentValue。

如下所示,每个map类都会继承Mapper类,在Mapper类中,run方法会调用InputFormat中的RecordReader来获得key、value

public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {
  /**
   * Expert users can override this method for more complete control over the
   * execution of the Mapper.
   * @param context
   * @throws IOException
   */
  public void run(Context context) throws IOException, InterruptedException {
    setup(context);
    try {
      while (context.nextKeyValue()) {
        map(context.getCurrentKey(), context.getCurrentValue(), context);
      }
    } finally {
      cleanup(context);
    }
  }
}

你可能感兴趣的:(InputFormat到key-value生成流程)