运行hadoop程序，如何在map中获取输入数据的全路径（fullpath）

可以利用override map函数的第三个参数。map函数如下：
public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, Reporter reporter)

String path = ((FileSplit) reporter.getInputSplit()).getPath().toString();   

即可取得全路径。

 

下面这个方法可以做文件选取：

public class Filter implements PathFilter {
     public boolean accept(Path path) {
       return !(path.toString().indexOf("abc") > -1);  
      }
}

jobConf .set( "mapred.input.pathFilter.class" , "Filter" );

org.apache.hadoop.fs
Interface PathFilter

All Known Implementing Classes:

OutputLogFilter

public interface PathFilter

 

Method Summary
boolean	accept(Path path)            Tests whether or not the specified abstract pathname should be included in a pathname list.

Method Detail

accept

boolean accept(Path path)

Tests whether or not the specified abstract pathname should be included in a pathname list.

 

Parameters:

path - The abstract pathname to be tested

Returns:

true if and only if pathname should be included