Hadoop Configuration 加载资源文件

Configuration 的loadResources()方法 (采用lazy加载)
1.解释了为什么会加载一些默认文件(*.site.xml)

2.测试了一下用file的方式去读取hdfs文件,不行

      else if (name instanceof Path)中,会转换为本地文件,最后变成d:\\hdfs地址
       linux下也试了一下,同样是不行的,这种方式暂时行不通,以后这样的还是用文件流的方式来读取,
因为最后都是转换InputStream流的方式来读取,然后交给Document去解析xml文件



3.addResources(Path)这个函数针对的本地文件,注释如下
file-path of resource to be added, the local filesystem is examined directly to find the resource, without referring to the classpath.

所以如果一定要用的话可以这样用
Path tmp = new Path("file:///D://shili//src//main//java//com//pangu//site.xml");
conf.addResource(tmp);


4.对于远程HDFS上以hdfs:// hdfs://yfw-0249:29000/pipeline/pets/pets_1.0/20130422/_logs/history/YFW-0337_1364622621869_job_201303301350_99243_conf.xml 

   采用文件流的方式来读取


以后方法调用时先读下方法注释



demo如下:

package com.pangu;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class Java4Test {
	private static FSDataInputStream fsInput;
	/**
	 * @param args
	 */
	public static void main(String[] args) {
		Configuration conf = new Configuration();
		conf.set("fs.default.name", "hdfs://YFW-0249:29000");
		
		conf.addResource("site.xml");
		String target = "mapred.reduce.tasks";
		String path = "hdfs://yfw-0249:29000/pipeline/pets/pets_1.0/20130422/_logs/history/YFW-0337_1364622621869_job_201303301350_99243_conf.xml";
		Path tmp = new Path("file:///D://shili//src//main//java//com//pangu//site.xml");
//		 ClassLoader.getSystemClassLoader().getParent() ;\
		try {
			FileSystem fs = FileSystem.get(conf);
			fsInput = fs.open(new Path(path));
		} catch (Exception e) {
			
		}
		/*Path tmp = new Path(path);
		 
		
		System.out.println(tmp.toUri().toString());
		
		
		File file = new File(tmpPath);
		System.out.println(file.getAbsolutePath());
		if (!file.exists()) {
			System.out.println("test  ");
		}*/
		conf.addResource(tmp);
		System.err.println(conf.getInt(target, 4));
	}

}




你可能感兴趣的:(hadoop)