hdfs 常用java API---代码篇（一）
一、hadoop环境准备（我安装的hadoop版本是hadoop-1.2.1)
首先要配置hadoop环境变量HADOOP_CLASSPATH(我的hadoop安装在了/home/grid/hadoop-1.2.1)：
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/usr/jdk1.6.0_45
# Extra Java CLASSPATH elements. Optional.
export HADOOP_CLASSPATH=/home/grid/hadoop-1.2.1/myclass
我在hadoop安装目录下创建了myclass目录
确认hadoop_classpath环境变量生效
[grid@h1 conf]$ source hadoop-env.sh
[grid@h1 conf]$ echo $HADOOP_CLASSPATH
/home/grid/hadoop-1.2.1/myclass
二、编写并编译URLCat源代码
1、代码功能
调用hadoop的java API接口，读取hdfs中的文件
hadoop的javaAPI说明可以参考http://hadoop.apache.org/docs/stable/api/index.html
2、代码内容
[grid@h1 myclass]$ cat URLCat.java
import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;
import java.io.InputStream;
import java.net.URL;
public class URLCat {
static
{
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
}
public static void main(String[] args) throws Exception
{
InputStream in = null; //定义一个输入stream in
try {
in = new URL(args[0]).openStream(); // 传递参数给 in
IOUtils.copyBytes(in, System.out, 4096, false); //读取IO in的内容并打印到屏幕（System.out)
}
finally {
IOUtils.closeStream(in);
}
}
}
3、代码编译及运行
[grid@h1 myclass]$ javac URLCat.java
URLCat.java:1: 软件包 org.apache.hadoop.fs 不存在
import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
^
URLCat.java:2: 软件包 org.apache.hadoop.io 不存在
import org.apache.hadoop.io.IOUtils;
^
URLCat.java:9: 找不到符号
符号： 类 FsUrlStreamHandlerFactory
位置： 类 URLCat
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
^
URLCat.java:16: 找不到符号
符号： 变量 IOUtils
位置： 类 URLCat
IOUtils.copyBytes(in, System.out, 4096, false);
^
URLCat.java:19: 找不到符号
符号： 变量 IOUtils
位置： 类 URLCat
IOUtils.closeStream(in);
^
5 错误
需要指定classpath进行编译
[grid@h1 myclass]$ javac -cp /home/grid/hadoop-1.2.1/hadoop-core-1.2.1.jar URLCat.java
[grid@h1 myclass]$ ls
URLCat.class URLCat.java
编译成功在$HADOOP_CLASSPATH目录下产生了.class的编译文件
运行代码：
[grid@h1 myclass]$ hadoop fs -ls in
Found 4 items
-rw-r--r-- 2 grid supergroup 101 2013-08-24 12:50 /user/grid/in/VERSION
-rw-r--r-- 2 grid supergroup 7 2013-08-23 21:30 /user/grid/in/test3.txt
-rw-r--r-- 2 grid supergroup 12 2013-08-23 21:30 /user/grid/in/text1.txt
-rw-r--r-- 2 grid supergroup 13 2013-08-23 21:30 /user/grid/in/text2.txt
[grid@h1 myclass]$ hadoop fs -cat in/test3.txt
hadoop
[grid@h1 myclass]$ hadoop URLCat hdfs://h1:9000/user/grid/in/test3.txt
hadoop
4、代码用到的API解读
1、hadoop基础类
org.apache.hadoop.fs Class FsUrlStreamHandlerFactory
java.lang.Object
org.apache.hadoop.fs.FsUrlStreamHandlerFactory
All Implemented Interfaces:
URLStreamHandlerFactory
public class FsUrlStreamHandlerFactory
extends Object
implements URLStreamHandlerFactory
Factory for URL stream handlers. There is only one handler whose job is to create UrlConnections. A FsUrlConnection relies on FileSystem to choose the appropriate FS implementation. Before returning our handler, we make sure that FileSystem knows an implementation for the requested scheme/protocol.
FsUrlStreamHandlerFactory：URL流处理工具，具体的作用要使用java.net.URL的 setURLStreamHandlerFactory()方法设置URLStreamHandlerFactory，这时需要传递一个 FsUrlStreamHandlerFactory。这个操作对一个jvm只能使用一次。
org.apache.hadoop.io Class IOUtils
java.lang.Object
org.apache.hadoop.io.IOUtils
public class IOUtils
extends Object
An utility class for I/O related functionality.
一个I/O相关功能的实用类