hadoop libhdfs的使用方法

博客说明：
博客内容用于学习与分享，有问题欢迎大家讨论留言。

关于作者：
程序员：杨洪（ellende）
blog: http://blog.csdn.net/ellende
email: yangh.personal@qq.com

转载请注明出处，引用部分网上博客，若有侵权还请作者联系与我。

本文主要介绍libhdfs.so 使用

1.使用环境如下：

hadoop 2.7.2

idk 1.7

centos

2.libhdfs.so使用

该共享库在hadoop-2.7.2/lib/native/libhdfs.so

还需要jdk的一个库jdk1.7.0_79/jre/lib/amd64/server/libjvm.so

hdfs.h头文件在
hadoop-2.7.2/include/hdfs.h

3.设置共享库路径

vim  /etc/ld.so.conf

修改如下：

/home/yj/HadoopFile/jdk1.7.0_79/jre/lib
/home/yj/HadoopFile/jdk1.7.0_79/jre/lib/amd64/server
/home/yj/HadoopFile/hadoop-2.7.2/lib/native

然后调用linux系统指令：

ldconfig

4.设置CLASSPATH不然会报NoClassDefFoundError exception: ExceptionUtils的错误

vim ~/.profile

修改如下：

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
CLASSPATH=`$HADOOP_INSTALL/bin/hadoop classpath`:$CLASSPATH
CLASSPATH=`find $HADOOP_INSTALL/share/hadoop/ -name *.jar | awk '{printf("%s:", $0);}'`$CLASSPATH

extern CLASSPATH

启用source ~/.profile

5.编写c文件

vim  above_sample.c

内容如下：

#include
#include
#include
#include "hdfs.h"


int main(int argc, char** argv)
{
printf("start main!\n");


//hdfsFS fs = hdfsConnect("default", 0);  //error!
hdfsFS fs = hdfsConnectAsUser("localhost", 9000, "hadoop");


//const char* writePath = "./tempfile/testfile.txt";   //at /user/hadoop/tempfile/testfile.txt
const char* writePath = "hdfs://localhost:9000/tmp/testfile.txt";  //at /tmp/testfile.txt


printf("open file!\n");
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0);
if (!writeFile)
{
fprintf(stderr, "Failed to open %s for writing!\n", writePath);
exit(-1);
}


printf("write file!\n");
char *buffer = "Hello, World\n";
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1);
if (hdfsFlush(fs, writeFile))
{
fprintf(stderr, "Failed to 'flush' %s\n", writePath);
exit(-1);
}


hdfsCloseFile(fs, writeFile);
printf("close file!\n");


return 0;
}

6.编译

gcc ./above_sample.c  -I ../hadoop-2.7.2/include/ -L ../hadoop-2.7.2/lib/native/ -lhdfs -L ../jdk1.7.0_79/jre/lib/amd64/server/ -ljvm  -o above_sample

7.运行

./above_sample

提示如下：

gcc ./above_sample.c  -I ../hadoop-2.7.2/include/ -L ../hadoop-2.7.2/lib/native/ -lhdfs -L ../jdk1.7.0_79/jre/lib/amd64/server/ -ljvm  -o above_sample
./above_sample
start main!
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/yj/HadoopFile/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/yj/HadoopFile/hadoop-2.7.2/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/yj/HadoopFile/hadoop-2.7.2/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2016-07-31 01:03:33,451 WARN  util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
open file!
write file!
close file!

通过hadoop fs -cat /tmp/testfile.txt

Hello, World

运行成功！