使用python访问hdfs
使用非Java语言访问hdfs有两种方法，一种是利用libhdfs.so来访问hdfs，另一种是使用thrift通信框架来访问，这里暂先介绍libhdfs
1、先安装libhdfs
# 前提是安装jdk6、jre6，利用cloudera.repo来安装hadoop-0.20
sudo yum –y install libhdfs*
2、安装python-devel(2.6+), gcc
sudo yum –y install python-devel gcc
3、下载libpyhdfs源码, 准备依赖包
svn checkout http://libpyhdfs.googlecode.com/svn/trunk/ libpyhdfs
cd libpyhdfs
cp /usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u0.jar lib/hadoop-0.20.1-core.jar
cp /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar lib/
cp /usr/lib/libhdfs.so.0 lib/
ln –s lib/libhdfs.so.0 lib/libhdfs.so
4、配置setup.py, 修改Java环境路径
vim setup.py
include_dirs = ['/usr/lib/jvm/java-6-sun/include/']
-> include_dirs = ['/usr/java/jdk1.6.0_24/include/']
runtime_library_dirs = ['/usr/local/lib/pyhdfs', '/usr/lib/jvm/java-6-sun/jre/lib/i386/server']
-> runtime_library_dirs = ['/usr/local/lib/pyhdfs', '/usr/java/jdk1.6.0_24/jre/lib/i386/server'],
5、修改jdk1.6.0_24/include/jni.h
#include "jni_md.h"
-> #include "linux/jni_md.h"
6、安装libpyhdfs
sudo python setup.py install --prefix="/usr/local"
# 测试
python pyhdfs_test.py
# 十分悲剧的报错了。。
import pyhdfs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libjvm.so: cannot enable executable stack as shared object requires: Permission denied
# 查了好久，最后发现是selinux的问题，暂时没有别的办法，把丫关了吧
sudo vim /etc/selinux/config
# 修改 SELINUX=disabled, 然后非重启关闭
sudo setenforce 0
# 检查selinux状态
sudo getenforce
再试试！！