hadoop无法加载本地库文件的原因 Unable to load native-hadoop library for your platform

      在suse11sp3 中安装了hadoop-2.7.1.tar.gz 版本， 安装好后，执行：  

    

     linux-xingyuncao7:/home/hadoop/bin #./hadoop fs -ls

    15/08/27 22:16:37 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes    where applicable

    ls: `.': No such file or directory

  报错： 不能加载本地库文件。由于suse11sp3 和 hadoop 都是64位的，排除是位数不正确导致的。

 

打开DEBUG日志：export HADOOP_ROOT_LOGGER=DEBUG,console后查看日志：

 

linux-xingyuncao7:/home/hadoop/lib/native# exportHADOOP_ROOT_LOGGER=DEBUG,console

 

linux-xingyuncao7:/home/hadoop/bin # ./hadoop fs -ls ; tail -f../logs/hadoop-root-namenode-linux-xingyuncao7.log

15/08/27 22:39:59 DEBUG util.Shell: setsidexited with exit code 0

15/08/27 22:39:59 DEBUG conf.Configuration:parsing URLjar:file:/home/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar!/core-default.xml

15/08/27 22:39:59 DEBUG conf.Configuration:parsing input streamsun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@b1f3c4e

15/08/27 22:40:00 DEBUG conf.Configuration:parsing URL file:/home/hadoop/etc/hadoop/core-site.xml

15/08/27 22:40:00 DEBUG conf.Configuration:parsing input stream java.io.BufferedInputStream@33af751c

15/08/27 22:40:00 DEBUGlib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRateorg.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess withannotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate ofsuccessful kerberos logins and latency (milliseconds)], about=, valueName=Time,type=DEFAULT, always=false, sampleName=Ops)

15/08/27 22:40:00 DEBUGlib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRateorg.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure withannotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of failedkerberos logins and latency (milliseconds)], about=, valueName=Time,type=DEFAULT, always=false, sampleName=Ops)

15/08/27 22:40:00 DEBUGlib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRateorg.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups withannotation @org.apache.hadoop.metrics2.annotation.Metric(value=[GetGroups],about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)

15/08/27 22:40:00 DEBUG impl.MetricsSystemImpl:UgiMetrics, User and group related metrics

15/08/27 22:40:00 DEBUG util.KerberosName:Kerberos krb5 configuration not found, setting default realm to empty

15/08/27 22:40:00 DEBUGsecurity.Groups: Creating new Groupsobject

15/08/27 22:40:00 DEBUGutil.NativeCodeLoader: Trying to load the custom-built native-hadoop library...

15/08/27 22:40:00 DEBUGutil.NativeCodeLoader:Failed to load native-hadoopwith error: java.lang.UnsatisfiedLinkError:/home/hadoop/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version`GLIBC_2.12' not found (required by /home/hadoop/lib/native/libhadoop.so.1.0.0)

15/08/27 22:40:00 DEBUGutil.NativeCodeLoader: java.library.path=/home/hadoop/lib/native

15/08/27 22:40:00 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable

15/08/27 22:40:00 DEBUGutil.PerformanceAdvisory: Falling back to shell based

15/08/27 22:40:00 DEBUGsecurity.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping

发现 libhadoop.so.1.0.0 加载的 libc.so.6 需要 2.12 版本。

查看系统的 glibc版本：是 2.11.3 版本的。

linux-xingyuncao7:/home/hadoop/bin # rpm-qa|grep glibc

glibc-2.11.3-17.54.1

glibc-32bit-2.11.3-17.54.1

glibc-i18ndata-2.11.3-17.54.1

glibc-locale-2.11.3-17.54.1

glibc-locale-32bit-2.11.3-17.54.1

glibc-devel-2.11.3-17.54.1

glibc-devel-32bit-2.11.3-17.54.1

查看  libhadoop.so 依赖的库文件：

linux-xingyuncao7:/home/hadoop/lib/native #ldd libhadoop.so

./libhadoop.so: /lib64/libc.so.6: version`GLIBC_2.12' not found (required by ./libhadoop.so)

./libhadoop.so: /lib64/libc.so.6: version`GLIBC_2.14' not found (required by ./libhadoop.so)

 

linux-xingyuncao7:/home/hadoop/bin #./hadoop checknative -a

15/08/27 22:20:41 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable

Native library checking:

hadoop: false

zlib:   false

snappy: false

lz4:    false

bzip2:  false

openssl: false

15/08/27 22:20:41 INFO util.ExitUtil:Exiting with status 1

需要把 glibc 升级到 2.14 版本。

但是在suse11sp3 上编译 glibc 通过后，无法安装， 执行make install 后，无法把 /lib64 目录下的 libc.so.6 文件的链接改到 新编译出来的 libc 库文件，而且  glibc 包也无法卸载。

原因找到了，下载hadoop的 源码。在suse11sp3 上重新编译 hadoop 。执行 mvn package -Pdist,native -DskipTests -Dtar 重新编译hadoop 。

linux-l8bm:/lib64 # ls -lrt *libc*
-rwxr-xr-x 1 root root   19016 Nov  5  2011 libcap.so.2.11
-rwxr-xr-x 1 root root   56440 Feb  3  2012 libcryptsetup.so.0.0.0
-rwxr-xr-x 1 root root   14720 Dec 22  2012 libcom_err.so.2.1
-rwxr-xr-x 1 root root  197116 May 10  2013 libcidn-2.11.3.so
-rwxr-xr-x 1 root root 1759139 May 10  2013 libc-2.11.3.so
-rwxr-xr-x 1 root root   57723 May 10  2013 libcrypt-2.11.3.so
lrwxrwxrwx 1 root root      18 Aug  8 10:26 libcrypt.so.1 -> libcrypt-2.11.3.so
lrwxrwxrwx 1 root root      17 Aug  8 10:26 libcidn.so.1 -> libcidn-2.11.3.so
lrwxrwxrwx 1 root root      14 Aug  8 10:26 libcap.so.2 -> libcap.so.2.11
lrwxrwxrwx 1 root root      17 Aug  8 10:26 libcom_err.so.2 -> libcom_err.so.2.1
lrwxrwxrwx 1 root root      22 Aug  8 10:31 libcryptsetup.so.0 -> libcryptsetup.so.0.0.0
-rwxr-xr-x 1 root root 8914645 Aug 27 23:39 libc-2.12.2.so
lrwxrwxrwx 1 root root      14 Aug 28 00:59 libc.so.6 -> libc-2.11.3.so