运行hadoop自带的pipes examples没有问题,自己写个却在jobtracker界面中报了Server failed to authenticate. Exiting错误.去日志中看下,完整异常如下:
1.job日志
java.io.IOException at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188) at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194) at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) at org.apache.hadoop.mapred.Child.main(Child.java:262)
2.task日志
2012-10-31 15:37:10,622 ERROR org.apache.hadoop.mapred.pipes.BinaryProtocol: java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:299) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:320) at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:121) 2012-10-31 15:37:10,626 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2012-10-31 15:37:10,667 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188) at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194) at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:149) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) at org.apache.hadoop.mapred.Child.main(Child.java:262) 2012-10-31 15:37:12,224 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
看来是权限的问题,那为什么我运行hadoop自带的pipes example就不会有问题呢?如此应该是环境方面的问题了.于是跑到$HADOOP_HOME/src/examples/org/pipes中的Makefile.in和configure看到的hadoop环境是默认的/usr/local/include和/usr/local/lib(前提是pipes,libhdfs,utils等已经安装).而我自己用c++写的mapreduce所配置的hadoop编译环境是$HADOOP_HOME/c++/include和$HADOOP_HOME/c++/lib,这个是hadoop默认的。于是改下自己的hadoop编译环境,发现便可以正常运行。
这里也有一篇解决方法,但针对的是你用hadoop自带的pipes example也报这个错误。则需要采用这个方法了。http://www.linuxquestions.org/questions/linux-software-2/hadoop-1-0-3-pipes-server-failed-to-authenticate-4175429779/。英文的,同时为了防止被删除或者被墙,我这里也记录下。
1.在/etc/profile添加一个变量,命令如下:
export LIB=-lcrypto
2.修改$HADOOP_HOME/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:
patch如下:
Index: src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java =================================================================== --- src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java (revision 1340233) +++ src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java (working copy) @@ -613,10 +613,10 @@ } } - private <T> String getEnumValues(Enum<? extends T>[] e) { + private String getEnumValues(Enum<?>[] e) { StringBuilder sb = new StringBuilder(); String sep = ""; - for (Enum<? extends T> v : e) { + for (Enum<?> v : e) { sb.append(sep); sb.append(v.name()); sep = "|";
3.安装c++/utils,到$(HADOOP_HOME)/src/c++/utils运行如下命令:
./configure make install
4.安装c++/pipes,到$(HADOOP_HOME)/src/c++/pipes运行如下命令:
./configure make install
5.再你的Makefile中使用如下设置:
-I$(HADOOP_HOME)/src/c++/install/include -L$(HADOOP_HOME)/src/c++/install/lib -lhadooputils -lhadooppipes -lcrypto -lssl -lpthread
或者你不用修改你的Makefile,只要将$(HADOOP_HOME)/src/c++/install下的东西替换到$HADOOP_HOME/c++即可.