在hadoop（1.0.4）中使用pipes需要注意的2个问题

阅读更多

 

如果完全按照hadoop权威指南中的使用pipe来实现c++的mapreduce，会遇到一些问题

 

问题1：链接生成二进制文件的时候失败

/home/robby/software/hadoop-1.0.4/src/c++/install/lib/libhadooppipes.a(HadoopPipes.o): In function `HadoopPipes::BinaryProtocol::createDigest(std::basic_string, std::allocator >&, std::basic_string, std::allocator >&)':
/home/robby/software/hadoop-1.0.4/src/c++/pipes/impl/HadoopPipes.cc:425: undefined reference to `EVP_sha1'
/home/robby/software/hadoop-1.0.4/src/c++/pipes/impl/HadoopPipes.cc:425: undefined reference to `HMAC_Init'
/home/robby/software/hadoop-1.0.4/src/c++/pipes/impl/HadoopPipes.cc:426: undefined reference to `HMAC_Update'
/home/robby/software/hadoop-1.0.4/src/c++/pipes/impl/HadoopPipes.cc:428: undefined reference to `HMAC_Final'
/home/robby/software/hadoop-1.0.4/src/c++/pipes/impl/HadoopPipes.cc:429: undefined reference to `HMAC_CTX_cleanup'
/home/robby/software/hadoop-1.0.4/src/c++/pipes/impl/HadoopPipes.cc:435: undefined reference to `BIO_f_base64'

 

解决方法：在编译的时候，加上参数 -lcrypto

 

问题2：运行mapreduce的时候报Authenticatio错误

at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188
at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194)
at org.apache.hadoop.mapred.pipes.Application.(Application.java:149)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)

 

解决办法：下载对应的hadoop源代码，然后编译c++部分，产生新的头文件和静态库，然后重新编译自己的c++代码，使用新的头文件和静态库