hadoop2.5.2学习及实践笔记（二）—— 编译源代码及导入源码至eclipse

生产环境中hadoop一般会选择64位版本，官方下载的hadoop安装包中的native库是32位的，因此运行64位版本时，需要自己编译64位的native库，并替换掉自带native库。

源码包下的BUILDING.txt，包含官方介绍，如：如何在linux、windows下编译；编译过程中的错误处理；将源码导入eclipse中的步骤等，推荐看一看。

 

本文环境同：hadoop2.5.2学习及实践笔记（一）—— 伪分布式学习环境搭建

 

一.编译源码

 

附：常用工具包网盘路径（部分包非编译时使用）：http://pan.baidu.com/s/1c0evtgW

 

 

BUILDING.txt文档列举的linux中编译条件如下：

* Unix System

* JDK 1.6+

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6 or newer (if compiling native code)

* Zlib devel (if compiling native code)

* openssl devel ( if compiling native hadoop-pipes )

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

 

 

注：jdk建议使用1.7，楼主使用1.8编译时报错，换回1.7编译成功。

 

 

编译过程

 

 

　　1.  linux系统包安装

 

# yum install cmake lzo-devel  zlib-devel  gcc autoconf automake libtool   ncurses-devel openssl-devel

　　

　　2. 安装Maven、Ant、Findbugs

　　>解压tar包，并将解压后的目录移动到安装路径

　　>配置环境变量

# vim /etc/profile



export MAVEN_HOME=/opt/apache-maven-3.1.4
export ANT_HOME=/opt/apache-ant-1.9.4

export FINDBUGS_HOME=/opt/findbugs-2.5.0
export PATH=$PATH:$MAVEN_HOME/bin:$ANT_HOME/bin:$FINDBUGS_HOME/bin

　　>环境变量生效

# source /etc/rofile

　　>maven国内镜像配置

　　修改MAVEN_HOME/conf/settings.xml文件<mirrors>内容

 <mirror>      

      <id>nexus-osc</id>  

      <mirrorOf>*</mirrorOf>  

      <name>Nexus osc</name>  

      <url>http://maven.oschina.net/content/groups/public/</url>  

 </mirror> 

 

　　3.从源码安装protobuf

　　>解压tar包，并进入解压后的根目录

　　>编译并安装

# ./configure

# make

# sudo make install

　　>验证安装

# protoc --version

　　

　　4.编译hadoop源码

　　>解压源码tar包，并进入解压后的根目录

$ mvn package -DskipTests -Pdist,native -Dtar

　　>其他编译选项

　　附：BUILDING.txt中原文

　　Create binary distribution without native code and without documentation:

　　$ mvn package -Pdist -DskipTests -Dtar

 

　　Create binary distribution with native code and with documentation:

　　$ mvn package -Pdist,native,docs -DskipTests -Dtar

 

　　Create source distribution:

　　$ mvn package -Psrc -DskipTests

 

　　Create source and binary distributions with native code and documentation:

　　$ mvn package -Pdist,native,docs,src -DskipTests -Dtar

 

　　Create a local staging version of the website (in /tmp/hadoop-site)

　　$ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

 

 

　　>编译过程中内存溢出错误处理

　　编译前设置MAVEN_OPTS环境变量

export MAVEN_OPTS="-Xms256m -Xmx512m" --内存大小自己调节

 

　　

　　附：BUILDING.txt中原文

　　 If the build process fails with an out of memory error, you should be able to fix it by increasing the memory used by maven

　　which can be done via the environment variable MAVEN_OPTS.

　　Here is an example setting to allocate between 256 and 512 MB of heap space to Maven

　　export MAVEN_OPTS="-Xms256m -Xmx512m"

 

　　5.替换native库

　　>编译后native路径：Hadoop-2.5.2-src/hadoop-dist/target/hadoop-2.5.2/lib/native

 　　未替换前，调用相关脚本会报警告,如：

[hadoop@localhost hadoop-2.5.2]$ ./sbin/stop-dfs.sh

15/03/22 05:45:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Stopping namenodes on [localhost]

localhost: stopping namenode

localhost: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode

15/03/22 05:45:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

　　编译后，替换掉native目录下文件后，不会再提示警告

[hadoop@localhost hadoop-2.5.2]$ ./sbin/start-dfs.sh

Starting namenodes on [localhost]

localhost: starting namenode, logging to /opt/modules/hadoop-2.5.2/logs/hadoop-hadoop-namenode-localhost.localdomain.out

localhost: starting datanode, logging to /opt/modules/hadoop-2.5.2/logs/hadoop-hadoop-datanode-localhost.localdomain.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /opt/modules/hadoop-2.5.2/logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out

 

 

 

二. 导入源码到eclipse 

  　　1. 安装hadoop-maven-plugins

进入源码下hadoop-maven-plugins目录，执行命令：

$ mvn install

 

　　2. 生成eclipse项目文件

返回源码根目录，执行：

$ mvn eclipse:eclipse -DskipTests

 

3. 导入eclipse

[File] > [Import] > [Existing Projects into Workspace]

 

 　　4. 导入后错误解决

 　　>hadoop-streaming中build path错误:

　　Java Build Path->Source:删除 hadoop-yarn-server-resourcemanager/conf

　　

　　>AvroRecord相关错误

　　a. 下载avro-tools-1.7.4.jar

　　b. 执行命令

$ cd hadoop-2.5.2-src/hadoop-common-project/hadoop-common/src/test/avro

$ java -jar avro-tools所在目录/avro-tools-1.7.4.jar compile schema avroRecord.avsc ../java

　　c. eclipse刷新项目 

　　

　　>protobuf相关错误

　　a. 执行命令

$ cd hadoop-2.5.2-src/hadoop-common-project/hadoop-common/src/test/proto

$ protoc --java_out=../java *.proto

　　b. eclipse刷新项目

 

 

　　附：BUILDING.txt中原文

　　Importing projects to eclipse

　　When you import the project to eclipse, install hadoop-maven-plugins at first.

　　$ cd hadoop-maven-plugins

　　$ mvn install 

　　Then, generate eclipse project files.

　　$ mvn eclipse:eclipse -DskipTests

　　At last, import to eclipse by specifying the root directory of the project via

　　[File] > [Import] > [Existing Projects into Workspace].