hadoop2.x源码编译

转载请标明出处：
http://blog.csdn.net/zwto1/article/details/50733753；

介绍

本篇主要会涉及以下内容:

1. 学会编译hadoop2.x源码
2. 明白如何将hadoop项目导入eclipse中

编译hadoop源码

为什么要编译

由于hadoop2.x 只支持Linux 32位，针对Linux 64位没有 native lib 库的支持，必须手动编译。所以下载的二进制包，当启动hadoop的时候，会出现如下警告：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

如何编译

在64位Linux下编译
首先打开hadoop2.x源码下的build.txt 查看编译说明：

Requirements:
* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

 Maven main modules:
  hadoop                              (Main Hadoop project)
         -hadoop-project   (Parent POM for all Hadoop Maven modules.        )
                                     (All plugins & dependencies versions are defined here.)
        -hadoop-project-dist (Parent POM for modules that generate distributions.)
        - hadoop-annotations (Generates the Hadoop doclet used to generated the Javadocs)
         - hadoop-assemblies         (Maven assemblies used by the different modules)
         - hadoop-common-project     (Hadoop Common)
         - hadoop-hdfs-project         (Hadoop HDFS)
         - hadoop-mapreduce-project   (Hadoop MapReduce)
         - hadoop-tools               (Hadoop tools like Streaming, Distcp, etc.)
         - hadoop-dist                (Hadoop distribution assembler)

编译选项

Building distributions:
Create binary distribution without native code and without documentation:
  $ mvn package -Pdist -DskipTests -Dtar
Create binary distribution with native code and with documentation:
  $ mvn package -Pdist,native,docs -DskipTests -Dtar
Create source distribution:
  $ mvn package -Psrc -DskipTests
Create source and binary distributions with native code and documentation:
  $ mvn package -Pdist,native,docs,src -DskipTests -Dtar
Create a local staging version of the website (in /tmp/hadoop-site)
  $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

编译前安装依赖包

yum install autoconf automake  libtool cmake
yum install ncurses-devel
yum install openssl-devel
yum install lzo-devel zlib-devel gcc gcc-c++

查软件包是否没有装上：

rpm  -qa | grep autoconf

编译前安装maven：
下载：apache-maven-3.0.5-bin.tar.gz

解压：

tar  -zxvf apache-maven-3.0.5-bin.tar.gz

设置环境变量：

export MAVEN_HOME = /usr/local/maven
export PATH=$PATH:$MAVEN_HOME/bin

执行命令：source /etc/profile或者 . /etc/profile
验证：mvn -v

编译前安装protobuf
下载：protobuf-2.5.0.tar.gz
解压:

tar  -zxvf protobuf-2.5.0.tar.gz

编译安装：
进入安装目录，进行配置，执行命令：

./configure

安装命令：

make & make install

编译前安装findbugs
下载：findbugs-3.0.0.tar.gz
解压:

tar  -zxvf findbugs-3.0.0.tar.gz

设置环境变量：

export FINDBUGS_HOME = /usr/local/findbugs
export PATH=$PATH:$FINDBUGS_HOME/bin

执行命令：source /etc/profile或者 . /etc/profile
验证：findbugs -version

MAVEN国内镜像配置

1、进入安装目录 /opt/modules/apache-maven-3.0.5/conf，编辑 settings.xml 文件

 修改<mirrors>内容：
    <mirror>  
        <id>nexus-osc</id>  
        <mirrorOf>*</mirrorOf>  
        <name>Nexus osc</name>  
        <url>http://maven.oschina.net/content/groups/public/</url>  
    </mirror> 

修改<profiles>内容：

<profile>  
    <id>jdk-1.8</id>  
    <activation>  
        <jdk>1.8</jdk>  
    </activation>  
    <repositories>  
        <repository>  
            <id>nexus</id>  
            <name>local private nexus</name>  
            <url>http://maven.oschina.net/content/groups/public/</url>  
            <releases>  
                <enabled>true</enabled>  
            </releases>  
            <snapshots>  
                <enabled>false</enabled>  
            </snapshots>  
        </repository>  
    </repositories> 
    <pluginRepositories>  
        <pluginRepository>  
            <id>nexus</id>  
            <name>local private nexus</name>  
            <url>http://maven.oschina.net/content/groups/public/</url>  
            <releases>  
                <enabled>true</enabled>  
            </releases>  
            <snapshots>  
                <enabled>false</enabled>  
            </snapshots>  
        </pluginRepository>  
    </pluginRepositories>  

2、复制配置
将该配置文件复制到用户目录，使得每次对maven创建时，都采用该配置
* 查看用户目录【/home/hadoop】是否存在【.m2】文件夹，如没有，则创建

$ cd /home/hadoop  
$ mkdir .m2
* 复制文件
$ cp /opt/modules/apache-maven-3.0.5/conf/settings.xml ~/.m2/

3.配置DNS
修改： vi /etc/resolv.conf

nameserver 8.8.8.8
nameserver 8.8.4.4

开始编译
Hadoop2.2 .0 有bug ，编译 之前需要先 fix bug
下载 hadoop2.x源码
解压
源码打patch,编辑目录$HADOOP_SRC_HOME/hadoop-common-project/hadoop-auth中的文件pom.xml,在第55行下增加以下内容：

<dependency>
   <groupId>org.mortbay.jetty</groupId>
    <artfactId>jetty-util<artfactId>
    <scope>test</scope>
  </dependency>

这里编译的是hadoop2.7.1版本

mvn package –DskipTests –Pdist,native,docs -Dtar

报错

1.编译出错的话，如下有解决方式：

export MAVEN_OPTS=”-Xms256m -Xmx512m”

2.报错：java heap size

把虚拟机内存调大，然后重启虚拟机，重新编译

3.报错还有可能是maven下载jar的时候，网络出现问题，重新编译

CentOS编译成功界面：

上面是Cetos下的编译，ubuntu编译：

首先要安装的软件：
JDk
Maven
Findbugs
G++
Protobuf
Openssl
CMake
Ant

Ubuntu编译成功界面：

将hadoop项目导入eclipse中

Importing projects to eclipse
When you import the project to eclipse, install hadoop-maven-plugins at first.
  $ cd hadoop-maven-plugins
  $ mvn install
Then, generate eclipse project files.
  $ mvn eclipse:eclipse -DskipTests
At last, import to eclipse by specifying the root directory of the project via
[File] > [Import] > [Existing Projects into Workspace].

导入eclipse没有导，步骤如上面官方文档，遇到的问题稍后补上

未完待续 ….