大数据Hadoop篇：源码编译

一、环境配置

1. CentOS联网
   配置CentOS能连接外网。Linux虚拟机ping www.baidu.com 是畅通的
   注意：采用root角色编译，减少文件夹权限出现问题
2. jar包准备(hadoop源码、JDK8、maven、ant 、protobuf)
   （1）hadoop-2.7.2-src.tar.gz
   （2）jdk-8u144-linux-x64.tar.gz
   （3）apache-ant-1.9.9-bin.tar.gz（build工具，打包用的）
   （4）apache-maven-3.0.5-bin.tar.gz
   （5）protobuf-2.5.0.tar.gz（序列化的框架）
   把以上文件用winscp传到/usr/local/software文件夹下

1.1、 jar包安装

注意：所有操作必须在root用户下完成

1. JDK解压、配置环境变量 JAVA_HOME和PATH，验证java-version(如下都需要验证是否配置成功)

 tar -zxf jdk-8u144-linux-x64.tar.gz -C /usr/local/

 vi ~/.bashrc
#JAVA_HOME：
export JAVA_HOME=/usr/local/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin

source ~/.bashrc

验证命令：java -version

1.2.、Maven解压、配置 MAVEN_HOME和PATH

1.2.1、解压压缩包

tar -zxvf apache-maven-3.0.5-bin.tar.gz -C /usr/local/

1.2.2、修改配置文件

vi conf/settings.xml

Maven镜像改为阿里云

    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |

       mirrorId</id>
       repositoryId</mirrorOf>
       Human Readable Name for this Mirror.</name>
       http://my.repository.com/repo/path</url>
      </mirror>
     -->
        
                nexus-aliyun</id>
                central</mirrorOf>
                Nexus aliyun</name>
                http://maven.aliyun.com/nexus/content/groups/public</url>
        </mirror>
</mirrors>

1.2.3、配置环境变量

 vi ~/.bashrc
#MAVEN_HOME
export MAVEN_HOME=/usr/local/maven
export PATH=:$MAVEN_HOME/bin

source ~/.bashrc

验证命令：mvn -version

1.3、ant解压、配置 ANT _HOME和PATH

1.3.1、解压ant

tar -zxvf apache-ant-1.9.9-bin.tar.gz -C /usr/local

1.3.2、配置环境变量

vi ~/.bashrc
#ANT_HOME
export ANT_HOME=/usr/local/ant
export PATH=:$ANT_HOME/bin

source ~/.bashrc

验证命令：ant -version

1.4、安装 glibc-headers 和 g++ 命令如下

yum install glibc-headers
yum install gcc-c++

安装make和cmake

yum install make
yum install cmake

解压protobuf ，进入到解压后protobuf主目录，/usr/local/protobuf，然后相继执行命令

tar -zxvf protobuf-2.5.0.tar.gz -C /usr/local/

分别执行下面每行命令

cd /usr/local/protobuf
./configure 
make 
make check 
make install 
ldconfig 

配置环境变量

vi ~/.bashrc
#LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/protobuf
export PATH=:$LD_LIBRARY_PATH

source ~/.bashrc

验证命令：protoc --version

安装openssl库yum install openssl-devel
安装 ncurses-devel库yum install ncurses-devel
到此，编译工具安装基本完成。

二、编译源码

1. 解压源码到/usr/local/software/hadoop_z/目录

tar -zxvf hadoop-2.7.2-src.tar.gz  -C /usr/local/software/hadoop_z

2. 进入到hadoop源码主目录

cd /usr/local/software/hadoop_z/hadoop-2.7.2-src

3. 通过maven执行编译命令(需要连网)

mvn package -Pdist,native -DskipTests -Dtar

漫长的等待，最终成功是全部SUCCESS

4. 成功的64位hadoop包在hadoop-2.7.2-src/hadoop-dist/target下

三、 编译源码过程中遇到的问题

（1）MAVEN install时候JVM内存溢出
处理方式：在环境配置文件和maven的执行文件均可调整MAVEN_OPT的heap大小。（详情查阅MAVEN 编译 JVM调优问题）
（2）编译期间maven报错。可能网络阻塞问题导致依赖库下载不完整导致，多次执行命令（一次通过比较难）：
mvn package -Pdist,nativeN -DskipTests -Dtar
（3）报ant、protobuf等错误，插件下载未完整或者插件版本问题，最开始链接有较多特殊情况，同时推荐
版本的问题汇总帖子

3.1、内存不足

在Building Apache Hadoop HDFS时报错

Compiling 678 source files to /usr/local/hadoopsrc/hadoop-2.7.2-src/hadoop-hdfs-project/hadoop-hdfs/target/classes

解决方法：
找到maven的目录 可以通过mvn -v 看到maven的目录

vi /bin/mvn

添加

MAVEN_OPTS="$MAVEN_OPTS -Xms256m -Xmx1024m -XX:MaxPermSize=128m -XX:ReservedCodeCacheSize=64m"

3.2、下载的jar包有问题

错误原因：下载的jar包有问题（可能不完整），所以报“error in opening zip file”

解决办法：从本机将对应的jar包下载来，覆盖linux上maven库中有问题的jar包。

查看 hadoop-common-project/hadoop-kms/target/antrun/build-main.xml 文件中使用的apache-tomcat版本号，并下载（文件中有地址），然后放到目录 hadoop2.6.4/hadoop-common-project/hadoop-kms/downloads下

声明：文章内容来自尚硅谷，B站有免费视屏。

[INFO] Executing tasks

main:
      [get] Destination already exists (skipping): /usr/local/software/hadoop_z/hadoop-2.7.2-src/hadoop-common-project/hadoop-kms/downloads/apache-tomcat-6.0.44.tar.gz
   [delete] Deleting directory /usr/local/software/hadoop_z/hadoop-2.7.2-src/hadoop-common-project/hadoop-kms/target/tomcat.exp
    [mkdir] Created dir: /usr/local/software/hadoop_z/hadoop-2.7.2-src/hadoop-common-project/hadoop-kms/target/tomcat.exp
     [exec] 
     [exec] gzip: ../../downloads/apache-tomcat-6.0.44.tar.gz: unexpected end of file
     [exec] tar: Unexpected EOF in archive
     [exec] tar: Unexpected EOF in archive
     [exec] tar: Error is not recoverable: exiting now
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................ SUCCESS [0.700s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [0.503s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [1.120s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.066s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [1.301s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [1.344s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [1.043s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [1.328s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [1.581s]
[INFO] Apache Hadoop Common .............................. SUCCESS [1:10.754s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [2.340s]
[INFO] Apache Hadoop KMS ................................. FAILURE [1.685s]
[INFO] Apache Hadoop Common Project ...................... SKIPPED
[INFO] Apache Hadoop HDFS ................................ SKIPPED
[INFO] Apache Hadoop HttpFS .............................. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] hadoop-yarn ....................................... SKIPPED
[INFO] hadoop-yarn-api ................................... SKIPPED
[INFO] hadoop-yarn-common ................................ SKIPPED
[INFO] hadoop-yarn-server ................................ SKIPPED
[INFO] hadoop-yarn-server-common ......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-yarn-client ................................ SKIPPED
[INFO] hadoop-yarn-server-sharedcachemanager ............. SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-yarn-registry .............................. SKIPPED
[INFO] hadoop-yarn-project ............................... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Gridmix ............................. SKIPPED
[INFO] Apache Hadoop Data Join ........................... SKIPPED
[INFO] Apache Hadoop Ant Tasks ........................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Pipes ............................... SKIPPED
[INFO] Apache Hadoop OpenStack support ................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support ......... SKIPPED
[INFO] Apache Hadoop Azure support ....................... SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:24.732s
[INFO] Finished at: Thu Jul 02 05:15:21 CST 2020
[INFO] Final Memory: 56M/230M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec dir="/usr/local/software/hadoop_z/hadoop-2.7.2-src/hadoop-common-project/hadoop-kms/target" executable="sh" failonerror="true">... @ 10:136 in /usr/local/software/hadoop_z/hadoop-2.7.2-src/hadoop-common-project/hadoop-kms/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-kms
You have new mail in /var/spool/mail/root