Hadoop源码学习-编译源码


编译源码非常简单,步骤如下:

  1. 下载源代码
  2. 安装依赖软件或库
  3. 编译打包

下载源码

官方下载并解压,过程略

查看编译要求

切换到解压后的hadoop源码根目录
Hadoop源码学习-编译源码_第1张图片

此目录下有个BUILDING.txt文件,里面有对building的详细说明,以Linux为例,要求如下:

  • Unix System
  • JDK 1.7+
  • Maven 3.0 or later
  • Findbugs 1.3.9 (if running findbugs)
  • ProtocolBuffer 2.5.0
  • CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
  • Zlib devel (if compiling native code)
  • openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
  • Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
  • Internet connection for first build (to fetch all Maven and Hadoop dependencies)

以上并不是每个都需要安装,看你编译的要求,我选择的在编译的时候,创建源码和二进制包,以及文档,我的环境如下:
* CentOS7
* JDK 1.8
* Maven 3.0.5
* Findbugs 1.3.9
* ProtocolBuffer 2.5.0
* CMake 3.6.3
* Zlib devel
* openssl devel
检查一下自己的系统,哪些安装过,哪些没有安装,安装之后需要可以直接在终端执行命令,所以,如果是那些网上手动下载解压的软件,需要对其配置PATH变量。

以上软件或依赖,请自行全部安装,否则,在build的时候会报错,以下是部分软件安装过程:

安装Protoc
下载地址:
http://pan.baidu.com/s/1pJlZubT

解压,切换到根目录,安装命令如下:

./configure
make && make install

默认情况下,protoco是安装在/usr/local/bin',/usr/local/man’下,但也可以指定安装到某个路径下面,例如,我要把protobuf安装在/home/yang/Soft/protobuf/目录下,则使用配置项prefix,如下:

 ./configure --prefix=/home/yang/Soft/protobuf/
 make && make install

配置环境变量:

[yang@master bin]$ vim ~/.bashrc 
export PATH=$PATH:/home/yang/Soft/protobuf/bin

[yang@master bin]$ source ~/.bashrc
[yang@master ~]$ protoc --version
libprotoc 2.5.0

安装Cmake(我之前安装过,只是没有配置PATH,这里只是配置一下环境变量)

export  PATH=$PATH:/home/yang/GitHub/json2avro/:/home/yang/Soft/protobuf/bin:/home/yang/Soft/cmake/cmake-3.6.3-Linux-x86_64/bin

安装findbugs
下载地址:https://sourceforge.net/projects/findbugs/files/findbugs/1.3.9/findbugs-1.3.9.tar.gz/download
这个也要配置环境变量
安装zlib

yum install zlib-devel.x86_64

安装openssl-devel

yum install openssl-devel.x86_64

编译打包

准备工作做完之后,切换到源码根目录,执行编译打包命令:

mvn package -Pdist,native,docs -DskipTests -Dtar

这个过程,时间会比较久,如果编译成功,有如下界面:
Hadoop源码学习-编译源码_第2张图片

BUILD SUCCESS
Congratulations

Exception

java.io.IOException: Cannot run program “cmake”

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory

解决方法:
安装Cmake

hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn’t exist

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (site) on project hadoop-common: An Ant BuildException has occured: stylesheet /home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn't exist.
[ERROR] around Ant part ...in="/home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/findbugsXml.xml" style="${env.FINDBUGS_HOME}/src/xsl/default.xsl" out="/home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/site/findbugs.html"/>... @ 43:261 in /home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml

编辑.bashrc文件,设置FINDBUGS_HOME

export FINDBUGS_HOME=/home/yang/Soft/findbugs-1.3.9

你可能感兴趣的:(Java,Linux,Hadoop)