Centos7下Hadoop3.x源码编译(Zstd问题已解决)

工具

组件	版本	下载地址
JDK	1.8.0_131	https://www.oracle.com/technetwork/java/javase/downloads/index.html
Hadoop	3.1.1	http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz
Maven	3.6.0	http://mirrors.hust.edu.cn/apache/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz
CMake	3.13.1	https://cmake.org/files/v3.13/cmake-3.13.1.tar.gz
ProtocolBuffer	2.5.0	https://pan.baidu.com/s/179vRCDoK1vp1u7S8NgZD6Q

编译完成生成的软件包

软件包名	下载地址	描述
hadoop-3.1.1.tar.gz	https://pan.baidu.com/s/1UFHq6dxoh5AiqO_cd85NTw	CentOS7.4下hadoop3.1.1源码编译
native-hadoop-3.1.1-with-sztd.tar.gz	https://download.csdn.net/download/weixin_43858859/10832098	hadoop3.1.1源码编译生成的native，含zstd

编译所需要工具和软件包
下载hadoop-3.1.1-src.tar.gz源码包，解压并查看BUILDING.txt

[hadoop@hadoop src]$cat BUILDING.txt
...
Requirements:
* Unix System
* JDK 1.8
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)
...

软件安装

安装JDK
下载jdk-8u131-linux-x64.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/java目录，安装jdk

[root@hadoop ~]# mkdir /usr/local/maven
[root@hadoop ~]#

解压jdk

[hadoop@hadoop src]$ tar -zxvf /home/hadoop/src/jdk-8u131-linux-x64.tar.gz -C /usr/java/

修改属主

[hadoop@hadoop ~]$ chown -R hadoop:hadoop /usr/java

配置环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD java environment
export JAVA_HOME=/usr/java/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证JDK

[hadoop@hadoop ~]$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
[hadoop@hadoop ~]$ 

安装Maven
下载apache-maven-3.6.0-bin.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/maven目录，安装maven

[root@hadoop ~]# mkdir /usr/local/maven  ##创建maven安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/maven  ##修改目录属主

解压maven

[hadoop@hadoop src]$ tar -zxvf apache-maven-3.6.0-bin.tar.gz -C /usr/local/maven/

配置maven环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD maven environment
export MAVEN_HOME=/usr/local/maven/apache-maven-3.6.0
export PATH=$MAVEN_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证Maven

[hadoop@hadoop ~]$ mvn -version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00)
Maven home: /usr/local/maven/apache-maven-3.6.0
Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix"
[hadoop@hadoop ~]$ 

配置国内maven仓库源

[hadoop@hadoop conf]$ vim /usr/local/maven/apache-maven-3.6.0/conf/settings.xml 
...

    
        nexus-aliyun
        central
        Nexus aliyun
     http://maven.aliyun.com/nexus/content/groups/public
    

...

安装工具包（root）
安装gcc make 工具

[root@hadoop ~]# yum install -y gcc* make

安装压缩工具

[root@hadoop ~]# yum -y install snappy*  bzip2* lzo* zlib*  lz4* gzip* 

安装一些基本工具

[root@hadoop ~]# yum -y install openssl* svn ncurses* autoconf automake libtool

安装openssl-devel

[root@hadoop ~]# yum -y install openssl-devel

安装zstd
安装扩展源 epel

[root@hadoop ~]# yum -y install epel-release

安装zstd

[root@hadoop ~]# yum -y install *zstd*

安装CMake
CMake 3.1 或者更新版本，此处我们使用3.13.1（cmake-3.13.1.tar.gz）
下载并上传cmake-3.13.1.tar.gz到/home/hadoop/src目录下
解压cmake

[hadoop@hadoop src]$ tar -zxvf cmake-3.13.1.tar.gz

切换到源码包目录

[hadoop@hadoop src]$ cd cmake-3.13.1

检查依赖

[hadoop@hadoop cmake-3.13.1]$ ./bootstrap

编译安装cmake（root）
使用root用户编译安装

[hadoop@hadoop cmake-3.13.1]$ su root
Password: 
[root@hadoop cmake-3.13.1]# make && make install

查看cmake版本

[hadoop@hadoop ~]$ cmake -version
cmake version 3.13.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[hadoop@hadoop ~]$ 

安装ProtocolBuffer
下载protobuf-2.5.0.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/protobuf目录，安装ProtocolBuffer

[root@hadoop ~]# mkdir /usr/local/protobuf  ##创建ProtocolBuffer安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/protobuf  ##修改目录属主

解压ProtocolBuffer

[hadoop@hadoop src]$ tar -zxvf protobuf-2.5.0.tar.gz -C /home/hadoop/src/

切换到/home/hadoop/src/protobuf-2.5.0目录

[hadoop@hadoop ~]$ cd /home/hadoop/src/protobuf-2.5.0
[hadoop@hadoop protobuf-2.5.0]$ 

检查库
/usr/local/protobuf是protobuf安装目录

[hadoop@hadoop protobuf-2.5.0]$ ./configure --prefix=/usr/local/protobuf

编译安装protobuf

[hadoop@hadoop protobuf-2.5.0]$ make && make install
...
make[3]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[2]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[1]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
[hadoop@hadoop protobuf-2.5.0]$ 

配置protobuf环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD protobuf  environment
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证protobuf

[hadoop@hadoop protobuf]$ protoc --version
libprotoc 2.5.0

编译Hadoop

下载hadoop-3.1.1-src.tar.gz并上传至/home/hadoop/src
解压hadoop源码包

[hadoop@hadoop src]$ tar -zxvf hadoop-3.1.1-src.tar.gz 

进入到hadoop-3.1.1-src目录

[hadoop@hadoop src]$ cd hadoop-3.1.1-src

修改pom文件（42 行）
将hadoop-annotations的值由provide改成compile

[hadoop@hadoop src]$ vim hadoop-common-project/hadoop-auth/pom.xml 
...

    
      
      org.apache.hadoop
      hadoop-annotations
      compile

...

务必保持网络畅通，如果出现缺少某个文件，则要先清理maven(使用命令 mvn clean) 再重新编译。

[root@hadoop hadoop-3.1.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar  ##使用root编译

出现下面结果则成功编译

编译完成后的包文件位置
/home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz

...
[INFO] --- exec-maven-plugin:1.3.1:exec (tar) @ hadoop-dist ---
$ tar cf hadoop-3.1.1.tar hadoop-3.1.1
$ gzip -f hadoop-3.1.1.tar

Hadoop dist tar available at: /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz

[INFO] 
...

hadoop native检查

进入到编译好软件包目录下

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$ 

检查native

[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 18:54:46,459 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 18:54:46,476 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 18:54:46,603 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
2018-12-06 18:54:46,803 INFO util.ExitUtil: Exiting with status 1: ExitException

问题解决(zstd)

安装zstd需要先安装扩展源，否则会报找不到zstd包，如下：

[root@hadoop ~]# yum -y install *zstd*
No package zstd available.
Error: Nothing to do
[root@hadoop ~]# 

没有安装zstd出现的问题
检查native时，缺失zstd

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 07:35:12,808 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 07:35:12,826 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 07:35:12,851 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /home/hadoop/src/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false 
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
2018-12-06 07:35:12,967 INFO util.ExitUtil: Exiting with status 1: ExitException
[hadoop@hadoop bin]$ 

转载：https://blog.csdn.net/weixin_43858859/article/details/84673249