Centos7下Hadoop3.x源码编译（支持zstd压缩方式）
工具安装
| 组件 | 版本 | 下载地址 |
|---|---|---|
| JDK | 1.8.0_131 | https://www.oracle.com/technetwork/java/javase/downloads/index.html |
| Hadoop | 3.1.1 | http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz |
| Maven | 3.6.0 | http://mirrors.hust.edu.cn/apache/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz |
| CMake | 3.13.1 | https://cmake.org/files/v3.13/cmake-3.13.1.tar.gz |
| ProtocolBuffer | 2.5.0 | https://pan.baidu.com/s/179vRCDoK1vp1u7S8NgZD6Q |
编译完成生成的软件包
| 软件包名 | 下载地址 | 描述 |
|---|---|---|
| hadoop-3.1.1.tar.gz | https://pan.baidu.com/s/1UFHq6dxoh5AiqO_cd85NTw | CentOS7.4下hadoop3.1.1源码编译 |
| native-hadoop-3.1.1-with-sztd.tar.gz | https://download.csdn.net/download/weixin_43858859/10832098 | hadoop3.1.1源码编译生成的native，含zstd |
编译所需要工具和软件包
下载hadoop-3.1.1-src.tar.gz源码包，解压并查看BUILDING.txt
[hadoop@hadoop src]$cat BUILDING.txt
...
Requirements:
* Unix System
* JDK 1.8
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)
...
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
软件安装
安装JDK
下载jdk-8u131-linux-x64.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/java目录，安装jdk
[root@hadoop ~]# mkdir /usr/local/maven
[root@hadoop ~]#
- 1
- 2
解压jdk
[hadoop@hadoop src]$ tar -zxvf /home/hadoop/src/jdk-8u131-linux-x64.tar.gz -C /usr/java/
- 1
修改属主
[hadoop@hadoop ~]$ chown -R hadoop:hadoop /usr/java
- 1
配置环境变量
[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD java environment
export JAVA_HOME=/usr/java/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH
- 1
- 2
- 3
- 4
source使环境变量生效
[hadoop@hadoop ~]$ source /etc/profile
- 1
验证JDK
[hadoop@hadoop ~]$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
[hadoop@hadoop ~]$
- 1
- 2
- 3
- 4
- 5
安装Maven
下载apache-maven-3.6.0-bin.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/maven目录，安装maven
[root@hadoop ~]# mkdir /usr/local/maven ##创建maven安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/maven ##修改目录属主
- 1
- 2
解压maven
[hadoop@hadoop src]$ tar -zxvf apache-maven-3.6.0-bin.tar.gz -C /usr/local/maven/
- 1
配置maven环境变量
[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD maven environment
export MAVEN_HOME=/usr/local/maven/apache-maven-3.6.0
export PATH=$MAVEN_HOME/bin:$PATH
- 1
- 2
- 3
- 4
source使环境变量生效
[hadoop@hadoop ~]$ source /etc/profile
- 1
验证Maven
[hadoop@hadoop ~]$ mvn -version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00)
Maven home: /usr/local/maven/apache-maven-3.6.0
Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix"
[hadoop@hadoop ~]$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
配置国内maven仓库源
[hadoop@hadoop conf]$ vim /usr/local/maven/apache-maven-3.6.0/conf/settings.xml
...
<mirrors>
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>central</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
</mirrors>
...
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
安装工具包（root）
安装gcc make 工具
[root@hadoop ~]# yum install -y gcc* make
- 1
安装压缩工具
[root@hadoop ~]# yum -y install snappy* bzip2* lzo* zlib* lz4* gzip*
- 1
安装一些基本工具
[root@hadoop ~]# yum -y install openssl* svn ncurses* autoconf automake libtool
- 1
安装zstd
安装扩展源 epel
[root@hadoop ~]# yum -y install epel-release
- 1
安装zstd
[root@hadoop ~]# yum -y install *zstd*
- 1
安装CMake
CMake 3.1 或者更新版本，此处我们使用3.13.1（cmake-3.13.1.tar.gz）
下载并上传cmake-3.13.1.tar.gz到/home/hadoop/src目录下
解压cmake
[hadoop@hadoop src]$ tar -zxvf cmake-3.13.1.tar.gz
- 1
切换到源码包目录
[hadoop@hadoop src]$ cd cmake-3.13.1
- 1
检查依赖
[hadoop@hadoop cmake-3.13.1]$ ./bootstrap
- 1
编译安装cmake（root）
使用root用户编译安装
[hadoop@hadoop cmake-3.13.1]$ su root
Password:
[root@hadoop cmake-3.13.1]# make && make install
- 1
- 2
- 3
查看cmake版本
[hadoop@hadoop ~]$ cmake -version
cmake version 3.13.1
CMake suite maintained and supported by Kitware (kitware.com/cmake).
[hadoop@hadoop ~]$
- 1
- 2
- 3
- 4
- 5
安装ProtocolBuffer
下载protobuf-2.5.0.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/protobuf目录，安装ProtocolBuffer
[root@hadoop ~]# mkdir /usr/local/protobuf ##创建ProtocolBuffer安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/protobuf ##修改目录属主
- 1
- 2
解压ProtocolBuffer
[hadoop@hadoop src]$ tar -zxvf protobuf-2.5.0.tar.gz -C /home/hadoop/src/
- 1
切换到/home/hadoop/src/protobuf-2.5.0目录
[hadoop@hadoop ~]$ cd /home/hadoop/src/protobuf-2.5.0
[hadoop@hadoop protobuf-2.5.0]$
- 1
- 2
检查库
/usr/local/protobuf是protobuf安装目录
[hadoop@hadoop protobuf-2.5.0]$ ./configure --prefix=/usr/local/protobuf
- 1
编译安装protobuf
[hadoop@hadoop protobuf-2.5.0]$ make && make install
...
make[3]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[2]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[1]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
[hadoop@hadoop protobuf-2.5.0]$
- 1
- 2
- 3
- 4
- 5
- 6
配置protobuf环境变量
[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD protobuf environment
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$PATH
- 1
- 2
- 3
- 4
source使环境变量生效
[hadoop@hadoop ~]$ source /etc/profile
- 1
验证protobuf
[hadoop@hadoop protobuf]$ protoc --version
libprotoc 2.5.0
- 1
- 2
编译Hadoop
下载hadoop-3.1.1-src.tar.gz并上传至/home/hadoop/src
解压hadoop源码包
[hadoop@hadoop src]$ tar -zxvf hadoop-3.1.1-src.tar.gz
- 1
进入到hadoop-3.1.1-src目录
[hadoop@hadoop src]$ cd hadoop-3.1.1-src
- 1
修改pom文件（42 行）
将hadoop-annotations的值由provide改成compile
[hadoop@hadoop src]$ vim hadoop-common-project/hadoop-auth/pom.xml
...
<dependencies>
<dependency>
<!-- Used, even though 'mvn dependency:analyze' doesn't find it -->
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
<scope>compile</scope>
</dependency>
...
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
务必保持网络畅通，如果出现缺少某个文件，则要先清理maven(使用命令 mvn clean) 再重新编译。
[root@hadoop hadoop-3.1.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar ##使用root编译
- 1
出现下面结果则成功编译
编译完成后的包文件位置
/home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz
...
[INFO] --- exec-maven-plugin:1.3.1:exec (tar) @ hadoop-dist ---
$ tar cf hadoop-3.1.1.tar hadoop-3.1.1
$ gzip -f hadoop-3.1.1.tar
Hadoop dist tar available at: /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz
[INFO]
….
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
hadoop native检查
进入到编译好软件包目录下
[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$
- 1
- 2
检查native
[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 18:54:46,459 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 18:54:46,476 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 18:54:46,603 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop: true /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
zstd : true /lib64/libzstd.so.1
snappy: true /lib64/libsnappy.so.1
lz4: true revision:10301
bzip2: true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L: false libhadoop was built without ISA-L support
2018-12-06 18:54:46,803 INFO util.ExitUtil: Exiting with status 1: ExitException
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
问题解决(zstd)
安装zstd需要先安装扩展源，否则会报找不到zstd包，如下：
[root@hadoop ~]# yum -y install *zstd*
No package zstd available.
Error: Nothing to do
[root@hadoop ~]#
- 1
- 2
- 3
- 4
没有安装zstd出现的问题
检查native时，缺失zstd
[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 07:35:12,808 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 07:35:12,826 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 07:35:12,851 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop: true /home/hadoop/src/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
zstd : false
snappy: true /lib64/libsnappy.so.1
lz4: true revision:10301
bzip2: true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L: false libhadoop was built without ISA-L support
2018-12-06 07:35:12,967 INFO util.ExitUtil: Exiting with status 1: ExitException
[hadoop@hadoop bin]$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16