Centos7下Hadoop3.x源码编译(支持zstd压缩方式)

工具安装

组件 版本 下载地址
JDK 1.8.0_131 https://www.oracle.com/technetwork/java/javase/downloads/index.html
Hadoop 3.1.1 http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz
Maven 3.6.0 http://mirrors.hust.edu.cn/apache/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz
CMake 3.13.1 https://cmake.org/files/v3.13/cmake-3.13.1.tar.gz
ProtocolBuffer 2.5.0 https://pan.baidu.com/s/179vRCDoK1vp1u7S8NgZD6Q

编译完成生成的软件包

软件包名 下载地址 描述
hadoop-3.1.1.tar.gz https://pan.baidu.com/s/1UFHq6dxoh5AiqO_cd85NTw CentOS7.4下hadoop3.1.1源码编译
native-hadoop-3.1.1-with-sztd.tar.gz https://download.csdn.net/download/weixin_43858859/10832098 hadoop3.1.1源码编译生成的native,含zstd

编译所需要工具和软件包
下载hadoop-3.1.1-src.tar.gz源码包,解压并查看BUILDING.txt

[hadoop@hadoop src]$cat BUILDING.txt
...
Requirements:
* Unix System
* JDK 1.8
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)
...

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16

软件安装

安装JDK

下载jdk-8u131-linux-x64.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/java目录,安装jdk

[root@hadoop ~]# mkdir /usr/local/maven
[root@hadoop ~]#

 
 
   
   
   
   
  • 1
  • 2

解压jdk

[hadoop@hadoop src]$ tar -zxvf /home/hadoop/src/jdk-8u131-linux-x64.tar.gz -C /usr/java/

 
 
   
   
   
   
  • 1

修改属主

[hadoop@hadoop ~]$ chown -R hadoop:hadoop /usr/java

 
 
   
   
   
   
  • 1

配置环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD java environment
export JAVA_HOME=/usr/java/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

 
 
   
   
   
   
  • 1

验证JDK

[hadoop@hadoop ~]$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
[hadoop@hadoop ~]$ 

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5

安装Maven

下载apache-maven-3.6.0-bin.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/maven目录,安装maven

[root@hadoop ~]# mkdir /usr/local/maven  ##创建maven安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/maven  ##修改目录属主

 
 
   
   
   
   
  • 1
  • 2

解压maven

[hadoop@hadoop src]$ tar -zxvf apache-maven-3.6.0-bin.tar.gz -C /usr/local/maven/

 
 
   
   
   
   
  • 1

配置maven环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD maven environment
export MAVEN_HOME=/usr/local/maven/apache-maven-3.6.0
export PATH=$MAVEN_HOME/bin:$PATH

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

 
 
   
   
   
   
  • 1

验证Maven

[hadoop@hadoop ~]$ mvn -version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00)
Maven home: /usr/local/maven/apache-maven-3.6.0
Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_131/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix"
[hadoop@hadoop ~]$ 

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

配置国内maven仓库源

[hadoop@hadoop conf]$ vim /usr/local/maven/apache-maven-3.6.0/conf/settings.xml 
...
<mirrors>
    <mirror>
        <id>nexus-aliyun</id>
        <mirrorOf>central</mirrorOf>
        <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
    </mirror>
</mirrors>
...

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11

安装工具包(root)

安装gcc make 工具

[root@hadoop ~]# yum install -y gcc* make

 
 
   
   
   
   
  • 1

安装压缩工具

[root@hadoop ~]# yum -y install snappy*  bzip2* lzo* zlib*  lz4* gzip* 

 
 
   
   
   
   
  • 1

安装一些基本工具

[root@hadoop ~]# yum -y install openssl* svn ncurses* autoconf automake libtool

 
 
   
   
   
   
  • 1

安装zstd

安装扩展源 epel

[root@hadoop ~]# yum -y install epel-release

 
 
   
   
   
   
  • 1

安装zstd

[root@hadoop ~]# yum -y install *zstd*

 
 
   
   
   
   
  • 1

安装CMake

CMake 3.1 或者更新版本,此处我们使用3.13.1(cmake-3.13.1.tar.gz)
下载并上传cmake-3.13.1.tar.gz到/home/hadoop/src目录下
解压cmake

[hadoop@hadoop src]$ tar -zxvf cmake-3.13.1.tar.gz

 
 
   
   
   
   
  • 1

切换到源码包目录

[hadoop@hadoop src]$ cd cmake-3.13.1

 
 
   
   
   
   
  • 1

检查依赖

[hadoop@hadoop cmake-3.13.1]$ ./bootstrap

 
 
   
   
   
   
  • 1

编译安装cmake(root)
使用root用户编译安装

[hadoop@hadoop cmake-3.13.1]$ su root
Password: 
[root@hadoop cmake-3.13.1]# make && make install

 
 
   
   
   
   
  • 1
  • 2
  • 3

查看cmake版本

[hadoop@hadoop ~]$ cmake -version
cmake version 3.13.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[hadoop@hadoop ~]$

  • 1
  • 2
  • 3
  • 4
  • 5

安装ProtocolBuffer

下载protobuf-2.5.0.tar.gz并上传至/home/hadoop/src
使用root用户创建/usr/local/protobuf目录,安装ProtocolBuffer

[root@hadoop ~]# mkdir /usr/local/protobuf  ##创建ProtocolBuffer安装目录
[root@hadoop ~]# chown -R hadoop:hadoop /usr/local/protobuf  ##修改目录属主

 
 
   
   
   
   
  • 1
  • 2

解压ProtocolBuffer

[hadoop@hadoop src]$ tar -zxvf protobuf-2.5.0.tar.gz -C /home/hadoop/src/

 
 
   
   
   
   
  • 1

切换到/home/hadoop/src/protobuf-2.5.0目录

[hadoop@hadoop ~]$ cd /home/hadoop/src/protobuf-2.5.0
[hadoop@hadoop protobuf-2.5.0]$ 

 
 
   
   
   
   
  • 1
  • 2

检查库
/usr/local/protobuf是protobuf安装目录

[hadoop@hadoop protobuf-2.5.0]$ ./configure --prefix=/usr/local/protobuf

 
 
   
   
   
   
  • 1

编译安装protobuf

[hadoop@hadoop protobuf-2.5.0]$ make && make install
...
make[3]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[2]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
make[1]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
[hadoop@hadoop protobuf-2.5.0]$ 

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

配置protobuf环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
## ADD protobuf  environment
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$PATH

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

 
 
   
   
   
   
  • 1

验证protobuf

[hadoop@hadoop protobuf]$ protoc --version
libprotoc 2.5.0

 
 
   
   
   
   
  • 1
  • 2

编译Hadoop

下载hadoop-3.1.1-src.tar.gz并上传至/home/hadoop/src
解压hadoop源码包

[hadoop@hadoop src]$ tar -zxvf hadoop-3.1.1-src.tar.gz 

 
 
   
   
   
   
  • 1

进入到hadoop-3.1.1-src目录

[hadoop@hadoop src]$ cd hadoop-3.1.1-src

 
 
   
   
   
   
  • 1

修改pom文件(42 行)
将hadoop-annotations的值由provide改成compile

[hadoop@hadoop src]$ vim hadoop-common-project/hadoop-auth/pom.xml 
...
<dependencies>
    <dependency>
      <!-- Used, even though 'mvn dependency:analyze' doesn't find it -->
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-annotations</artifactId>
      <scope>compile</scope>
</dependency>
...

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10

务必保持网络畅通,如果出现缺少某个文件,则要先清理maven(使用命令 mvn clean) 再重新编译。

[root@hadoop hadoop-3.1.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar  ##使用root编译

 
 
   
   
   
   
  • 1

出现下面结果则成功编译
在这里插入图片描述
编译完成后的包文件位置
/home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz

...
[INFO] --- exec-maven-plugin:1.3.1:exec (tar) @ hadoop-dist ---
$ tar cf hadoop-3.1.1.tar hadoop-3.1.1
$ gzip -f hadoop-3.1.1.tar

Hadoop dist tar available at: /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz

[INFO]
.

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9

hadoop native检查

进入到编译好软件包目录下

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$ 

 
 
   
   
   
   
  • 1
  • 2

检查native

[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 18:54:46,459 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 18:54:46,476 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 18:54:46,603 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  true /lib64/libzstd.so.1
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
2018-12-06 18:54:46,803 INFO util.ExitUtil: Exiting with status 1: ExitException

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14

问题解决(zstd)

安装zstd需要先安装扩展源,否则会报找不到zstd包,如下:

[root@hadoop ~]# yum -y install *zstd*
No package zstd available.
Error: Nothing to do
[root@hadoop ~]# 

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4

没有安装zstd出现的问题
检查native时,缺失zstd

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
[hadoop@hadoop bin]$ ./hadoop checknative -a
2018-12-06 07:35:12,808 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-12-06 07:35:12,826 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-12-06 07:35:12,851 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /home/hadoop/src/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false 
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
2018-12-06 07:35:12,967 INFO util.ExitUtil: Exiting with status 1: ExitException
[hadoop@hadoop bin]$ 

 
 
   
   
   
   
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16

你可能感兴趣的:(Hadoop,hadoop,zstd)