Hadoop2.6.0源码编译支持Snappy压缩

环境准备

• 系统:Centos6.5 64位
• JDK1.7 因为pom文件里面很多注明了JDK1.7，所以最好用JDK1.7编译
• Maven：3.6.0
• Hadoop源码：Hadoop2.6.0-cdh5.7.0.src.tar.gz
• Protocol Buffer 2.5.0
• Jar依赖包

---

安装Protocol Buffer 2.5.0

• root权限安装依赖包

# yum install -y gcc gcc-c++ make cmake

• 指定安装目录

# ./configure --prefix=/home/hadoop/app/protobuf-2.5.0

• 开始安装

# make && make install

• 配置更改系统环境变量

# vi ~/.bash_profile

# export PROTOC_HOME=/home/hadoop/app/protobuf-2.5.0
# export PATH=$PROTOC_HOME/bin:$PATH

# source ~/.bash_profile

• 验证安装

# protoc --version

---

切换root用户安装其他类库

# yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
# yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

---

• 解压Jar依赖包

# tar -zxvf repo.tar.gz -C ~/repo/

---

• 更改Maven仓库为阿里Maven仓库

<mirror>
	<id>alimavenid>
	<name>aliyun mavenname>
	<url>http://maven.aliyun.com/nexus/content/groups/public/url>
	<mirrorOf>centralmirrorOf>
mirror>

• 仓库地址指向repo目录
  

---

• 解压Hadoop源码，并进入解压源码的根目录，执行编译命令
  

$ tar -zxvf 
$ mvn clean package -Pdist,native -DskipTests -Dtar

[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/hadoop-dist-2.6.0-cdh5.7.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.6.0-cdh5.7.0:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.739 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.752 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.330 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.358 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.494 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.868 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  3.347 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  3.433 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.263 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:14 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  6.671 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 12.120 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.023 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:11 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 21.243 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  7.253 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.534 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.042 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.038 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 59.239 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 19.573 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.037 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  7.043 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 13.277 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.277 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  4.412 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 12.304 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  1.268 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  4.144 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.027 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  1.841 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.774 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.028 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  3.980 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  3.694 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.098 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 15.339 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 13.593 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.875 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  6.275 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  4.309 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  4.149 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.476 s]
[INFO] hadoop-mapreduce-client-nativetask ................. SUCCESS [ 54.730 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.152 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  3.763 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  5.388 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.731 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  5.601 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.641 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.735 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.845 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.034 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.570 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.004 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  4.821 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.722 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  4.530 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.333 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  3.644 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  2.320 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.090 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  5.750 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.031 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:16 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11:04 min
[INFO] Finished at: 2019-01-12T22:16:28+08:00
[INFO] ------------------------------------------------------------------------

• 至此编译成功，编译好的hadoop安装包存放目录如下

[hadoop@192 target]$ pwd
/home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target

• 安装完编译好的Hadoop后检查是否已支持Snappy

[hadoop@192 hadoop-2.6.0-cdh5.7.0]$ hadoop checknative -a
19/01/12 23:13:21 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
19/01/12 23:13:21 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/lib64/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

---

编译报错解决

以下错误是因为没有更改阿里的Maven仓库造成，按照本文更改阿里Maven仓库即可

some problems were encountered while processing the poms:
[error] unresolveable build extension: 
plugin org.apache.felix:maven-bundle-plugin:2.4.0 or one of its dependencies could not be resolved: 
failed to read artifact descriptor for org.apache.felix:maven-bundle-plugin:jar:2.4.0

---

• 如图示无法下载apache-tomcat-6.0.44.tar.gz

---

[hadoop@192 conf]$ cd /home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads
[hadoop@192 downloads]$ pwd
/home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads
[hadoop@192 downloads]$ wget http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/bin/apache-tomcat-6.0.44.tar.gz

• 重新执行编译即可

---