写在前面的话:
hadoop默认下载镜像里面的提供两种选择:源码包,编译好的。比如猫哥为了图方便,从下面链接里直接找了一个编译好的。
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/
下载、配置一路还算可以,只是put什么的就出问题了。报错内容:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
查了查,貌似是32位、64位问题。就是说,官网上提供的是32位机上编译好的。64位系统的话,需要自己编译。
CentOS 6.6
jdk-1.7.0-x86_64
执行如下命令:
yum install autoconfautomake libtool cmake ncurses-devel openssl-devel gcc*
2.1 直接官网下载一个最新版编译好的,比如:http://mirrors.hust.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
2.2 解压缩,放到随便哪个位置。
tar xf apache-maven-3.3.9-bin.tar.gz
mv apache-maven-3.3.9 /data/maven339
2.3 编辑环境变量:vi /etc/profile,在最下面添加如下内容:
export MAVEN_HOME=/letv/maven339
export PATH=$PATH:$MAVEN_HOME/bin:
source /etc/profile使之生效。
2.4 验证:执行如下命令,有结果则说明maven装好了。
mvn --version
记得Protocbuf必须是2.5.0版本。本来猫哥找到一个最新版的,结果编译不过去,查看报错原因,说不是“2.5.0”。也是醉了。搜索一下哪里有这个版本吧。
3.1 安装很简单,依次执行:
./configure --prefix=/usr/local/protoc
make
make check
make install
3.2 编辑环境变量,vi /etc/profile,使之生效:
最懒的办法,直接在最后添加PATH的地方补一段,记得冒号“:”分开就可以了。
export PATH=$PATH:$MAVEN_HOME/bin:/usr/local/protoc/bin
3.3 让环境变量生效
source /etc/profile
4.1 下载源码包:
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/
上面链接,找带“src”那个就可以。
4.2 编译
tar xf hadoop-2.7.2-src.tar.gz
cd hadoop-2.7.2-src
mvn package -Pdist,native -DskipTests -Dtar
mvn过程会很长,20分钟算很幸运的!反正猫哥这么一行命令敲下去,再没报错。
编译好的结果看起来很酷:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.116 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.715 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.838 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.128 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.521 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.044 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.604 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 2.623 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.420 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:34 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.375 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 14.680 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.024 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:30 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 18.526 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 5.891 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.811 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.026 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.023 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [01:34 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 30.626 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.026 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 8.511 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 14.287 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 2.476 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 5.793 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 15.043 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 4.242 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 5.157 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 2.664 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.038 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 1.903 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 1.328 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.021 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 4.723 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 3.674 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.910 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 15.041 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 13.080 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 3.921 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 6.833 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 4.313 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 6.208 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 4.336 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 5.000 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 3.334 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 4.060 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 7.985 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.673 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.210 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.296 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.966 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.728 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.245 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 8.479 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 3.518 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 10.013 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.229 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 7.260 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.053 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.415 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 7.485 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.038 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 31.595 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:09 min
[INFO] Finished at: 2017-03-15T16:50:46+08:00
[INFO] Final Memory: 213M/1044M
[INFO] ------------------------------------------------------------------------
编译结果应该是在 hadoop-2.7.2-src/hadoop-dist/target/hadoop-2.7.2/。
剩下的就是Hadoop配置问题了。