0.安装JDK,使用 java -version 查看jdk版本,确定JDK版本是64位。
a. 解压jdk
$ tar -xvzf jdk-7u60-linux-x64.tar.gz
b. 设置环境变量 vim /etc/profile
export JAVA_HOME=/usr/local/jdk1.7 export HADOOP_HOME=/usr/local/hadoop-2.6.0
export MAVEN_HOME=/opt/apache-maven
export FINDBUG_HOME=/opt/findbugs-3.0.0
export ANT_HOME=/opt/apache-ant-1.9.4
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
:$MAVEN_HOME/bin:$FINDBUG_HOME/bin:$ANT_HOME/bin (备注:不能换行)
c.使配置文件生效
$ source /etc/profile
1.安装gcc|gc++
yum install gcc
yum install gcc-c++
验证
2.安装Apache-Maven。
tar -zxvf apache-maven-3.2.1.tar.gz
配置环境变量 vim /etc/profile
export MAVEN_HOME=/opt/apache-maven
export PATH=.:PATH:$MAVEN_HOME/bin
验证: mvn --version
3.安装Apache-ant(重要).
tar -zxvf apache-ant-1.9.4-bin.tar.gz
配置环境变量 vim /etc/profile
export MAVEN_HOME=/opt/apache-maven
export PATH=$PATH:$
MAVEN_HOME/bin
3.安装protobuf(goole序列化工具)
tar -zxvf protobuf-2.5.0.tar.gz
./configuration
make #编译
make install
验证:protoc --version
4.安装CMake2.6 or newer
安装 yum install cmake
安装 yum install openssl-devel
安装 yum install ncurses-devel
验证:cmake --version
5.安装make
yum install make
验证: make --version
6.Hadoop - hadoop-common-project中的pom.xml添加依赖(hadoop-2.2.0需要修改,hadoop2.6.0版本不需要)
org.mortbay.jetty
test
7.在编译之前防止 java.lang.OutOfMemoryError: Java heap space 堆栈问题,在centos系统中执行命令:
$ export MAVEN_OPTS="-Xms256m -Xmx512m"
8.解压压缩包
tar -zxvf hadoop-2.6.0-src.tar.gz
a.执行命令 cd ${hostname_Local}/hadoop-2.6.0/ 目录下
b.编译
mvn package -DskipTests -Pdist,native
c.
编译好的项目放在 hadoop-2.6.0-src/hadoop-dist/target目录下。
/root/Downloads/hadoop-2.6.0-src/hadoop-dist/target 即
hadoop-2.6.0就是编译好的包。
=====================================================================
编译日志:
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-dist --- [WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /root/Downloads/hadoop-2.6.0-src/hadoop-dist/target/hadoop-dist-2.6.0.jar [INFO]
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO] No sources in project. Archive not created. [INFO]
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO] No sources in project. Archive not created. [INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-dist --- [INFO]
[INFO] --- maven-antrun-plugin:1.7:run (tar) @ hadoop-dist ---
[INFO] Executing tasks main: [INFO] Executed tasks [INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /root/Downloads/hadoop-2.6.0-src/hadoop-dist/target/hadoop-dist-2.6.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 13.582 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 9.846 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 24.408 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 1.967 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 6.443 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 20.692 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 14.250 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 23.716 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 13.714 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [08:46 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 47.127 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 48.790 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.316 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [14:58 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [11:10 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [01:43 min]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 27.438 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.146 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.165 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [07:03 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [03:31 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.827 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [01:11 min]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [02:25 min]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 17.129 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 39.350 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [01:44 min]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 32.941 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 44.664 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.197 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 15.165 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 9.604 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.149 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 31.971 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 22.195 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.673 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [02:08 min]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [01:38 min]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 24.796 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [01:02 min]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 43.043 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [01:09 min]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 9.662 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 40.439 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 13.894 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 32.797 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [01:00 min]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 11.333 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 35.122 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 22.939 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 17.568 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 12.339 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 18.325 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 27.889 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 30.148 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [01:28 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 25.086 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.657 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 25.302 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 23.268 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.156 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:06 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:17 h [INFO] Finished at: 2014-12-29T20:45:54-08:00
[INFO] Final Memory: 94M/193M [INFO] ------------------------------------------------------------------------
[root@Master hadoop-2.6.0-src]#
1.在hadoop2.0中通常由两个NameNode组成,一个处于active状态,另一个处于standby状态。Active NameNode对外提供服务,而Standby NameNode则不对外提供服务,仅同步active namenode的状态,以便能够在它失败时快速进行切换。
hadoop2.0官方提供了两种HDFS HA的解决方案,一种是NFS,另一种是QJM。这里我们使用简单的QJM。在该方案中,主备NameNode之间通过一组JournalNode同步元数据信息,一条数据只要成功写入多数JournalNode即认为写入成功。通常配置奇数个JournalNode
这里还配置了一个zookeeper集群,用于ZKFC(DFSZKFailoverController)故障转移,当Active NameNode挂掉了,会自动切换Standby NameNode为standby状态
2.hadoop-2.2.0中依然存在一个问题,就是ResourceManager只有一个,存在单点故障,hadoop-2.4.1解决了这个问题,有两个ResourceManager,一个是Active,一个是Standby,状态由zookeeper进行协调