Hadoop2.2.0(yarn)编译部署手册

Created on 2014-3-30
URL : http://www.cnblogs.com/zhxfl/p/3633919.html
@author: zhxfl

 

 

Hadoop-2.2编译

由于Hadoop-2.2只发布了32位的版本，所以如果是在64位操作系统上，需要重新编译

安装 maven

安装maven，查找其安装目录

sudo apt-get intall maven

Find /usr -name “*maven*” 

根据其安装目录添加环境变量

export M2_HOME=/usr/shared/maven

export PATH=$PATH:$M2_HOME/bin 

export MAVEN_OPTS="-Xms256m -Xmx512m"

安装 google protobuf

wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz



/configure--prefix=/usr/local/protobuf   

sudo make && sudo make install

sudo vim /etc/ld.so.conf   [添加/usr/local/lib]

ldconfig

安装Cmake

sudo apt-get install cmake

安装依赖库

libglib2.0-dev libssl-dev

pom.xml 中有个bug，添加下面patch即可

参考 https://issues.apache.org/jira/browse/HADOOP-10110

 

Index: hadoop-common-project/hadoop-auth/pom.xml



===================================================================



--- hadoop-common-project/hadoop-auth/pom.xml     (revision 1543124)



+++ hadoop-common-project/hadoop-auth/pom.xml   (working copy)



@@ -54,6 +54,11 @@



     </dependency>



     <dependency>



       <groupId>org.mortbay.jetty</groupId>



+      <artifactId>jetty-util</artifactId>



+      <scope>test</scope>



+    </dependency>



+    <dependency>



+      <groupId>org.mortbay.jetty</groupId>



       <artifactId>jetty</artifactId>



       <scope>test</scope>



     </dependency>

开始编译：

mvn package -Pdist,native -DskipTests -Dtar 

 

常见错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-hdfs: An Ant BuildException has occured: exec returned: 1 -> [Help 1]

安装libglib2.0-dev

 

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1 -> [Help 1]

安装libssl-dev

 

[ERROR] /home/yarn/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/j                                                                                        ava/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:                                                                                        [86,13] cannot access org.mortbay.component.AbstractLifeCycle

 

参考 https://issues.apache.org/jira/browse/HADOOP-10110

 

最后在目录~/hadoop-2.2.0-src/hadoop-dist/target中有一个hadoop-2.2.0的目录就是编译出来的版本

 

Hadoop-2.2环境配置

添加用户

每个节点都添加yarn用户

 

添加用户

sudo adduser yarn

 

把用户添加到hadoop组中（如果你没有一个hadoop的组，需要新建这个组）

sudo gpasswd -a yarn hadoop

 

给yarn用户添加sudo权限

sudo vim /etc/sudoers

 

添加如下语句

yarn ALL=(ALL:ALL) ALL

 

ssh配置

针对master

sudo apt-get install openssh-server

ssh-keygen(一直按enter即可)

在~/.ssh目录下，有一个id_rsa（私钥），一个id_rsa.pub（公钥）

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

输入ssh localhost确认是否需要输入密码

 

拷贝authorized_keys 到slave1~slave3节点

 

scp authorized_keys yarn@slave1:~/.ssh/

scp authorized_keys yarn@slave2:~/.ssh/

scp authorized_keys yarn@slave3:~/.ssh/

 

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/

针对salves

都要执行ssh-keygen

最后在master节点上使用ssh slave1等测试

 

其他

每个节点上面都需要添加如下ip

vim /etc/hosts

219.219.216.48 master

219.219.216.47 slave1

219.219.216.45 slave2

219.219.216.46 slave3

参考

http://dongxicheng.org/mapreduce-nextgen/hadoop-yarn-install/

 

配置文件

hadoop-env.sh

添加JAVA_HOME环境变量

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/

Core-site.xml

 

<configuration>

        <property>

                <name>fs.defaultFS</name>

                <value>hdfs://master:8020</value>

                <final>true</final>

        </property>

        <property>

                <name>dfs.replication</name>

                <value>2</value>

        </property>

        <property>

                <name>hadoop.tmp.dir</name>

                <value>/home/yarn/hadoop-files/tmp</value>

        </property>

</configuration>

Hdfs-site.xml

<configuration>

        <property>

        <name>dfs.namenode.name.dir</name>

                <value>/home/yarn/hadoop-files/name</value>

        </property>

        <property>

                <name>dfs.datanode.data.dir</name>

                <value>/home/yarn/hadoop-files/dfs/data</value>

        </property>

        <property>

                <name>dfs.replication</name>

                <value>2</value>

        </property>

        <property>

                <name>hadoop.tmp.dir</name>

                <value>/home/yarn/hadoop-files/tmp/</value>

                <description>A base for other temporary directories.</description>

        </property>

</configuration>

红色部分根据你实际的目录而定。

 

mapred-site.xml

<configuration>

        <property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

        </property>

        <property>

                <name>mapred.child.java.opts</name>

                <value>-Xmx1024m</value>

        </property>

</configuration>

yarn-site.xml

<configuration>

   <property>

     <name>yarn.nodemanager.aux-services</name>

     <value>mapreduce_shuffle</value>

  </property>

  <property>

     <name>yarn.resourcemanager.address</name>

     <value>master:8032</value>

  </property>

  <property>

      <name>yarn.resourcemanager.resource-tracker.address</name>

      <value>master:8031</value>

  </property>

  <property>

      <name>yarn.resourcemanager.admin.address</name>

      <value>master:8033</value>

  </property>

  <property>

      <name>yarn.resourcemanager.scheduler.address</name>

      <value>master:8030</value>

  </property>



  <property>

      <name>yarn.web-proxy.address</name>

      <value>master:8888</value>

  </property>

  <property>

     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

     <value>org.apache.hadoop.mapred.ShuffleHandler</value>

  </property>

  <property>

     <name>yarn.nodemanager.loacl-dirs</name>

     <value>/home/yarn/hadoop-2.2.0/hadoop-files/hadoop-loacl-dirs/</value>

     <final>true</final>

  </property>

</configuration>

 

 

其他可能需要设置的环境变量

export M2_HOME=/usr/share/maven

export PATH=$PATH:$M2_HOME/bin:~/hadoop-2.2.0/bin

export MAVEN_OPTS="-Xms2048m -Xmx2048m"

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/

export HADOOP_HOME="/home/yarn/hadoop-2.2.0"

export HADOOP_PREFIX="/home/yarn/hadoop-2.2.0"

export YARN_HOME=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/

export YARN_CONF_DIR=$HADOOP_CONF_DIR

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

export SCALA_HOME=/usr/share/scala/

export PATH=$SCALA_HOME/bin/:$PATH

 

测试

改变文件用户组属性

./hdfs dfs -chgrp -R yarn

./hdfs dfs -mkdir /yarn

这样yarn文件夹就属于用户yarn的了

编译和提交任务错误参考：

http://www.cnblogs.com/lucius/p/3435296.html