Hadoop分布式集群部署

在阿里云临时租了三台服务器，做hadoop部署使用.

集群角色分工

第1台：hadoop-master

   角色：NameNode;ResourceManager

   Addr：172.26.73.102

第2台：hadoop-slave1

   角色：DataNode;NodeManager

   Addr：172.26.73.103

第3台：hadoop-slave2

   角色：DataNode;NodeManager

   Addr：172.26.73.104

上传文件

 hadoop2.4.1和jdk1.7目录上传到hadoop-master的／root下面

免密SSH登录配置

      分别在三台机器上执行此命令

      ssh-keygen -t rsa  

      ssh-copy-id -i .ssh/id_rsa.pub root@hadoop-slave1

      ssh-copy-id -i .ssh/id_rsa.pub root@hadoop-slave2

      ssh-copy-id -i .ssh/id_rsa.pub root@hadoop-master

配置环境变量

三台机器上配置HADOOP_HOME相关参数，修改/etc/profile

JAVA_HOME=/root/jdk1.7.0_80/

export JAVA_HOME

PATH=$PATH:$JAVA_HOME/bin

export PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

HADOOP_HOME=/root/hadoop-2.4.1

PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_HOME PATH

修改配置文件

hadoop-env.sh 

修改JAVA_HOME

export JAVA_HOME=/root/jdk1.7.0_80

hdfs-site.xml

    dfs.replication

    2

 

 

    dfs.permissions

    false

 

core-site.xml

    fs.defaultFS

    hdfs://172.26.73.102:9000

 

 

    hadoop.tmp.dir

    /root/hadoop-2.4.1/tmp

 

slaves

172.26.73.103

172.26.73.104

mapred-site.xml

    mapreduce.framework.name

    yarn

 

yarn-site.xml

        yarn.resourcemanager.hostname

        172.26.73.102

        yarn.nodemanager.aux-services

        mapreduce_shuffle

格式化namenode目录

hdfs namenode -format

传送文件

把hadoop-master的JDK目录和hadoop目录，传到hadoop-slave1和hadoop-slave1的/root下面

scp -r /root/* root@hadoop-slave1:/root

 scp -r /root/* root@hadoop-slave2:/root

启动hadoop

hadoop-master上运行start-all.sh，可能出现类似下面问题：

fix the library with 'execstack -c ', or link it with '-z noexecstack' 

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /root/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.

请在hadoop-env.sh和yarn-env.sh添加

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib" 

启动成功用，可以在hadoop-master运行jps，看到：

13958 SecondaryNameNode

13782 NameNode

14478 Jps

14107 ResourceManager

hadoop-slave1和hadoop-slave2，运行jps，看到：

13958 SecondaryNameNode

13782 NameNode

14492 Jps

14107 ResourceManager