Hadoop的HDFS集群和YARN集群的搭建过程

安装Hadoop单机模式：

         1.在家目录下创建“bigdata”目录。

         2.将jdk.tar.gz和hadoop.tar.gz移动到家目录下的bigdata目录中。

         3.解压jdk.tar.gz。

         bigdata>tar zxvf jdk.tar.gz

         4.给jdk1.8.0_152创建软链接到jdk

         bigdata>ln -snf /home/zhujuan/bigdata/jdk1.8.0_152 /home/zhujuan/bigdata/jdk

 

         5.配置环境变量

         JAVA_HOME

         JRE_HOME

         CLASSPATH

         PATH

 

         bigdata>vi ~/.profile

 

         exportJAVA_HOME=/home/zhujuan/bigdata/jdk

         exportJRE_HOME=$JAVA_HOME/jre

         exportCLASSPATH=.:$JAVA_HOME/lib

         exportPATH=$PATH:$JAVA_HOME/bin

 

         source~/.profile

 

         6.解压hadoop.tar.gz

         bigdata>tar zxvf hadoop.tar.gz

 

         7.创建Hadoop的软链接

         ln-snf /home/zhujuan/bigdata/hadoop-2.6.0 /home/zhujuan/bigdata/hadoop

 

         8.配置Hadoop的环境变量

         HADOOP_USER_NAME

         HADOOP_HOME

         PATH

 

         bigdata>vi ~/.profile

 

         exportHADOOP_USER_NAME=zhujuan

         exportHADOOP_HOME=/home/zhujuan/bigdata/hadoop

         exportPATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

 

         9.HDFS集群做配置：

         $HADOOP_HOME/etc/hadoop

 

         core-site.xml

                 

                         

                                            fs.defaultFS

                                            hdfs://127.0.0.1:9000

                         

                 

 

         mkdir-p /home/zhujuan/bigdata/data/hadoop/hdfs/nn

         mkdir-p /home/zhujuan/bigdata/data/hadoop/hdfs/snn

         mkdir-p /home/zhujuan/bigdata/data/hadoop/hdfs/dn

 

         hdfs-site.xml

                 

                         

                                   dfs.nameservices

                                   hadoop-cluster

                         

                         

                                   dfs.replication

                                   1

                         

                         

                                   dfs.blocksize

                                   32M

                         

                         

                                   dfs.namenode.name.dir

                                   file:///home/zhujuan/bigdata/data/hadoop/hdfs/nn

                         

                         

                                   dfs.namenode.checkpoint.dir

                                   file:///home/zhujuan/bigdata/data/hadoop/hdfs/snn

                         

                         

                                   dfs.namenode.checkpoint.edits.dir

                                   file:///home/zhujuan/bigdata/data/hadoop/hdfs/snn

                         

                         

                                   dfs.datanode.data.dir

                                   file:///home/zhujuan/bigdata/data/hadoop/hdfs/dn

                         

                 

         10.格式化HDFS集群

                  hdfsnamenode -format

         11.启动HDFS集群

                  启动主节点：hadoop-daemon.sh start namenode

                  启动从节点：hadoop-daemon.sh start datanode

         12.WEB页面访问HDFS

                  127.0.0.1:50070

         13.配置YARN集群

 

                  mapred-site.xml

                         

                                  

                                            mapreduce.framework.name

                                            yarn

                                  

                         

 

 

                  创建目录：

                          mkdir-p ~/bigdata/data/hadoop/yarn/nm

 

                  yarn-site.xml

                         

                                   yarn.resourcemanager.hostname

                                   localhost

                         

                         

                                   yarn.nodemanager.aux-services

                                   mapreduce_shuffle

                         

                         

                                   yarn.nodemanager.local-dirs

                                   file:///home/zhujuan/bigdata/data/hadoop/yarn/nm

                         

         启动YARN集群：

                  主节点：yarn-daemon.shstart resourcemanager

                  从节点：yarn-daemon.shstart nodemanager

 

                  启动MapReduce作业的历史日志服务器：

                  mr-jobhistory-daemon.shstart historyserver

 

         YARN集群的WEB监控页面：

                  1.0.0.80:8088

 

         提交一个简单的mr程序：

         yarnjar /home/zhujuan/bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.3.jarpi 16 100

 

         vi~/bigdata/hadoop/etc/hadoop/slaves