hadoop3.1.3分布式集群搭建

环境准备：
jdk1.8
centos6.10
zookeeper3.4.5
三台服务器免密登录配置完成(此处省略步骤)

1、上传Hadoop3.1.3安装包并解压
这里我是安装在/usr/local 这个路径下

解压：tar -zxvf /usr/local/hadoop-3.1.3.tar.gz

2、首先提前创建好需要的文件夹

mkdir -p /usr/local/hadoop/data
mkdir -p /usr/local/hadoop/dfs
mkdir -p /usr/local/hadoop/tmp

3、vi /etc/profile
配置hadoop的安装路径

export HADOOP_HOME=/usr/local/hadoop-3.1.3
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile 才会生效

4、vi /etc/hosts
配置三台服务器的主机名和ip

192.168.130.28   myhbase
192.168.130.29   xmhl
192.168.130.30   xmhl30

5、vi core-site.xml

	fs.defaultFS
	hdfs://xmhl:9000
	       tfs://
		   gfs://
		   file:///




	hadoop.tmp.dir
	/usr/local/hadoop/tmp



    hadoop.proxyuser.root.hosts
    *


    hadoop.proxyuser.root.groups
    *

6、vi hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.8.0_141

7、vi mapred-site.xml

	mapreduce.framework.name
	yarn

8、vi yarn-site.xml

        yarn.nodemanager.aux-services
        mapreduce_shuffle
    
    
        yarn.resourcemanager.webapp.address
        xmhl:8088
    


        yarn.resourcemanager.hostname
        xmhl


   yarn.application.classpath
/usr/local/hadoop-3.1.3/etc/hadoop:/usr/local/hadoop-3.1.3/share/hadoop/common/lib/*:/usr/local/hadoop-3.1.3/share/hadoop/common/*:/usr/local/hadoop-3.1.3/share/hadoop/hdfs:/usr/local/hadoop-3.1.3/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.3/share/hadoop/hdfs/*:/usr/local/hadoop-3.1.3/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.3/share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.3/share/hadoop/yarn:/usr/local/hadoop-3.1.3/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.3/share/hadoop/yarn/*



	 yarn.nodemanager.pmem-check-enabled
	false


	 yarn.nodemanager.vmem-check-enabled
	false

9、vi hdfs-site.xml

        dfs.namenode.http-address
        
        xmhl:50070
    
    
        dfs.namenode.name.dir
        /usr/local/hadoop/dfs/name
     
     
         dfs.datanode.data.dir
         /usr/local/hadoop/dfs/data
     
     
        dfs.replication
        2
    


 	dfs.namenode.secondary.http-address
 	
  	myhbase:50090            

10、vi workers
配置三台服务器的主机名

xmhl
myhbase
xmhl30

以上1-10步骤三台服务器都要进行配置

11、第一次启动需要格式化namenode

hdfs namenode -format

12、启动hadoop

先启动HDFS  sbin/start-dfs.sh
再启动YARN  sbin/start-yarn.sh

或者一起启动  sbin/start-all.sh

13、验证是否启动成功

使用jps命令验证，出现以下进程即为启动成功
27408 NameNode
28218 Jps
27643 SecondaryNameNode
28066 NodeManager
27803 ResourceManager
27512 DataNode

hadoop自带的UI界面：
http://192.xx.xx.xx:50070 （HDFS管理界面）
http://192.xx.xx.xx:8088 （MR管理界面）