Hadoop2.7.5双主基本配置（记录）

hadoop集群搭建首先需要JAVA环境，还有要zookeeper集群，可以参考JAVA和zookeeper安装文档

准备

软件准备

请到hadoop官网下载hadoop2.7.5版本：https://hadoop.apache.org/releases.html，可以下载binary包直接安装hadoop-2.7.5.tar.gz，或者下载source包编译安装

环境准备

服务器列表：
10.242.154.185 master001
10.242.154.186 master002
10.242.154.187 slave001
10.242.154.188 slave002
10.242.154.189 slave003
10.242.154.190 slave004
10.242.154.191 slave005

zookeeper:
10.242.167.200
10.242.167.201
10.242.167.202

安装

配置SSH免密登录，可以参考脚本
生产秘钥命令：ssh-keygen -t rsa
配置相关文件：
配置core-site.xml

        
                fs.defaultFS
                hdfs://master001
        
        
                hadoop.tmp.dir
                /data/tmp
        
        
                ha.zookeeper.quorum
                10.242.167.200:2181,10.242.167.201:2181,10.242.167.202:2181
        
        
                hadoop.proxyuser.root.groups
                root
         
         
                hadoop.proxyuser.root.hosts
                *
         

配置hadoop-env.sh

基本配置
export HADOOP_LOG_DIR=/log/hadoop/
export JAVA_HOME=/usr/local/jdk1.8.0_121

配置hdfs-site.xml

        
                dfs.nameservices
                master001
        
        
                dfs.ha.namenodes.master001
                master001,master002
        
        
                dfs.namenode.rpc-address.master001.master001
                master001:8020
        
        
                dfs.namenode.http-address.master001.master001
                master001:50070
        
        
                dfs.namenode.rpc-address.master001.master002
                master002:8020
        
        
                dfs.namenode.http-address.master001.master002
                master002:50070
        
        
                dfs.namenode.shared.edits.dir
                qjournal://slave003:8485;slave002:8485;slave01:8485/master002
        
        
                dfs.journalnode.edits.dir
                /usr/local/hadoop-2.7.5/journal
        
        
                dfs.ha.automatic-failover.enabled
                true
        
        
            dfs.permissions
            false
        
        
                dfs.client.failover.proxy.provider.masters
                org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
        
        
                dfs.ha.fencing.methods
                
                        sshfence
                        shell(/bin/true)
                
        
        
                dfs.ha.fencing.ssh.private-key-files
                /home/hadoop/.ssh/id_rsa
        
        
                dfs.ha.fencing.ssh.connect-timeout
                30000
        
        
                dfs.hosts.exclude
                /usr/local/hadoop/etc/hadoop/excludes 
        

配置mapred-site.xml

	
		mapreduce.framework.name
		yarn
	
	
		mapreduce.map.memory.mb
		4096
		
	
		mapreduce.reduce.memory.mb
		4096
		
	
		yarn.app.mapreduce.am.resource.mb
		4096
		
	
		mapreduce.task.io.sort.mb
		1536
		
	
		mapreduce.map.java.opts
		-Xmx3072m
		
	
		mapreduce.reduce.java.opts
		-Xmx3072m
		
	
		yarn.app.mapreduce.am.command-opts
		-Xmx3072m
			

配置slaves

slave001
slave002
slave003
slave004
slave005

配置yarn-site.xml

        
                yarn.resourcemanager.ha.enabled
                true
        
        
                yarn.resourcemanager.cluster-id
                RM_HA_ID
        
        
                yarn.resourcemanager.ha.rm-ids
                rm1,rm2
        
        
                yarn.resourcemanager.hostname.rm1
                master001
        
        
                yarn.resourcemanager.hostname.rm2
                master002
        
        
                yarn.resourcemanager.recovery.enabled
                true
        
        
                yarn.resourcemanager.store.class
                org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
        
        
                yarn.resourcemanager.zk-address
                10.242.167.200:2181,10.242.167.201:2181,10.242.167.202:2181
        
        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        
        
                yarn.scheduler.minimum-allocation-mb
                4096
        
        
                yarn.scheduler.maximum-allocation-mb
                12288
        		
        
                yarn.nodemanager.resource.memory-mb
                12288
        				

然后在各个节点建好相关目录
将/usr/local/hadoop 整个目录scp到其他节点
然后进入/usr/local/hadoop/sbin
执行`./start-all.sh`
启动集群

访问WEB端
http://10.242.154.186:50070
active

standby
http://10.242.154.185:50070

基本搭建结束