8.大数据学习之旅——hadoop-Hadoop完全分布式配置

Hadoop完全分布式配置

---

1. 关闭防火墙
2. 修改主机名
3. 配置hosts文件。将需要搭建集群的主机全部配置到hosts文件中

192.168.32.138 hadoop01
192.168.32.139 hadoop02
192.168.32.140 hadoop03

4. 配置免密登录
5. 安装jdk
6. 安装zk
7. 配置Hadoop
   1. 编辑hadoop-env.sh，并且重新生效
   2. 编辑core-site.xml

<property>
<name>fs.defaultFSname>        
<value>hdfs://nsvalue>
property>

<property>
<name>hadoop.tmp.dirname>
<value>/home/software/hadoop-2.7.1/tmpvalue>
property>

<property>
<name>ha.zookeeper.quorumname>
<value>hadoop01:2181,hadoop02:2181,hadoop03:2181value>
property>

8. 编辑hdfs-site.xml

<property>
<name>dfs.nameservicesname>
<value>nsvalue>
property>

<property>
<name>dfs.ha.namenodes.nsname>
<value>nn1,nn2value>
property>

<property>
<name>dfs.namenode.rpc-address.ns.nn1name>
<value>hadoop01:9000value>
property>

<property>
<name>dfs.namenode.http-address.ns.nn1name>
<value>hadoop01:50070value>
property>

<property>
<name>dfs.namenode.rpc-address.ns.nn2name>
<value>hadoop02:9000value>
property>

<property>
<name>dfs.namenode.http-address.ns.nn2name>
<value>hadoop02:50070value>
property>

<property>
<name>dfs.namenode.shared.edits.dirname>
<value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/nsvalue>
property>

<property>
<name>dfs.journalnode.edits.dirname>
<value>/home/software/hadoop-2.7.1/tmp/journalvalue>
property>

<property>
<name>dfs.ha.automatic-failover.enabledname>
<value>truevalue>
property>

<property>
<name>dfs.client.failover.proxy.provider.nsname>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvid
ervalue>
property>

<property>
<name>dfs.ha.fencing.methodsname>
<value>sshfencevalue>
property>

<property>
<name>dfs.ha.fencing.ssh.private-key-filesname>
<value>/root/.ssh/id_rsavalue>
property>

<property>  
<name>dfs.namenode.name.dirname>  
<value>file:///home/software/hadoop-2.7.1/tmp/hdfs/namevalue>  
property>  

<property>  
<name>dfs.datanode.data.dirname>  
<value>file:///home/software/hadoop-2.7.1/tmp/hdfs/datavalue>  
property>
  
<property>  
<name>dfs.replicationname>  
<value>3value>  
property> 
          
                       
<property>  
<name>dfs.permissionsname>  
<value>falsevalue>  
property>

4. 编辑mapred-site.xml

<property>  
<name>mapreduce.framework.namename>  
<value>yarnvalue>  
property>

5. 编辑yarn-site.xml

<property>
<name>yarn.resourcemanager.ha.enabledname>
<value>truevalue>
property>

<property>
<name>yarn.resourcemanager.ha.rm-idsname>
<value>rm1,rm2value>
property>

<property>
<name>yarn.resourcemanager.hostname.rm1name>
<value>hadoop01value>
property>

<property>
<name>yarn.resourcemanager.hostname.rm2name>
<value>hadoop03value>
property>

<property>
<name>yarn.resourcemanager.recovery.enabledname>
<value>truevalue>
property>

<property>
<name>yarn.resourcemanager.store.classname>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
property>

<property>
<name>yarn.resourcemanager.zk-addressname>
<value>hadoop01:2181,hadoop02:2181,hadoop03:2181value>
property>

<property>
<name>yarn.resourcemanager.cluster-idname>
<value>ns-yarnvalue>
property>

<property>  
<name>yarn.nodemanager.aux-servicesname>  
<value>mapreduce_shufflevalue>  
property> 

<property>
<name>yarn.resourcemanager.hostnamename>
<value>hadoop03value>
property>

6. 编辑slaves

hadoop01
hadoop02
hadoop03

7. 拷贝到其他节点上
8. 启动zookeeper
9. 格式化zookeeper

hdfs zkfc -formatZK

10. 启动journalnode

hadoop-daemon.sh start journalnode

11. 在第一台节点上格式化并启动namenode

hadoop namenode -format
hadoop-daemon.sh start namenode

12. 在第二个节点上格式化启动namenode

hdfs namenode -bootstrapStandby
hadoop-daemon.sh start namenode

13. 三台节点启动datanode

hadoop-daemon.sh start datanode

14. 在第一台节点和第二节点上启动zkfc（FailoverController）

hadoop-daemon.sh start zkfc

15. 在第一个节点上启动yarn

start-yarn.sh

16. 在第三个节点上启动resourcemanager

yarn-daemon.sh start resourcemanager

需要hadoop集群的请在博客中搜索
Hadoop集群中需要编辑的文件.part1.rar
Hadoop集群中需要编辑的文件.part2.rar
两部分压缩包，然后解压

上一篇 7.大数据学习之旅——hadoop-MapReduce