Hadoop-入门篇环境搭建（四）

转载请注明原文出处

hadoop搭建

阿牛的资料下载
Hadoop-入门篇环境搭建（一）
Hadoop-入门篇环境搭建（二）
Hadoop-入门篇环境搭建（三）
Hadoop-入门篇环境搭建（四）

机器配置

+所有机器都有这个步骤
这里有4台机器
hadoop 在/opt/soft/hadoop 下面

node1

+ vi ~/.bash_profile
  export HADOOP_PREFIX=/opt/soft/hadoop
+ source ~/.bash_profile

# cd /opt/soft/hadoop/etc/hadoop

# vi mapred-env.sh
  export JAVA_HOME=/usr/java/jdk1.7.0_79
#vi hadoop-env.sh
  export JAVA_HOME=/usr/java/jdk1.7.0_79

# vi slaves
  node2
  node3
  node4

+ mkdir -p /opt/data/hadoop
+ mkdir  -p /opt/data/journalnode
#vi hdfs-site.xml
#集群名字ID mycluster请替换为下面的value=hadoop

  dfs.nameservices
  hadoop

#集群有哪些机器

  dfs.ha.namenodes.hadoop
  nn1,nn2
#集群的namenode ip

  dfs.namenode.rpc-address.hadoop.nn1
  node1:8020


  dfs.namenode.rpc-address.hadoop.nn2
  node2:8020


#连接页面

  dfs.namenode.http-address.hadoop.nn1
  node1:50070


  dfs.namenode.http-address.hadoop.nn2
  node2:50070



  dfs.namenode.shared.edits.dir
  qjournal://node2:8485;node3:8485;node4:8485/hadoop


  dfs.client.failover.proxy.provider.hadoop



  dfs.ha.fencing.methods
  sshfence



  dfs.ha.fencing.ssh.private-key-files
  /root/.ssh/id_dsa




  dfs.ha.fencing.ssh.connect-timeout
  60000

#mkdir -p /opt/data/journalnode

  dfs.journalnode.edits.dir
  /opt/data/journalnode

#自动选举
 
   dfs.ha.automatic-failover.enabled
   true
 
#zk节点的位置
  
   ha.zookeeper.quorum
   node1:2181,node2:2181,node3:2181
 

#in your core-site.xml file:
#
#  fs.defaultFS
#  hdfs://hadoop
#
#


vi core-site.xml

 
   ipc.client.connect.max.retries
    20
    
    
  

  
   ipc.client.connect.retry.interval
    5000
    
      Indicates the number of milliseconds aclient will wait for before retrying to establish a server connection.
    
  



        ha.zookeeper.quorum
        node1:2181,node2:2181,node3:2181
    
    
        hadoop.tmp.dir
        /opt/data/hadoop
    


  fs.defaultFS
  hdfs://hadoop



hdfs高可用搭建完成
yarn搭建

cd /opt/soft/hadoop/etc/hadoop
#vi yarn-site.xml

   yarn.resourcemanager.ha.enabled
   true
 
 #this is yarn id not hdfs's id
 
   yarn.resourcemanager.cluster-id
   hadoop-yarn
 
 #this is resourcemanager id how many resourcemanager ,if you want to set RSM please set this
 
   yarn.resourcemanager.ha.rm-ids
   rm1,rm2
 
 
   yarn.resourcemanager.hostname.rm1
   node3
 
 
   yarn.resourcemanager.hostname.rm2
   node4
 
 #there is ZK ,please synchronous your's ZK cluster
 
   yarn.resourcemanager.zk-address
   node1:2181,node2:2181,node3:2181
 
 #this is a defects in yarn ,You have to remember
 #please mapreduce_shuffle
  
   yarn.nodemanager.aux-services
   mapreduce_shuffle
 

16年的学习笔记，留言相互交流。

关注老铁的公众号，从小菜比到老菜比

你假笨

462563010加QQ群一起学习
欢迎关注spring4all