hadoop-hdfs-ha 搭建及一些常见错误

hadoop-hdfs-ha 搭建

ps.省略了基础环境的配置（没有ssh免密以及主机名称，jdk）

1. zookeeper配置
   到zookeeper的conf目录下

   cp zoo_sample.cfg zoo.cfg
   

   对zoo.cfg进行配置

   # The number of milliseconds of each tick
   tickTime=2000
   # The number of ticks that the initial 
   # synchronization phase can take
   initLimit=10
   # The number of ticks that can pass between 
   # sending a request and getting an acknowledgement
   syncLimit=5
   # the directory where the snapshot is stored.
   # do not use /tmp for storage, /tmp here is just 
   # example sakes.
   dataDir=/var/zookeeper
   # the port at which the clients will connect
   clientPort=2181
   # the maximum number of client connections.
   # increase this if you need to handle more clients
   #maxClientCnxns=60
   #
   # Be sure to read the maintenance section of the 
   # administrator guide before turning on autopurge.
   #
   # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
   #
   # The number of snapshots to retain in dataDir
   #autopurge.snapRetainCount=3
   # Purge task interval in hours
   # Set to "0" to disable auto purge feature
   #autopurge.purgeInterval=1
   server.1=192.168.41.11:2888:3888 #第二个端口用来选举leader和follower
   server.2=192.168.41.12:2888:3888
   server.3=192.168.41.13:2888:3888
   
   

   注意我这里省略了分发的步骤，分发步骤见4

   mkdir -p /var/zookeeper #递归的创建目录
   

   echo 1 > /var/zookeeper/myid #这里注意myid和zookeeper服务器一一对应eg.第一台myid就1 ，第二台就2以此类推。
   

2. hdfs-site.xml配置

   
       
           fs.replication
           2
       
       
     		dfs.nameservices
     		mycluster
       
       
   		dfs.ha.namenodes.mycluster
     		nn1,nn2
       
       
    		dfs.namenode.rpc-address.mycluster.nn1
     		master:8020
       
       
     		dfs.namenode.rpc-address.mycluster.nn2
     		slave1:8020
       
       
     		dfs.namenode.http-address.mycluster.nn1
      		master:50070
       
       
     		dfs.namenode.http-address.mycluster.nn2
   		slave1:50070
       
       
           dfs.namenode.shared.edits.dir
           qjournal://master:8485;slave1:8485;slave2:8485/mycluster
       
       
           dfs.journalnode.edits.dir
           /var/hadoop/journalnode
       
       
    	 dfs.client.failover.proxy.provider.mycluster
          	 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
       
       
         dfs.ha.fencing.methods
         sshfence
       
       
         dfs.ha.fencing.ssh.private-key-files
         /root/.ssh/id_rsa
       
       
         dfs.ha.automatic-failover.enabled
         true
       
   
   
   

3. core-site.xml配置

   
       
   	fs.defaultFS
   	hdfs://mycluster
       
       
   	hadoop.tmp.dir
   	/var/hadoop-2.6/ha
       
       
      	ha.zookeeper.quorum
      	slave1:2181,slave2:2181,slave3:2181
       
   
   

4. slaves文件配置
   在这个文件下添加datanode的主机名

5. 配置文件分发
   到目标目录下eg.我hadoop，zookeeper存放在/opt 下接下来bash命令以我的位置为例

   scp -r  ./zk root@目标主机:`pwd`/zk
   
   

   scp -r  ./hadoop root@目标主机:`pwd`/hadoop
   

6. 启动集群
   ~先启动journalnode

   hadoop-daemon.sh start journalnode
   

   ~格式化master的namenode并启动

   hdfs namenode -format
   

   hadoop-daemon.sh start namenode
   

   ~开启slave1 的namenode（注意千万别格式化）

   hdfs namenode -bootStrapStandby
   

   ~zk

   在zookeeper服务器下启动

   zkServer.sh start
   

   在master下格式化zookeeper

   hdfs zkfc -formatZK
   

   ~启动角色

   start-dfs.sh
   

master

slave1