Hadoop+Hbase高可用集群搭建

Hadoop配置

1、core-site.xml配置：

        fs.defaultFS
        hdfs://ns1



        hadoop.tmp.dir
        /usr/local/hadoop-2.7.6/tmp

 

        ha.zookeeper.quorum
        hadoop02:2181,hadoop03:2181,hadoop04:2181


   ipc.client.connect.max.retries
    20
    
      Indicates the number of retries a clientwill make to establisha server connection.
    
   

   ipc.client.connect.retry.interval
    5000
    
      Indicates the number of milliseconds aclient will wait for before retrying to establish a server connection.
    
  

    io.file.buffer.size
    262144
  

2、hdfs-site.xml配置：

        dfs.nameservices
        ns1



        dfs.ha.namenodes.ns1
        nn1,nn2



        dfs.namenode.rpc-address.ns1.nn1
        hadoop05:9000



        dfs.namenode.http-address.ns1.nn1
        hadoop05:50070


        
        dfs.namenode.rpc-address.ns1.nn2
         hadoop06:9000



         dfs.namenode.http-address.ns1.nn2
         hadoop06:50070



         dfs.namenode.shared.edits.dir                    
         qjournal://hadoop02:8485;hadoop03:8485;hadoop04:8485/ns1



         dfs.journalnode.edits.dir
         /usr/local/hadoop-2.7.6/journaldata


         dfs.namenode.name.dir
         /usr/local/hadoop-2.7.6/name


         dfs.datanode.data.dir
         /usr/local/hadoop-2.7.6/data




         dfs.ha.automatic-failover.enabled
         true



         dfs.client.failover.proxy.provider.ns1                    org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider



         dfs.ha.fencing.methods
         
                sshfence
                shell(/bin/true)
         



         dfs.ha.fencing.ssh.private-key-files
         /root/.ssh/id_rsa

         

         dfs.ha.fencing.ssh.connect-timeout
         30000


 
    dfs.namenode.handler.count
    50
  
   
    dfs.datanode.handler.count
    8
  
   
    dfs.datanode.max.transfer.threads 
    8192 
 


      dfs.client.read.shortcircuit     
      true   


      dfs.domain.socket.path
      /var/lib/hadoop-hdfs/dn.socket

3、mapred-site.xml配置：

          mapreduce.framework.name
          yarn

4、yarn-site.xml配置：

          yarn.resourcemanager.ha.enabled
          true



          yarn.resourcemanager.cluster-id
          yrc



          yarn.resourcemanager.ha.rm-ids
          rm1,rm2



          yarn.resourcemanager.hostname.rm1
          hadoop05


          yarn.resourcemanager.hostname.rm2
          hadoop06



          yarn.resourcemanager.zk-address
          hadoop02:2181,hadoop03:2181,hadoop04:2181


          yarn.nodemanager.aux-services
          mapreduce_shuffle

5、slaves配置

hadoop02
hadoop03
hadoop04

6、建立文件夹

mkdir tmp journaldata name data logs pids

7、拷贝到其他节点上

scp -r hadoop-2.7.6 root@hadoop03:/usr/local/

8、从结点下分别启动各个JournalNode

./hadoop-daemon.sh start journalnode

9、登陆主节点hadoop05，进入到hadoop-2.7.6/bin目录下，格式化文件系统 

./hadoop namenode -format

10、启动第2步初始化好的NameNode服务

./hadoop-daemon.sh start namenode

11、在master2 bin中同步master1

hdfs namenode -bootstrapStandby

12、edit日志初始化到JN节点

./hdfs namenode -initializeSharedEdits

13、在master1初始化ZKFC

hdfs zkfc -formatZK

14、启动Hadoop的HA集群

./start-dfs.sh
./start-yarn.sh

Hbase配置

1、hbase-env.sh配置

export JAVA_HOME=/usr/local/jdk1.8.0_191
export HBASE_CLASSPATH=/usr/local/hadoop-2.7.6/etc/Hadoop
export HBASE_PID_DIR=/usr/local/hbase-1.4.4/pids
export HBASE_MANAGES_ZK=false

2、hbase-site.xml配置

        
            hbase.rootdir
            hdfs://ns1/hbase
        
        
            hbase.cluster.distributed
            true
        
        
            hbase.master
            60000
        
        
            hbase.tmp.dir
            /usr/local/hbase-1.4.4/tmp
        
        
            hbase.zookeeper.quorum
            hadoop02,hadoop03,hadoop04
        
        
            hbase.zookeeper.property.dataDir
            /usr/local/hbase-1.4.4/data
        
        
            hbase.zookeeper.property.clientPort
            2181
        
        
            zookeeper.session.timeout
            120000
        
        
            hbase.regionserver.restart.on.zk.expire
            true
        

        
            hbase.regionserver.handler.count
            100
        
 
        
            hbase.client.scanner.caching
            100
        

                     
hbase.hregion.memstore.mslab.enabled
            true
       

        
            hbase.client.retries.number  
            11 
        

        
           hbase.ipc.client.tcpnodelay  
           true 
         
 
       
           hbase.client.scanner.timeout.period
           120000
       
 
       
          hbase.rpc.timeout
          180000
       
           
          hbase.hregion.memstore.block.multiplier    
          8    
       
       
          hbase.hstore.blockingStoreFiles
          30
       
       
          hbase.client.write.buffer
          5242880
       

3、conf下新建backup-masters输入hadoop06

vim backup-masters

4、配置regionservers

hadoop02
hadoop03
hadoop04

5、创建目录

mkdir pids tmp

6、解决hbase和hadoop的LSF4J包冲突njnu-bigdata-01  bin]$ ./

mv slf4j-log4j12-1.7.5.jar slf4j-log4j12-1.7.5.jar.bk

7、拷贝hadoop的core-site.xml和hdfs-site.xml拷贝到hbase的conf下

8、拷贝到其他节点上

9、运行