hive 3.1.1 高可用集群搭建（与zookeeper集成）搭建笔记

一、简介

          1.  hive

                  三个节点 分别在hdp01、hdp02 、hdp03

          2.  zookeeper

                 5个节点  分别在 hdp04、hdp05、hdp06 、hdp07 、hdp08

          3. hadoop 

                 7个节点   ： namenode    hdp01 、hdp02      

                                      datanode    hdp03、hdp04、hdp05 、hdp06、hdp07 、hdp08

 

二、搭建步骤

           1.安装mysql

                   rpm -ivh  MySQL-server-5.6.26-1.linux_glibc2.5.x86_64.rpm   

                   rpm -ivh  MySQL-client-5.6.26-1.linux_glibc2.5.x86_64.rpm

                  可能会缺少perl

                  yum install perl

          注意要点：

                  首次登陆完成之后，注意要直接配置好root可以远程登录，否则还需要进行其他的修改。

          如何修改在另外一篇文章中有记录

                https://mp.csdn.net/postedit/89081368

           2.安装hive与mysql 、zk集成

                 这里安装步骤省略。主要是修改配置文件，这里只说明一下hive的安装文件的配置以及说明

                        scp hive-env.sh.template hive-env.sh

                        scp hive-default.xml.template hive-site.xml

                 修改hive-env.sh

                        具体的配置根据自己的情况来定，我的配置是这样

                        HADOOP_HOME=/usr/hadoop/hadoop-2.8.1/
                        HIVE_CONF_DIR=/root/app/apache-hive-3.1.1-bin/conf
                        HIVE_AUX_JARS_PATH=/root/app/apache-hive-3.1.1-bin/lib

                修改hive-site.xml

   

-----------------------------------------数据库集成配置-----------------------------------------

javax.jdo.option.ConnectionURL
jdbc:mysql://hdp01:3306/hivedb?createDatabaseIfNotExist=true
JDBC connect string for a JDBC metastore

 

javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
Driver class name for a JDBC metastore

 

javax.jdo.option.ConnectionUserName
root
username to use against metastore database

 

javax.jdo.option.ConnectionPassword
root
password to use against metastore database

--------------------------------------------hive工作目录的配置--------------------------------------------------

 环境变量最好配置一下，否则获取不了的话，生成的文件名很杂乱，比如下面的我就设置了

 
    hive.exec.local.scratchdir
    /root/app/apache-hive-3.1.1-bin/tmp/hiveuser
    Local scratch space for Hive jobs
 
 
    hive.downloaded.resources.dir
    /root/app/apache-hive-3.1.1-bin/tmp/${hive.session.id}_resources    
    Temporary local directory for added resources in the remote file system.
 

-------------------------------------------元数据的配置创建很重要，不配置的话与mysql集成有可能会出问题------------

    datanucleus.schema.autoCreateAll
    true
    Auto creates necessary schema on a startup if one doesn't exist. Set this to false, after creating it once.To enable auto create also set hive.metastore.schema.verification=false. Auto creation is not recommended for production use cases, run schematool command instead.
 
 
    hive.metastore.schema.verification
    false
   
      Enforce metastore schema version consistency.
      True: Verify that version information stored in is compatible with one from Hive jars.  Also disable automatic
            schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
            proper metastore schema migration. (Default)
      False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
   
 

--------------------------------------连接mysql数据库的用户名密码---------------------------

    javax.jdo.option.ConnectionUserName
    hiveuser
    Username to use against metastore database
 

    javax.jdo.option.ConnectionPassword
    123456789
    password to use against metastore database
 

---------------------------------------其他配置--------------------------------------------------

 
    hive.querylog.location
    /root/app/apache-hive-3.1.1-bin/tmp/qrylog
    Location of Hive run time structured log file
 

---------------------------------------zookeeper---------------------------------------------------

 
    hive.zookeeper.quorum
   
       hdp04:2181,hdp05:2181,hdp06:2181,hdp07:2181,hdp08:2181
   
   
      List of ZooKeeper servers to talk to. This is needed for:
      1. Read/write locks - when hive.lock.manager is set to
      org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager,
      2. When HiveServer2 supports service discovery via Zookeeper.
      3. For delegation token storage if zookeeper store is used, if
      hive.cluster.delegation.token.store.zookeeper.connectString is not set
      4. LLAP daemon registry service
      5. Leader selection for privilege synchronizer
   
 

 
    hive.server2.support.dynamic.service.discovery
    true
    Whether HiveServer2 supports dynamic service discovery for its clients. To support this, each instance of HiveServer2 currently uses ZooKeeper to register itself, when it is brought up. JDBC/ODBC clients should use the ZooKeeper ensemble: hive.zookeeper.quorum in their connection string.
 
 
    hive.server2.zookeeper.namespace
    hiveserver2_zk
    The parent node in ZooKeeper used by HiveServer2 when supporting dynamic service discovery.
 
 
    hive.server2.zookeeper.publish.configs
    true
    Whether we should publish HiveServer2's configs to ZooKeeper.
 

--------------------------------------hiveserver的日志路径配置-----------------------------------------------

    hive.server2.logging.operation.log.location
    /root/app/apache-hive-3.1.1-bin/tmp/operation_logs
    Top level directory where operation logs are stored if logging functionality is enabled
 

 
    hive.server2.thrift.client.user
    root
    Username to use against thrift client
 
 
    hive.server2.thrift.client.password
    root
    Password to use against thrift client
 

    hive.server2.thrift.port
    10000
    Port number of HiveServer2 Thrift interface when hive.server2.transport.mode is 'binary'.
 

    hive.server2.transport.mode
    binary
   
      Expects one of [binary, http].
      Transport mode of HiveServer2.
   
 
 
    hive.server2.thrift.bind.host
    hdp01
    Bind host on which to run the HiveServer2 Thrift service.
 

------------------------------------------------关于hdfs的相关配置-------------------------------------------

    hive.exec.scratchdir
    /tmp/hive
    HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.
 
 
    hive.repl.rootdir
    /user/hive/repl/
    HDFS root dir for all replication dumps.
 

   hdfs-site.xml  文件配置

dfs.webhdfs.enabled
true

core-site.xml 配置 这里配置很重要  还有要注意这里的name页签中的root 这里是hdfs登录的具体的用户名，写错了就会报错，访问hive的时候会报错

     hadoop.proxyuser.root.hosts
     *
   
   
    hadoop.proxyuser.root.groups
    *
   

 

             配置完成之后，需要有一个jdbc的jar包驱动

  

 三、操作指令

      1.后台启动服务. 在hive节点上启动即可

          nohup hiveserver2 -hiveconf hive.root.logger=DEBUG,console  1> hive.log 2>&1 &

     2.客户端访问  belline 敲入以下指令登录

       !connect jdbc:hive2://hdp04:2181,hdp05:2181,hdp06:2181,hdp07:2181,hdp08:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk root  "root"