[1] Hive3.x 安装与debug

1 下载安装hive3.1.1

下载地址

修改hive-env.sh
如：

HADOOP_HOME=/Users/xxx/software/hadoop/hadoop-2.7.4
export HIVE_CONF_DIR=/Users/xxx/software/hive/conf
export HIVE_AUX_JARS_PATH=/Users/xxx//software/hive/lib

建立hive-site.xml
拷贝hive-default.xml.template修改为hive-site.xml文件
开头添加

   system:java.io.tmpdir
    /tmp/hive3/java
  
  
    system:user.name
    ${user.name}
  

其他修改部分

hive.exec.scratchdir
/tmp/hive3

hive.metastore.warehouse.dir
/user/hive3/warehouse

hive.repl.rootdir
/user/hive3/repl/

hive.repl.cmrootdir
/user/hive3/cmroot/

hive.repl.replica.functions.root.dir
/user/hive3/repl/functions/

hive.query.results.cache.directory
/tmp/hive3/_resultscache_

hive.exec.local.scratchdir
/Users/didi/Documents/software/hive3/data0/hive/${user.name}

以上目录可以使用默认（这里本机已经有hive2.1，把hive改成hive3了）

在hdfs中建立相应文件目录：  
/bin/hdfs dfs -mkdir -p /user/hive3/warehouse  
/bin/hdfs dfs -mkdir -p /tmp/hive3/  
hdfs dfs -chmod 777 /user/hive/warehouse  
hdfs dfs -chmod 777 /tmp/hive  
hadoop fs -chmod 777 /user/hive/warehouse   
hadoop fs -chmod 777 /tmp/hive  
同上建立并设置  
/bin/hdfs dfs -mkdir -p /user/hive3/repl  
hdfs dfs -chmod 777 /user/hive3/repl
hadoop fs -chmod 777 /user/hive3/repl

/bin/hdfs dfs -mkdir -p /user/hive3/cmroot  
hdfs dfs -chmod 777 /user/hive3/cmroot
hadoop fs -chmod 777 /user/hive3/cmroot

/bin/hdfs dfs -mkdir -p /user/hive3/repl/functions    
hdfs dfs -chmod 777 /user/hive3/repl/functions 
hadoop fs -chmod 777 /user/hive3/repl/functions  


/bin/hdfs dfs -mkdir -p /tmp/hive3/_resultscache_
hdfs dfs -chmod 777 /tmp/hive3/_resultscache_
hadoop fs -chmod 777 /tmp/hive3/_resultscache_

其他修改

hive.metastore.db.type
mysql

hive.metastore.uris
thrift://localhost:9083

javax.jdo.option.ConnectionPassword
root #mysql 密码

javax.jdo.option.ConnectionUserName
root  #mysql user

javax.jdo.option.ConnectionURL
jdbc:mysql://localhost:3306/hive3?createDatabaseIfNotExist=true&characterEncoding=UTF-8

javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver

datanucleus.connectionPoolingType
BONECP

hive.local.time.zone
Asia/Shanghai

hive.mapred.mode
nonstrict

其他参考：Mac-单机Hive安装与测试 将hive安装好，初始化完元数据

2 Hive命名行并启动debug模式

/..../software/hive3/hive$ bin/hive --debug
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Listening for transport dt_socket at address: 8000

这里没有修改监听端口，默认8000

3-IDEA配置remote-debug

• 同版本的hive源码并导入IDEA中提前编译好（具体参见hive wiki）
• 配置远端debug：Once you see this message, in Eclipse right click on the project you want to debug, go to “Debug As -> Debug Configurations -> Remote Java Application” and hit the “+” sign on far left top. This should bring up a dialog box. Make sure that the host is the host on which the Beeline CLI is running and the port is “8000”. Before you start debugging, make sure that you have set appropriate debug breakpoints in the code. Once ready, hit “Debug”. The remote debugger should attach to Beeline and proceed.
  

4-点击debug配置好的远端debug

IDEA console显示如下表示连接成功

Connected to the target VM, address: 'localhost:8000', transport: 'socket'

5-同时Hive cli进入命令cli状态

hive3/hive$ bin/hive --debug
Listening for transport dt_socket at address: 8000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/..../software/hive3/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/.../software/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 378e1844-e7fe-40c7-ada5-7de6a30508a5

Logging initialized using configuration in jar:file:/.../software/hive3/hive/lib/hive-common-3.1.1.jar!/hive-log4j2.properties Async: true
Hive Session ID = 152f6855-de8d-4a5d-b251-b6b0347c4c08
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>

6-在IDEA 添加断点

如在CliDriver添加断点

public int processCmd(String cmd) {
   在该方法中添加断点，等待命令行输入
}

7-Hive命令行输入

n engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;

此时IDEA中运行至断点处

参考

• Mac-单机Hive安装与测试
• How do I debug my queries in Hive?
• Hive wiki