hive 使用tez

1,编译tez 见https://www.jianshu.com/p/b2569796dd27

2,将 编译后的tez-0.9.2.tar.gz 上传到hdfs上. tez-site.xml中会使用到.见tez.lib.uris属性.
3,在 $HADOOP_HOME/etc/hadoop 下新建 tez-site.xml.内容如下

 
    tez.lib.uris
    ${fs.defaultFS}/apps/tez/tez-0.9.2.tar.gz
  


     tez.use.cluster.hadoop-libs
     false


     tez.am.resource.memory.mb
     1024


     tez.am.resource.cpu.vcores
     1


     tez.container.max.java.heap.fraction
     0.4


     tez.task.resource.memory.mb
     1024


     tez.task.resource.cpu.vcores
     1

4,设置 hive. 修改hive-site.xml文件

  
    hive.execution.engine
    tez
  

  
    hive.server2.authentication
    NONE
  

5,设置客户端的tez. 将 tez-0.9.2.tar.gz 解压到本地 /usr/tez下.

//设置软连接方便日后升级.
$ ln -s tez-0.9.2 default
//配置本地环境变量
export TEZ_CONF_DIR=$HADOOP_CONF_DIR
export TEZ_HOME=/usr/tez/default
export TEZ_JARS=$TEZ_HOME/*:$TEZ_HOME/lib/*
export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH

6,重启服务
会有一下一些错误.
错误1 :cause: org.apache.hadoop.service.ServiceStateException: java.lang.NoClassDefFoundError: com/google/common/net/UrlEscapers

2020-06-20T08:57:06,834 INFO [main] org.apache.hadoop.service.AbstractService - Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED; cause: org.apache.hadoop.service.ServiceStateException: java.lang.NoClassDefFoundError: com/google/common/net/UrlEscapers
org.apache.hadoop.service.ServiceStateException: java.lang.NoClassDefFoundError: com/google/common/net/UrlEscapers
    at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) ~[hadoop-common-2.7.7.jar:?]
    at org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1945) ~[tez-dag-0.9.2.jar:0.9.2]
    at org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:2012) ~[tez-dag-0.9.2.jar:0.9.2]
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) [hadoop-common-2.7.7.jar:?]
    at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2663) [tez-dag-0.9.2.jar:0.9.2]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_91]
    at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_91]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) [hadoop-common-2.7.7.jar:?]
    at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2659) [tez-dag-0.9.2.jar:0.9.2]
    at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2464) [tez-dag-0.9.2.jar:0.9.2]

解决: 我是将$HIVE_HOME/lib中的guava-14.0.1.jar 升级到guava-19.0.jar. 这个问题解决.但是日志中还是有些错误,不影响结果的产出.

image.png

错误2:
使用hive cli 的方式可以正常提交sql.并且在yarn上可以看到作业的运行情况.
但是使用 ** beeline -u jdbc:hive2://localhost:10000 ** .在yarn上看不到作业运行,并且报错. 错误日志只能在hive ui上查看.

解决:
在 local log中的hive.log中查看到是文件权限问题.

beeline 的时候需要指定用户名.
beeline -u jdbc:hive2://localhost:10000 -n wl