HiveServer2运行慢，不定时宕机，问题排查
HiveServer2运行慢，不定时宕机，问题排查
文章目录
- HiveServer2运行慢，不定时宕机，问题排查
- 1. 问题描述
- 2. 原因分析（分析方法）
- 3. 解决方案
- 总结
1. 问题描述
大数据组件 HiveServer2性能问题，有时运行很慢，几天宕机一次，需要不定时重启。
2. 原因分析（分析方法）
分析思路： 运行慢时，jstat 查看 gc回收情况
root@tbds-1:~>ps -ef|grep HiveServer2
hive 137791 1 99 Jan22 ? 7-10:47:32 /usr/jdk64/jdk1.8.0_111/bin/java -Xmx16384m -Dhdp.version=2.2.0.0-2041 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Dhadoop.log.dir=/data/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/data/bigdata/tbds/usr/hdp/2.2.0.0-2041/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/data/bigdata/tbds/usr/hdp/2.2.0.0-2041/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx16384m -Djava.security.auth.login.config=/etc/hive/conf.server/kafka_client_for_ranger_jaas.conf -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=70 -XX:+PrintGCDateStamps -XX:GCLogFileSize=512M -verbose:gc -XX:+PrintGCDetails -Xloggc:/data/var/log/hive/hive_gc.log -Dlog4j.configurationFile=/etc/hive/conf/hive-log4j2.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.2.0.0-2041/hive/lib/hive-service-2.2.0.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-server-extensions-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-server-extensions.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-streaming-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-streaming.jar -hiveconf hive.metastore.uris=
root 215376 214471 0 16:43 pts/6 00:00:00 grep --color=auto HiveServer2
root@tbds-10-232-2-37:~>jstat -gcutil 137791 2000
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.98 33.61 13.77 22802 6534.194 292 9846.938 16381.132
0.00 0.00 0.00 99.99 33.61 13.77 22804 6534.537 292 9882.885 16417.422
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 0.00 99.99 33.61 13.77 22808 6535.752 293 9882.885 16418.637
0.00 0.00 31.49 100.00 33.61 13.77 22808 6535.752 293 9919.457 16455.208
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 100.00 33.62 13.77 22809 6536.053 294 9919.457 16455.510
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 0.00 99.99 33.62 13.77 22811 6536.596 295 9953.494 16490.090
0.00 0.00 84.44 99.98 33.62 13.77 22812 6536.596 295 9988.835 16525.430
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
0.00 0.00 0.00 99.98 33.62 13.77 22813 6537.770 296 9988.835 16526.604
更深层次的问题：16G内存为什么不够？
至于16G内存为什么会不够，需要做个heap dump分析，这个jvm宕机了，没有写dump文件。。。下次喽…
3. 解决方案
解决方案： 调大内存
如果是社区版：在$HIVE_HOME/conf/hive-env.sh 中，修改Xmx大小，翻倍配置。
如果是cloudera管理，在管理页面改
总结
分析程序古怪问题，其中一种思路： 黑盒分析。直接使用JVM工具，看看jvm内部各个内存区的运行情况，确认问题后，在fix。
这里有个问题：我这次是运气好，jstat看到正在FullGC, 如果是刚重启完，jvm没什么问题，但很确定过几天会有问题，如何处理这种场景，给大家思考一下？