一日一坑之：hadoop3.x VS hive3.x
最近安装了新版本的hadoop3.x版本，按照老思路此时需要安装hive3.x用来搭配我们崭新的hadoop，重新翻出来我的泛黄的hive2.x版本的安装说明书安装了一遍，就在此时坑来了。
坑一：FAILED: ParseException line 1:5 cannot recognize input near 'show' 'database' '' in ddl statement
错误信息：
NoViableAltException(70@[651:1: ddlStatement : ( createDatabaseStatement | switchDatabaseStatement | dropDatabaseStatement | createTableStatement | dropTableStatement | truncateTableStatement | alterStatement | descStatement | showStatement | metastoreCheck | createViewStatement | dropViewStatement | createFunctionStatement | createMacroStatement | createIndexStatement | dropIndexStatement | dropFunctionStat ent | dropMacroStatement | analyzeStatement | lockStatement | unlockStatement | lockDatabase | unlockDatabase | createRoleStatement | d▽opRoleStatement | grantPrivileges | revokePrivileges | showGrants | showRoleGrants | showRolePrincipals | showRoles | grantRole | revokeRole | setRole | showCurrentRole );])
at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
at org.antlr.runtime.DFA.predict(DFA.java:116)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2084)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1392)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1030)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1026)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:952)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:800)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
FAILED: ParseException line 1:5 cannot recognize input near 'show' 'database' '' in ddl statement
错误原因：忘加s了
解决方法：将 show database 改成 show databases
坑二：FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStore
错误信息：
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStore错误原因：没有初始化元数据数据库
解决办法：schematool -dbType mysql -initSchema 没配置环境变量，去bin下找
该内容执行过程中也会出问题
坑三：这种坑没有错误，要看说明
错误信息：在hive-default.xml.template中
错误原因：没有hive-site.xml
解决办法：cp hive-default.xml.template hive-site.xml
坑四：Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
错误信息：
Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: **Illegal character entity: expansion character (code 0x8
at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]**
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3003)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2931)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4990)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5063)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5150)
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:5098)
at org.apache.hive.beeline.HiveSchemaTool.(HiveSchemaTool.java:96)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java:2456)
at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java:2403)
at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java:2369)
at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1515)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2828)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3257)
at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3063)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2986)
... 15 more 错误原因：配置文件中多了一个违法字符
解决办法：vi hive-site.xml 也有可能还没到这一步 vi hive-default.xml
跳到3215行或者3232行，删除掉
坑五：Error: Syntax error: Encountered "" at line 1, column 64.
错误信息：
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Connecting to jdbc:derby:;databaseName=metastore_db;create=true
Connected to: Apache Derby (version 10.10.2.0 - (1582446))
Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:derby:> !autocommit on
Autocommit status: true
0: jdbc:derby:> /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */
Error: Syntax error: Encountered "" at line 1, column 64. (state=42X01,code=30000)
Closing: 0: jdbc:derby:;databaseName=metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
错误原因：没有指定元数据库位置和连接信息或错误连接信息
网上一些解决办法都是这样的，在hive-site.xml中配置这个信息，但还会出现同样错误
javax.jdo.option.ConnectionURL
jdbc:mysql://hadoop1:3306/hive?useSSL=false
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
javax.jdo.option.ConnectionUserName
root
javax.jdo.option.ConnectionPassword
123456
hive.metastore.schema.verification
false
解决办法：vi hive-site.xml
按照配置去找 568行，584行，797行，1101行，1126行。
具体找法是进入vi不进入编辑状态，输入行号加G例如568G
坑6：Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
错误信息：
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/app/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 43f139fe-0200-4f89-906b-2fe8109c3b86
Logging initialized using configuration in file:/home/hadoop/app/hive/conf/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
at org.apache.hadoop.fs.Path.initialize(Path.java:259)
at org.apache.hadoop.fs.Path.(Path.java:217)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:710)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:627)
at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:256)
... 12 more 错误原因：没有设置数据临时数据位置或临时数据位置解析失败
解决办法：
第一步：在$HOME_HIVE也就是hive家目录创建三个文件夹tmp，logs，download分别存储临时文件，查询日志，下载的资源
cd $HIVE_HOME
mkdir tmp
mkdir logs
mkdir download第二步：参考下面内容配置参数
hive.exec.local.scratchdir
$HIVE_HOME/tmp
Local scratch space for Hive jobs
hive.querylog.location
$HIVE_HOME/logs
Location of Hive run time structured log file
hive.downloaded.resources.dir
$HIVE_HOME/download
Temporary local directory for added resources in the remote file system.
具体操作：vi hive-site.xml
跳至142行，147行，1845行，修改内容
如果在show databases还出现错误，看下一个
坑7：IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI:
错误信息：
hive> show databases;
FAILED: IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI: file:./$HIVE_HOME/tmp/1ef383fc-eff9-475c-bd54-bd87a5d16414/hive_2019-09-03_22-46-44_530_11458349错误原因：解析不了$HIVE_HOME
解决办法：错误6中的$HIVE_HOME都改成真正对应的路径如/home/hadoop/app/hive即可
坑8：FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
错误信息：
hive> select * from xixi;
OK
Time taken: 4.745 seconds
hive> insert into xixi values(1);
Query ID = hadoop_20190910153718_1860e7ec-88d5-441b-9854-81e6872ce034
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapreduce.job.reduces=
Starting Job = job_1567403212809_0004, Tracking URL = http://hadoop1:8088/proxy/application_1567403212809_0004/
Kill Command = /home/hadoop/app/hadoop/bin/mapred job -kill job_1567403212809_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-09-10 15:38:42,612 Stage-1 map = 0%, reduce = 0%
2019-09-10 15:38:44,268 Stage-1 map = 100%, reduce = 100%
Ended Job = job_1567403212809_0004 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1567403212809_0004_m_000000 (and more) from job job_1567403212809_0004
Task with the most failures(4):
-----
Task ID:
task_1567403212809_0004_m_000000
URL:
http://hadoop1:8088/taskdetails.jsp?jobid=job_1567403212809_0004&tipid=task_1567403212809_0004_m_000000
-----
Diagnostic Messages for this Task:
[2019-09-10 15:38:42.526]Container [pid=19303,containerID=container_1567403212809_0004_01_000005] is running 443861504B beyond the 'VIRTUAL' memory limit. Current usage: 64.2 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1567403212809_0004_01_000005 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 19315 19303 19303 19303 (java) 522 330 2582822912 16141 /data/app/java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1567403212809_0004/container_1567403212809_0004_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.85.130 36324 attempt_1567403212809_0004_m_000000_3 5
|- 19303 19302 19303 19303 (bash) 0 0 115896320 306 /bin/bash -c /data/app/java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1567403212809_0004/container_1567403212809_0004_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.85.130 36324 attempt_1567403212809_0004_m_000000_3 5 1>/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005/stdout 2>/home/hadoop/app/hadoop/logs/userlogs/application_1567403212809_0004/container_1567403212809_0004_01_000005/stderr
[2019-09-10 15:38:42.963]Container killed on request. Exit code is 143
[2019-09-10 15:38:43.258]Container exited with a non-zero exit code 143.
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL 错误原因：每个map分配的内存过小
解决办法：
hive> set mapreduce.map.memory.mb=2048;