起因
最近公司增加了多台hive客户端来访问hive元数据，半夜的时候hive报错出现如下信息：
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
... 7 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
... 12 more
Caused by: MetaException(message:Metastore contains multiple versions)
at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:356)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:171)
... 17 more 如报错提示，hive metastore出现了多个版本，去hive的元数据库查看表VERSION，发现多了一条记录
select * from VERSION;
1 1.1.0 Set by MetaStore [email protected]
2 1.1.0 Set by MetaStore [email protected] #这条是多出的解决过程
查资料
google了一些资料，网上也有人提过，如HIVE-9543，网上大家说的解决方法有如下
设置datanucleus.autoCreateSchema=false
此配置官网介绍：
Default Value: true
Added In: Hive 0.7.0
Removed In: Hive 2.0.0 with HIVE-6113, replaced by datanucleus.schema.autoCreateAll
Creates necessary schema on a startup if one does not exist. Set this to false, after creating it once.
#意思就是这个参数再hive元数据初始化的时候用到，之后就可以设置为false禁用设置此参数为false后，继续观察，错误还是会再次出现
查看日志及报错
#查看hive运行日志发现多版本出现的时候，有如下日志
Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version
#意思是说在metastore中未找到版本信息，hive.metastore.schema.verification未禁用，因此记录下版本信息，也就是往版本表中插入一条记录
#接着结合之前的报错，看看报错类如下：
Caused by: MetaException(message:Metastore contains multiple versions)
at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)查看源代码
通过查看源代码发现，相关代码如下
##同步方法checkSchema()
private synchronized void checkSchema() throws MetaException {
// recheck if it got verified by another thread while we were waiting
if (isSchemaVerified.get()) {
return;
}
//获取hive配置，也就是hive.metastore.schema.verification的值
boolean strictValidation =
HiveConf.getBoolVar(getConf(), HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
// read the schema version stored in metastore db
//读取metastore的版本信息
String schemaVer = getMetaStoreSchemaVersion();
if (schemaVer == null) {
//如果版本信息未找到，这个时候strictValidation为true的时候直接抛出异常
if (strictValidation) {
throw new MetaException("Version information not found in metastore. ");
} else {
//否则，调用方法插入版本信息，也就是之前日志所说的
LOG.warn("Version information not found in metastore. "
+ HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
" is not enabled so recording the schema version " +
MetaStoreSchemaInfo.getHiveSchemaVersion());
setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
"Set by MetaStore " + USER + "@" + HOSTNAME);
}
}
##setMetaStoreSchemaVersion方法如下
public void setMetaStoreSchemaVersion(String schemaVersion, String comment) throws MetaException {
MVersionTable mSchemaVer;
boolean commited = false;
//此参数控制了记录version信息
boolean recordVersion =
HiveConf.getBoolVar(getConf(), HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION_RECORD_VERSION);
//参数为false，则返回，不记录版本信息，否则将插入版本信息
if (!recordVersion) {
LOG.warn("setMetaStoreSchemaVersion called but recording version is disabled: " +
"version = " + schemaVersion + ", comment = " + comment);
return;
}
try {
mSchemaVer = getMSchemaVersion();
} catch (NoSuchObjectException e) {
// if the version doesn't exist, then create it
mSchemaVer = new MVersionTable();
}
##查看HiveConf中METASTORE_SCHEMA_VERIFICATION_RECORD_VERSION可知hive.metastore.schema.verification.record.version默认为true,则允许记录版本信息 解决方案
通过以上的源码查看，解决方案已经出来了，其实方法有几种，我选取的做法是将hive.metastore.schema.verification.record.version设置为fals
当然你也可以关闭版本校验
遗留问题
- 看网上说的hive多版本问题似乎是并发、网络引起的，源代码中为什么没有获取到metastore的版本schema信息，这是一个问题，还有待源码探究
- 几个参数都可以起到在代码流程中阻断记录版本信息的操作，哪种是无风险的，还有待深究
感受
- 发现问题第一时间上谷歌查看似乎是我现在的习惯，总想着网上有现成的解决方案，但是有些东西拿过来还是自己要思考一下
- 大数据组件繁多，对于源码的探究永无止境，不能只停留于表面，要花时间去一点点积累