Hive 关于修改表结构其他客户端不同步的问题

项目混用了多个HiveServer2 和 HiveCli。一段时间以后，出现了一个奇怪的问题。一个客户端修改了表结构以后，其他客户的访问的时候，却发现表结构并没有被同步修改。

 

查了不少资料，最后发现是因为HiveServer2 服务器开启了缓存的问题，记录如下：

 

问题出现原因是由于 L2 cache 这种缓存机制不支持分布式缓存。

 

关闭 L2 cache 后问题解决

 

 

 datanucleus.cache.level2.type 

  none 

 

因没有 HiveServer2 的配置权限，发现一个临时的解决办法。登陆到执行作业的 HiveServer2 上，使用 set 语句关闭这个参数，然后访问修改的表（desc 一下就可以），退出。下次再访问的时候就发现缓存已经更新了。

 

set datanucleus.cache.level2.type=none;

Hive 默认是关闭了这个参数的：

 

datanucleus.cache.level2.type       Default Value: none in Hive 0.9 and later; SOFT in Hive 0.7 to 0.8.1     Added In: Hive 0.7.0   NONE = disable the datanucleus level 2 cache, SOFT = soft reference based cache, WEAK = weak reference based cache. Warning note: For most Hive installations, enabling the datanucleus cache can lead to correctness issues, and is dangerous. This should be left as "none".

 

JDO Cache(高速缓存)：

 

Cachingis an essential mechanism in providing efficient usage of resources in manysystems. Data management using JDO is no different and provides a definition ofcaching at 2 levels. Caching allows objects to be retained and returned rapidlywithout having to make an extra call to the datastore. The 2 levels of cachingavailable with DataNucleus are

 

1.Level 1 Cache - mandated by the JDO specification, and represents the cachingof instances within a PersistenceManager

 

2.Level 2 Cache - represents the caching of instances within aPersistenceManagerFactory (across multiple PersistenceManager's)

 

Youcan think of a cache as a Map, with values referred to by keys. In the case ofJDO, the key is the object identity (identity is unique in JDO).

 

 

Bydefault the Level 2 Cache is enabled. The user can configure the Level 2 Cacheif they so wish. This is controlled by use of the persistence propertydatanucleus.cache.level2.type. You set this to "type" of cacherequired. With the Level 2 Cache you currently have the following options.

 

1.none - turn OFF Level 2 caching.

 

2.weak - use the internal (weak reference based) L2 cache. Provides support forthe JDO 2 interface of being able to pin objects into the cache, and unpin themwhen required.This option does notsupport distributed caching, solely running within the JVM of the clientapplication. Weak references are held to non pinned objects.

 

3.soft - use the internal (soft reference based) L2 cache. Provides support forthe JDO 2 interface of being able to pin objects into the cache, and unpin themwhen required.This option does notsupport distributed caching, solely running within the JVM of the clientapplication. Soft references are held to non pinned objects.

 

 

参考：

http://blog.csdn.net/lalaguozhe/article/details/9184593

https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties