hive0.13整合hbase 0.96.2 hadoop2.2.0

最近研究了一下hive与hbase的整合，使用的都是各自的最新release， hive0.13, hbase0.96.2，整合的过程其实挺简单的，大致需要注意的地方如下：

1. hive的配置文件hive-site.xml需要添加的内容：

<property>
  <name>hive.aux.jars.path</name>
  <value>file:///home/grid/hive/lib/hive-hbase-handler-0.13.0.jar,file:///home/grid/hive/lib/hbase-client-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-common-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-common-0.96.2-hadoop2-tests.jar,file:///home/grid/hive/lib/hbase-protocol-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/hbase-server-0.96.2-hadoop2.jar,file:///home/grid/hive/lib/htrace-core-2.04.jar,file:///home/grid/hive/lib/zookeeper-3.4.6.jar,file:///home/grid/hive/lib/protobuf-java-2.5.0.jar,file:///home/grid/hive/lib/guava-11.0.2.jar</value>
</property>

 

<property>
<name>hbase.zookeeper.quorum</name>
<value>server1,server2</value>
</property>

 

2.安装hbase的过程不再累述，可以参考http://blog.csdn.net/codestinity/article/details/6947464

 

最后在双方查询都能查到数据，但是在hive中向hbase插入数据的时候出现了问题，错误信息如下：

java.io.FileNotFoundException: File does not exist: hdfs://*.*.*.*:9000/home/grid/hbase/lib/hbase-hadoop-compat-0.96.2-hadoop2.jar
 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
 at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
 at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
 at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:99)
 at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
 at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
 at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)
 at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
 at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
 at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
 at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
 at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
 at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
 at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:740)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://*.*.*.*:9000/home/grid/hbase/lib/hbase-hadoop-compat-0.96.2-hadoop2.jar)'
Execution failed with exit status: 1
Obtaining error information

Task failed!
Task ID:
  Stage-0

Logs:

/tmp/root/hive.log
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

 

这是需要将使用的jar包上传到hdfs文件系统中，所以，需要哪个就上传哪个吧，用put命令。

最后成功实现hive与hbase的整合。