2. 修改hive/conf下hive-site.xml文件,在底部添加如下内容:
[html] view plaincopy <!-- <property> <name>hive.exec.scratchdir</name> <value>/usr/local/hive/tmp</value> </property> --> <property> <name>hive.querylog.location</name> <value>/usr/local/hive/logs</value> </property> <property> <name>hive.aux.jars.path</name> <value>file:///usr/local/hive/lib/hive-hbase-handler-0.8.0.jar,file:///usr/local/hive/lib/hbase-0.90.5.jar,file:///usr/local/hive/lib/zookeeper-3.3.2.jar</value> </property>
[html] view plaincopy
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and
then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information. at org.apache.hadoop.
hbase.zookeeper.ZooKeeperWatcher.
参考:http://blog.sina.com.cn/s/blog_410d18710100vlbq.html
现在可以尝试启动Hive了。
单节点启动:
|
> bin/hive -hiveconf hbase.master=master: 60000 |
集群启动:
> bin/hive -hiveconf hbase.zookeeper.quorum=slave
如何hive-site.xml文件中没有配置hive.aux.jars.path,则可以按照如下方式启动。
> bin/hive --auxpath /usr/local/hive/lib/hive-hbase-handler-0.8.0.jar, /usr/local/hive/lib/hbase-0.90.5.jar, /usr/local/hive/lib/zookeeper-3.3.2.jar -hiveconf hbase.zookeeper.quorum=slave
接下来可以做一些测试了。
1.创建hbase识别的数据库:
CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz");hbase.table.name 定义在hbase的table名称
hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE
pokes;
c) 使用sql导入hbase_table_1
[sql] view plaincopy
hive> INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=86;
3. 查看数据
[sql] view plaincopy
hive> select * from hbase_table_1;
这时可以登录Hbase去查看数据了.
> /usr/local/hbase/bin/hbase shell
hbase(main):001:0> describe 'xyz'
hbase(main):002:0> scan 'xyz'
hbase(main):003:0> put 'xyz','100','cf1:val','www.360buy.com'
这时在Hive中可以看到刚才在Hbase中插入的数据了。
hive> select * from hbase_table_1
4. hive访问已经存在的hbase
使用CREATE EXTERNAL TABLE
[sql] view plaincopy
CREATE EXTERNAL TABLE hbase_table_2(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:val")
TBLPROPERTIES("hbase.table.name" = "some_existing_table");
多列和多列族(Multiple Columns and Families)
1.创建数据库
Java代码
CREATE TABLE hbase_table_2(key int, value1 string, value2 int, value3 int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" = ":key,a:b,a:c,d:e"
);
2.插入数据
Java代码
INSERT OVERWRITE TABLE hbase_table_2 SELECT foo, bar, foo+1, foo+2
FROM pokes WHERE foo=98 OR foo=100;
这个有3个hive的列(value1和value2,value3),2个hbase的列族(a,d)
Hive的2列(value1和value2)对应1个hbase的列族(a,在hbase的列名称b,c),hive的另外1列(value3)对应列(e)位于列族(d)
3.登录hbase查看结构
Java代码
hbase(main):003:0> describe "hbase_table_2"
DESCRIPTION ENABLED
{NAME => 'hbase_table_2', FAMILIES => [{NAME => 'a', COMPRESSION => 'N true
ONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_M
EMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'd', COMPRESSION =>
'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN
_MEMORY => 'false', BLOCKCACHE => 'true'}]}
1 row(s) in 1.0630 seconds
4.查看hbase的数据
Java代码
hbase(main):004:0> scan 'hbase_table_2'
ROW COLUMN+CELL
100 column=a:b, timestamp=1297695262015, value=val_100
100 column=a:c, timestamp=1297695262015, value=101
100 column=d:e, timestamp=1297695262015, value=102
98 column=a:b, timestamp=1297695242675, value=val_98
98 column=a:c, timestamp=1297695242675, value=99
98 column=d:e, timestamp=1297695242675, value=100
2 row(s) in 0.0380 seconds
5.在hive中查看
Java代码
hive> select * from hbase_table_2;
OK
100 val_100 101 102
98 val_98 99 100
Time taken: 3.238 seconds
参考资料:
http://running.iteye.com/blog/898399
http://heipark.iteye.com/blog/1150648
http://www.javabloger.com/article/apache-hadoop-hive-hbase-integration.html
2. 修改hive/conf下hive-site.xml文件,在底部添加如下内容:
[html] view plaincopy <!-- <property> <name>hive.exec.scratchdir</name> <value>/usr/local/hive/tmp</value> </property> --> <property> <name>hive.querylog.location</name> <value>/usr/local/hive/logs</value> </property> <property> <name>hive.aux.jars.path</name> <value>file:///usr/local/hive/lib/hive-hbase-handler-0.8.0.jar,file:///usr/local/hive/lib/hbase-0.90.5.jar,file:///usr/local/hive/lib/zookeeper-3.3.2.jar</value> </property>
[html] view plaincopy
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and
then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information. at org.apache.hadoop.
hbase.zookeeper.ZooKeeperWatcher.
参考:http://blog.sina.com.cn/s/blog_410d18710100vlbq.html
现在可以尝试启动Hive了。
单节点启动:
|
> bin/hive -hiveconf hbase.master=master: 60000 |
集群启动:
> bin/hive -hiveconf hbase.zookeeper.quorum=slave
如何hive-site.xml文件中没有配置hive.aux.jars.path,则可以按照如下方式启动。
> bin/hive --auxpath /usr/local/hive/lib/hive-hbase-handler-0.8.0.jar, /usr/local/hive/lib/hbase-0.90.5.jar, /usr/local/hive/lib/zookeeper-3.3.2.jar -hiveconf hbase.zookeeper.quorum=slave
接下来可以做一些测试了。
1.创建hbase识别的数据库:
CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz");hbase.table.name 定义在hbase的table名称
hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE
pokes;
c) 使用sql导入hbase_table_1
[sql] view plaincopy
hive> INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=86;
3. 查看数据
[sql] view plaincopy
hive> select * from hbase_table_1;
这时可以登录Hbase去查看数据了.
> /usr/local/hbase/bin/hbase shell
hbase(main):001:0> describe 'xyz'
hbase(main):002:0> scan 'xyz'
hbase(main):003:0> put 'xyz','100','cf1:val','www.360buy.com'
这时在Hive中可以看到刚才在Hbase中插入的数据了。
hive> select * from hbase_table_1
4. hive访问已经存在的hbase
使用CREATE EXTERNAL TABLE
[sql] view plaincopy
CREATE EXTERNAL TABLE hbase_table_2(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:val")
TBLPROPERTIES("hbase.table.name" = "some_existing_table");
多列和多列族(Multiple Columns and Families)
1.创建数据库
Java代码
CREATE TABLE hbase_table_2(key int, value1 string, value2 int, value3 int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" = ":key,a:b,a:c,d:e"
);
2.插入数据
Java代码
INSERT OVERWRITE TABLE hbase_table_2 SELECT foo, bar, foo+1, foo+2
FROM pokes WHERE foo=98 OR foo=100;
这个有3个hive的列(value1和value2,value3),2个hbase的列族(a,d)
Hive的2列(value1和value2)对应1个hbase的列族(a,在hbase的列名称b,c),hive的另外1列(value3)对应列(e)位于列族(d)
3.登录hbase查看结构
Java代码
hbase(main):003:0> describe "hbase_table_2"
DESCRIPTION ENABLED
{NAME => 'hbase_table_2', FAMILIES => [{NAME => 'a', COMPRESSION => 'N true
ONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_M
EMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'd', COMPRESSION =>
'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN
_MEMORY => 'false', BLOCKCACHE => 'true'}]}
1 row(s) in 1.0630 seconds
4.查看hbase的数据
Java代码
hbase(main):004:0> scan 'hbase_table_2'
ROW COLUMN+CELL
100 column=a:b, timestamp=1297695262015, value=val_100
100 column=a:c, timestamp=1297695262015, value=101
100 column=d:e, timestamp=1297695262015, value=102
98 column=a:b, timestamp=1297695242675, value=val_98
98 column=a:c, timestamp=1297695242675, value=99
98 column=d:e, timestamp=1297695242675, value=100
2 row(s) in 0.0380 seconds
5.在hive中查看
Java代码
hive> select * from hbase_table_2;
OK
100 val_100 101 102
98 val_98 99 100
Time taken: 3.238 seconds
参考资料:
http://running.iteye.com/blog/898399
http://heipark.iteye.com/blog/1150648
http://www.javabloger.com/article/apache-hadoop-hive-hbase-integration.html