Hive HBase 整合(中文)

hive hbase整合，要求比较多，1.hive的得是0.6.0（当前最新的版本）
2.hive本身要求hadoop的最高版本是hadoop-0.20.2
3.要求hbase的版本是0.20.3，其他版本需要重新编译hive_hbase-handler
但是新版的hbase（0.90）变动特别大，根本无法从编译。这点比较恶心，hbase目前升级比较快，当前是0.90（从0.20.6直接跳到0.89），至于为什么这样跳跃，参考官方的解释http://wiki.apache.org/hadoop/Hbase/HBaseVersions


1)启动Hbase,
要求hbase-0.20.3，zookeeper-3.2.2
如果使用的不是hbase-0.20.3需要重新编译hive_hbase-handler.jar

2)单节点HBase的连接
./bin/hive -hiveconf hbase.master=master:60000

3)集群HBase的连接
1.启动zookeeper
2.启动hbase
3.启动hive，添加zookeeper的支持

./bin/hive -hiveconf hbase.zookeeper.quorum= master,slave-A,slave-B

./bin/hive -hiveconf hbase.zookeeper.quorum= master,slave-A,slave-B

./bin/hive -hiveconf hbase.zookeeper.quorum= master,slave-A,slave-B
//所有的zookeeper节点

二、插入数据
启动

./bin/hive --auxpath /data/soft/hive/lib/hive_hbase-handler.jar,/data/soft/hive/lib/hbase-0.20.3.jar,/data/soft/hive/lib/zookeeper-3.2.2.jar  -hiveconf hbase.zookeeper.quorum=slave-001,slave-002,slave-003

./bin/hive --auxpath /data/soft/hive/lib/hive_hbase-handler.jar,/data/soft/hive/lib/hbase-0.20.3.jar,/data/soft/hive/lib/zookeeper-3.2.2.jar  -hiveconf hbase.zookeeper.quorum=slave-001,slave-002,slave-003

./bin/hive --auxpath /data/soft/hive/lib/hive_hbase-handler.jar,/data/soft/hive/lib/hbase-0.20.3.jar,/data/soft/hive/lib/zookeeper-3.2.2.jar -hiveconf hbase.zookeeper.quorum=slave-001,slave-002,slave-003

hive
1.创建hbase识别的数据库

 CREATE TABLE hbase_table_1(key int, value string)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
 TBLPROPERTIES ("hbase.table.name" = "xyz");

CREATE TABLE hbase_table_1(key int, value string)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
 TBLPROPERTIES ("hbase.table.name" = "xyz");

CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz");
hbase.table.name 定义在hbase的table名称
hbase.columns.mapping 定义在hbase的列族



2.使用sql导入数据
i.预先准备数据
a)新建hive的数据表

CREATE TABLE pokes (foo INT, bar STRING);

CREATE TABLE pokes (foo INT, bar STRING);

CREATE TABLE pokes (foo INT, bar STRING);
b）批量插入数据

hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;

hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;

hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;
这个文件位于hive的安装目录下， examples/files/kv1.txt

ii.使用sql导入hbase_table_1

ii.使用sql导入hbase_table_1

ii.使用sql导入hbase_table_1

INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=86;

INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=86;

INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=86;
注意，默认的启动会报错的
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
启动的时候要添加

-auxpath /data/soft/hive/lib/hive_hbase-handler.jar,/data/soft/hive/lib/hbase-0.20.3.jar,/data/soft/hive/lib/zookeeper-3.2.2.jar

-auxpath /data/soft/hive/lib/hive_hbase-handler.jar,/data/soft/hive/lib/hbase-0.20.3.jar,/data/soft/hive/lib/zookeeper-3.2.2.jar

-auxpath /data/soft/hive/lib/hive_hbase-handler.jar,/data/soft/hive/lib/hbase-0.20.3.jar,/data/soft/hive/lib/zookeeper-3.2.2.jar

3查看数据

hive> select * from  hbase_table_1;

hive> select * from  hbase_table_1;

hive> select * from hbase_table_1;
会显示刚刚插入的数据
86      val_86

hbase
1.登录hbase

[root@master hbase]# ./bin/hbase shell

[root@master hbase]# ./bin/hbase shell

[root@master hbase]# ./bin/hbase shell

2.查看表结构

 hbase(main):001:0> describe 'xyz'
 DESCRIPTION                                                             ENABLED
  {NAME => 'xyz', FAMILIES => [{NAME => 'cf1', COMPRESSION => 'NONE', VE true
  RSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
   'false', BLOCKCACHE => 'true'}]}
 1 row(s) in 0.7460 seconds

hbase(main):001:0> describe 'xyz'
DESCRIPTION                                                             ENABLED
  {NAME => 'xyz', FAMILIES => [{NAME => 'cf1', COMPRESSION => 'NONE', VE true
  RSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
   'false', BLOCKCACHE => 'true'}]}
 1 row(s) in 0.7460 seconds

hbase(main):001:0> describe 'xyz' DESCRIPTION ENABLED {NAME => 'xyz', FAMILIES => [{NAME => 'cf1', COMPRESSION => 'NONE', VE true RSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]} 1 row(s) in 0.7460 seconds

3.查看加载的数据

 hbase(main):002:0> scan 'xyz'
 ROW                          COLUMN+CELL
  86                          column=cf1:val, timestamp=1297690405634, value=val_86

hbase(main):002:0> scan 'xyz'
ROW                          COLUMN+CELL
  86                          column=cf1:val, timestamp=1297690405634, value=val_86

hbase(main):002:0> scan 'xyz' ROW COLUMN+CELL 86 column=cf1:val, timestamp=1297690405634, value=val_86                                                     
1 row(s) in 0.0540 seconds
可以看到，在hive中添加的数据86，已经在hbase中了

4.添加数据

 ' hbase(main):008:0> put 'xyz','100','cf1:val','www.360buy.com'
 0 row(s) in 0.0630 seconds

' hbase(main):008:0> put 'xyz','100','cf1:val','www.360buy.com'
 0 row(s) in 0.0630 seconds

' hbase(main):008:0> put 'xyz','100','cf1:val','www.360buy.com' 0 row(s) in 0.0630 seconds

Hive
参看hive中的数据

 hive> select * from hbase_table_1;
 OK
 100     www.360buy.com
 86      val_86
 Time taken: 8.661 seconds

hive> select * from hbase_table_1;
OK
 100     www.360buy.com
 86      val_86
 Time taken: 8.661 seconds

hive> select * from hbase_table_1; OK 100 www.360buy.com 86 val_86 Time taken: 8.661 seconds
刚刚在hbase中插入的数据，已经在hive里了

hive访问已经存在的hbase
使用CREATE EXTERNAL TABLE

 CREATE EXTERNAL TABLE hbase_table_2(key int, value string)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:val")
 TBLPROPERTIES("hbase.table.name" = "some_existing_table");

CREATE EXTERNAL TABLE hbase_table_2(key int, value string)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:val")
 TBLPROPERTIES("hbase.table.name" = "some_existing_table");

CREATE EXTERNAL TABLE hbase_table_2(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:val") TBLPROPERTIES("hbase.table.name" = "some_existing_table");



三、多列和多列族（Multiple Columns and Families）
1．创建数据库

 CREATE TABLE hbase_table_2(key int, value1 string, value2 int, value3 int)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES (
 "hbase.columns.mapping" = ":key,a:b,a:c,d:e"
);

CREATE TABLE hbase_table_2(key int, value1 string, value2 int, value3 int)
 STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
 "hbase.columns.mapping" = ":key,a:b,a:c,d:e"
);

CREATE TABLE hbase_table_2(key int, value1 string, value2 int, value3 int) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( "hbase.columns.mapping" = ":key,a:b,a:c,d:e" );
2.插入数据

 INSERT OVERWRITE TABLE hbase_table_2 SELECT foo, bar, foo+1, foo+2
 FROM pokes WHERE foo=98 OR foo=100;

INSERT OVERWRITE TABLE hbase_table_2 SELECT foo, bar, foo+1, foo+2
 FROM pokes WHERE foo=98 OR foo=100;

INSERT OVERWRITE TABLE hbase_table_2 SELECT foo, bar, foo+1, foo+2 FROM pokes WHERE foo=98 OR foo=100;

这个有3个hive的列（value1和value2，value3），2个hbase的列族（a，d）
Hive的2列（value1和value2）对应1个hbase的列族（a，在hbase的列名称b，c），hive的另外1列（value3）对应列（e）位于列族（d）

3.登录hbase查看结构

 hbase(main):003:0> describe "hbase_table_2"
 DESCRIPTION                                                             ENABLED
  {NAME => 'hbase_table_2', FAMILIES => [{NAME => 'a', COMPRESSION => 'N true
  ONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_M
  EMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'd', COMPRESSION =>
  'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN
  _MEMORY => 'false', BLOCKCACHE => 'true'}]}
 1 row(s) in 1.0630 seconds

hbase(main):003:0> describe "hbase_table_2"
DESCRIPTION                                                             ENABLED
  {NAME => 'hbase_table_2', FAMILIES => [{NAME => 'a', COMPRESSION => 'N true
  ONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_M
  EMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'd', COMPRESSION =>
  'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN
  _MEMORY => 'false', BLOCKCACHE => 'true'}]}
 1 row(s) in 1.0630 seconds

hbase(main):003:0> describe "hbase_table_2" DESCRIPTION ENABLED {NAME => 'hbase_table_2', FAMILIES => [{NAME => 'a', COMPRESSION => 'N true ONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_M EMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'd', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN _MEMORY => 'false', BLOCKCACHE => 'true'}]} 1 row(s) in 1.0630 seconds

4.查看hbase的数据

 hbase(main):004:0> scan 'hbase_table_2'
 ROW                          COLUMN+CELL
  100                         column=a:b, timestamp=1297695262015, value=val_100
  100                         column=a:c, timestamp=1297695262015, value=101
  100                         column=d:e, timestamp=1297695262015, value=102
  98                          column=a:b, timestamp=1297695242675, value=val_98
  98                          column=a:c, timestamp=1297695242675, value=99
  98                          column=d:e, timestamp=1297695242675, value=100
 2 row(s) in 0.0380 seconds

hbase(main):004:0> scan 'hbase_table_2'
ROW                          COLUMN+CELL
  100                         column=a:b, timestamp=1297695262015, value=val_100
  100                         column=a:c, timestamp=1297695262015, value=101
  100                         column=d:e, timestamp=1297695262015, value=102
  98                          column=a:b, timestamp=1297695242675, value=val_98
  98                          column=a:c, timestamp=1297695242675, value=99
  98                          column=d:e, timestamp=1297695242675, value=100
 2 row(s) in 0.0380 seconds

hbase(main):004:0> scan 'hbase_table_2' ROW COLUMN+CELL 100 column=a:b, timestamp=1297695262015, value=val_100 100 column=a:c, timestamp=1297695262015, value=101 100 column=d:e, timestamp=1297695262015, value=102 98 column=a:b, timestamp=1297695242675, value=val_98 98 column=a:c, timestamp=1297695242675, value=99 98 column=d:e, timestamp=1297695242675, value=100 2 row(s) in 0.0380 seconds

5.在hive中查看

 hive> select * from hbase_table_2;
 OK
 100     val_100 101     102
 98      val_98  99      100
 Time taken: 3.238 seconds