sqoop命令，oracle导入到hdfs、hbase、hive

1.测试Oracle连接

bin/sqoop list-databases --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password

2.导入hdfs

3.1 导入

bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --target-dir /sqoop/oracle/esb_service_param --split-by PARAM_ID

3.2验证

hdfs dfs -ls /sqoop/oracle/esb_service_param
hdfs dfs -cat /sqoop/oracle/esb_service_param/part-m-00000

4.导入hbase

4.0直接新建并导入

bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-create-table --hbase-table orcl_esb_service_param --hbase-row-key PARAM_ID --column-family info

报错

16/09/07 14:05:30 INFO mapreduce.HBaseImportJob: Creating missing HBase table orcl_esb_service_param
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)V

原因

版本不匹配，环境上版本固定，所以选用手动新建hbase表，再导入数据

4.1新建hbase表

hbase shell
create 'orcl_esb_service_param', 'region'

4.2导入mysql数据到hbase

bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-table orcl_esb_service_param --hbase-row-key PARAM_ID --column-family region

额外例子，导入hbase生成复合的rowKey

bin/sqoop import -D sqoop.hbase.add.row.key=true --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-table lsy_test_multiple_table --hbase-row-key PARAM_ID,SERVICE_ID --column-family data

4.3验证

scan 'orcl_esb_service_param'
count 'orcl_esb_service_param'

5.导入hive

5.1新建hive表，并导入数据

bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hive-import --create-hive-table --target-dir /user/hive/warehouse/orcl_esb_service_param --hive-table orcl_esb_service_param

5.2 验证

show tables;
select * from orcl_esb_service_param;
select count(*) from orcl_esb_service_param;

6.增量

增量到hdfs

bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --target-dir /sqoop/oracle/esb_service_param --split-by PARAM_ID --incremental append --check-column PARAM_ID --last-value 33

返回

16/09/07 15:27:17 INFO tool.ImportTool: --incremental append
16/09/07 15:27:17 INFO tool.ImportTool: --check-column PARAM_ID
16/09/07 15:27:17 INFO tool.ImportTool: --last-value 34
16/09/07 15:27:17 INFO tool.ImportTool: (Consider saving this with 'sqoop job --create')

校验

hdfs dfs -cat /sqoop/oracle/esb_service_param/part-m-00004

增量到hbase

bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-table orcl_esb_service_param --hbase-row-key PARAM_ID --column-family region --incremental append --check-column PARAM_ID --last-value 33

返回

16/09/07 15:59:49 INFO tool.ImportTool: --incremental append
16/09/07 15:59:49 INFO tool.ImportTool: --check-column PARAM_ID
16/09/07 15:59:49 INFO tool.ImportTool: --last-value 34
16/09/07 15:59:49 INFO tool.ImportTool: (Consider saving this with 'sqoop job --create')

校验

scan 'orcl_esb_service_param'

7.参考

Sqoop 1.4.6 UserGuide