连接Oracle与Hadoop(3) 使用OLH加载Hbase到Oracle

OLH是Oracle Loader for Hadoop的缩写，Oracle的大数据连接器（BDC）的一个组件，可将多种数据格式从HDFS上加载到Oracle数据库库中。

 

本文在同一台服务器上模拟oracle数据库与hadoop集群，实验目标：使用OLH从Hadoop端的Hbase表加载数据到Oracle表中。

对于加载Hbase表场景，需要通过Hive创建一个基于Hbase的映射表，再通过Hive接口导入到Oracle表中。

 

Oracle端：

服务器	系统用户	安装软件	软件安装路径
Server1	oracle	Oracle Database 12.1.0.2	/u01/app/oracle/product/12.1.0/dbhome_1

 

Hadoop集群端：

服务器	系统用户	安装软件	软件安装路径
Server1	hadoop	Hadoop 2.6.2	/home/hadoop/hadoop-2.6.2
		Hive 1.1.1	/home/hadoop/hive-1.1.1
		Hbase 1.1.2	/home/hadoop/hbase-1.1.2
		jdk1.8.0_65	/home/hadoop/jdk1.8.0_65
		OLH 3.5.0	/home/hadoop/oraloader-3.5.0-h2

 

• 部署Hadoop/Hive/Hbase/OLH软件

将Hadoop/Hive/Hbase/OLH软件解压到相应目录

1. [hadoop@server1 ~]$ tree -L 1
2. ├── hadoop-2.6.2
3. ├── hbase-1.1.2
4. ├── hive-1.1.1
5. ├── jdk1.8.0_65
6. ├── oraloader-3.5.0-h2

 

• 配置Hadoop/Hive/Hbase/OLH环境变量

1. export JAVA_HOME=/home/hadoop/jdk1.8.0_65
2.  
3. export HADOOP_USER_NAME=hadoop
4. export HADOOP_YARN_USER=hadoop
5. export HADOOP_HOME=/home/hadoop/hadoop-2.6.2
6. export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
7. export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
8. export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec
9. export HADOOP_COMMON_HOME=${HADOOP_HOME}
10. export HADOOP_HDFS_HOME=${HADOOP_HOME}
11. export HADOOP_MAPRED_HOME=${HADOOP_HOME}
12. export HADOOP_YARN_HOME=${HADOOP_HOME}
13. export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
14. export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
15.  
16. export HIVE_HOME=/home/hadoop/hive-1.1.1
17. export HIVE_CONF_DIR=${HIVE_HOME}/conf
18.  
19. export HBASE_HOME=/home/hadoop/hbase-1.1.2
20. export HBASE_CONF_DIR=/home/hadoop/hbase-1.1.2/conf
21.  
22. export OLH_HOME=/home/hadoop/oraloader-3.5.0-h2
23.  
24. export HADOOP_CLASSPATH=/usr/share/java/mysql-connector-java.jar
25. export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*:$HIVE_CONF_DIR:$HBASE_HOME/lib/*:$HBASE_CONF_DIR
26. export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$OLH_HOME/jlib/*
27.  
28. export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$PATH

 

• 配置Hadoop/Hive/Hbase软件

core-site.xml

1. <configuration>
2. <property>
3.         <name>fs.defaultFS</name>
4.         <value>hdfs://server1:8020</value>
5. </property>
6. <property>
7.         <name>fs.checkpoint.period</name>
8.         <value>3600</value>
9. </property>
10. <property>
11.         <name>fs.checkpoint.size</name>
12.         <value>67108864</value>
13. </property>
14. <property>
15.         <name>hadoop.proxyuser.hadoop.groups</name>
16.         <value>*</value>
17. </property>
18. <property>
19.         <name>hadoop.proxyuser.hadoop.hosts</name>
20.         <value>*</value>
21. </property>
22. </configuration>

 

hdfs-site.xml

1. <configuration>
2. <property>
3.         <name>hadoop.tmp.dir</name>
4.         <value>file:///home/hadoop</value>
5. </property>
6. <property>
7.         <name>dfs.namenode.name.dir</name>
8.         <value>file:///home/hadoop/dfs/nn</value>
9. </property>
10. <property>
11.         <name>dfs.datanode.data.dir</name>
12.         <value>file:///home/hadoop/dfs/dn</value>
13. </property>
14. <property>
15.         <name>dfs.namenode.checkpoint.dir</name>
16.         <value>file:///home/hadoop/dfs/sn</value>
17. </property>
18. <property>
19.         <name>dfs.replication</name>
20.         <value>1</value>
21. </property>
22. <property>
23.         <name>dfs.permissions.superusergroup</name>
24.         <value>supergroup</value>
25. </property>
26. <property>
27.         <name>dfs.namenode.http-address</name>
28.         <value>server1:50070</value>
29. </property>
30. <property>
31.         <name>dfs.namenode.secondary.http-address</name>
32.         <value>server1:50090</value>
33. </property>
34. <property>
35.         <name>dfs.webhdfs.enabled</name>
36.         <value>true</value>
37. </property>
38. </configuration>

 

yarn-site.xml

1. <configuration>
2. <property>
3.         <name>yarn.resourcemanager.scheduler.address</name>
4.         <value>server1:8030</value>
5. </property>
6. <property>
7.         <name>yarn.resourcemanager.resource-tracker.address</name>
8.         <value>server1:8031</value>
9. </property>
10. <property>
11.         <name>yarn.resourcemanager.address</name>
12.         <value>server1:8032</value>
13. </property>
14. <property>
15.         <name>yarn.resourcemanager.admin.address</name>
16.         <value>server1:8033</value>
17. </property>
18. <property>
19.         <name>yarn.resourcemanager.webapp.address</name>
20.         <value>server1:8088</value>
21. </property>
22. <property>
23.         <name>yarn.nodemanager.local-dirs</name>
24.         <value>file:///home/hadoop/yarn/local</value>
25. </property>
26. <property>
27.         <name>yarn.nodemanager.log-dirs</name>
28.         <value>file:///home/hadoop/yarn/logs</value>
29. </property>
30. <property>
31.         <name>yarn.log-aggregation-enable</name>
32.         <value>true</value>
33. </property>
34. <property>
35.         <name>yarn.nodemanager.remote-app-log-dir</name>
36.         <value>/yarn/apps</value>
37. </property>
38. <property>
39.         <name>yarn.app.mapreduce.am.staging-dir</name>
40.         <value>/user</value>
41. </property>
42. <property>
43.         <name>yarn.nodemanager.aux-services</name>
44.         <value>mapreduce_shuffle</value>
45. </property>
46. <property>
47.         <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
48.         <value>org.apache.hadoop.mapred.ShuffleHandler</value>
49. </property>
50. <property>
51.         <name>yarn.log.server.url</name>
52.         <value>http://server1:19888/jobhistory/logs/</value>
53. </property>
54. </configuration>

 

• 配置Hive

hive-site.xml

1. <configuration>
2. <property>
3.         <name>javax.jdo.option.ConnectionURL</name>
4.         <value>jdbc:mysql://server1:3306/metastore?createDatabaseIfNotExist=true</value>
5. </property>
6. <property>
7.         <name>javax.jdo.option.ConnectionDriverName</name>
8.         <value>com.mysql.jdbc.Driver</value>
9. </property>
10. <property>
11.         <name>javax.jdo.option.ConnectionUserName</name>
12.         <value>hive</value>
13. </property>
14. <property>
15.         <name>javax.jdo.option.ConnectionPassword</name>
16.         <value>123456</value>
17. </property>
18. <property>
19.         <name>hbase.master</name>
20.         <value>server1:16000</value>
21. </property>
22. <property>
23.         <name>hive.cli.print.current.db</name>
24.         <value>true</value>
25. </property>
26. </configuration>

 

• 配置Hbase

Hbase-site.xml

1. <configuration>
2. <property>
3.         <name>hbase.rootdir</name>
4.         <value>hdfs://server1/hbase</value>
5. </property>
6. <property>
7.         <name>hbase.cluster.distributed</name>
8.         <value>true</value>
9. </property>
10. <property>
11.         <name>hbase.master.port</name>
12.         <value>16000</value>
13. </property>
14. </configuration>

 

• 创建一张Hbase表

1. create 'WLSSERVER' , 'LOG'
2. put 'WLSSERVER', 'row1', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:16-PM-PDT'
3. put 'WLSSERVER', 'row1', 'LOG:CATEGORY', 'Notice'
4. put 'WLSSERVER', 'row1', 'LOG:TYPE', 'WebLogicServer'
5. put 'WLSSERVER', 'row1', 'LOG:SERVERNAME', 'AdminServer'
6. put 'WLSSERVER', 'row1', 'LOG:CODE', 'BEA-000365'
7. put 'WLSSERVER', 'row1', 'LOG:MSG', 'Server state changed to STANDBY'
8. put 'WLSSERVER', 'row2', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:17-PM-PDT'
9. put 'WLSSERVER', 'row2', 'LOG:CATEGORY', 'Notice'
10. put 'WLSSERVER', 'row2', 'LOG:TYPE', 'WebLogicServer'
11. put 'WLSSERVER', 'row2', 'LOG:SERVERNAME', 'AdminServer'
12. put 'WLSSERVER', 'row2', 'LOG:CODE', 'BEA-000365'
13. put 'WLSSERVER', 'row2', 'LOG:MSG', 'Server state changed to STARTING'
14. put 'WLSSERVER', 'row3', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:18-PM-PDT'
15. put 'WLSSERVER', 'row3', 'LOG:CATEGORY', 'Notice'
16. put 'WLSSERVER', 'row3', 'LOG:TYPE', 'WebLogicServer'
17. put 'WLSSERVER', 'row3', 'LOG:SERVERNAME', 'AdminServer'
18. put 'WLSSERVER', 'row3', 'LOG:CODE', 'BEA-000365'
19. put 'WLSSERVER', 'row3', 'LOG:MSG', 'Server state changed to ADMIN'
20. put 'WLSSERVER', 'row4', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:19-PM-PDT'
21. put 'WLSSERVER', 'row4', 'LOG:CATEGORY', 'Notice'
22. put 'WLSSERVER', 'row4', 'LOG:TYPE', 'WebLogicServer'
23. put 'WLSSERVER', 'row4', 'LOG:SERVERNAME', 'AdminServer'
24. put 'WLSSERVER', 'row4', 'LOG:CODE', 'BEA-000365'
25. put 'WLSSERVER', 'row4', 'LOG:MSG', 'Server state changed to RESUMING'
26. put 'WLSSERVER', 'row5', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:20-PM-PDT'
27. put 'WLSSERVER', 'row5', 'LOG:CATEGORY', 'Notice'
28. put 'WLSSERVER', 'row5', 'LOG:TYPE', 'WebLogicServer'
29. put 'WLSSERVER', 'row5', 'LOG:SERVERNAME', 'AdminServer'
30. put 'WLSSERVER', 'row5', 'LOG:CODE', 'BEA-000331'
31. put 'WLSSERVER', 'row5', 'LOG:MSG', 'Started WebLogic AdminServer'
32. put 'WLSSERVER', 'row6', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:21-PM-PDT'
33. put 'WLSSERVER', 'row6', 'LOG:CATEGORY', 'Notice'
34. put 'WLSSERVER', 'row6', 'LOG:TYPE', 'WebLogicServer'
35. put 'WLSSERVER', 'row6', 'LOG:SERVERNAME', 'AdminServer'
36. put 'WLSSERVER', 'row6', 'LOG:CODE', 'BEA-000365'
37. put 'WLSSERVER', 'row6', 'LOG:MSG', 'Server state changed to RUNNING'
38. put 'WLSSERVER', 'row7', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:22-PM-PDT'
39. put 'WLSSERVER', 'row7', 'LOG:CATEGORY', 'Notice'
40. put 'WLSSERVER', 'row7', 'LOG:TYPE', 'WebLogicServer'
41. put 'WLSSERVER', 'row7', 'LOG:SERVERNAME', 'AdminServer'
42. put 'WLSSERVER', 'row7', 'LOG:CODE', 'BEA-000360'
43. put 'WLSSERVER', 'row7', 'LOG:MSG', 'Server started in RUNNING mode'

 

• 创建Hive表映射到Hbase表

1. CREATE EXTERNAL TABLE wlsserver_hbase
2. ( key string
3. TIME_STAMP string,
4. CATEGORY string,
5. TYPE string,
6. SERVERNAME string,
7. CODE string,
8. MSG string
9. )
10. STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
11. WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,LOG:TIME_STAMP,LOG:CATEGORY,LOG:TYPE,LOG:SERVERNAME,LOG:CODE,LOG:MSG")
12. TBLPROPERTIES("hbase.table.name" = "WLSSERVER");

 

• 创建Oracle加载目标表

1. CREATE TABLE wlsserver
2. (
3.     time_stamp VARCHAR2(4000),
4.     Category VARCHAR2(4000),
5.     Type VARCHAR2(4000),
6.     servername VARCHAR2(4000),
7.     code VARCHAR2(4000),
8.     msg VARCHAR2(4000)
9.   );

 

• 配置Oracle Loader的配置文件

1. <?xml version="1.0" encoding="UTF-8" ?>
2. <configuration>
3.  
4.    <!-- Input settings -->
5.    <property>
6.       <name>mapreduce.inputformat.class</name>
7.       <value>oracle.hadoop.loader.lib.input.HiveToAvroInputFormat</value>
8.    </property>
9.    <property>
10.       <name>oracle.hadoop.loader.input.hive.databaseName</name>
11.       <value>default</value>
12.    </property>
13.    <property>
14.       <name>oracle.hadoop.loader.input.hive.tableName</name>
15.       <value>wlsserver_hbase</value>
16.    </property>
17.  
18.    <!-- Output settings -->
19.    <property>
20.       <name>mapreduce.job.outputformat.class</name>
21.       <value>oracle.hadoop.loader.lib.output.JDBCOutputFormat</value>
22.    </property>
23.    <property>
24.       <name>mapreduce.output.fileoutputformat.outputdir</name>
25.       <value>oraloadout</value>
26.    </property>
27.  
28.    <!-- Table information -->
29.    <property>
30.       <name>oracle.hadoop.loader.loaderMap.targetTable</name>
31.       <value>WLSSERVER</value>
32.    </property>
33.  
34.    <!-- Connection information -->
35.    <property>
36.       <name>oracle.hadoop.loader.connection.url</name>
37.       <value>jdbc:oracle:thin:@${HOST}:${TCPPORT}:${SID}</value>
38.    </property>
39.    <property>
40.       <name>TCPPORT</name>
41.       <value>1521</value>
42.    </property>
43.    <property>
44.       <name>HOST</name>
45.       <value>server1</value>
46.    </property>
47.    <property>
48.       <name>SID</name>
49.       <value>orcl</value>
50.    </property>
51.    <property>
52.       <name>oracle.hadoop.loader.connection.user</name>
53.       <value>baron</value>
54.    </property>
55.    <property>
56.       <name>oracle.hadoop.loader.connection.password</name>
57.       <value>baron</value>
58.    </property>
59. </configuration>

 

• 使用OLH加载 Hbase表到Oracle数据库

1. hadoop jar $OLH_HOME/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf /home/hadoop/OraLoadJobConf-hbase.xml -libjars \
2. $OLH_HOME/jlib/oraloader.jar,$HIVE_HOME/lib/hive-exec-1.2.1.jar,\
3. $HIVE_HOME/lib/hive-metastore-1.2.1.jar,$HIVE_HOME/lib/libfb303-0.9.2.jar,\
4. $HIVE_HOME/lib/hive-hbase-handler-1.2.1.jar,$HBASE_HOME/lib/hbase-client-1.1.2.jar,\
5. $HBASE_HOME/lib/hbase-server-1.1.2.jar,$HBASE_HOME/lib/hbase-protocol-1.1.2.jar,\
6. $HBASE_HOME/lib/htrace-core-3.1.0-incubating.jar,$HBASE_HOME/lib/hbase-common-1.1.2.jar,\
7. $HBASE_HOME/lib/netty-all-4.0.23.Final.jar

 

输出结果如下：

1. Oracle Loader for Hadoop Release 3.5.0 - Production
2. Copyright (c) 2011, 2015, Oracle and/or its affiliates. All rights reserved.
3. 15/12/21 10:37:22 INFO loader.OraLoader: Oracle Loader for Hadoop Release 3.5.0 - Production
4. Copyright (c) 2011, 2015, Oracle and/or its affiliates. All rights reserved.
5. 15/12/21 10:37:22 INFO loader.OraLoader: Built-Against: hadoop-2.2.0 hive-0.13.0 avro-1.7.6 jackson-1.8.8
6. 15/12/21 10:37:50 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: WLSSERVER is not partitioned
7. 15/12/21 10:37:50 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided
8. 15/12/21 10:37:50 INFO loader.OraLoader: Reduce tasks set to 0 because of no partitioning or sorting. Loading will be done in the map phase.
9. 15/12/21 10:37:50 INFO output.DBOutputFormat: Setting map tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat
10. 15/12/21 10:37:52 WARN loader.OraLoader: Sampler is disabled because the number of reduce tasks is less than two. Job will continue without sampled information.
11. 15/12/21 10:37:52 INFO loader.OraLoader: Submitting OraLoader job OraLoader
12. 15/12/21 10:37:52 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.56.101:8032
13. 15/12/21 10:37:57 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
14. 15/12/21 10:37:57 INFO metastore.ObjectStore: ObjectStore, initialize called
15. 15/12/21 10:37:57 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16. 15/12/21 10:37:57 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
17. 15/12/21 10:38:00 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
19. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
20. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21. 15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
22. 15/12/21 10:38:03 INFO DataNucleus.Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
23. 15/12/21 10:38:04 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
24. 15/12/21 10:38:04 INFO metastore.ObjectStore: Initialized ObjectStore
25. 15/12/21 10:38:04 INFO metastore.HiveMetaStore: Added admin role in metastore
26. 15/12/21 10:38:04 INFO metastore.HiveMetaStore: Added public role in metastore
27. 15/12/21 10:38:04 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
28. 15/12/21 10:38:05 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=wlsserver_hbase
29. 15/12/21 10:38:05 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_table : db=default tbl=wlsserver_hbase
30. 15/12/21 10:38:06 INFO input.HiveToAvroInputFormat: Row filter: null
31. 15/12/21 10:38:07 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x62a54948 connecting to ZooKeeper ensemble=localhost:2181
32. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
33. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:host.name=server1
34. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_65
35. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
36. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.home=/home/hadoop/jdk1.8.0_65/jre
37. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/hadoop-2.6.2/lib/native
38. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
39. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
40. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
41. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
42. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.version=3.8.13-55.1.6.el7uek.x86_64
43. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
44. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
45. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
46. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x62a549480x0, quorum=localhost:2181, baseZNode=/hbase
47. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
48. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
49. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x151c51c160b000d, negotiated timeout = 90000
50. 15/12/21 10:38:07 WARN mapreduce.TableInputFormatBase: You are using an HTable instance that relies on an HBase-managed Connection. This is usually due to directly creating an HTable, which is deprecated. Instead, you should create a Connection object and then request a Table instance from it. If you don't need the Table instance for your own use, you should instead use the TableInputFormatBase.initalizeTable method directly.
51. 15/12/21 10:38:07 INFO mapreduce.TableInputFormatBase: Creating an additional unmanaged connection because user provided one can't be used for administrative actions. We'll close it when we close out the table.
52. 15/12/21 10:38:07 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5b1efaaf connecting to ZooKeeper ensemble=localhost:2181
53. 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5b1efaaf0x0, quorum=localhost:2181, baseZNode=/hbase
54. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
55. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
56. 15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x151c51c160b000e, negotiated timeout = 90000
57. 15/12/21 10:38:07 INFO util.RegionSizeCalculator: Calculating region sizes for table "WLSSERVER".
58. 15/12/21 10:38:09 INFO metastore.HiveMetaStore: 0: Shutting down the object store...
59. 15/12/21 10:38:09 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=Shutting down the object store...
60. 15/12/21 10:38:09 INFO metastore.HiveMetaStore: 0: Metastore shutdown complete.
61. 15/12/21 10:38:09 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=Metastore shutdown complete.
62. 15/12/21 10:38:09 INFO mapreduce.JobSubmitter: number of splits:1
63. 15/12/21 10:38:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1450710904519_0002
64. 15/12/21 10:38:10 INFO impl.YarnClientImpl: Submitted application application_1450710904519_0002
65. 15/12/21 10:38:10 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1450710904519_0002/
66. 15/12/21 10:38:28 INFO loader.OraLoader: map 0% reduce 0%
67. 15/12/21 10:39:12 INFO loader.OraLoader: map 100% reduce 0%
68. 15/12/21 10:39:17 INFO loader.OraLoader: Job complete: OraLoader (job_1450710904519_0002)
69. 15/12/21 10:39:17 INFO loader.OraLoader: Counters: 30
70.         File System Counters
71.                 FILE: Number of bytes read=0
72.                 FILE: Number of bytes written=125961
73.                 FILE: Number of read operations=0
74.                 FILE: Number of large read operations=0
75.                 FILE: Number of write operations=0
76.                 HDFS: Number of bytes read=16889
77.                 HDFS: Number of bytes written=22053
78.                 HDFS: Number of read operations=5
79.                 HDFS: Number of large read operations=0
80.                 HDFS: Number of write operations=4
81.         Job Counters
82.                 Launched map tasks=1
83.                 Data-local map tasks=1
84.                 Total time spent by all maps in occupied slots (ms)=45057
85.                 Total time spent by all reduces in occupied slots (ms)=0
86.                 Total time spent by all map tasks (ms)=45057
87.                 Total vcore-seconds taken by all map tasks=45057
88.                 Total megabyte-seconds taken by all map tasks=46138368
89.         Map-Reduce Framework
90.                 Map input records=7
91.                 Map output records=7
92.                 Input split bytes=16889
93.                 Spilled Records=0
94.                 Failed Shuffles=0
95.                 Merged Map outputs=0
96.                 GC time elapsed (ms)=400
97.                 CPU time spent (ms)=8080
98.                 Physical memory (bytes) snapshot=196808704
99.                 Virtual memory (bytes) snapshot=2101088256
100.                 Total committed heap usage (bytes)=123797504
101.         File Input Format Counters
102.                 Bytes Read=0
103.         File Output Format Counters
104.                 Bytes Written=22053

 

• 在Oracle中，验证加载结果

1. SQL >select count(*) from wlsserver;
2.  
3.   COUNT(*)
4. ----------
5.          7

 

• 问题总结

1. java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.exec.Utilities.copyTableJobPropertiesToConf

   workaround：最初选用了hive-1.2.1版本，该版本太高，降级到hive-1.1.1版本

2. java.lang.NullPointerException

   workaround：OLH 3.4.0 版本 Bug 21392316 ,升级到 OLH 3.5.0版本

3. java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSplit not found

   workaround：

   for i in *.jar; do jar -tvf "$i" | grep -Hsi HBaseSplit && echo "$i"; done #找到缺少类的包，并将包加到 -libjars参数后面