连接Oracle与Hadoop(3) 使用OLH加载Hbase到Oracle
OLH是Oracle Loader for Hadoop的缩写，Oracle的大数据连接器（BDC）的一个组件，可将多种数据格式从HDFS上加载到Oracle数据库库中。
本文在同一台服务器上模拟oracle数据库与hadoop集群，实验目标：使用OLH从Hadoop端的Hbase表加载数据到Oracle表中。
对于加载Hbase表场景，需要通过Hive创建一个基于Hbase的映射表，再通过Hive接口导入到Oracle表中。
Oracle端：
| 服务器 |
系统用户 |
安装软件 |
软件安装路径 |
| Server1 |
oracle |
Oracle Database 12.1.0.2 |
/u01/app/oracle/product/12.1.0/dbhome_1 |
Hadoop集群端：
| 服务器 |
系统用户 |
安装软件 |
软件安装路径 |
| Server1 |
hadoop |
Hadoop 2.6.2 |
/home/hadoop/hadoop-2.6.2 |
| Hive 1.1.1 |
/home/hadoop/hive-1.1.1 |
||
| Hbase 1.1.2 |
/home/hadoop/hbase-1.1.2 |
||
| jdk1.8.0_65 |
/home/hadoop/jdk1.8.0_65 |
||
| OLH 3.5.0 |
/home/hadoop/oraloader-3.5.0-h2 |
- 部署Hadoop/Hive/Hbase/OLH软件
将Hadoop/Hive/Hbase/OLH软件解压到相应目录
-
[hadoop@server1 ~]$ tree -L 1
-
├── hadoop-2.6.2
-
├── hbase-1.1.2
-
├── hive-1.1.1
-
├── jdk1.8.0_65
-
├── oraloader-3.5.0-h2
- 配置Hadoop/Hive/Hbase/OLH环境变量
-
export JAVA_HOME=/home/hadoop/jdk1.8.0_65
-
-
export HADOOP_USER_NAME=hadoop
-
export HADOOP_YARN_USER=hadoop
-
export HADOOP_HOME=/home/hadoop/hadoop-2.6.2
-
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
-
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
-
export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec
-
export HADOOP_COMMON_HOME=${HADOOP_HOME}
-
export HADOOP_HDFS_HOME=${HADOOP_HOME}
-
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
-
export HADOOP_YARN_HOME=${HADOOP_HOME}
-
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
-
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
-
-
export HIVE_HOME=/home/hadoop/hive-1.1.1
-
export HIVE_CONF_DIR=${HIVE_HOME}/conf
-
-
export HBASE_HOME=/home/hadoop/hbase-1.1.2
-
export HBASE_CONF_DIR=/home/hadoop/hbase-1.1.2/conf
-
-
export OLH_HOME=/home/hadoop/oraloader-3.5.0-h2
-
-
export HADOOP_CLASSPATH=/usr/share/java/mysql-connector-java.jar
-
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*:$HIVE_CONF_DIR:$HBASE_HOME/lib/*:$HBASE_CONF_DIR
-
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$OLH_HOME/jlib/*
-
-
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$PATH
- 配置Hadoop/Hive/Hbase软件
core-site.xml
-
<configuration>
-
<property>
-
<name>fs.defaultFS</name>
-
<value>hdfs://server1:8020</value>
-
</property>
-
<property>
-
<name>fs.checkpoint.period</name>
-
<value>3600</value>
-
</property>
-
<property>
-
<name>fs.checkpoint.size</name>
-
<value>67108864</value>
-
</property>
-
<property>
-
<name>hadoop.proxyuser.hadoop.groups</name>
-
<value>*</value>
-
</property>
-
<property>
-
<name>hadoop.proxyuser.hadoop.hosts</name>
-
<value>*</value>
-
</property>
-
</configuration>
hdfs-site.xml
-
<configuration>
-
<property>
-
<name>hadoop.tmp.dir</name>
-
<value>file:///home/hadoop</value>
-
</property>
-
<property>
-
<name>dfs.namenode.name.dir</name>
-
<value>file:///home/hadoop/dfs/nn</value>
-
</property>
-
<property>
-
<name>dfs.datanode.data.dir</name>
-
<value>file:///home/hadoop/dfs/dn</value>
-
</property>
-
<property>
-
<name>dfs.namenode.checkpoint.dir</name>
-
<value>file:///home/hadoop/dfs/sn</value>
-
</property>
-
<property>
-
<name>dfs.replication</name>
-
<value>1</value>
-
</property>
-
<property>
-
<name>dfs.permissions.superusergroup</name>
-
<value>supergroup</value>
-
</property>
-
<property>
-
<name>dfs.namenode.http-address</name>
-
<value>server1:50070</value>
-
</property>
-
<property>
-
<name>dfs.namenode.secondary.http-address</name>
-
<value>server1:50090</value>
-
</property>
-
<property>
-
<name>dfs.webhdfs.enabled</name>
-
<value>true</value>
-
</property>
-
</configuration>
yarn-site.xml
-
<configuration>
-
<property>
-
<name>yarn.resourcemanager.scheduler.address</name>
-
<value>server1:8030</value>
-
</property>
-
<property>
-
<name>yarn.resourcemanager.resource-tracker.address</name>
-
<value>server1:8031</value>
-
</property>
-
<property>
-
<name>yarn.resourcemanager.address</name>
-
<value>server1:8032</value>
-
</property>
-
<property>
-
<name>yarn.resourcemanager.admin.address</name>
-
<value>server1:8033</value>
-
</property>
-
<property>
-
<name>yarn.resourcemanager.webapp.address</name>
-
<value>server1:8088</value>
-
</property>
-
<property>
-
<name>yarn.nodemanager.local-dirs</name>
-
<value>file:///home/hadoop/yarn/local</value>
-
</property>
-
<property>
-
<name>yarn.nodemanager.log-dirs</name>
-
<value>file:///home/hadoop/yarn/logs</value>
-
</property>
-
<property>
-
<name>yarn.log-aggregation-enable</name>
-
<value>true</value>
-
</property>
-
<property>
-
<name>yarn.nodemanager.remote-app-log-dir</name>
-
<value>/yarn/apps</value>
-
</property>
-
<property>
-
<name>yarn.app.mapreduce.am.staging-dir</name>
-
<value>/user</value>
-
</property>
-
<property>
-
<name>yarn.nodemanager.aux-services</name>
-
<value>mapreduce_shuffle</value>
-
</property>
-
<property>
-
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
-
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
-
</property>
-
<property>
-
<name>yarn.log.server.url</name>
-
<value>http://server1:19888/jobhistory/logs/</value>
-
</property>
-
</configuration>
- 配置Hive
hive-site.xml
-
<configuration>
-
<property>
-
<name>javax.jdo.option.ConnectionURL</name>
-
<value>jdbc:mysql://server1:3306/metastore?createDatabaseIfNotExist=true</value>
-
</property>
-
<property>
-
<name>javax.jdo.option.ConnectionDriverName</name>
-
<value>com.mysql.jdbc.Driver</value>
-
</property>
-
<property>
-
<name>javax.jdo.option.ConnectionUserName</name>
-
<value>hive</value>
-
</property>
-
<property>
-
<name>javax.jdo.option.ConnectionPassword</name>
-
<value>123456</value>
-
</property>
-
<property>
-
<name>hbase.master</name>
-
<value>server1:16000</value>
-
</property>
-
<property>
-
<name>hive.cli.print.current.db</name>
-
<value>true</value>
-
</property>
-
</configuration>
- 配置Hbase
Hbase-site.xml
-
<configuration>
-
<property>
-
<name>hbase.rootdir</name>
-
<value>hdfs://server1/hbase</value>
-
</property>
-
<property>
-
<name>hbase.cluster.distributed</name>
-
<value>true</value>
-
</property>
-
<property>
-
<name>hbase.master.port</name>
-
<value>16000</value>
-
</property>
-
</configuration>
- 创建一张Hbase表
-
create 'WLSSERVER' , 'LOG'
-
put 'WLSSERVER', 'row1', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:16-PM-PDT'
-
put 'WLSSERVER', 'row1', 'LOG:CATEGORY', 'Notice'
-
put 'WLSSERVER', 'row1', 'LOG:TYPE', 'WebLogicServer'
-
put 'WLSSERVER', 'row1', 'LOG:SERVERNAME', 'AdminServer'
-
put 'WLSSERVER', 'row1', 'LOG:CODE', 'BEA-000365'
-
put 'WLSSERVER', 'row1', 'LOG:MSG', 'Server state changed to STANDBY'
-
put 'WLSSERVER', 'row2', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:17-PM-PDT'
-
put 'WLSSERVER', 'row2', 'LOG:CATEGORY', 'Notice'
-
put 'WLSSERVER', 'row2', 'LOG:TYPE', 'WebLogicServer'
-
put 'WLSSERVER', 'row2', 'LOG:SERVERNAME', 'AdminServer'
-
put 'WLSSERVER', 'row2', 'LOG:CODE', 'BEA-000365'
-
put 'WLSSERVER', 'row2', 'LOG:MSG', 'Server state changed to STARTING'
-
put 'WLSSERVER', 'row3', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:18-PM-PDT'
-
put 'WLSSERVER', 'row3', 'LOG:CATEGORY', 'Notice'
-
put 'WLSSERVER', 'row3', 'LOG:TYPE', 'WebLogicServer'
-
put 'WLSSERVER', 'row3', 'LOG:SERVERNAME', 'AdminServer'
-
put 'WLSSERVER', 'row3', 'LOG:CODE', 'BEA-000365'
-
put 'WLSSERVER', 'row3', 'LOG:MSG', 'Server state changed to ADMIN'
-
put 'WLSSERVER', 'row4', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:19-PM-PDT'
-
put 'WLSSERVER', 'row4', 'LOG:CATEGORY', 'Notice'
-
put 'WLSSERVER', 'row4', 'LOG:TYPE', 'WebLogicServer'
-
put 'WLSSERVER', 'row4', 'LOG:SERVERNAME', 'AdminServer'
-
put 'WLSSERVER', 'row4', 'LOG:CODE', 'BEA-000365'
-
put 'WLSSERVER', 'row4', 'LOG:MSG', 'Server state changed to RESUMING'
-
put 'WLSSERVER', 'row5', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:20-PM-PDT'
-
put 'WLSSERVER', 'row5', 'LOG:CATEGORY', 'Notice'
-
put 'WLSSERVER', 'row5', 'LOG:TYPE', 'WebLogicServer'
-
put 'WLSSERVER', 'row5', 'LOG:SERVERNAME', 'AdminServer'
-
put 'WLSSERVER', 'row5', 'LOG:CODE', 'BEA-000331'
-
put 'WLSSERVER', 'row5', 'LOG:MSG', 'Started WebLogic AdminServer'
-
put 'WLSSERVER', 'row6', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:21-PM-PDT'
-
put 'WLSSERVER', 'row6', 'LOG:CATEGORY', 'Notice'
-
put 'WLSSERVER', 'row6', 'LOG:TYPE', 'WebLogicServer'
-
put 'WLSSERVER', 'row6', 'LOG:SERVERNAME', 'AdminServer'
-
put 'WLSSERVER', 'row6', 'LOG:CODE', 'BEA-000365'
-
put 'WLSSERVER', 'row6', 'LOG:MSG', 'Server state changed to RUNNING'
-
put 'WLSSERVER', 'row7', 'LOG:TIME_STAMP', 'Apr-8-2014-7:06:22-PM-PDT'
-
put 'WLSSERVER', 'row7', 'LOG:CATEGORY', 'Notice'
-
put 'WLSSERVER', 'row7', 'LOG:TYPE', 'WebLogicServer'
-
put 'WLSSERVER', 'row7', 'LOG:SERVERNAME', 'AdminServer'
-
put 'WLSSERVER', 'row7', 'LOG:CODE', 'BEA-000360'
-
put 'WLSSERVER', 'row7', 'LOG:MSG', 'Server started in RUNNING mode'
- 创建Hive表映射到Hbase表
-
CREATE EXTERNAL TABLE wlsserver_hbase
-
( key string
-
TIME_STAMP string,
-
CATEGORY string,
-
TYPE string,
-
SERVERNAME string,
-
CODE string,
-
MSG string
-
)
-
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
-
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,LOG:TIME_STAMP,LOG:CATEGORY,LOG:TYPE,LOG:SERVERNAME,LOG:CODE,LOG:MSG")
-
TBLPROPERTIES("hbase.table.name" = "WLSSERVER");
- 创建Oracle加载目标表
-
CREATE TABLE wlsserver
-
(
-
time_stamp VARCHAR2(4000),
-
Category VARCHAR2(4000),
-
Type VARCHAR2(4000),
-
servername VARCHAR2(4000),
-
code VARCHAR2(4000),
-
msg VARCHAR2(4000)
-
);
- 配置Oracle Loader的配置文件
-
<?xml version="1.0" encoding="UTF-8" ?>
-
<configuration>
-
-
<!-- Input settings -->
-
<property>
-
<name>mapreduce.inputformat.class</name>
-
<value>oracle.hadoop.loader.lib.input.HiveToAvroInputFormat</value>
-
</property>
-
<property>
-
<name>oracle.hadoop.loader.input.hive.databaseName</name>
-
<value>default</value>
-
</property>
-
<property>
-
<name>oracle.hadoop.loader.input.hive.tableName</name>
-
<value>wlsserver_hbase</value>
-
</property>
-
-
<!-- Output settings -->
-
<property>
-
<name>mapreduce.job.outputformat.class</name>
-
<value>oracle.hadoop.loader.lib.output.JDBCOutputFormat</value>
-
</property>
-
<property>
-
<name>mapreduce.output.fileoutputformat.outputdir</name>
-
<value>oraloadout</value>
-
</property>
-
-
<!-- Table information -->
-
<property>
-
<name>oracle.hadoop.loader.loaderMap.targetTable</name>
-
<value>WLSSERVER</value>
-
</property>
-
-
<!-- Connection information -->
-
<property>
-
<name>oracle.hadoop.loader.connection.url</name>
-
<value>jdbc:oracle:thin:@${HOST}:${TCPPORT}:${SID}</value>
-
</property>
-
<property>
-
<name>TCPPORT</name>
-
<value>1521</value>
-
</property>
-
<property>
-
<name>HOST</name>
-
<value>server1</value>
-
</property>
-
<property>
-
<name>SID</name>
-
<value>orcl</value>
-
</property>
-
<property>
-
<name>oracle.hadoop.loader.connection.user</name>
-
<value>baron</value>
-
</property>
-
<property>
-
<name>oracle.hadoop.loader.connection.password</name>
-
<value>baron</value>
-
</property>
-
</configuration>
- 使用OLH加载 Hbase表到Oracle数据库
-
hadoop jar $OLH_HOME/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf /home/hadoop/OraLoadJobConf-hbase.xml -libjars \
-
$OLH_HOME/jlib/oraloader.jar,$HIVE_HOME/lib/hive-exec-1.2.1.jar,\
-
$HIVE_HOME/lib/hive-metastore-1.2.1.jar,$HIVE_HOME/lib/libfb303-0.9.2.jar,\
-
$HIVE_HOME/lib/hive-hbase-handler-1.2.1.jar,$HBASE_HOME/lib/hbase-client-1.1.2.jar,\
-
$HBASE_HOME/lib/hbase-server-1.1.2.jar,$HBASE_HOME/lib/hbase-protocol-1.1.2.jar,\
-
$HBASE_HOME/lib/htrace-core-3.1.0-incubating.jar,$HBASE_HOME/lib/hbase-common-1.1.2.jar,\
-
$HBASE_HOME/lib/netty-all-4.0.23.Final.jar
输出结果如下：
-
Oracle Loader for Hadoop Release 3.5.0 - Production
-
Copyright (c) 2011, 2015, Oracle and/or its affiliates. All rights reserved.
-
15/12/21 10:37:22 INFO loader.OraLoader: Oracle Loader for Hadoop Release 3.5.0 - Production
-
Copyright (c) 2011, 2015, Oracle and/or its affiliates. All rights reserved.
-
15/12/21 10:37:22 INFO loader.OraLoader: Built-Against: hadoop-2.2.0 hive-0.13.0 avro-1.7.6 jackson-1.8.8
-
15/12/21 10:37:50 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: WLSSERVER is not partitioned
-
15/12/21 10:37:50 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided
-
15/12/21 10:37:50 INFO loader.OraLoader: Reduce tasks set to 0 because of no partitioning or sorting. Loading will be done in the map phase.
-
15/12/21 10:37:50 INFO output.DBOutputFormat: Setting map tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat
-
15/12/21 10:37:52 WARN loader.OraLoader: Sampler is disabled because the number of reduce tasks is less than two. Job will continue without sampled information.
-
15/12/21 10:37:52 INFO loader.OraLoader: Submitting OraLoader job OraLoader
-
15/12/21 10:37:52 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.56.101:8032
-
15/12/21 10:37:57 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
-
15/12/21 10:37:57 INFO metastore.ObjectStore: ObjectStore, initialize called
-
15/12/21 10:37:57 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
-
15/12/21 10:37:57 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
-
15/12/21 10:38:00 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
-
15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
-
15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
-
15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
-
15/12/21 10:38:03 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
-
15/12/21 10:38:03 INFO DataNucleus.Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
-
15/12/21 10:38:04 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
-
15/12/21 10:38:04 INFO metastore.ObjectStore: Initialized ObjectStore
-
15/12/21 10:38:04 INFO metastore.HiveMetaStore: Added admin role in metastore
-
15/12/21 10:38:04 INFO metastore.HiveMetaStore: Added public role in metastore
-
15/12/21 10:38:04 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
-
15/12/21 10:38:05 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=wlsserver_hbase
-
15/12/21 10:38:05 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_table : db=default tbl=wlsserver_hbase
-
15/12/21 10:38:06 INFO input.HiveToAvroInputFormat: Row filter: null
-
15/12/21 10:38:07 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x62a54948 connecting to ZooKeeper ensemble=localhost:2181
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:host.name=server1
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_65
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.home=/home/hadoop/jdk1.8.0_65/jre
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client 15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/hadoop-2.6.2/lib/native
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:os.version=3.8.13-55.1.6.el7uek.x86_64
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x62a549480x0, quorum=localhost:2181, baseZNode=/hbase
-
15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
-
15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
-
15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x151c51c160b000d, negotiated timeout = 90000
-
15/12/21 10:38:07 WARN mapreduce.TableInputFormatBase: You are using an HTable instance that relies on an HBase-managed Connection. This is usually due to directly creating an HTable, which is deprecated. Instead, you should create a Connection object and then request a Table instance from it. If you don't need the Table instance for your own use, you should instead use the TableInputFormatBase.initalizeTable method directly.
-
15/12/21 10:38:07 INFO mapreduce.TableInputFormatBase: Creating an additional unmanaged connection because user provided one can't be used for administrative actions. We'll close it when we close out the table.
-
15/12/21 10:38:07 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5b1efaaf connecting to ZooKeeper ensemble=localhost:2181
-
15/12/21 10:38:07 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5b1efaaf0x0, quorum=localhost:2181, baseZNode=/hbase
-
15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
-
15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
-
15/12/21 10:38:07 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x151c51c160b000e, negotiated timeout = 90000
-
15/12/21 10:38:07 INFO util.RegionSizeCalculator: Calculating region sizes for table "WLSSERVER".
-
15/12/21 10:38:09 INFO metastore.HiveMetaStore: 0: Shutting down the object store...
-
15/12/21 10:38:09 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=Shutting down the object store...
-
15/12/21 10:38:09 INFO metastore.HiveMetaStore: 0: Metastore shutdown complete.
-
15/12/21 10:38:09 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=Metastore shutdown complete.
-
15/12/21 10:38:09 INFO mapreduce.JobSubmitter: number of splits:1
-
15/12/21 10:38:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1450710904519_0002
-
15/12/21 10:38:10 INFO impl.YarnClientImpl: Submitted application application_1450710904519_0002
-
15/12/21 10:38:10 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1450710904519_0002/
-
15/12/21 10:38:28 INFO loader.OraLoader: map 0% reduce 0%
-
15/12/21 10:39:12 INFO loader.OraLoader: map 100% reduce 0%
-
15/12/21 10:39:17 INFO loader.OraLoader: Job complete: OraLoader (job_1450710904519_0002)
-
15/12/21 10:39:17 INFO loader.OraLoader: Counters: 30
-
File System Counters
-
FILE: Number of bytes read=0
-
FILE: Number of bytes written=125961
-
FILE: Number of read operations=0
-
FILE: Number of large read operations=0
-
FILE: Number of write operations=0
-
HDFS: Number of bytes read=16889
-
HDFS: Number of bytes written=22053
-
HDFS: Number of read operations=5
-
HDFS: Number of large read operations=0
-
HDFS: Number of write operations=4
-
Job Counters
-
Launched map tasks=1
-
Data-local map tasks=1
-
Total time spent by all maps in occupied slots (ms)=45057
-
Total time spent by all reduces in occupied slots (ms)=0
-
Total time spent by all map tasks (ms)=45057
-
Total vcore-seconds taken by all map tasks=45057
-
Total megabyte-seconds taken by all map tasks=46138368
-
Map-Reduce Framework
-
Map input records=7
-
Map output records=7
-
Input split bytes=16889
-
Spilled Records=0
-
Failed Shuffles=0
-
Merged Map outputs=0
-
GC time elapsed (ms)=400
-
CPU time spent (ms)=8080
-
Physical memory (bytes) snapshot=196808704
-
Virtual memory (bytes) snapshot=2101088256
-
Total committed heap usage (bytes)=123797504
-
File Input Format Counters
-
Bytes Read=0
-
File Output Format Counters
-
Bytes Written=22053
- 在Oracle中，验证加载结果
-
SQL >select count(*) from wlsserver;
-
-
COUNT(*)
-
----------
-
7
- 问题总结
-
java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.exec.Utilities.copyTableJobPropertiesToConf
workaround：最初选用了hive-1.2.1版本，该版本太高，降级到hive-1.1.1版本
-
java.lang.NullPointerException
workaround：OLH 3.4.0 版本 Bug 21392316 ,升级到 OLH 3.5.0版本
-
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSplit not found
workaround：
for i in *.jar; do jar -tvf "$i" | grep -Hsi HBaseSplit && echo "$i"; done #找到缺少类的包，并将包加到 -libjars参数后面