spark sql读取hive数据时报找不到mysql驱动的解决办法

一、问题现象：

scala> hiveContext.hql("show databases").collect().foreach(println)
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
14/12/08 23:04:31 INFO ParseDriver: Parsing command: show databases
14/12/08 23:04:31 INFO ParseDriver: Parse Completed
14/12/08 23:04:31 INFO Driver:
14/12/08 23:04:31 INFO Driver:
14/12/08 23:04:31 INFO Driver:
14/12/08 23:04:31 INFO Driver:
14/12/08 23:04:31 INFO ParseDriver: Parsing command: show databases
14/12/08 23:04:31 INFO ParseDriver: Parse Completed
14/12/08 23:04:31 INFO Driver:
14/12/08 23:04:31 INFO Driver:
14/12/08 23:04:32 INFO Driver: Semantic Analysis Completed
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO ListSinkOperator: Initializing Self 0 OP
14/12/08 23:04:32 INFO ListSinkOperator: Operator 0 OP initialized
14/12/08 23:04:32 INFO ListSinkOperator: Initialization Done 0 OP
14/12/08 23:04:32 INFO Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver: Starting command: show databases
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
14/12/08 23:04:32 INFO ObjectStore: ObjectStore, initialize called
14/12/08 23:04:32 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/home/hadoop/cdh4/spark-1.1.1-bin-cdh4/lib/datanucleus-rdbms-3.2.1.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/home/hadoop/cdh4/spark/lib/datanucleus-rdbms-3.2.1.jar."
14/12/08 23:04:32 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/home/hadoop/cdh4/spark/lib/datanucleus-core-3.2.2.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/home/hadoop/cdh4/spark-1.1.1-bin-cdh4/lib/datanucleus-core-3.2.2.jar."
14/12/08 23:04:32 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/home/hadoop/cdh4/spark-1.1.1-bin-cdh4/lib/datanucleus-api-jdo-3.2.1.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/home/hadoop/cdh4/spark/lib/datanucleus-api-jdo-3.2.1.jar."
14/12/08 23:04:32 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
14/12/08 23:04:32 ERROR DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1074)
        at org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:2198)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:328)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
        at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:318)
        at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:284)
        at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
        at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
        at org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
        at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:382)
        at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:382)
        at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
        at org.apache.spark.sql.SchemaRDD.(SchemaRDD.scala:103)
        at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:106)
        at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:110)
        at $line9.$read$$iwC$$iwC$$iwC$$iwC.(:15)
        at $line9.$read$$iwC$$iwC$$iwC.(:20)
        at $line9.$read$$iwC$$iwC.(:22)
        at $line9.$read$$iwC.(:24)
        at $line9.$read.(:26)
        at $line9.$read$.(:30)
        at $line9.$read$.()
        at $line9.$eval$.(:7)
        at $line9.$eval$.()
        at $line9.$eval.$print()
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:823)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:868)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:780)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:625)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:633)
        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:638)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:963)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:911)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:911)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:911)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1006)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
        at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1072)
        ... 59 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
        ... 64 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:781)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:275)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:304)
        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:234)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:209)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
        at org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:64)
        at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:286)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:121)
        ... 69 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
        at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:281)
        at org.datanucleus.store.AbstractStoreManager.(AbstractStoreManager.java:239)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:292)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
        at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1069)
        at org.datanucleus.NucleusContext.initialise(NucleusContext.java:359)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:768)
        ... 98 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:237)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:110)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.(ConnectionFactoryImpl.java:82)
        ... 116 more
Caused by: org.datanucleus.store.rdbms.datasource.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.datasource.AbstractDataSourceFactory.loadDriver(AbstractDataSourceFactory.java:58)
        at org.datanucleus.store.rdbms.datasource.BoneCPDataSourceFactory.makePooledDataSource(BoneCPDataSourceFactory.java:61)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:217)
        ... 118 more

14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 ERROR Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 INFO Driver:
14/12/08 23:04:32 ERROR HiveContext: 
======================
HIVE FAILURE OUTPUT
======================
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

======================
END HIVE FAILURE OUTPUT
======================
          
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:322)
        at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:284)
        at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
        at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
        at org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
        at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:382)
        at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:382)
        at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
        at org.apache.spark.sql.SchemaRDD.(SchemaRDD.scala:103)
        at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:106)
        at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:110)
        at $iwC$$iwC$$iwC$$iwC.(:15)
        at $iwC$$iwC$$iwC.(:20)
        at $iwC$$iwC.(:22)
        at $iwC.(:24)
        at (:26)
        at .(:30)
        at .()
        at .(:7)
        at .()
        at $print()
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:823)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:868)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:780)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:625)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:633)
        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:638)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:963)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:911)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:911)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:911)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1006)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

scala>            

    
二、解决办法：
1、$HIVE_HOME/conf/hive-site.xml中增加关于 hive.metastore.uris 的配置信息，如下：

  hive.metastore.uris
  thrift://namenode1:9083
  IP address (or fully-qualified domain name) and port of the metastore host

2、执行:$HIVE_HOME/bin/hive --service metastore
3、将$HIVE_HOME/conf/hive-site.xml拷贝至$SPARK_HOME/conf/目录下。至此问题可以解决。
4、启动spark-shell进行验证:$SPARK_HOME/bin/spark-shell --master namenode1:7077