Spark 使用Phoenix往HBase插入数据报错 PhoenixOutputFormat not found

集群版本：cdh5.8

Phoenix版本：4.7.0

项目依赖管理：Maven

描述：

     在工作中，需要使用Phoenix往HBase写数据，但是调试的时候出现问题

17/06/12 09:35:46 ERROR kerberos.SaveAsPhoenixApp$: save to phoenix error :java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:985)
at org.apache.phoenix.spark.ProductRDDFunctions.saveToPhoenix(ProductRDDFunctions.scala:51)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp$.saveRDDToPhoenixSIMPLELOG(SaveAsPhoenixApp.scala:144)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp$.saveAsphoenixProcess(SaveAsPhoenixApp.scala:93)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp$.main(SaveAsPhoenixApp.scala:156)
at com.yjf.phoenix.kerberos.SaveAsPhoenixApp.main(SaveAsPhoenixApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197) ... 29 more

各种调试，最后成功解决！

解决方案：

增加 phoenix library jar paths in /etc/spark/conf/classpath.txt

/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-core-4.7.0-clabs-ph‌​oenix1.3.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-spark-4.7.0-clabs-p‌​hoenix1.3.0.jar
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/calcite‌​-avatica-1.6.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/calcite‌​-avatica-server-1.6.‌​0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/commons‌​-csv-1.0.jar
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/hbase-p‌​refix-tree-1.2.0-cdh‌​5.7.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/hbase-p‌​rocedure-1.2.0-cdh5.‌​7.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-flume-4.7.0-clabs-p‌​hoenix1.3.0.jar
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-pherf-4.7.0-clabs-p‌​hoenix1.3.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-pherf-4.7.0-clabs-p‌​hoenix1.3.0-minimal.‌​jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-pig-4.7.0-clabs-pho‌​enix1.3.0.jar
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-server-4.7.0-clabs-‌​phoenix1.3.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-server-4.7.0-clabs-‌​phoenix1.3.0-runnabl‌​e.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/phoenix‌​-server-client-4.7.0‌​-clabs-phoenix1.3.0.‌​jar
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/twill-a‌​pi-0.6.0-incubating.‌​jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/twill-c‌​ommon-0.6.0-incubati‌​ng.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/twill-c‌​ore-0.6.0-incubating‌​.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/twill-d‌​iscovery-api-0.6.0-i‌​ncubating.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/twill-d‌​iscovery-core-0.6.0-‌​incubating.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/twill-z‌​ookeeper-0.6.0-incub‌​ating.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/tephra-‌​api-0.7.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/tephra-‌​core-0.7.0.jar 
/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/lib/tephra-‌​hbase-compat-0.98-0.‌​7.0.jar