Spark+hadoop 开发总结
错误总结: 1.Caused by: java.lang.ClassNotFoundException: scala.xml.MetaData 解决:POM配置依赖 <dependency> dependency> 2.版本冲突 Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$scope()Lscala/xml/TopScope$; at org.apache.spark.ui.jobs.StagePage.(StagePage.scala:44) at org.apache.spark.ui.jobs.StagesTab.(StagesTab.scala:34) at org.apache.spark.ui.SparkUI.(SparkUI.scala:62) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157) at org.apache.spark.SparkContext.(SparkContext.scala:443) at com.shentu.cn.SimpleALS$.main(SimpleALS.scala:24) at com.shentu.cn.SimpleALS.main(SimpleALS.scala)
环境：
scala-2.11.8
依赖：
"org.apache.spark"% "spark-core_2.10" % "2.0.0"
解决方案：
"org.apache.spark"% "spark-core_2.11" % "2.0.0"
3.spark 2.0安装后求PI测试 启动master : master 节点 ./start-master.sh ./start-slave.sh spark://hadoop:7077 cd $SPARK_HOME/bin ./run-example SparkPi 10 --master spark://master_ip:7077 4.Spark程序启动时hadoop 依赖过低 7/09/30 11:58:53 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(0,0,ResultTask,ExceptionFailure(java.lang.NoSuchMethodError,org.apache.hadoop.mapred.TaskID.(Lorg/apache/hadoop/mapreduce/JobID;Lorg/apache/hadoop/mapreduce/TaskType;I)V,[Ljava.lang.StackTraceElement;@2310a1a5,java.lang.NoSuchMethodError: org.apache.hadoop.mapred.TaskID. at org.apache.spark.rdd.HadoopRDD$.addLocalConfiguration(HadoopRDD.scala:372) 经查是在引入spark-hive,spark-sql相关依赖时,hadoop 版本过低所致,hadoop 2.x后无hadoop-core依赖,分散为 hadoop-mapreduce-client-core hadoop-hdfs hadoop-common hadoop-client< hadoop-annotations hadoop-auth 等依赖 解决,删除本地仓库对应hadoop-core目录,在对应的引用目录,剔除相应版本的依赖(红色标注) 1. rm -rf /home/jerry/.m2/repository/org/apache/hadoop/hadoop-core/ 2.<dependency> dependency> 5.log4g ClassNotFoundException,spark-core依赖无法导入 问题:是引入的依赖配置有无 按照maven repository 中央仓库引入的无法使用,需要修改去掉scope部分 <dependency>(Lorg/apache/hadoop/mapreduce/JobID;Lorg/apache/hadoop/mapreduce/TaskType;I)V provided dependency> 6.spark-submit 提交应用报错:classNotFoundException 经查是配置在spark-env.sh中的Java_HOME报错,无法找到相应的mainClass 7.mahout0.13+hadoop2.8+spark2.0集成后允许mahout 出现错误 归根结底还是相应软件的classpath下无对应的jar包,需要在spark jars目录复制过来进行添加 1.无法找到sparkconf主类: Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 解决:复制/usr/spark/jars目录下的 spark-core_2.11-2.0.0.jar spark-examples_2.11-2.0.0.jar 2.无法找到scala依赖 ut/spark/spark.csv --output /out/mahout/ --master yarn-client --inDelim "," --itemIDPosition 2 --rowIDPosition 1 --filterPosition 2 Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more 解决:同样复制/usr/spark/jars目录下的scala依赖 cp /usr/spark/jars/scala* scala-xml_2.11-1.0.2.jar scala-reflect-2.11.8.jar scala-parser-combinators_2.11-1.0.4.jar scalap-2.11.8.jar scala-library-2.11.8.jar scala-compiler-2.11.8.jar 3.无法找到序列化,压缩类 17/10/26 23:01:27 INFO Utils: Successfully started service 'sparkDriver' on port 39291. Exception in thread "main" java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/Serializer at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:225) at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:238) at org.apache.spark.SparkEnv$.instantiateClassFromConf$1(SparkEnv.scala:259) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:262) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:165) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:259) at org.apache.spark.SparkContext.(SparkContext.scala:423) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.kryo.Serializer at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 18 more 解决:同样复制/usr/spark/jars目录下的kryo相关依赖到/usr/mahout/lib 4. Exception in thread "main" java.lang.NoClassDefFoundError: com/twitter/chill/KryoBase at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:225) at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:238) at org.apache.spark.SparkEnv$.instantiateClassFromConf$1(SparkEnv.scala:259) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:262) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:165) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:259) at org.apache.spark.SparkContext.(SparkContext.scala:423) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: com.twitter.chill.KryoBase at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 解决:如果使用idea,maven管理依赖,double shift 搜索kryoBase所在jar包 在本地仓库位置后 将该jar包复制到/usr/mahout/lib/ 5 Exception in thread "main" java.lang.NoClassDefFoundError: org/json4s/JsonAST$JValue at org.apache.spark.ui.jobs.StagesTab.(StagesTab.scala:33) at org.apache.spark.ui.SparkUI.(SparkUI.scala:62) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157) at org.apache.spark.SparkContext.(SparkContext.scala:443) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: org.json4s.JsonAST$JValue at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 14 more 将maven 中仓库中的对应jar包复制到mahout classpath 目录 sudo cp json4s-ast_2.10-3.2.10.jar /usr/mahout/lib/ 6. Exception in thread "main" java.lang.NoClassDefFoundError: javax/servlet/FilterRegistration at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:142) at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:135) at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:129) at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:99) at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:128) at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:115) at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:80) at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:64) at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:64) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:64) at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:68) at org.apache.spark.ui.SparkUI.(SparkUI.scala:81) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157) at org.apache.spark.SparkContext.(SparkContext.scala:443) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: javax.servlet.FilterRegistration at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 26 more 17/10/29 21:57:05 INFO DiskBlockManager: Shutdown hook called 解决: sudo cp javax.servlet-api-3.1.0.jar /usr/mahout/lib/ 7. Exception in thread "main" java.lang.NoClassDefFoundError: org/glassfish/jersey/servlet/ServletContainer at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:193) at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:75) at org.apache.spark.ui.SparkUI.(SparkUI.scala:81) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157) at org.apache.spark.SparkContext.(SparkContext.scala:443) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.servlet.ServletContainer at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 解决 sudo cp jersey-container-servlet-core-2.22.2.jar /usr/mahout /lib/ 8 Exception in thread "main" java.lang.NoClassDefFoundError: org/glassfish/jersey/server/spi/Container at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:193) at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:75) at org.apache.spark.ui.SparkUI.(SparkUI.scala:81) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157) at org.apache.spark.SparkContext.(SparkContext.scala:443) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.server.spi.Container 解决 sudo cp jersey-server-2.22.2.jar /usr/mahout/lib/ 9 Exception in thread "main" java.lang.NoClassDefFoundError: org/glassfish/jersey/server/spi/Container at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:193) at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:75) at org.apache.spark.ui.SparkUI.(SparkUI.scala:81) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157) at org.apache.spark.SparkContext.(SparkContext.scala:443) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.server.spi.Container 解决 sudo cp jersey-server-2.22.2.jar /usr/mahout/lib/ 10 javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter] at org.spark_project.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1530) at org.spark_project.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1560) at org.spark_project.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1233) at org.spark_project.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1101) at org.spark_project.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:367) at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1$$anonfun$apply$6.apply(JettyUtils.scala:214) at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1$$anonfun$apply$6.apply(JettyUtils.scala:214) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1.apply(JettyUtils.scala:214) at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1.apply(JettyUtils.scala:192) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at org.apache.spark.ui.JettyUtils$.addFilters(JettyUtils.scala:192) at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$$anonfun$org$apache$spark$scheduler$cluster$YarnSchedulerBackend$$addWebUIFilter$3.apply(YarnSchedulerBackend.scala:161) at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$$anonfun$org$apache$spark$scheduler$cluster$YarnSchedulerBackend$$addWebUIFilter$3.apply(YarnSchedulerBackend.scala:161) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.org$apache$spark$scheduler$cluster$YarnSchedulerBackend$$addWebUIFilter(YarnSchedulerBackend.scala:161) at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receive$1.applyOrElse(YarnSchedulerBackend.scala:258) at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117) at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205) at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:211) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: MultiException[javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter] at org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:842) at org.spark_project.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1526) ... 25 more Suppressed: javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter at org.spark_project.jetty.servlet.BaseHolder.doStart(BaseHolder.java:102) at org.spark_project.jetty.servlet.FilterHolder.doStart(FilterHolder.java:93) at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:888) ... 26 more Caused by: javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter at org.spark_project.jetty.servlet.BaseHolder.doStart(BaseHolder.java:102) at org.spark_project.jetty.servlet.FilterHolder.doStart(FilterHolder.java:93) at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:851) 解决 sudo cp hadoop-yarn-server-web-proxy-2.8.0.jar /usr/mahout/lib/ 11 Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/Module at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:225) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:191) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:187) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:187) at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:101) at org.apache.spark.SparkContext.(SparkContext.scala:510) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.Module at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 解决 sudo cp jackson-databind-2.6.5.jar /usr/mahout/lib/ 12 Exception in thread "main" java.lang.NoClassDefFoundError: com/codahale/metrics/json/MetricsModule at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:225) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:191) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:187) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:187) at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:101) at org.apache.spark.SparkContext.(SparkContext.scala:510) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: om.codahale.metrics.json.MetricsModule 解决 sudo cp metrics-json-3.1.2.jar /usr/mahout/lib/ 13 Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:193) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:187) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:187) at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:101) at org.apache.spark.SparkContext.(SparkContext.scala:510) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonAutoDetect at com.fasterxml.jackson.databind.introspect.VisibilityChecker$Std.(VisibilityChecker.java:170) at com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:261) at org.apache.spark.metrics.sink.MetricsServlet.(MetricsServlet.scala:48) ... 23 more Caused by: java.lang.ClassNotFoundException:com.fasterxml.jackson.annotation.JsonAutoDetect 解决 sudo cp jackson-annotations-2.4.4.jar /usr/mahout/lib/ 14 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED! 7/10/29 23:11:43 ERROR SparkContext: Error initializing SparkContext. java.lang.IllegalStateException: Spark context stopped while waiting for backend at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581) at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162) at org.apache.spark.SparkContext.(SparkContext.scala:549) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) 17/10/29 23:11:43 INFO SparkUI: Stopped Spark web UI at http://192.168.1.110:4040 17/10/29 23:11:43 INFO SparkContext: SparkContext already stopped. Exception in thread "main" java.lang.IllegalStateException: Spark context stopped while waiting for backend 解决 vim yarn-site.xmlyarn.nodemanager.vmem-pmem-ratio 3 15 Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/DefaultScalaModule$ at org.apache.spark.SparkContext.withScope(SparkContext.scala:682) at org.apache.spark.SparkContext.textFile(SparkContext.scala:800) at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:61) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39) at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372) at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException:com.fasterxml.jackson.module.scala.DefaultScalaModule$ 解决 sudo cp jackson-module-scala_2.10-2.4.4.jar /usr/mahout/lib/ 16 Exception in thread "main" java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at com.fasterxml.jackson.module.scala.ser.IteratorSerializerModule$class.$init$(IteratorSerializerModule.scala:70) at com.fasterxml.jackson.module.scala.DefaultScalaModule.(DefaultScalaModule.scala:19) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala:35) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala) at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala:82) at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala) at org.apache.spark.SparkContext.withScope(SparkContext.scala:682) at org.apache.spark.SparkContext.textFile(SparkContext.scala:800) at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:61) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39) at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372) at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) 解决:版本复制错误 2.6.5 ==>2.4.4 sudo cp jackson-module-scala_2.10-2.4.4.jar /usr/mahout/lib/ 17. Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/paranamer/ParanamerAnnotationIntrospector at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospectorModule$class.$init$(ScalaAnnotationIntrospectorModule.scala:105) at com.fasterxml.jackson.module.scala.DefaultScalaModule.(DefaultScalaModule.scala:20) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala:37) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala) at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala:82) at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala) at org.apache.spark.SparkContext.withScope(SparkContext.scala:682) at org.apache.spark.SparkContext.textFile(SparkContext.scala:800) at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:61) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:312) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39) at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372) at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:146) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.module.paranamer.ParanamerAnnotationIntrospector 解决 sudo cp jackson-module-paranamer-2.6.5.jar /usr/mahout/lib/ 17 mvn dependency:tree >> /path/to/dependency.log 查看依赖关系,排除错误版本
现象
异常：当看到异常信息里有NoSuchMethodError时，你应该很高兴，因为这种错误一般是因为jar包冲突引起的。例如异常信息
Invocationof init method failed; nested exception isjava.lang.NoSuchMethodError:org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.
解决方法
排除jar包依赖
步骤
第一步：查看应用最终打成的war包里依赖了哪几个跟netty相关的jar包，执行命令ls |grep *netty* 得到
jboss.jboss-netty-3.2.5.Final.jar
netty-3.2.1.Final.jar
netty-3.6.3.Final.jar
这说明有三个netty相关的jar
第二步：是哪些jar包引入上面三个跟netty相关的jar的？
利用mavendependency:tree插件分析，执行mavendependency:tree>dependency.log里,vimdependency.log得到
org.jboss.netty:netty:jar:3.2.1.Final:compile
io.netty:netty:jar:3.6.3.Final:compile
com.alibaba.external:jboss.jboss-netty:jar:3.2.5.Final:compile
第三步：排除依赖
按照mavendependency:tree里计算出来的maven坐标exclue掉低版本，保留高版本，因此排除掉下面两个jar包
注意：虽然第二步结果中，三个jar包名字
例如Company1.Unit1.Project1:App1:jar,Company2.Unit2.Project2:App2:jar ,它们依赖了同一个类org.jboss.netty.channel.socket.nio.NioClientSocketChannel，
而这两个所在的jar包版本不一致。
由于jar包的加载是随机的，如果加载了低版本就会出现异常NoSuchMethodError。这也是为什么在开发机和daily环境都可以启动，但预发布失败的原因。
18: Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class at org.apache.spark.SparkConf$DeprecatedConfig.(SparkConf.scala:762) at org.apache.spark.SparkConf$.(SparkConf.scala:615) at org.apache.spark.SparkConf$.(SparkConf.scala) at org.apache.spark.SparkConf.set(SparkConf.scala:84) at org.apache.spark.SparkConf.set(SparkConf.scala:73) at org.apache.spark.SparkConf.setMaster(SparkConf.scala:105) at com.shentu.cn.LitchiProcess$.main(LitchiProcess.scala:20) at com.shentu.cn.LitchiProcess.main(LitchiProcess.scala) Caused by: java.lang.ClassNotFoundException: scala.Product$class at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 8 more 解决:引入插件时,使用了高版本编译的scala SDK, 需要选择同等版本的Scala SDK Spark 读取多目录,目录下多文件:
一.多目录
valinputPath = List("hdfs://localhost:9000/test/hiveTest","hdfs://localhost:9000/test/hiveTest2")
.mkString(",")
sparkContext
.textFile( inputPath )
或者是(读取的文件是spark,hadoop作业产生的规则文件)
sc.textFile("hdfs://localhost:9000/test/*/part-*")
二.多文件
sc.textFile("hdfs://localhost:9000/test/hiveTesst2/part-*")