Spark+hadoop 开发总结

错误总结:
1.Caused by: java.lang.ClassNotFoundException: scala.xml.MetaData

解决:POM配置依赖
<dependency>



dependency>
2.版本冲突
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$scope()Lscala/xml/TopScope$;
	at org.apache.spark.ui.jobs.StagePage.(StagePage.scala:44)
	at org.apache.spark.ui.jobs.StagesTab.(StagesTab.scala:34)
	at org.apache.spark.ui.SparkUI.(SparkUI.scala:62)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215)
	at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157)
	at org.apache.spark.SparkContext.(SparkContext.scala:443)
	at com.shentu.cn.SimpleALS$.main(SimpleALS.scala:24)
	at com.shentu.cn.SimpleALS.main(SimpleALS.scala)

环境:

scala-2.11.8

依赖:

"org.apache.spark"% "spark-core_2.10" % "2.0.0"



解决方案:

"org.apache.spark"% "spark-core_2.11" % "2.0.0"

3.spark 2.0安装后求PI测试


启动master :
master 节点
./start-master.sh
./start-slave.sh spark://hadoop:7077

cd $SPARK_HOME/bin
./run-example  SparkPi 10 --master spark://master_ip:7077








4.Spark程序启动时hadoop 依赖过低
7/09/30 11:58:53 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(0,0,ResultTask,ExceptionFailure(java.lang.NoSuchMethodError,org.apache.hadoop.mapred.TaskID.(Lorg/apache/hadoop/mapreduce/JobID;Lorg/apache/hadoop/mapreduce/TaskType;I)V,[Ljava.lang.StackTraceElement;@2310a1a5,java.lang.NoSuchMethodError: org.apache.hadoop.mapred.TaskID.(Lorg/apache/hadoop/mapreduce/JobID;Lorg/apache/hadoop/mapreduce/TaskType;I)V
	at org.apache.spark.rdd.HadoopRDD$.addLocalConfiguration(HadoopRDD.scala:372)


经查是在引入spark-hive,spark-sql相关依赖时,hadoop 版本过低所致,hadoop 2.x后无hadoop-core依赖,分散为
hadoop-mapreduce-client-core
hadoop-hdfs
hadoop-common
hadoop-client<
hadoop-annotations
hadoop-auth
等依赖

解决,删除本地仓库对应hadoop-core目录,在对应的引用目录,剔除相应版本的依赖(红色标注)

1. rm -rf /home/jerry/.m2/repository/org/apache/hadoop/hadoop-core/
2.<dependency>









dependency>

















5.log4g ClassNotFoundException,spark-core依赖无法导入

问题:是引入的依赖配置有无

按照maven repository 中央仓库引入的无法使用,需要修改去掉scope部分


<dependency>



   provided
dependency>

6.spark-submit 提交应用报错:classNotFoundException 

经查是配置在spark-env.sh中的Java_HOME报错,无法找到相应的mainClass




7.mahout0.13+hadoop2.8+spark2.0集成后允许mahout 出现错误

归根结底还是相应软件的classpath下无对应的jar,需要在spark jars目录复制过来进行添加


1.无法找到sparkconf主类:  
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

解决:复制/usr/spark/jars目录下的
spark-core_2.11-2.0.0.jar
spark-examples_2.11-2.0.0.jar


2.无法找到scala依赖
ut/spark/spark.csv   --output /out/mahout/      --master    yarn-client      --inDelim   ","      --itemIDPosition   2      --rowIDPosition   1        --filterPosition   2   
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more


解决:同样复制/usr/spark/jars目录下的scala依赖
cp /usr/spark/jars/scala*
scala-xml_2.11-1.0.2.jar
scala-reflect-2.11.8.jar
scala-parser-combinators_2.11-1.0.4.jar
scalap-2.11.8.jar
scala-library-2.11.8.jar
scala-compiler-2.11.8.jar




3.无法找到序列化,压缩类
17/10/26 23:01:27 INFO Utils: Successfully started service 'sparkDriver' on port 39291.
Exception in thread "main" java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/Serializer
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
	at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:238)
	at org.apache.spark.SparkEnv$.instantiateClassFromConf$1(SparkEnv.scala:259)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:262)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:165)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:259)
	at org.apache.spark.SparkContext.(SparkContext.scala:423)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.kryo.Serializer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 18 more

解决:同样复制/usr/spark/jars目录下的kryo相关依赖到/usr/mahout/lib


4.
Exception in thread "main" java.lang.NoClassDefFoundError: com/twitter/chill/KryoBase
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
	at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:238)
	at org.apache.spark.SparkEnv$.instantiateClassFromConf$1(SparkEnv.scala:259)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:262)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:165)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:259)
	at org.apache.spark.SparkContext.(SparkContext.scala:423)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: com.twitter.chill.KryoBase
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)


解决:如果使用idea,maven管理依赖,double shift 搜索kryoBase所在jar包 在本地仓库位置后
将该jar包复制到/usr/mahout/lib/




5
Exception in thread "main" java.lang.NoClassDefFoundError: org/json4s/JsonAST$JValue
	at org.apache.spark.ui.jobs.StagesTab.(StagesTab.scala:33)
	at org.apache.spark.ui.SparkUI.(SparkUI.scala:62)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215)
	at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157)
	at org.apache.spark.SparkContext.(SparkContext.scala:443)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: org.json4s.JsonAST$JValue
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more


maven 中仓库中的对应jar包复制到mahout classpath 目录
sudo cp json4s-ast_2.10-3.2.10.jar /usr/mahout/lib/

6.
Exception in thread "main" java.lang.NoClassDefFoundError: javax/servlet/FilterRegistration
	at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:142)
	at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:135)
	at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:129)
	at org.spark_project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:99)
	at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:128)
	at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:115)
	at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:80)
	at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:64)
	at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:64)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:64)
	at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:68)
	at org.apache.spark.ui.SparkUI.(SparkUI.scala:81)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215)
	at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157)
	at org.apache.spark.SparkContext.(SparkContext.scala:443)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: javax.servlet.FilterRegistration
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 26 more
17/10/29 21:57:05 INFO DiskBlockManager: Shutdown hook called


解决:
sudo cp javax.servlet-api-3.1.0.jar /usr/mahout/lib/


7.


 Exception in thread "main" java.lang.NoClassDefFoundError: org/glassfish/jersey/servlet/ServletContainer
	at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:193)
	at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:75)
	at org.apache.spark.ui.SparkUI.(SparkUI.scala:81)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215)
	at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157)
	at org.apache.spark.SparkContext.(SparkContext.scala:443)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.servlet.ServletContainer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

解决

 sudo cp jersey-container-servlet-core-2.22.2.jar /usr/mahout
/lib/

8

 Exception in thread "main" java.lang.NoClassDefFoundError: org/glassfish/jersey/server/spi/Container
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:193)
	at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:75)
	at org.apache.spark.ui.SparkUI.(SparkUI.scala:81)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215)
	at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157)
	at org.apache.spark.SparkContext.(SparkContext.scala:443)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.server.spi.Container


解决

sudo cp jersey-server-2.22.2.jar /usr/mahout/lib/


9
Exception in thread "main" java.lang.NoClassDefFoundError: org/glassfish/jersey/server/spi/Container
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:193)
	at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:75)
	at org.apache.spark.ui.SparkUI.(SparkUI.scala:81)
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:215)
	at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:157)
	at org.apache.spark.SparkContext.(SparkContext.scala:443)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.server.spi.Container

解决
 sudo cp jersey-server-2.22.2.jar /usr/mahout/lib/



10

javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter]
	at org.spark_project.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1530)
	at org.spark_project.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1560)
	at org.spark_project.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1233)
	at org.spark_project.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1101)
	at org.spark_project.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:367)
	at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1$$anonfun$apply$6.apply(JettyUtils.scala:214)
	at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1$$anonfun$apply$6.apply(JettyUtils.scala:214)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1.apply(JettyUtils.scala:214)
	at org.apache.spark.ui.JettyUtils$$anonfun$addFilters$1.apply(JettyUtils.scala:192)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at org.apache.spark.ui.JettyUtils$.addFilters(JettyUtils.scala:192)
	at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$$anonfun$org$apache$spark$scheduler$cluster$YarnSchedulerBackend$$addWebUIFilter$3.apply(YarnSchedulerBackend.scala:161)
	at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$$anonfun$org$apache$spark$scheduler$cluster$YarnSchedulerBackend$$addWebUIFilter$3.apply(YarnSchedulerBackend.scala:161)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.org$apache$spark$scheduler$cluster$YarnSchedulerBackend$$addWebUIFilter(YarnSchedulerBackend.scala:161)
	at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receive$1.applyOrElse(YarnSchedulerBackend.scala:258)
	at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:211)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: MultiException[javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter]
	at org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:842)
	at org.spark_project.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1526)
	... 25 more
	Suppressed: javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
		at org.spark_project.jetty.servlet.BaseHolder.doStart(BaseHolder.java:102)
		at org.spark_project.jetty.servlet.FilterHolder.doStart(FilterHolder.java:93)
		at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
		at org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:888)
		... 26 more
Caused by: javax.servlet.UnavailableException: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
	at org.spark_project.jetty.servlet.BaseHolder.doStart(BaseHolder.java:102)
	at org.spark_project.jetty.servlet.FilterHolder.doStart(FilterHolder.java:93)
	at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:851)



解决
sudo cp hadoop-yarn-server-web-proxy-2.8.0.jar /usr/mahout/lib/



11


Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/Module
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
	at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:191)
	at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:187)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
	at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:187)
	at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:101)
	at org.apache.spark.SparkContext.(SparkContext.scala:510)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.Module
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

解决
sudo cp jackson-databind-2.6.5.jar /usr/mahout/lib/




12


Exception in thread "main" java.lang.NoClassDefFoundError: com/codahale/metrics/json/MetricsModule
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
	at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:191)
	at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:187)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
	at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:187)
	at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:101)
	at org.apache.spark.SparkContext.(SparkContext.scala:510)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: om.codahale.metrics.json.MetricsModule



解决
sudo cp metrics-json-3.1.2.jar /usr/mahout/lib/

13 
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:193)
	at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:187)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
	at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:187)
	at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:101)
	at org.apache.spark.SparkContext.(SparkContext.scala:510)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonAutoDetect
	at com.fasterxml.jackson.databind.introspect.VisibilityChecker$Std.(VisibilityChecker.java:170)
	at com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:261)
	at org.apache.spark.metrics.sink.MetricsServlet.(MetricsServlet.scala:48)
	... 23 more
Caused by: java.lang.ClassNotFoundException:com.fasterxml.jackson.annotation.JsonAutoDetect
	 

解决
sudo cp jackson-annotations-2.4.4.jar /usr/mahout/lib/


14

 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!

7/10/29 23:11:43 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
	at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581)
	at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162)
	at org.apache.spark.SparkContext.(SparkContext.scala:549)
	at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:99)
	at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:87)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:118)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:199)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
17/10/29 23:11:43 INFO SparkUI: Stopped Spark web UI at http://192.168.1.110:4040
17/10/29 23:11:43 INFO SparkContext: SparkContext already stopped.
Exception in thread "main" java.lang.IllegalStateException: Spark context stopped while waiting for backend

解决
vim yarn-site.xml


yarn.nodemanager.vmem-pmem-ratio

3








15
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/DefaultScalaModule$
	at org.apache.spark.SparkContext.withScope(SparkContext.scala:682)
	at org.apache.spark.SparkContext.textFile(SparkContext.scala:800)
	at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:61)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382)
	at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39)
	at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException:com.fasterxml.jackson.module.scala.DefaultScalaModule$


解决
sudo cp jackson-module-scala_2.10-2.4.4.jar /usr/mahout/lib/



16

Exception in thread "main" java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at com.fasterxml.jackson.module.scala.ser.IteratorSerializerModule$class.$init$(IteratorSerializerModule.scala:70)
	at com.fasterxml.jackson.module.scala.DefaultScalaModule.(DefaultScalaModule.scala:19)
	at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala:35)
	at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala)
	at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala:82)
	at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala)
	at org.apache.spark.SparkContext.withScope(SparkContext.scala:682)
	at org.apache.spark.SparkContext.textFile(SparkContext.scala:800)
	at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:61)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382)
	at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39)
	at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)


解决:版本复制错误 2.6.5 ==>2.4.4
sudo cp jackson-module-scala_2.10-2.4.4.jar /usr/mahout/lib/

17.
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/paranamer/ParanamerAnnotationIntrospector
	at com.fasterxml.jackson.module.scala.introspect.ScalaAnnotationIntrospectorModule$class.$init$(ScalaAnnotationIntrospectorModule.scala:105)
	at com.fasterxml.jackson.module.scala.DefaultScalaModule.(DefaultScalaModule.scala:20)
	at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala:37)
	at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala)
	at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala:82)
	at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala)
	at org.apache.spark.SparkContext.withScope(SparkContext.scala:682)
	at org.apache.spark.SparkContext.textFile(SparkContext.scala:800)
	at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:61)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75)
	at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:312)
	at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382)
	at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39)
	at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
	at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
	at scala.Option.map(Option.scala:146)
	at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
	at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.module.paranamer.ParanamerAnnotationIntrospector

解决
sudo cp jackson-module-paranamer-2.6.5.jar /usr/mahout/lib/



17 mvn dependency:tree >> /path/to/dependency.log 查看依赖关系,排除错误版本

现象

异常:当看到异常信息里有NoSuchMethodError时,你应该很高兴,因为这种错误一般是因为jar包冲突引起的。例如异常信息

Invocationof init method failed; nested exception isjava.lang.NoSuchMethodError:org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(Ljava/util/concurrent/Executor;Ljava/util/concurrent/Executor;II)V



解决方法

    排除jar包依赖



步骤

第一步:查看应用最终打成的war包里依赖了哪几个跟netty相关的jar包,执行命令ls |grep *netty* 得到

    jboss.jboss-netty-3.2.5.Final.jar
   netty-3.2.1.Final.jar
    netty-3.6.3.Final.jar

  这说明有三个netty相关的jar



第二步:是哪些jar包引入上面三个跟netty相关的jar的?

  利用mavendependency:tree插件分析,执行mavendependency:tree>dependency.log,vimdependency.log得到 
 org.jboss.netty:netty:jar:3.2.1.Final:compile
 io.netty:netty:jar:3.6.3.Final:compile
 com.alibaba.external:jboss.jboss-netty:jar:3.2.5.Final:compile



第三步:排除依赖

按照mavendependency:tree里计算出来的maven坐标exclue掉低版本,保留高版本,因此排除掉下面两个jar

 
com.alibaba.external
jboss.jboss-netty

  
org.jboss.netty
netty

 注意:虽然第二步结果中,三个jar包名字不同,但类的全路径org.jboss.netty.channel.socket.nio.NioClientSocketChannel是一样的。 

 例如Company1.Unit1.Project1:App1:jar,Company2.Unit2.Project2:App2:jar ,它们依赖了同一个类org.jboss.netty.channel.socket.nio.NioClientSocketChannel

 而这两个所在的jar包版本不一致。



由于jar包的加载是随机的,如果加载了低版本就会出现异常NoSuchMethodError。这也是为什么在开发机和daily环境都可以启动,但预发布失败的原因。



18:
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
	at org.apache.spark.SparkConf$DeprecatedConfig.(SparkConf.scala:762)
	at org.apache.spark.SparkConf$.(SparkConf.scala:615)
	at org.apache.spark.SparkConf$.(SparkConf.scala)
	at org.apache.spark.SparkConf.set(SparkConf.scala:84)
	at org.apache.spark.SparkConf.set(SparkConf.scala:73)
	at org.apache.spark.SparkConf.setMaster(SparkConf.scala:105)
	at com.shentu.cn.LitchiProcess$.main(LitchiProcess.scala:20)
	at com.shentu.cn.LitchiProcess.main(LitchiProcess.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 8 more

解决:引入插件时,使用了高版本编译的scala SDK, 需要选择同等版本的Scala SDK

 

Spark 读取多目录,目录下多文件:

.多目录

 valinputPath = List("hdfs://localhost:9000/test/hiveTest","hdfs://localhost:9000/test/hiveTest2")
                  .mkString(",")

  sparkContext
     .textFile( inputPath )


或者是(读取的文件是spark,hadoop作业产生的规则文件)

sc.textFile("hdfs://localhost:9000/test/*/part-*")

.多文件

sc.textFile("hdfs://localhost:9000/test/hiveTesst2/part-*")

你可能感兴趣的:(Spark,hadoop,hadoop,spark)