Spark 使用过程中遇见的问题

参考：https://blog.csdn.net/SCGH_Fx/article/details/72529132

-------------------------------------------------------------------------------------------------------------------------------------------------------------

1、idea启动出现无效的源发行版：10

解决：intellij：file->project structure->project，Project Language Level，下拉列表选和本机JDK版本一致的，我的JDK1.8，这里选8，ok

 

2、java.lang.ClassNotFoundException: scala.Cloneable

解决：下载scala-sdk

 

3、java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream

解决：将hadoop-common-0.23.9.jar复制到lib目录下。

下载链接：http://www.java2s.com/Code/Jar/h/Downloadhadoopcommon0239jar.htm

 

4、java.lang.ClassNotFoundException: org.spark_project.guava.cache.CacheLoader

解决：将spark-network-common_2.11-2.4.4.jar复制到lib目录下。

下载链接：https://www.mvnjar.com/org.apache.spark/spark-network-common_2.11/2.4.4/detail.html

 

5、java.lang.NoClassDefFoundError: org/apache/commons/lang3/SystemUtils

解决：将commons-lang3-3.9.jar复制到lib目录下。

下载链接：https://archive.apache.org/dist/commons/lang/binaries/

 

6、java.lang.NoClassDefFoundError: scala/Product$class

原因：Spark是基于scala 2.11编译的，而环境使用了最新的2.12的scala的编译版本；

解决：重新安装scala 2.11版本

下载链接：https://www.scala-lang.org/download/2.11.12.html

 

7、java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder

解决：将slf4j-simple-1.7.25.jar和slf4j-api-1.7.25.jar复制到lib目录下。

下载链接：https://mvnrepository.com/artifact/org.slf4j

 

8、java.lang.NoClassDefFoundError: org/apache/log4j/Level

解决：删除slf4j-log4j12.jar依赖

 

9、java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration

解决：将commons-configuration-1.9.jar复制到lib目录下。

 

10、java.lang.NoClassDefFoundError: com/google/common/collect/Maps

解决：将google-collections-1.0.jar复制到lib目录下。

下载链接：http://www.java2s.com/Code/Jar/c/Downloadcommonsconfiguration19jar.htm

 

11、java.lang.NoClassDefFoundError: org/apache/hadoop/security/authentication/util/KerberosName

解决：我这里是有jar包版本冲突，hadoop-common-0.23.9.jar和hadoop-common-2.9.2.jar，删除了前者

 

12、java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName

解决：将hadoop-auth-2.7.2.jar复制到lib目录下。

下载链接：https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-auth/2.7.2

 

13、java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper

解决：将woodstox-core-asl-4.4.1.jar 复制到lib目录下

下载链接：https://mvnrepository.com/artifact/org.codehaus.woodstox/woodstox-core-asl/4.4.1

 

14、java.lang.NoClassDefFoundError: org/codehaus/stax2/XMLInputFactory2

解决：将stax2-2.1.jar 复制到lib目录下

下载链接：http://www.java2s.com/Code/Jar/s/Downloadstax221jar.htm

 

15、java.lang.NoClassDefFoundError: org/codehaus/stax2/ri/evt/BaseEventImpl

解决：将stax2-api-3.0.1.jar复制到lib目录下

下载链接：http://www.java2s.com/Code/Jar/s/Downloadstax2api301jar.htm

 

16、java.lang.NoClassDefFoundError: com/google/common/collect/Interners

解决：将guava-27.0.1-jre.jar复制到lib目录下

下载链接：https://repo1.maven.org/maven2/com/google/guava/guava/27.0.1-jre/

 

17、java.lang.NoSuchMethodError: com.google.common.collect.MapMaker.keyEquivalence(Lcom/google/common/base/Equivalence;)Lcom/google/common/collect/MapMaker;

解决：删除google-collections-xx.jar依赖

 

18、java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosTicket(Ljavax/security/auth/Subject

解决：版本冲突了，hadoop-common降低版本到2.7，添加hadoop-common-2.7.4.jar依赖

下载链接：https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/2.7.4

 

19、java.lang.NoClassDefFoundError: org/apache/spark/network/shuffle/ShuffleClient

解决：将spark-network-shuffle_2.11-2.1.2.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/org.apache.spark/spark-network-shuffle_2.11/2.1.2

 

20、java.lang.ClassNotFoundException: io.netty.channel.Channel

解决：将netty-all-4.1.42.Final.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/io.netty/netty-all/4.1.42.Final

 

21、java.lang.ClassNotFoundException: com.codahale.metrics.MetricSet

解决：将metrics-core-3.0.2.jar复制到lib目录下

下载链接：https://repo1.maven.org/maven2/com/codahale/metrics/metrics-core/3.0.2/

 

22、java.lang.ClassNotFoundException: com.esotericsoftware.kryo.Serializer

解决：将kryo-4.0.0.jar复制到lib目录下

下载链接：https://repo1.maven.org/maven2/com/esotericsoftware/kryo/4.0.0/

 

23、java.lang.ClassNotFoundException: com.twitter.chill.KryoBase

解决：将chill_2.12-0.9.2.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/com.twitter/chill_2.12/0.9.2

 

24、java.lang.ClassNotFoundException: org.apache.spark.unsafe.array.ByteArrayMethods

解决：将spark-unsafe_2.12-2.4.0.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/org.apache.spark/spark-unsafe_2.12/2.4.0

 

25、java.lang.ClassNotFoundException: javax.ws.rs.WebApplicationException

解决：将javax.ws.rs-api-2.0.1.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api/2.0.1

 

26、java.lang.ClassNotFoundException: org.json4s.JsonAST$JValue

解决：将json4s-ast_2.12-3.2.11.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/org.json4s/json4s-ast_2.12/3.2.11

 

27、java.lang.ClassNotFoundException: scala.xml.MetaData

解决：将scala-xml-2.11.0-M4.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/org.scala-lang/scala-xml/2.11.0-M4

 

28、java.lang.ClassNotFoundException: javax.servlet.Servlet

解决：将javax.servlet-api-3.1.0.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/javax.servlet/javax.servlet-api/3.1.0

 

29、java.lang.ClassNotFoundException: org.glassfish.jersey.servlet.ServletContainer

解决：将jersey-container-servlet-core-2.25.1.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/org.glassfish.jersey.containers/jersey-container-servlet-core/2.25.1

 

30、java.lang.ClassNotFoundException: org.glassfish.jersey.server.spi.Container

解决：将jersey-server-2.25.1.jar复制到lib目录下

下载链接：https://mvnrepository.com/artifact/org.glassfish.jersey.core/jersey-server/2.25.1

 

31、java.lang.ClassNotFoundException: org.apache.spark.launcher.LauncherProtocol$Message

解决：将spark-launcher_2.11-2.2.0.jar复制到lib下

下载链接：https://mvnrepository.com/artifact/org.apache.spark/spark-launcher_2.11/2.2.0

 

32、java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.Module

解决：将jackson-databind-2.9.8.jar复制到lib下

下载链接：https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.9.8/

 

33、java.lang.ClassNotFoundException: com.fasterxml.jackson.core.Versioned

解决：将jackson-core-2.9.8.jar复制到lib下

下载链接：https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.9.8/

 

34、java.lang.ClassNotFoundException: com.codahale.metrics.json.MetricsModule

解决：将metrics-json-3.0.2.jar复制到lib下

下载链接：https://repo1.maven.org/maven2/com/codahale/metrics/metrics-json/3.0.2/

 

35、java.lang.ClassNotFoundException: com.fasterxml.jackson.annotation.JsonView

解决：将jackson-annotations-2.9.8.jar复制到lib下

下载链接：https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.9.8/