[Spark streaming举例]-- yarn-cluster运行spark streaming问题
一、本地运行spark出现的错误
Exception in thread "main" java.lang.VerifyError:
class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.
(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;
Lcom/fasterxml/jackson/databind/JsonSerializer;)
Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
at java.lang.ClassLoader.defineClass1(Native Method)
解决：jar版本冲突！、缺少jar包
二、集群运行kafka+spark streaming 报错：
16/07/16 19:21:38 ERROR ApplicationMaster: Uncaught exception:
java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
Log Type: stderr
Log Upload Time: Sat Jul 16 20:06:15 +0800 2016
Log Length: 4090
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data1/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/lib/hadoop/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/07/16 20:06:09 INFO ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
16/07/16 20:06:10 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1464573394111_39135_000001
16/07/16 20:06:11 INFO SecurityManager: Changing view acls to: yarn,e_lvbin
16/07/16 20:06:11 INFO SecurityManager: Changing modify acls to: yarn,e_lvbin
16/07/16 20:06:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, e_lvbin); users with modify permissions: Set(yarn, e_lvbin)
16/07/16 20:06:11 INFO ApplicationMaster: Starting the user application in a separate Thread
16/07/16 20:06:11 ERROR ApplicationMaster: Uncaught exception:
java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
at java.util.jar.JarVerifier.update(JarVerifier.java:228)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
at java.util.jar.JarFile.getInputStream(JarFile.java:415)
at sun.misc.JarIndex.getJarIndex(JarIndex.java:137)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:674)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:666)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:665)
at sun.misc.URLClassPath$JarLoader.
at sun.misc.URLClassPath$3.run(URLClassPath.java:366)
at sun.misc.URLClassPath$3.run(URLClassPath.java:356)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:355)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:332)
at sun.misc.URLClassPath.getResource(URLClassPath.java:198)
at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:519)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:290)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:155)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:625)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:623)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
16/07/16 20:06:11 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes)
16/07/16 20:06:11 INFO ShutdownHookManager: Shutdown hook called
Log Type: stdout
Log Upload Time: Sat Jul 16 20:06:15 +0800 2016
Log Length: 0
解决：
1、注意以后引入包时一定使用maven引入，除了必要的以外，不要自己下载包，所以我把原有的包通过maven引入后，
再打包上传至集群，提交任务
2、下载json-lib，需要这样写
然而集群运行仍然报错：
Log Type: stderr
Log Upload Time: Sun Jul 17 10:03:41 +0800 2016
Log Length: 129904
Showing 4096 bytes of 129904 total. Click here for the full log.
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:525)
16/07/17 10:03:40 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.SparkException: java.nio.channels.UnresolvedAddressException
org.apache.spark.SparkException: Couldn't find leader offsets for Set([20160710aa,0]))
16/07/17 10:03:40 INFO SparkContext: Invoking stop() from shutdown hook
16/07/17 10:03:40 INFO SparkUI: Stopped Spark web UI at http://10.37.54.14:37161
16/07/17 10:03:40 INFO DAGScheduler: Stopping DAGScheduler
16/07/17 10:03:40 INFO YarnClusterSchedulerBackend: Shutting down all executors
16/07/17 10:03:40 INFO YarnClusterSchedulerBackend: Asking each executor to shut down
16/07/17 10:03:40 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. host21.slave.cluster.enn.cn:50376
16/07/17 10:03:40 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. host16.slave.cluster.enn.cn:36144
16/07/17 10:03:40 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/07/17 10:03:40 INFO MemoryStore: MemoryStore cleared
16/07/17 10:03:40 INFO BlockManager: BlockManager stopped
16/07/17 10:03:40 INFO BlockManagerMaster: BlockManagerMaster stopped
16/07/17 10:03:40 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/07/17 10:03:40 INFO SparkContext: Successfully stopped SparkContext
16/07/17 10:03:40 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: org.apache.spark.SparkException: java.nio.channels.UnresolvedAddressException
org.apache.spark.SparkException: Couldn't find leader offsets for Set([20160710aa,0]))
16/07/17 10:03:40 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/07/17 10:03:40 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/07/17 10:03:40 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
16/07/17 10:03:40 INFO Remoting: Remoting shut down
16/07/17 10:03:40 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
16/07/17 10:03:40 INFO ApplicationMaster: Deleting staging directory .sparkStaging/application_1464573394111_39734
16/07/17 10:03:41 INFO ShutdownHookManager: Shutdown hook called
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data9/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-69c591f3-d723-47bb-b2e1-bfc77c052e58
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data8/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-7765f20d-f307-4c1a-8811-fdb48993b702
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data7/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-15c74518-c9f0-4edd-b643-460f502b7f20
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data5/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-e6651b39-9817-44e3-a469-f80ac57d9a2b
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data6/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-9a0870db-9708-48d5-b6c1-bc2914eedde9
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data2/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-a01e9f2c-4c00-48e3-ae1e-b4fd5aba99a2
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data10/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-217ea605-f5ec-47d9-99d0-e06ff9d9988f
16/07/17 10:03:41 INFO ShutdownHookManager: Deleting directory /data4/yarn/nm/usercache/e_lvbin/appcache/application_1464573394111_39734/spark-68e961c9-d244-4484-bed5-4dfe17d61400
Log Type: stdout
Log Upload Time: Sun Jul 17 10:03:41 +0800 2016
Log Length: 0
解决：注意红色字体，在设置kafka的broker时，由于spark streaming通过ZK去获取kafka的地址, 但是zk中保存的kafka是以域名的方式保存的, 而新集群上没有配置相关的hosts, 所以只需要在新集群的spark机器上的/etc/hosts加上对应的kafka hosts域名和ip即可解决。
另外可能是：安装kafka时，配置的不是ip，而是hostname.