浅谈spark yarn模式的问题调试

    spark运行的时候，有可能发生崩溃，而在spark console里面看到的堆栈，很可能不是发生问题的实际堆栈，这个时候需要看yarn日志来定位问题。

    我在调试spark程序的时候，采用如下命令启动spark驱动程序：

/usr/local/spark-1.3.1-bin-hadoop2.6/bin/spark-submit --supervise --class spark_security.Sockpuppet  --name "testperf"  --executor-memory 4096M --num-executors 8 --driver-memory 8096M --master yarn-client /home/www/spark_Security-1.0-SNAPSHOT.jar

这个时候报如下错误：

15/07/03 14:35:01 INFO scheduler.DAGScheduler: Stopping DAGScheduler
15/07/03 14:35:01 INFO scheduler.DAGScheduler: Job 10 failed: foreachRDD at Sockpuppet.scala:80, took 2.226514 s
15/07/03 14:35:01 INFO scheduler.DAGScheduler: Stage 20 (map at Sockpuppet.scala:57) failed in 2.192 s
15/07/03 14:35:01 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
15/07/03 14:35:01 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut down
15/07/03 14:35:01 ERROR scheduler.JobScheduler: Error running job streaming job 1435905299000 ms.0
org.apache.spark.SparkException: Job cancelled because SparkContext was shut down
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:699)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:698)
        at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
        at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:698)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1411)
        at org.apache.spark.util.EventLoop.stop(EventLoop.scala:84)
        at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1346)
        at org.apache.spark.SparkContext.stop(SparkContext.scala:1380)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:143)
Exception in thread "main" org.apache.spark.SparkException: Job cancelled because SparkContext was shut down
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:699)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:698)
        at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
        at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:698)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1411)
        at org.apache.spark.util.EventLoop.stop(EventLoop.scala:84)
        at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1346)
        at org.apache.spark.SparkContext.stop(SparkContext.scala:1380)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$$anon$1.run(YarnClientSchedulerBackend.scala:143)
15/07/03 14:35:01 INFO cluster.YarnClientSchedulerBackend: Stopped
15/07/03 14:35:01 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
15/07/03 14:35:01 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
15/07/03 14:35:01 INFO storage.MemoryStore: MemoryStore cleared
15/07/03 14:35:01 INFO storage.BlockManager: BlockManager stopped

可是在我的驱动程序中，实在是没有出错导致异常的地方。这个时候我采用如下命令

yarn logs -applicationId application_1436175803684_0004 >execption

看了一下yarn日志。

yarn中出错信息有2个，第一个是如下堆栈信息：

 at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

第二个是如下错误：

15/07/06 10:49:54 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM

我最先根据

 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM

这条出错信息查找解决方案，尝试了各种解决方案，花费了很长的时间，问题还是没有解决。

然后根据yarn里面的这条堆栈信息：

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

    找到了解决方案，问题的原因是hdp版本不对，在spark-defaults.conf文件里面加上如下两行：

spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0–2041
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0–2041

问题解决

总结一下：

1. 当spark console打印的堆栈很可能只是表面现象，导致问题出现的堆栈信息很可能在yarn的日志里面

2. yarn日志里面的堆栈错误，要优先排查解决