Spark基于Yarn-Client提交任务报错以及解决方案

Spark基于Yarn-client模式提交人物产生的错误

初次在Yarn-client模式下提交时，第一次运行报错，主要为一下三部分

• client token: N/A
  	 diagnostics: Application application_1568870310822_0001 failed 2 times due to AM Container for appattempt_1568870310822_0001_000002 exited with
    exitCode: 1For more detailed output, check application tracking page:http://node03:8088/proxy/application_1568870310822_0001/Then, click on links to logs of each a
  ttempt.Diagnostics: Exception from container-launch.
  Container id: container_1568870310822_0001_02_000001
  Exit code: 1
  Stack trace: ExitCodeException exitCode=1: 
  	at org.apache.hadoop.util.Shell.runCommand(Shell.java:575)
  	at org.apache.hadoop.util.Shell.run(Shell.java:478)
  	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766)
  	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
  	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
  	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
  	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  	at java.lang.Thread.run(Thread.java:745)
  

• 2019-09-19 13:19:54 INFO  Client:54 - Deleted staging directory hdfs://mycluster/user/root/.sparkStaging/application_1568870310822_0001
  2019-09-19 13:19:54 ERROR SparkContext:91 - Error initializing SparkContext.
  org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
  	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
  	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
  	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
  	at org.apache.spark.SparkContext.(SparkContext.scala:500)
  	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
  	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:933)
  	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:924)
  	at scala.Option.getOrElse(Option.scala:121)
  	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:924)
  	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
  	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:498)
  	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
  	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
  	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
  	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
  	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  
  

• Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch applicatio
  n master.	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
  	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
  	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
  	at org.apache.spark.SparkContext.(SparkContext.scala:500)
  	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
  	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:933)
  	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:924)
  	at scala.Option.getOrElse(Option.scala:121)
  	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:924)
  	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
  	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:498)
  	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
  	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
  	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
  	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
  	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  

解决方法

• 在resourcemanager中查看日志，显示编译出来的class版本，与jdk版本不匹配，则检查hadoop-env.sh,yarn-env.sh,mapred-env.sh中jdk版本是否一致，不一致在需要在每个node中更改

• 再次更改之后记得重启集群！再次进行测试，仍然失败，在resourcemanager的webui上查看显示内存爆了

• 在yarn-site.xml中添加配置

  	<property>
          <name>yarn.nodemanager.pmem-check-enabledname>
          <value>falsevalue>
      property>
  
      <property>
          <name>yarn.nodemanager.vmem-check-enabledname>
          <value>falsevalue>
      property>
  
  

• 重启集群，再次提交任务，终于成功！