spark提交任务时出错

错误:1. 运行:

spark-submit \

--master yarn \

--deploy-mode client \

--num-executors 3 \

--executor-memory 2G \

--class cn.itcast.conversion.PageConversion \

/export/SparkJar/PageConversion.jar /page_conversion/user_conversion.json

 找不到类

spark提交任务时出错_第1张图片

2. 运行:

spark-submit \

--master yarn \

--deploy-mode cluster \

--num-executors 3 \

--executor-memory 2G \

--class cn.itcast.conversion.PageConversion \

/export/SparkJar/PageConversion.jar /page_conversion/user_conversion.json报错。

Container exited with a non-zero exit code 13

Failing this attempt. Failing the application.

 ApplicationMaster host: N/A

 ApplicationMaster RPC port: -1

 queue: default

 start time: 1668946860193

 final status: FAILED

 tracking URL: http://spark01:8088/cluster/app/application_1668945046099_0002

 user: root

Exception in thread "main" org.apache.spark.SparkException: Application application_1668945046099_0002 finished with failed status

at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)

at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

2022-11-20 20:21:26 INFO  ShutdownHookManager:54 - Shutdown hook called

2022-11-20 20:21:26 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-4979d5d9-3a88-4453-90b5-6952b49184c3

2022-11-20 20:21:26 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-65b0a91d-13d8-4714-b909-aa1c7cb24888

原因:文件错放到test测试层

spark提交任务时出错_第2张图片

 解决方法:应该修改到main方法下的java测试层(蓝色文件夹)

再提交测试

spark-submit \

--master yarn \

--deploy-mode cluster \

--num-executors 3 \

--executor-memory 2G \

--class cn.itcast.conversion.PageConversion \

/export/SparkJar/PageConversion.jar /page_conversion/user_conversion.json

spark提交任务时出错_第3张图片spark提交任务时出错_第4张图片

 就解决了~~~

你可能感兴趣的:(hadoop)