Spark任务异常The auxService spaark_shuffle does not exist

问题现象

Spark任务抛出如下异常:

org.apache.hadoop.yarm.exception.InvalidAuxServiceException: The auxService:spark_shuffle does not exist

问题解决方案

于yarn-site.xml修改或添加如下配置:


    yarn.nodemanager.aux-services
    spark_shuffle,mapreduce_shuffle


    yarn.nodemanager.aux-services.spark_shuffle.class
    org.apache.spark.network.yarn.YarnShuffleService

你可能感兴趣的:(Spark任务异常The auxService spaark_shuffle does not exist)