spark yarn-cluster模式参数配置

现象：在提交spark任务时指定executor-memory 2g时报错

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Required executor memory (2048+384 MB) is above the max threshold (2048 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
java.lang.IllegalArgumentException: Required executor memory (2048+384 MB) is above the max threshold (2048 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

executor memory的上限时2g，参数配置中executor memory 2g但是集群自身的jvm也需要内存导致资源超出上限；

 

先改了

yarn.scheduler.maximum-allocation-mb

为4g但是仍然报同样的错误；

又改参数

yarn.nodemanager.resource.memory-mb

顺利跑完 ；