Spark- 由于 dynamicAllocation 导致的 executor 不断增加的坑

 

参考文章 ：

解决CDH SparkStreaming任务启动之后executor不停增长的问题,num-executors配置不管用。...

https://blog.csdn.net/arli_xu/article/details/83034577

 

spark中 Dynamic Allocation 以及 num-executors 的问题

https://blog.csdn.net/enterpine/article/details/82664370

 

Spark Dynamic Allocation 分析

https://www.jianshu.com/p/826991ba9375

 

spark 运行环境 ： cdh 5.14 默认的 spark 版本

 

      最近在提交作业的时候遇到了一个问题，特点是 executor 会动态增加。 

       经过研究，我发现主要的问题是，spark 该环境下会 默认设置 spark.dynamicAllocation.enabled 为 true , 该值会导致 executor 数量的动态变化。

 

动态扩容的相关配置参数：

https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation

Spark 2.4.0  latest

Dynamic Allocation

Property Name	Default	Meaning
spark.dynamicAllocation.enabled	false	Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload. For more detail, see the description here.  This requires spark.shuffle.service.enabled to be set. The following configurations are also relevant:spark.dynamicAllocation.minExecutors,spark.dynamicAllocation.maxExecutors, andspark.dynamicAllocation.initialExecutorsspark.dynamicAllocation.executorAllocationRatio
spark.dynamicAllocation.executorIdleTimeout	60s	If dynamic allocation is enabled and an executor has been idle for more than this duration, the executor will be removed. For more detail, see this description.
spark.dynamicAllocation.cachedExecutorIdleTimeout	infinity	If dynamic allocation is enabled and an executor which has cached data blocks has been idle for more than this duration, the executor will be removed. For more details, see this description.
spark.dynamicAllocation.initialExecutors	spark.dynamicAllocation.minExecutors	Initial number of executors to run if dynamic allocation is enabled.  If `--num-executors` (or `spark.executor.instances`) is set and larger than this value, it will be used as the initial number of executors.
spark.dynamicAllocation.maxExecutors	infinity	Upper bound for the number of executors if dynamic allocation is enabled.
spark.dynamicAllocation.minExecutors	0	Lower bound for the number of executors if dynamic allocation is enabled.
spark.dynamicAllocation.executorAllocationRatio	1	By default, the dynamic allocation will request enough executors to maximize the parallelism according to the number of tasks to process. While this minimizes the latency of the job, with small tasks this setting can waste a lot of resources due to executor allocation overhead, as some executor might not even do any work. This setting allows to set a ratio that will be used to reduce the number of executors w.r.t. full parallelism. Defaults to 1.0 to give maximum parallelism. 0.5 will divide the target number of executors by 2 The target number of executors computed by the dynamicAllocation can still be overridden by the spark.dynamicAllocation.minExecutors andspark.dynamicAllocation.maxExecutors settings
spark.dynamicAllocation.schedulerBacklogTimeout	1s	If dynamic allocation is enabled and there have been pending tasks backlogged for more than this duration, new executors will be requested. For more detail, see this description.
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout	schedulerBacklogTimeout	Same as spark.dynamicAllocation.schedulerBacklogTimeout, but used only for subsequent executor requests. For more detail, see this description.

 

 

解决方法 ：

 

设置 该值 为 false, 

并显式指定  driver-cores , executor-cores 等属性

    --driver-memory 2G \
    --driver-cores 1 \
    --num-executors 6 \
    --executor-cores 2 \
    --executor-memory 2G \

 

完整提交命令 :

 

nohup /usr/bin/spark2-submit \
    --class ${class_name} \
    --name ${JOB_NAME} \
    --files ${config} \
    --master yarn \
    --conf spark.dynamicAllocation.enabled=false \
    --driver-memory 2G \
    --driver-cores 1 \
    --num-executors 6 \
    --executor-cores 2 \
    --executor-memory 2G \
    --jars ${classpath} \
    ${ROOT_PATH}/libs/${APP_NAME}-${libVersion}-SNAPSHOT.jar online ${config} \
    > ${ROOT_PATH}/logs/start.error 2> ${ROOT_PATH}/logs/start.log &