Spark On YARN自动调整Executor数量配置 - Dynamic Resource Allocation

Spark 1.5.2版本支持为Spark On YARN模式的Spark Application根据Task自动调整Executor数,要启用该功能,需做以下操作:

一:在所有的NodeManager中,修改yarn-site.xml,为yarn.nodemanager.aux-services添加spark_shuffle值,设置yarn.nodemanager.aux-services.spark_shuffle.class值为org.apache.spark.network.yarn.YarnShuffleService,如下:
修改:

点击(此处)折叠或打开

  1. <property>
  2. <name>yarn.nodemanager.aux-services</name>
  3. <value>mapreduce_shuffle,spark_shuffle</value>
  4. </property>
添加:

点击(此处)折叠或打开

  1. <property>
  2. <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
  3. <value>org.apache.spark.network.yarn.YarnShuffleService</value>
  4. </property>

  5. <property>
  6. <name>spark.shuffle.service.port</name>
  7. <value>7337</value>
  8. </property>

二:将 $SPARK_HOME/lib/spark-1.5.2-yarn-shuffle.jar 文件拷贝到hadoop/lib (hadoop lib)目录下,我的是/usr/lib/hadoop/lib/这个路径下,有些文章还说拷贝到/usr/lib/hadoop-yarn/lib/(yarn lib)或者软连接处理

备注:
这个操作在所有的 nodemanager 中操作


三:配置 $SPARK_HOME/conf/spark-defaults.conf,内容如下

spark.dynamicAllocation.minExecutors 1 #最小Executor数
spark.dynamicAllocation.maxExecutors 100 #最大Executor数

开启自动调节
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true

四:执行时开启自动调整Executor数开关,以spark-sql yarn client模式为例:

spark-sql  --master yarn-client --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=true -e "SELECT COUNT(*) FROM xx"

这里的--conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=true 如果在spark-defaults.conf设置开启就可以省略了

对于使用spark-submit也是一样:
    spark-submit \
    --class SySpark.SqlOnSpark \
    --master yarn-client \
    --conf spark.shuffle.service.enabled=true \
    --conf spark.dynamicAllocation.enabled=true \
    /data/jars/SqlOnSpark.jar \
    "SELECT COUNT(*) FROM xx"

参考:
http://blog.chinaunix.net/uid-22570852-id-5182664.html
http://lxw1234.com/archives/2015/12/593.htm
https://spark.apache.org/docs/1.5.2/job-scheduling.html#default-behavior-of-pools

你可能感兴趣的:(HADOOP,Spark)