spark-submit提交到yarn上常见错误

#!/bin/bash
systime=`date -d "-24 hour" +%Y%m-%d-%H-%M`
ym=`echo ${systime} | awk -F '-' '{print $1}'`
d=`echo ${systime} | awk -F '-' '{print $2}'`
h=`echo ${systime} | awk -F '-' '{print $3}'`
syst_print=`date`
echo ------------------------ $syst_print ------------------------------ >> /data/module/testcxb/rec_match.log

nohup /home/modules/spark-2.1.1/bin/spark-submit --master yarn --deploy-mode cluster --class com.emg.rec_match.RecMatch --name match --driver-memory 2G --executor-memory 7G --executor-cores 2 --num-executors 15 /data/module/testcxb/rec_match-1.0-SNAPSHOT.jar ${ym}/${d}

 

修改为yarn模式可能会出现内存不够的问题

19/01/24 10:03:00 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
Exception in thread "main" java.lang.IllegalArgumentException: Required executor memory (9216+921 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

是因为默认executor-memory的内存为8G  需要在yarn-site.xml中添加（具体参数看机器内存）

        
  		yarn.nodemanager.resource.memory-mb
  		24576