spark-submit脚本

此spark-submit脚本可以配合定时任务，定时任务做定时5分钟（时间随你定，建议大于1分钟），用于监控spark实时任务，若spark实时任务挂掉，5分钟后会重新启动

spark-submit脚本：

#!/bin/bash
 count=`ps -ef | grep totalUriStreaming | grep -v grep | wc -l` 
 kinit -kt /root/hdfs.keytab hdfs/[email protected]  #此为kebros安全认证，若集群中没有kebros安全认证则不需要
 if [ $count == 0 ]; then      
     echo "Start totalLogUriNums ..."      
     nohup spark-submit \
     --class com.awifi.bigdata.syslog.totalLogUriNums.totalUriStreaming \
     --master yarn \
     --driver-memory 2G \
     --executor-memory 4G \
     --deploy-mode client \
     --num-executors 12 \
     --executor-cores 2 \
     --jars /opt/gofunTest/jars/commons-pool2-2.0.jar,/opt/gofunTest/jars/jedis-2.8.1.jar,/opt/gofunTest/jars/fastjson-1.2.30.jar,/opt/gofunTest/jars/mysql-connector-java-5.1.41-bin.jar \
     /opt/gofunTest/totalUriNums/sparkAnalysis.jar > /opt/gofunTest/totalUriNums/totalUriNums.out 2>&1 & 
 fi

定时任务：(可以看我另一篇关于定时任务的博客)

*/5 * * * * root /xxx/xxx/xxx/spark-submit.sh