解决spark streaming日志不断增长问题

spark streaming在yarn-client或者本地local执行的时候，由于长时间运行日志会越来越多，查看起来很不方便，可以利用log4j对spark streaming的日志进行配置。
配置步骤如下：

• 1.新建driver端的log4j配置文件driver-log4j.properties

log4j.rootLogger =INFO,stdout,I,E

### output to console ###
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm} %5p %t %c{2}:%L - %m%n

### output error to files ####
log4j.appender.E=org.apache.log4j.DailyRollingFileAppender
log4j.appender.E.layout=org.apache.log4j.PatternLayout
log4j.appender.E.layout.conversionPattern=%-d{yyyy-MM-dd HH:mm:ss} %5p %t %c{2}:%L - %m%n
log4j.appender.E.maxFileSize=100MB
log4j.appender.E.maxBackupIndex=5
log4j.appender.E.Append = true
log4j.appender.E.Threshold = ERROR
log4j.appender.E.file=/home/root/log/streaming/stderror.log
log4j.appender.E.encoding=UTF-8

### output info to files ####
log4j.appender.I=org.apache.log4j.DailyRollingFileAppender
log4j.appender.I.layout=org.apache.log4j.PatternLayout
log4j.appender.I.layout.conversionPattern=%-d{yyyy-MM-dd HH:mm:ss} %5p %t %c{2}:%L - %m%n
log4j.appender.I.maxFileSize=100MB
log4j.appender.I.maxBackupIndex=5
log4j.appender.I.Append = true
log4j.appender.I.Threshold = INFO
log4j.appender.I.file=/home/root/log/streaming/stdout.log
log4j.appender.I.encoding=UTF-8

以上分别配置了控制台，info日志输出和error的输出

执行脚本如下：

BASEDIR=$(dirname $(readlink -f $0))
    echo $BASEDIR

 nohup  spark-submit  \
        --class com.MainFunc \
        --master local[8] \
        --driver-memory 2g \
        --executor-memory 3g \
        --conf spark.eventLog.enabled=false  \
        --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:driver-log4j.properties \
        --files $BASEDIR/driver-log4j.properties \
        $BASEDIR/../jars/my-streaming-1.0.jar  > /dev/null 2>&1 &

注意结尾，输出到 /dev/null。