Flume远程实时采集Windows产生的log4j产生的数据

一、配置log4j.properties

在IDEA的resource下面新建log4j.properties

配置如下：主要定义了控制台和flume里面的日志格式，flumeAppender配置参考Flume官网

log4j.rootLogger=INFO,stdout,flume
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = 192.168.116.10
log4j.appender.flume.Port = 41414
log4j.appender.flume.UnsafeMode = true
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
log4j.appender.flume.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n

二、配置依赖

我把整个scala项目依赖都写进来的，其实生产日志只需要配置flume-ng-log4jappender就够了

  4.0.0
  com.imooc.spark
  sparktrain
  1.0
  2008
  
    2.11.4
    1.0.0
    2.4.0
    2.6.1
    1.2.6
  
  
    
      org.scala-lang
      scala-library
      ${scala.version}
    
    
      org.apache.kafka
      kafka_2.11
      1.0.0
    
    
      org.apache.hadoop
      hadoop-client
      ${hadoop.version}
    
    
      io.netty
      netty-all
      4.1.17.Final
    
    
      com.fasterxml.jackson.module
      jackson-module-scala_2.11
      2.9.1
    
    
      mysql
      mysql-connector-java
      5.1.31
    
    
      org.apache.hbase
      hbase-client
      ${hbase.version}
    
    
      org.apache.spark
      spark-sql_2.11
      2.4.0
    
    
      org.apache.spark
      spark-streaming-flume_2.11
      2.3.0
    
    
      org.apache.commons
      commons-lang3
      3.5
    
    
      org.apache.spark
      spark-streaming-flume-sink_2.11
      2.3.0
    
    
      org.slf4j
      slf4j-log4j12
      1.7.21
    
    
      log4j
      log4j
      1.2.16
    
    
      org.apache.flume.flume-ng-clients
      flume-ng-log4jappender
      1.6.0
    
    
      org.apache.spark
      spark-streaming-kafka-0-8_2.11
      2.4.0
    
    
      org.apache.spark
      spark-streaming_2.11
      2.4.0
    
  

  
    src/main/scala
    src/test/scala
    
      
        org.scala-tools
        maven-scala-plugin
        
          
            
              compile
              testCompile
            
          
        
        
          ${scala.version}
          
            -target:jvm-1.5
          
        
      
      
        org.apache.maven.plugins
        maven-eclipse-plugin
        
          true
          
            ch.epfl.lamp.sdt.core.scalabuilder
          
          
            ch.epfl.lamp.sdt.core.scalanature
          
          
            org.eclipse.jdt.launching.JRE_CONTAINER
            ch.epfl.lamp.sdt.launching.SCALA_CONTAINER
          
        
      
    
  
  
    
      
        org.scala-tools
        maven-scala-plugin
        
          ${scala.version}
        
      
    
  

三、模拟产生日志

import org.apache.log4j.Logger;

public class LoggerGenerator {
    private static Logger logger=Logger.getLogger(LoggerGenerator.class.getName());
    public static void main(String[] args)throws Exception{
        int i=0;
        while (1==1){
            Thread.sleep(1000);
            logger.info("now is "+i);
            i++;
        }

    }
}

四、配置Flume

a1.sources = r1
a1.sinks = k1
a1.channels = c1
# 配置监控文件
a1.sources.r1.type = avro
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.port=41414
# 配置sink
#a1.sinks.k1.type =logger
a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.directory =/usr/tmp/stream
a1.sinks.k1.sink.rollInterval=300
a1.sinks.k1.sink.batchSize=100
a1.sinks.k1.sink.serializer=TEXT
a1.sinks.k1.sink.serializer.appendNewline = false
#-------------
# 配置channel
a1.channels.c1.type = memory
# 将三者串联
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

五、启动Flume

 flume-ng agent -n a1 -c /usr/local/src/apache-flume-1.6.0-bin/conf -f /usr/local/src/apache-flume-1.6.0-bin/conf/streaming.conf -Dflume.root.logger=INFO,console

六、启动java程序

直接右键run

七、测试

可以发现在定义的文件夹下有新生成的日志文件，关闭Flume之后，日志已经写进去了