ELK日志处理之使用logstash收集log4J日志 和log4j日志properties配置

Log4j 主要由三部分组成：

1. loggers：负责采集日志信息。
2. appenders：负责将日志信息发布到不同地方。
3. layouts：负责以各种风格格式化日志信息。

创建maven工程，pom中导入日志依赖：

    log4j
    log4j
    1.2.17

 在resource目录下新建log4j.properties,加入以下配置:

### 设置###
log4j.rootLogger = debug,stdout,D,E,logstash

### 输出信息到控制抬 ###
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = [%-5p] %d{yyyy-MM-dd HH:mm:ss,SSS} method:%l%n%m%n

### 输出DEBUG 级别以上的日志到=/Users/elk/log4j/debug.log###
log4j.appender.D = org.apache.log4j.DailyRollingFileAppender
log4j.appender.D.File = /Users/bee/Documents/elk/log4j/debug.log
log4j.appender.D.Append = true
log4j.appender.D.Threshold = DEBUG 
log4j.appender.D.layout = org.apache.log4j.PatternLayout
log4j.appender.D.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n

### 输出ERROR 级别以上的日志到=/Users/elk/log4j/error.log  ###
log4j.appender.E = org.apache.log4j.DailyRollingFileAppender
log4j.appender.E.File =/Users/bee/Documents/elk/log4j/error.log 
log4j.appender.E.Append = true
log4j.appender.E.Threshold = ERROR 
log4j.appender.E.layout = org.apache.log4j.PatternLayout
log4j.appender.E.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n

#输出日志到logstash
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.RemoteHost=127.0.0.1
log4j.appender.logstash.port=4560
log4j.appender.logstash.ReconnectionDelay=60000
log4j.appender.logstash.LocationInfo=true

配置文件中，把日志输出了四份：

1. 第一份输出到控制台
2. 第二份把DEBUG 级别以上的日志到文件
3. 第三份把输出ERROR 级别以上的日志到文件
4. 第四份输出到logstash

在java目录下添加TestLoggerMain.java，内容如下: 

import java.util.ArrayList;
import java.util.List;

import org.apache.log4j.Logger;

public class TestLoggerMain {
	public static final Logger logger = Logger.getLogger(TestLoggerMain.class);

	public static void main(String[] args) {

		logger.debug("This is a debug message!");
		logger.info("This is info message!");
		logger.warn("This is a warn message!");
		logger.error("This is error message!");
		List list = new ArrayList();
		list.add(1);
		list.add(2);
		list.add(3);

		try {
			for (Integer num : list) {
				list.remove(num);
			}

		} catch (Exception e) {
			logger.error(e);
		}
	}

}

配置logstash

创建logstash.conf配置文件

input {                                      #日志数据输入来源log4j
    log4j {
        host => "127.0.0.1"
        port => 4561
    }

}



output {
    stdout {
      codec => rubydebug
    }
    elasticsearch{
        hosts => ["localhost:9200"]
        index => "log4j-%{+YYYY.MM.dd}"
        document_type => "log4j_type"
    }
}

首先启动Elasticsearch，然后启动logstash和kibaba

logstash输出：

进入kibaba查询：

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 16,
    "successful": 16,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.2.0",
        "_score": 1,
        "_source": {
          "buildNum": 14695
        }
      },
      {
        "_index": "log4j-2019.01.08",
        "_type": "log4j_type",
        "_id": "AWgsBq4xsF3SfWsXEnpg",
        "_score": 1,
        "_source": {
          "method": "queryData",
          "thread": "http-nio-8025-exec-1",
          "priority": "ERROR",
          "type": "simple",
          "message": "nested exception is org.apache.ibatis.type.TypeException: Could not set parameters for mapping: ParameterMapping{property='level', mode=IN, javaType=class java.lang.Integer, jdbcType=null, numericScale=null, resultMapId='null', jdbcTypeName='null', expression='null'}. Cause: org.apache.ibatis.type.TypeException: Error setting non null for parameter #1 with JdbcType null . Try setting a different JdbcType for this parameter or a different configuration property. Cause: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer",
          "path": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
          "@timestamp": "2019-01-08T05:54:20.018Z",
          "file": "JiucaiyunSolrServiceImpl.java:328",
          "@version": "1",
          "host": "127.0.0.1:56347",
          "logger_name": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
          "class": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
          "timestamp": 1546926859986,
        }
      }
    ]
  }
}

# 控制台(console)
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.Threshold=DEBUG
log4j.appender.console.ImmediateFlush=true
log4j.appender.console.Target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%p] %m%n

# 日志文件(logFile)
log4j.appender.logFile=org.apache.log4j.FileAppender
log4j.appender.logFile.Threshold=DEBUG
log4j.appender.logFile.ImmediateFlush=true
log4j.appender.logFile.Append=true
log4j.appender.logFile.File=D:/logs/log.log4j
log4j.appender.logFile.layout=org.apache.log4j.PatternLayout
log4j.appender.logFile.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%p] %m%n

# 滚动文件(rollingFile)
log4j.appender.rollingFile=org.apache.log4j.RollingFileAppender
log4j.appender.rollingFile.Threshold=DEBUG
log4j.appender.rollingFile.ImmediateFlush=true
log4j.appender.rollingFile.Append=true
log4j.appender.rollingFile.File=D:/logs/log.log4j
log4j.appender.rollingFile.MaxFileSize=200KB
log4j.appender.rollingFile.MaxBackupIndex=50
log4j.appender.rollingFile.layout=org.apache.log4j.PatternLayout
log4j.appender.rollingFile.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%p] %m%n

# 定期滚动日志文件(dailyFile)
log4j.appender.dailyFile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.dailyFile.Threshold=DEBUG
log4j.appender.dailyFile.ImmediateFlush=true
log4j.appender.dailyFile.Append=true
log4j.appender.dailyFile.File=D:/logs/log.log4j
log4j.appender.dailyFile.DatePattern='.'yyyy-MM-dd
log4j.appender.dailyFile.layout=org.apache.log4j.PatternLayout
log4j.appender.dailyFile.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%p] %m%n

以上介绍的配置都是全局的，整个工程的代码使用同一套配置，意味着所有的日志都输出在了相同的地方，你无法直接了当的去看数据库访问日志、用户登录日志、操作日志，它们都混在一起，因此，需要为包甚至是类配置单独的日志输出，下面给出一个例子，为“com.demo.test”包指定日志输出器“test”，“com.demo.test”包下所有类的日志都将输出到/log/test.log文件

log4j.logger.com.demo.test=DEBUG,test
log4j.appender.test=org.apache.log4j.FileAppender
log4j.appender.test.File=/log/test.log
log4j.appender.test.layout=org.apache.log4j.PatternLayout
log4j.appender.test.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%p] %m%n

也可以让同一个类输出不同的日志，为达到这个目的，需要在这个类中实例化两个logger

private static Log logger1 = LogFactory.getLog("myTest1");
private static Log logger2 = LogFactory.getLog("myTest2");

然后分别配置

log4j.logger.myTest1= DEBUG,test1
log4j.additivity.myTest1=false
log4j.appender.test1=org.apache.log4j.FileAppender
log4j.appender.test1.File=/log/test1.log
log4j.appender.test1.layout=org.apache.log4j.PatternLayout
log4j.appender.test1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%p] %m%n
  
log4j.logger.myTest2=DEBUG,test2
log4j.appender.test2=org.apache.log4j.FileAppender
log4j.appender.test2.File=/log/test2.log
log4j.appender.test2.layout=org.apache.log4j.PatternLayout
log4j.appender.test2.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%p] %m%n