Logstash收集json格式日志文件如何写配置文件

1、日志格式

{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}

我们如果收集这个日志只是做简单的配置。如下：

input {
    file {

        path => ["/home/elk/logstash-5.6.3/request"]
        type => "chenxun"
    }
}


output {

    stdout {
        codec => rubydebug
    }

    elasticsearch {
        hosts => "192.168.2.181:9200"

    }
}

那么收集到的结果是：

{
    "_index": "logstash-2017.11.22",
    "_type": "chenxun",
    "_id": "AV_iTR0AM1H1mf2je0nC",
    "_version": 1,
    "_score": 1,
    "_source": {
        "@version": "1",
        "host": "Ubuntu-20170424",
        "path": "/home/elk/logstash-5.6.3/request",
        "@timestamp": "2017-11-22T05:57:05.383Z",
        "message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",
        "type": "chenxun"
    }
}

即会将json记录做为一个字符串放到”message”下,这不是我们想要的结果，是让logstash自动解析json记录，将各字段放入elasticsearch中。下面介绍如何配置.

1.直接设置codec => json

input {
    file {

        path => ["/home/elk/logstash-5.6.3/request"]
        type => "chenxun"
        codec => json

    }   
}

这个时候看看结果： 已经把json解析到各个字段中去了

{
    "_index": "logstash-2017.11.22",
    "_type": "136587",
    "_id": "AV_iXHbGM1H1mf2jfF4d",
    "_version": 1,
    "_score": 1,
    "_source": {
        "48": "136587",
        "10018": 0,
        "10070": 0,
        "10071": 0,
        "10190": 0,
        "10191": 0,
        "path": "/home/elk/logstash-5.6.3/request",
        "@timestamp": "2017-11-22T06:13:51.361Z",
        "@version": "1",
        "host": "Ubuntu-20170424",
        "type": "136587"
    }
}

可以设置编码格式：（收集中文日志）

codec => json {
            charset => "UTF-8"
        }

2、使用filter json

配置如下：

input {
    file {

        path => ["/home/elk/logstash-5.6.3/request"]

    }
}

filter {
        json {
            source => "message"
            #target => "doc"
            #remove_field => ["message"]
        }        
}

output {

    stdout {
        codec => rubydebug
    }

    elasticsearch {
        hosts => "192.168.2.181:9200"

    }
}

输入结果：

{
    "_index": "logstash-2017.11.22",
    "_type": "136587",
    "_id": "AV_igupKM1H1mf2jfxm2",
    "_version": 1,
    "_score": 1,
    "_source": {
        "48": "136587",
        "10018": 0,
        "10070": 0,
        "10071": 0,
        "10190": 0,
        "10191": 0,
        "path": "/home/elk/logstash-5.6.3/request",
        "@timestamp": "2017-11-22T06:55:51.335Z",
        "@version": "1",
        "host": "Ubuntu-20170424",
        "message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",
    "type": "136587"
    }
}

可以看到，原始记录被保存，同时字段也被解析保存。如果确认不需要保存原始记录内容，可以加设置：remove_field => [“message”]

其中特别需要注意解析json数据的内容，logstash会在向es插入数据时默认会在_source下增加type,host,path三个字段，如果json内容中本身也含有type,host,path字段，那么解析后将覆盖掉logstash默认的这三个字段，尤其是type字段，这个同时也是做为index/type用的，覆盖掉后，插入进es中的index/type就是json数据记录中的内容，将不再是logstash config中配置的type值。