logstash@timestamp时间差8小时导致数据到前一天的索引中

问题:logstash@timestamp时间差8小时导致数据到前一天的索引中,比如凌晨00:57的数据跑到了前一天的索引中

很多文章的解决办法,是在原来的@timestamp的基础上进行+8小时来解决了,但我后来试过还是有问题。

给出我自己试验后的解决方案:

日志:

2022-04-01 00:57:49.979-192.168.0.107-stock [http-nio-8082-exec-8] INFO  com.yh.stock.controller.StockController - 库存减一19  -----

logstash配置:

input {
    beats{
	  port => 5044
    }
} 


filter {
	
	
	

    grok{
        match => ['message','%{TIMESTAMP_ISO8601:logdate}']
    }
	
	grok{
        match => { "logdate" => ["%{DATE_EU:ymd}"]}
    }
	
	date{
             match => [ "logdate", "yyyy-MM-dd HH:mm:ss.SSS" ]
             target => "@timestamp"
        }
	
	
	  

	  
	  
}
 
output {
    elasticsearch{
	    hosts => ["127.0.0.1:9200"]
		index => "logstash-test-%{ymd}"
    }
	
	stdout{
		codec=>rubydebug
	}
}

input:数据来源于filebeat

filter:

grok:插件

自带正则:https://github.com/garethr/logstash-patterns/blob/master/patterns/logstash

logstash输出结果:

{
         "agent" => {
        "ephemeral_id" => "12ff0840-e828-4bd8-be26-658625fd9c81",
            "hostname" => "DESKTOP-F55K9CL",
                  "id" => "776244d9-9922-4964-802c-a1423a61a3de",
                "name" => "DESKTOP-F55K9CL",
             "version" => "7.15.2",
                "type" => "filebeat"
    },
           "log" => {
        "offset" => 10326,
          "file" => {
            "path" => "D:\\yh\\ideaProjects\\springcouldalibaba\\stock\\src\\main\\resources\\logs\\20220405\\info.log"
        }
    },
           "ymd" => "22-04-01",
    "@timestamp" => 2022-03-31T16:57:49.979Z,
          "host" => {
        "name" => "DESKTOP-F55K9CL"
    },
           "ecs" => {
        "version" => "1.11.0"
    },
      "@version" => "1",
       "message" => "2022-04-01 00:57:49.979-192.168.0.107-stock [http-nio-8082-exec-8] INFO  com.yh.stock.controller.StockController - 库存减一19  -----",
         "event" => {
        "original" => "2022-04-01 00:57:49.979-192.168.0.107-stock [http-nio-8082-exec-8] INFO  com.yh.stock.controller.StockController - 库存减一19  -----"
    },
         "input" => {
        "type" => "log"
    },
       "logdate" => "2022-04-01 00:57:49.979",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}

日志里的时间是:2022-04-01 00:57:49.979

@timestamp的时间是:2022-03-31T16:57:49.979Z

ymd:22-04-01

因为我们output中配置的是index => "logstash-test-%{ymd}",所以虽然@timestamp变成了我们日志的里的时间减去8小时,但是他最后在es中生成的索引是

logstash@timestamp时间差8小时导致数据到前一天的索引中_第1张图片

达到了我们要的目的。最后,可能有人有疑问,那就是kibana中的时间@timestamp会不会也是减去8小时的时间呢,我们来看一下:

 logstash@timestamp时间差8小时导致数据到前一天的索引中_第2张图片

 没有问题,和我们日志中的时间是一致的

filebeat+elk刚接触,有问题,欢迎指出和讨论

你可能感兴趣的:(elasticsearch,elk,logstash)