debug date filter in logstash

Background

Recently some guys came by and were curious about the date filter in logstash (both 2.4 and 5.x, but I am using logstash 2.4 in this post). The basic function for date filter is to convert a free style date string to a timestamp (or datetime), which shall be used to set @timestamp of the this message.

But it seemed that people often came across issues during this procedure, so I post this passage to make this whole process easier.

PS. this post does not contain much information, and is quite easy for understanding. It serves as a quick help to new guys here.

Solution

In this post, I am going to set up a debug solution of debugging date filter in logstash. With this set up, we can:

• directly set the input to logstash
• set up filters in logstash, add debug into through ruby filter
• output in debug mode

Step 1 setup logstash.json

Download logstash 2.4 or logstash 5.X and unzip them to your local PC.
add a logstash.json in this folder (or anywhere you like). The content of this file may look like the following.

input {
  stdin {}
}

filter {
  # used for convert a customized date string to a timestamp
  # which will be set as '@timestamp' field of the event
  date {
    # message means the full body of a single incoming event
    match => [ "message", "dd/MMM/yyyy:HH:mm:ss Z"]
  }

  # used to add additional field to this event
  ruby {
    # convert timestamp to customized time string
    init => "require 'date'"
    # BUT be careful, for logstash 5.X, you cannot direct set event's field, but instead you can use event.set method to make it. Please refer here: https://www.elastic.co/guide/en/logstash/current/event-api.html
    code => "event['@testST'] = DateTime.strptime('1318996912','%s').strftime('%a, %d %b %Y')"
  }
}

output {
  stdout { codec => rubydebug }
}

I will explain some items of the configuration here.

input {
  stdin {}
}

This will take the input data directly from standard input. So later we can freely set the input data through terminal.

output {
  stdout { codec => rubydebug }
}

It sets the output with fairly debug info.

As for the other filters in the logstash, please refer to the official doc.

Step 2 off you go

With logstash.json created, now let’s play.

echo "14/Jul/2017:00:57:29 +0800" | bin/logstash -e -f ./logstash.json

In this demo, I am feeding logstash with a single piece of message 14/Jul/2017:00:57:29 +0800, which is referred as message in logstash configuration.

Our task here is to extract the date string and put that date as the @timestamp of this message. (Otherwise logstash is going to set @timestamp to when it receives this message, which might not meet our need)

You can check with the output and see whether your converting with date filter is correctly working. If it is not working as expected, you can easily change your logstash configuration file and test again! It saves a lot of time.

Contact me

If you get any question, you are welcome to contact me via:

• email: [email protected]
• wechat: nisxiya