Logstash之kafka数据入ElasticSearch

kafka里面的数据都是自定义拼接的字符串需要在logstash中filter做分割；

如果是json格式，则会被自动解析，无需分割。

下面样例：

input{
      kafka {
        zk_connect => "bdc41.hexun.com:2181,bdc40.hexun.com:2181,bdc46.hexun.com:2181,bdc54.hexun.com:2181,bdc53.hexun.com:2181"
        group_id => "logstash"
        topic_id => "CyLog"
        reset_beginning => false # boolean (optional)
        consumer_threads => 3  # number (optional)
        decorate_events => true
     }
}

filter {
     #用，分割
     ruby{
           init =>"@kname =['showflag','datetime','ip']"
           code =>"event.append(Hash[@kname.zip(event['message'].split(/,/))])" 
          # remove_field => ["message"]
     }

   #有汉字则去掉注解，防止中文乱码，<span style="font-family: Arial, Helvetica, sans-serif;">当然如果传入编码不统一，则会报错</span>
   #  urldecode {
   #      all_fields => true
   #  }
}

output{
    elasticsearch{
      hosts => [ "10.130.2.53:9200","10.130.2.46:9200","10.130.2.54:9200" ]
      flush_size=>50000
      workers => 5
      index=> "logstash-cylog"
    }   
}