rsyslog通过自定义json格式发送日志信息给logstash

需求：

通过自定义rsyslog的输出格式，通过json编码方式将日志信息发送给logstash进行处理。

Rsyslog配置：

1、将centos6.5的rsyslog升级到最新版

[root@centos-yum ~]# vi /etc/yum.repo.d/rsyslog.repo

[rsyslog_v8]

name=Adiscon CentOS-$releasever - local packages for $basearch

baseurl=http://rpms.adiscon.com/v8-stable/epel-$releasever/$basearch

enabled=1

gpgcheck=0

gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon

protect=1

[root@centos-yum ~]#yum update


[root@centos-yum ~]#yum install rsyslog




2、修改rsyslog的配置文件

[root@centos-yum ~]# vi /etc/rsyslog.conf 
*增加以下内容 
template(name="json_lines" type="list" option.json="on") {
  constant(value="{")
  constant(value="\"timestamp\":\"")
  property(name="timereported" dateFormat="rfc3339")
  constant(value="\",\"message\":\"")
  property(name="msg")
  constant(value="\",\"host\":\"")
  property(name="hostname")
  constant(value="\",\"severity\":\"")
  property(name="syslogseverity-text")
  constant(value="\",\"facility\":\"")
  property(name="syslogfacility-text")
  constant(value="\",\"app-name\":\"")
  property(name="programname")
  constant(value="\",\"procid\":\"")
  property(name="procid")
  constant(value="\"}\n")
}
 
action(
    type="omfwd"
    Target="10.20.20.67"
    Port="8515"
    Protocol="tcp"
    template="json_lines"
 )

rsyslog的json格式输出需要首先定义日志格式模板template，template通过以下格式定义：

template(parameters) { list-descriptions }

每个tmplate都必须有一个name参数，用于定义template名称，该template名称必须唯一；template还需要有一个参数type,用于定义template类型，rsyslog的template支持以下类型：

• list

• subtree

• string

• plugin

    本文只讨论在list的类型下输出json格式，其它类型不做讨论。

template的<options>部分为可选配置，template的options大小写敏感，目前有以下类型：

• option.sql - 将输出格式化为适合mysql的格式。

• option.stdsql - 将输出格式化为适合兼容标准sql的格式。

• option.json - 将输出格式化为json格式。

    本次我们采用option.json格式输出。

2.2 template中List定义

   在list类型的template中，通过一组constant和property描述定义该list，constant描述的是固定的文本信息，property定义了引用的rsyslog属性，例如：

template(name="tpl1" type="list") {    
    constant(value="Syslog MSG is: '")   
    property(name="msg")    
    constant(value="', ")    
    property(name="timereported" 
    dateFormat="rfc3339" caseConversion="lower")    
    constant(value="\n")    
}

property可以有很多参数选择，本例中我们property中的name参数。用于调用rsyslog的properties。

2.3 rsyslog的property

rsyslog中的data item被称为“properties”，需要通过调用对应的property来访问数据。

本例中基与name调用需要发送的日志信息，例如：

property(name="hostname")

详细的property可以参考文档最后的property参考。

2.4 定义对应的rsyslog输出action

rsyslog支持许多种输出方式，本文调用omfwd模块将日志输出到对应的logstash服务器，如下

action(    
    type="omfwd"    
    Target="10.20.20.67"    
    Port="8515"    
    Protocol="tcp"    
    template="json_lines" 
)

3、logstash对应配置文件

logstash对应的input定义如下：

input {    
    tcp {        
        port => "8515"        
        codec => "json"        
        type => "syslog-json"    
    }
}

可以观察到成功收到对应的syslog-json格式日志

    { 
      "timestamp" => "2015-11-26T10:10:59.257734+08:00",       
      "message" => " pam_unix(sshd:session): session opened for user root by (uid=0)",         
       "host" => "centos-yum",      
       "severity" => "info",      
       "facility" => "authpriv",    
       "syslog-tag" => "sshd[15251]:",      
       "app-name" => "sshd",        
       "procid" => "15251",      
       "@version" => "1",    
       "@timestamp" => "2015-11-26T02:10:58.013Z",          
       "type" => "syslog-json"
     }

附：rsyslog的property参考

rsyslog中的data item被称为“properties”，需要通过调用对应的property来访问数据。

Messgage Properties：

msg

           the MSG part of the message (aka “the message” ;))

• rawmsg

• the message excactly as it was received from the socket. Should be useful for debugging. It is also useful if a message should be forwarded totally unaltered.

• rawmsg-after-pri

• Almost the same as rawmsg, but the syslog PRI is removed. If no PRI was present, rawmsg-after-pri is identical to rawmsg. Note that the syslog PRI is header field that contains information on syslog facility and severity. It is enclosed in greater-than and less-than characters, e.g. “<191>”. This field is often not written to log files, but usually needs to be present for the receiver to properly classify the message. There are some rare cases where one wants the raw message, but not the PRI. You can use this property to obtain that. In general, you should know that you need this format, otherwise stay away from the property.

• hostname

• hostname from the message

• source

• alias for HOSTNAME

• fromhost

• hostname of the system the message was received from (in a relay chain, this is the system immediately in front of us and not necessarily the original sender). This is a DNS-resolved name, except if that is not possible or DNS resolution has been disabled.

• fromhost-ip

• The same as fromhost, but always as an IP address. Local inputs (like imklog) use 127.0.0.1 in this property.

• syslogtag

• TAG from the message

programname

    the “static” part of the tag, as defined by BSD syslogd. For example, when TAG is “named[12345]”, programname is “named”.

priPRI part of the message - undecoded (single value)pri-textthe PRI part of the message in a textual form with the numerical PRI appended in brackes (e.g. “local0.err<133>”)iutthe monitorware InfoUnitType - used when talking to a MonitorWare backend (also for Adiscon LogAnalyzer)syslogfacilitythe facility from the message - in numerical formsyslogfacility-textthe facility from the message - in text formsyslogseverityseverity from the message - in numerical formsyslogseverity-textseverity from the message - in text formsyslogpriorityan alias for syslogseverity - included for historical reasons (be careful: it still is the severity, not PRI!)syslogpriority-textan alias for syslogseverity-texttimegeneratedtimestamp when the message was RECEIVED. Always in high resolutiontimereportedtimestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds)timestampalias for timereportedprotocol-versionThe contents of the PROTCOL-VERSION field from IETF draft draft-ietf-syslog-protcolstructured-dataThe contents of the STRUCTURED-DATA field from IETF draft draft-ietf-syslog-protocolapp-nameThe contents of the APP-NAME field from IETF draft draft-ietf-syslog-protocolprocidThe contents of the PROCID field from IETF draft draft-ietf-syslog-protocolmsgidThe contents of the MSGID field from IETF draft draft-ietf-syslog-protocolinputnameThe name of the input module that generated the message (e.g. “imuxsock”, “imudp”). Note that not all modules necessarily provide this property. If not provided, it is an empty string. Also note that the input module may provide any value of its liking. Most importantly, it is not necessarily the module input name. Internal sources can also provide inputnames. Currently, “rsyslogd” is defined as inputname for messages internally generated by rsyslogd, for example startup and shutdown and error messages. This property is considered useful when trying to filter messages based on where they originated - e.g. locally generated messages (“rsyslogd”, “imuxsock”, “imklog”) should go to a different place than messages generated somewhere.

System Properties：

  System Proprties由rsyslog的core engine提供数据。

• $bom

• The UTF-8 encoded Unicode byte-order mask (BOM). This may be useful in templates for RFC5424 support, when the character set is know to be Unicode.

• $now

• The current date stamp in the format YYYY-MM-DD

• $year

• The current year (4-digit)

• $month

• The current month (2-digit)

• $day

• The current day of the month (2-digit)

• $hour

• The current hour in military (24 hour) time (2-digit)

• $hhour

• The current half hour we are in. From minute 0 to 29, this is always 0 while from 30 to 59 it is always 1.

• $qhour

• The current quarter hour we are in. Much like $HHOUR, but values range from 0 to 3 (for the four quater hours that are in each hour)

• $minute

• The current minute (2-digit)

• $myhostname

• The name of the current host as it knows itself (probably useful for filtering in a generic way)