Rsyslog采集linux日志及转发到Logstash

最新要做一个对Linux系统日志采集的需求，当然除了Linux的系统日志采集外，还需要转发Tomcat日志，或者Nginx日志等。所以就使用了rsyslog这个比较常用并且功能比较强大的工具。

版本：

Rsyslog V5

Logstash 5.2.2

一、就不做过多的介绍了直接贴测试通过的rsyslog.conf配置文件该配置文件的目录为：

/etc/rsyslog.conf

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad imklog  # provides kernel logging support (previously done by rklogd)

#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception

#$ModLoad imudp

#$UDPServerRun 514

# Provides TCP syslog reception

#$ModLoad imtcp

#$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.(内核)

kern.*                                                /dev/console

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!(记录的内核消息、各种服务的公共消息，报错信息等)

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.(包含验证和授权方面信息)

authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.(包含来着系统运行电子邮件服务器的日志信息)

mail.*                                                  -/var/log/maillog

# Log cron stuff(每当cron进程开始一个工作时，就会将相关信息记录在这个文件中)

cron.*                                                  /var/log/cron

# Everybody gets emergency messages

*.emerg                                                *

# Save news errors of level crit and higher in a special file.

uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log(自定义的消息)

local7.*                                                /var/log/boot.log

$ModLoad imfile #装载imfile模块

$InputFileName /opt/tomcat/apache-tomcat-8.5.15/logs/catalina.out #读取日志文件

$InputFileTag catalina: #日志写入日志附加标签字符串

$InputFileFacility local5 #日志类型

$InputFileSeverity info #日志等级

$InputFileStateFile ssologs.log_state #定义记录偏移量数据文件名

$InputFilePollInterval 1 #检查日志文件间隔（秒）

$InputFilePersistStateInterval 1 #回写偏移量数据到文件间隔时间（秒）

$InputRunFileMonitor #激活读取，可以设置多组日志读取，每组结束时设置本参数。以示生效。

$InputFileName /opt/tomcat/apache-tomcat-8.5.15/logs/localhost_access_log.%$year%-%$month%-%$day%.txt #读取日志文件

$InputFileTag access: #日志写入日志附加标签字符串

$InputFileFacility local6 #日志类型

$InputFileSeverity info #日志等级

$InputFileStateFile sssologs.log_state #定义记录偏移量数据文件名

$InputFilePollInterval 1 #检查日志文件间隔（秒）

$InputFilePersistStateInterval 1 #回写偏移量数据到文件间隔时间（秒）

$InputRunFileMonitor #激活读取，可以设置多组日志读取，每组结束时设置本参数。以示生效。

# ### begin forwarding rule ###

# The statement between the begin ... end define a SINGLE forwarding

# rule. They belong together, do NOT split them. If you create multiple

# forwarding rules, duplicate the whole block!

# Remote Logging (we use TCP for reliable delivery)

#

# An on-disk queue is created for this action. If the remote host is

# down, messages are spooled to disk and sent when it is up again.

#$WorkDirectory /var/lib/rsyslog # where to place spool files

#$ActionQueueFileName fwdRule1 # unique name prefix for spool files

#$ActionQueueMaxDiskSpace 1g  # 1gb space limit (use as much as possible)

#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown

#$ActionQueueType LinkedList  # run asynchronously

#$ActionResumeRetryCount -1    # infinite retries if host is down

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

*.* @10.255.0.167:514

# ### end of the forwarding rule ###

# A template to for higher precision timestamps + severity logging

$template SpiceTmpl,"%TIMESTAMP%.%TIMESTAMP:::date-subseconds% %syslogtag% %syslogseverity-text%:%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

:programname, startswith, "spice-vdagent"  /var/log/spice-vdagent.log;SpiceTmpl

上面文件中：

. @remote-host:514

.即表示转发所有设备的日志信息

@表示使用UDP协议传输

@@表示使用TCP协议传输

找到上面这句去掉前面的#号然后添加对应的IP和端口即可。

例：

. @10.255.0.165:514

如果你只想要转发服务器上的指定设备的日志消息，比如说内核设备，那么你可以在rsyslog配置文件中使用以下声明。

kern.* @10.255.0.165:514

修改完成后执行service rsyslog restart 重新启动rsyslong 即可。

二、Logstash配置

input {

    udp {

        port => 514

        type => syslog

    }

}

filter {

    if [type] == "syslog" {

            grok {

              patterns_dir => "/opt/logstash/logstash-5.2.2/patterns"

              match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

            }

    }

}

output{

    elasticsearch {

        hosts => ["10.255.0.167"]

        index => "rsyslog_test"

    }

    stdout{

        codec => rubydebug

    }

}

最后更新配置：/etc/init.d/rsyslog restart

附：Logstash的配置很简单只是监听514端口就可以了，但是使用grok切日志才是麻烦的，毕竟那么多种的日志每种都要写对应的正则O(∩_∩)O~