1、采集tomcat确实比之前的需求复杂很多，我在搭建了一个tomcat的环境，然后产生如下报错先贴出来：
Jan 05, 2017 10:53:35 AMorg.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allowsoptimal performance in production environments was not found on thejava.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jan 05, 2017 10:53:35 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8088"]
Jan 05, 2017 10:53:35 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jan 05, 2017 10:53:35 AM org.apache.coyote.AbstractProtocol init
SEVERE: Failed to initialize end point associated withProtocolHandler ["ajp-bio-8009"]
java.net.BindException: Address already in use (Bind failed):8009
atorg.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:413)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:665)
atorg.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:452)
atorg.apache.catalina.startup.Catalina.load(Catalina.java:667)
atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
atorg.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:400)
... 16 more
2、分析我们需要的结构：
通过上面的分析，我们需要的数据有:时间戳、类名、日志信息。
我们需要的操作就是先把相同时间和的多行日志数据合并到同一个事件里面再分析。
###提示，因为tomcat日志比较困难，我们可以参考默认的日志结构：
[root@monitor patterns]# pwd
/test/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.0.2/patterns
[root@monitor patterns]# catjava
JAVACLASS(?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
#Space is an allowed characterto match special cases like 'Native Method' or 'Unknown Source'
JAVAFILE (?:[A-Za-z0-9_. -]+)
#Allow special , methods
JAVAMETHOD(?:(<(?:cl)?init>)|[a-zA-Z$_][a-zA-Z$_0-9]*)
#Line number is optional inspecial cases 'Native method' or 'Unknown source'
JAVASTACKTRACEPART %{SPACE}at%{JAVACLASS:class}\.%{JAVAMETHOD:method}\(%{JAVAFILE:file}(?::%{NUMBER:line})?\)
# Java Logs
JAVATHREAD(?:[A-Z]{2}-Processor[\d]+)
JAVACLASS(?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+
JAVAFILE (?:[A-Za-z0-9_.-]+)
JAVASTACKTRACEPART at %{JAVACLASS:class}\.%{WORD:method}\(%{JAVAFILE:file}:%{NUMBER:line}\)
JAVALOGMESSAGE (.*)
# MMM dd, yyyy HH:mm:ss eg:Jan 9, 2014 7:13:13 AM
CATALINA_DATESTAMP %{MONTH}%{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
# yyyy-MM-dd HH:mm:ss,SSS ZZZeg: 2014-01-09 17:32:25,527 -0800
TOMCAT_DATESTAMP20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})%{ISO8601_TIMEZONE}
CATALINALOG%{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
# 2014-01-09 20:03:28,269-0800 | ERROR | com.example.service.ExampleService - something compeletelyunexpected happened...
TOMCATLOG%{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} -%{JAVALOGMESSAGE:logmessage}
通过对比我们可以很简单的先把日志相同时间的合并：
[root@controlleretc]# cat tomcat.conf
input{stdin{}}
filter {
multiline {
pattern =>"(^%{CATALINA_DATESTAMP})"
negate => true
what => "previous"
}
if "_grokparsefailure" in[tags] {
drop { }
}
grok {
match => ["message", "%{CATALINALOG}" ]
}
date {
match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS Z", "MMM dd,yyyy HH:mm:ss a" ]
}
}
output{stdout{codec=>rubydebug}}
##先看测试数据，要比较小一点的：
Jan 05, 2017 10:53:35 AMorg.apache.catalina.startup.Catalina load
INFO: Initialization processedin 728 ms
Jan 05, 2017 10:53:35 AMorg.apache.catalina.core.StandardService startInternal
INFO: Starting serviceCatalina
Jan 05, 2017 10:53:35 AMorg.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine:Apache Tomcat/7.0.73
测试效果：
"@timestamp" =>2017-01-05T03:45:46.749Z,
"@version" => "1",
"host" =>"controller",
"message" => "Jan 05,2017 10:53:35 AM org.apache.catalina.startup.Catalina load\nINFO:Initialization processed in 728 ms",
"tags" => [
[0] "multiline"
]
}
{
"@timestamp" =>2017-01-05T03:45:46.760Z,
"@version" => "1",
"host" =>"controller",
"message" => "Jan 05,2017 10:53:35 AM org.apache.catalina.core.StandardService startInternal\nINFO:Starting service Catalina",
"tags" => [
[0] "multiline"
]
}
{
"@timestamp" => 2017-01-05T03:45:46.780Z,
"@version" => "1",
"host" =>"controller",
"message" => "Jan 05,2017 10:53:35 AM org.apache.catalina.core.StandardEngine startInternal\nINFO:Starting Servlet Engine: Apache Tomcat/7.0.73",
"tags" => [
[0] "multiline"
]
}
3、之前用的都是系统默认的catalina文件管理日志，通过简化的方式我们可以使用log4j的方式。
1、安装log4j:
1、下载与Tomcat相应版本的tomcat-juli.jar和tomcat-juli-adapters.jar，及log4j-1.2.17.jar，放在tomcat/lib目录中附上网址：http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.73/bin/extras/下载时注意你的TOMCAT版本
再将tomcat-juli.jar复制到tomcat/bin目录中，替换掉原来的
2、修改Tomcat的conf/context.xml文件，将改为这步很重要。很多人会忘。
3、创建log4j.properties放在tomcat/lib中
[root@controller lib]# cat log4j.properties
log4j.rootLogger=info,Console,R
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
#log4j.appender.Console.layout.ConversionPattern=%d [%t] %-5p %c- %m%n
log4j.appender.Console.layout.ConversionPattern=%d{yy-MM-ddHH:mm:ss} %5p %c{1}:%L - %m%n
log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
log4j.appender.R.File=${catalina.home}/logs/tomcat.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%d{yyyy.MM.dd HH:mm:ss}%5p %c{1}(%L):? %m%n
log4j.logger.org.apache=info, R
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost]=DEBUG,R
log4j.logger.org.apache.catalina.core=info, R
log4j.logger.org.apache.catalina.session=info,R
4、重启看到log目录下生成tomcat.log文件说明已经安装成功了。
5、log4j当然可以指定生成日志文件的格式：
log4j.appender.R.layout.ConversionPattern={"debug_level":"%p","debug_timestamp":"%d{ISO8601}","debug_thread":"%t","debug_file":"%F","debug_line":"%L","debug_message":"%m"}%n
##生成日志之后直接解析成json即可。
6、当然也有一个比较优秀的插件，也是我们推荐的方式：log4j-jsonevent-layout：
这玩意儿的作用相当于我们在nginx中干的事儿，直接将log4j的日志格式定义成json的，有助于性能提升~
7、安装：
先上传一下几个包，已经从官方打包了几个jar包，确实的话很容易失败和报错：
commons-lang-2.6.jar
jsonevent-layout-1.8-SNAPSHOT.jar
json-smart-1.1.1.jar
7、修改log4j.properties，直接把日志发送到Logstash：
[root@controller lib]# cat log4j.properties
log4j.rootCategory=info, RollingLog###为了方便出日志我们用Info，线上大家可以用WARN
log4j.appender.RollingLog=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingLog.Threshold=TRACE
log4j.appender.RollingLog.File=${catalina.home}/logs/api.log
log4j.appender.RollingLog.DatePattern=.yyyy-MM-dd
log4j.appender.RollingLog.layout=net.logstash.log4j.JSONEventLayoutV1
###备注：重启后我们生成了相关日志在api.log下面，下面我们用json格式可以直接解析他了。
看一下我们需要做的匹配文件：
[root@controller etc]# cattomcat_log4j_layout.conf
input {
file{
codec => json
path => "/usr/local/src/apache-tomcat-7.0.73/logs/api.log"
type => "log4j"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output{
if[type] == "log4j"{
redis {
host => "192.168.0.46"
port => 6379
data_type => "list"
key => "logstash:log4j"
}
}
}
本文出自：http://www.roncoo.com/course/view/3c0710458fe347c2a0b31135bbbcb57b