ELK实战-Logstash multiline:识别错误堆栈

概述

在通过ELK收集日志的时候,通常需要对日志进行分析,例如实时监控错误堆栈,并进行告警。

通常错误堆栈都是多行的,但通常ELK默认都是识别单行的,怎么才能多行呢?

logstash的codec、filter中均有multiline插件,可以匹配单行内容,并于上下行作为1个输入。

本文主要讲述如何使用logstash的multiline插件来识别错误堆栈。

测试环境

1个CentOS7系统:
* ELK服务器

测试思路

  • logstash监控日志文件
  • logstash配置识别multiline
  • 手动向日志文件中写入python的错误堆栈

实战

logstash配置文件

logstash的配置文件(logstash.conf.stack)如下

input {
file {
    path => "[your_path]/python_stack.log"
    start_position => "beginning"
}
file {
    path => "/var/log/messages"
}

}

filter {
  multiline {
    pattern => ".*TRACE.*"
    what => "previous"
  }
}

output {
    elasticsearch { hosts => ["localhost:9200"] }
    stdout {}
}

配置文件说明:

  • 输入

    • 监控[your_path]/python_stack.log日志文件,
    • 监控/var/log/messages日志文件
  • 过滤

    • 识别每行中.*TRACE.*的内容,如果正则匹配,那么该行就与前一行是同一条日志,且合并内容后作为1条日志处理。
  • 输出:

    • 输出到elasticsearch
    • 输出到logstash的标准输出中

启动ELK

  • 启动elastaticsearch。参考启动elastaticsearch
  • 启动logstash:bin/logstash -f [your_path]/logstash.conf.stack。参考启动logstash
  • 启动kibana。参考启动kibana

向日志文件写入堆栈

向日志文件“(与logstash.conf.stack中的日志路径相同)写入python的堆栈,例如下面所示

2016-04-09 00:00:05.712 40113 DEBUG [your_model]._drivers.amqp [req-958e5439-7657-42ac-973e-616ab154f471 ] UNIQUE_ID is d8fa7221771f439bb7975dc7740bab29. _add_unique_id /usr/lib/py
2016-04-09 00:00:05.713 40113 DEBUG [your_model]._drivers.amqp [req-958e5439-7657-42ac-973e-616ab154f471 ] UNIQUE_ID is 63223f5d931b499b999fd5114495e6f4. _add_unique_id /usr/lib/py
2016-04-09 00:00:05.717 40113 DEBUG [your_model]._drivers.amqp [-] unpacked context: {u'read_deleted': u'no', u'project_name': None, u'user_id': None, u'roles': [u'admin'], u'tenan 2016-04-09 00:00:05.720 40113 ERROR [your.package.path] [req-958e5439-7657-42ac-973e-616ab154f471 ] Exception during message handling: No row was found for one() 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] Traceback (most recent call last): 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib/python2.7/site-packages/[your_package_path].py", line 142, in _dispatch_and_reply 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] executor_callback)) 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib/python2.7/site-packages/[your_package_path].py", line 186, in _dispatch 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] executor_callback) 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib/python2.7/site-packages/[your_package_path].py", line 130, in _do_dispatch 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] result = func(ctxt, **new_args) 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 62, in sync_routers 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] return self._sync_routers_inner(context, **kwargs) 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 77, in _sync_routers_inner 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] router_ids, sfrouter_versions) 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 344, in _filter_sync_routers 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] return self.l3plugin._filter_sync_routers(context, router_ids, sfrouter_versions) 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 227, in _filter_sync_routers 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] filter_by(router_id=id).one()) 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2401, in one 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] raise orm_exc.NoResultFound("No row was found for one()") 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] NoResultFound: No row was found for one() 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] 2016-04-09 00:00:05.720 40113 TRACE [your.package.path] END 2016-04-09 00:00:05.720 40113 INFO [your.package.path] TEST logstash multiline

验证

  • 在向日志文件写入堆栈后,便可以看到在logstash的标准输出中便可以看到”多行的堆栈信息”.
  • 在kibana中也可以看到对应的堆栈显示为1条记录。

至此,ELK作为识别multiline的实战完成了。

你可能感兴趣的:(logstash,ELK)