背景
最近在使用fluent-bit做日志解析,有部分日志解析不了,专门研究下怎么解析这种日志。做个笔记,以后再使用时直接拿出来使用。
日志格式
2019-02-06 22:32:30 [INFO ][:0] - start_time:1549209600, end_time: 1549468799
2019-02-06 22:32:30 [INFO ][:0] - start_time:1549209600, end_time: 1549468799
2019-02-06 22:32:30 [INFO ][:0] - [draw_machine_global] DoGmActivity self._acUniqueId:36, updateCfg->uniqueId:36, startTime:1549209600, endTime:1549814100.
2019-02-06 22:32:30 [INFO ][:0] - [SetQueueLoginGlobal] DoGmActivity maxCnt:0 warnCnt:0 allowCntMinute:0
2019-02-06 22:32:30 [INFO ][:0] - [QueueLoginMgr] OnSetQueueNum maxCut:2000 warnCnt:1700 allowCntMinute:6
2019-02-06 22:32:30 [ERROR][:0] - server state change, new state:0, effect time:1548840180, delay:1, interval:1, kick:False, tip:
2019-02-06 22:32:30 [INFO ][:0] - start_time:1549209720.0, end_time: 1549468740.0
2019-02-06 22:32:30 [INFO ][:0] - start_time:1549209720.0, end_time: 1549468740.0
2019-02-06 22:32:31 [ERROR][:0] - server state on effect timer, cnt:1
2019-02-06 22:32:31 [INFO ][:0] - ****** dirty_queue.py done, interval:60 seconds ******
通过ruby的正则编辑工具调试
https://rubular.com/r/C1vfTpxMIYS59T
fluent-bit parse配置如下:
[PARSER]
Name mwptest
Format regex
Regex ^(?
fluent-bit.conf 部分配置
[INPUT]
Name tail
Tag cmge.autotools
Path /data/log/test #这里不能为目录,只能匹配文件,如:*.log
Parser mwptest
[OUTPUT]
Name file
Match *
Path output.txt
解析出来的结果如下
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209600, end_time: 1549468799"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209600, end_time: 1549468799"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[draw_machine_global] DoGmActivity self._acUniqueId:36, updateCfg->uniqueId:36, startTime:1549209600, endTime:1549814100."}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[SetQueueLoginGlobal] DoGmActivity maxCnt:0 warnCnt:0 allowCntMinute:0"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[QueueLoginMgr] OnSetQueueNum maxCut:2000 warnCnt:1700 allowCntMinute:6"}]
cmge.autotools: [1549492350.000000, {"time1":"ERROR", "message":"server state change, new state:0, effect time:1548840180, delay:1, interval:1, kick:False, tip:"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209720.0, end_time: 1549468740.0"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209720.0, end_time: 1549468740.0"}]
cmge.autotools: [1549492351.000000, {"time1":"ERROR", "message":"server state on effect timer, cnt:1"}]
cmge.autotools: [1549492351.000000, {"time1":"INFO ", "message":"****** dirty_queue.py done, interval:60 seconds ******"}]
总结下
这里最主要是要学会利用ruby的正则编辑工具帮助做解析,可以减少很多自己在环境中调试时间。