ruby正则表达

背景

最近在使用fluent-bit做日志解析，有部分日志解析不了，专门研究下怎么解析这种日志。做个笔记，以后再使用时直接拿出来使用。

日志格式

2019-02-06 22:32:30 [INFO ][:0] -  start_time:1549209600, end_time: 1549468799
2019-02-06 22:32:30 [INFO ][:0] -  start_time:1549209600, end_time: 1549468799
2019-02-06 22:32:30 [INFO ][:0] - [draw_machine_global] DoGmActivity self._acUniqueId:36, updateCfg->uniqueId:36, startTime:1549209600, endTime:1549814100.
2019-02-06 22:32:30 [INFO ][:0] - [SetQueueLoginGlobal]  DoGmActivity maxCnt:0 warnCnt:0 allowCntMinute:0
2019-02-06 22:32:30 [INFO ][:0] - [QueueLoginMgr] OnSetQueueNum  maxCut:2000 warnCnt:1700 allowCntMinute:6
2019-02-06 22:32:30 [ERROR][:0] - server state change, new state:0, effect time:1548840180, delay:1, interval:1, kick:False, tip:
2019-02-06 22:32:30 [INFO ][:0] -  start_time:1549209720.0, end_time: 1549468740.0
2019-02-06 22:32:30 [INFO ][:0] -  start_time:1549209720.0, end_time: 1549468740.0
2019-02-06 22:32:31 [ERROR][:0] - server state on effect timer, cnt:1
2019-02-06 22:32:31 [INFO ][:0] - ****** dirty_queue.py done, interval:60 seconds ******

通过ruby的正则编辑工具调试

https://rubular.com/r/C1vfTpxMIYS59T
fluent-bit parse配置如下：

[PARSER]
    Name    mwptest
    Format  regex
    Regex    ^(?[^\]]*) \[(?[^\]]*)\]\[.*\] - (?.*)$
    Time_Key    time
    #Time_Format %Y-%m-%d:%H:%M:%S,%L
    #Time_Format %Y-%m-%dT%H:%M:%S,%L
    Time_Format %Y-%m-%d %H:%M:%S

fluent-bit.conf 部分配置

[INPUT]
    Name tail
    Tag  cmge.autotools
    Path /data/log/test    #这里不能为目录，只能匹配文件，如：*.log
    Parser mwptest
[OUTPUT]
    Name  file
    Match *
    Path output.txt

解析出来的结果如下

cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209600, end_time: 1549468799"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209600, end_time: 1549468799"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[draw_machine_global] DoGmActivity self._acUniqueId:36, updateCfg->uniqueId:36, startTime:1549209600, endTime:1549814100."}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[SetQueueLoginGlobal]  DoGmActivity maxCnt:0 warnCnt:0 allowCntMinute:0"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[QueueLoginMgr] OnSetQueueNum  maxCut:2000 warnCnt:1700 allowCntMinute:6"}]
cmge.autotools: [1549492350.000000, {"time1":"ERROR", "message":"server state change, new state:0, effect time:1548840180, delay:1, interval:1, kick:False, tip:"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209720.0, end_time: 1549468740.0"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":" start_time:1549209720.0, end_time: 1549468740.0"}]
cmge.autotools: [1549492351.000000, {"time1":"ERROR", "message":"server state on effect timer, cnt:1"}]
cmge.autotools: [1549492351.000000, {"time1":"INFO ", "message":"****** dirty_queue.py done, interval:60 seconds ******"}]

总结下

这里最主要是要学会利用ruby的正则编辑工具帮助做解析，可以减少很多自己在环境中调试时间。