graylog实现日志监控

graylog

graylog是一个轻量级的日志管理工具,依托elasticsearch作为日志存储中间件,MongoDB作为元数据信息存储中间件.自带WEB-UI界面,LDAP整合各种日志类型.提供了日志收集、日志查询、监控告警等相关功能。提供了graylog sidecar通过sidecar模式可以很方便的收集目标主机、容器的各种日志信息，无缝整合filebeat。搜索语法跟kibana类似,自带简单的监控告警功能提供了webhook email等方式。

部署

官方提供了docker docker-compose 手动部署等模式可供用户自行选择
本次采用docker部署

1. graylog依赖了ES和MongoDB 需要提前将ES 和 MongoDB部署完成并且保持与graylog之间网络互通

2. 生成**password_secret** 和 root_password_sha2

   #password 需要至少64位的随机数即可 如果有多个节点graylog需要保证password_secret是一致的
   pwgen -N 1 -s 96
   # root_password_sha2 作为WEB-UI 的管理员密码 默认用户名admin
   echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
   
   

3. 采用docker部署

   docker run
   -p 9000:9000 -p 12201:12201 -p 1514:1514 -p 5555:5555  -p 5044:5044
   -e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/"
   -e GRAYLOG_IS_MASTER="true"
   -e GRAYLOG_PASSWORD_SECRET="填入上面生成的password_secret"
   -e GRAYLOG_ROOT_PASSWORD_SHA2="填入上面生成的root_password"
   -e GRAYLOG_ELASTICSEARCH_HOSTS="http://username:[email protected]:9200"
   -e GRAYLOG_MONGODB_URI="mongodb://10.60.36.230:27017/graylog"
   -e GRAYLOG_ROOT_TIMEZONE=Asia/Shanghai
   --name=graylog
   -d graylog/graylog:5.0
   

4. 访问graylog WEB-UI localhost:9000 默认端口9000 输入用户名admin和上面设置的root_password密码的原文

   

此上graylog-server部署完成

graylog sidecar

sidecar类似一个agent的模式作为日志的采集者发送数据给graylog-server也是官方比较推荐的一种日志收集方式

1. 选择符合graylog-server版本的 sidecar https://go2docs.graylog.org/5-0/getting_in_log_data/graylog_sidecar.html?tocpath=Getting%20in%20Log%20Data%7CGraylog%20Sidecar%7C_____0

2. 以windows为例 其他安装参考官方文档

3. 通过WEB-UI 生成一个API token

   

4. 根据步骤逐步安装 修改graylog API的URL 和刚刚生成API token 还有实例名称(默认用hostname)

   

5. 如果上述配置错了可以再 C:\\Program Files\\Graylog\\sidecar\\sidecar.yml. 修改其中的配置重启

6. 此时可以在WEB-UI看见sidecar

   

7. 添加INPUT BEAT数据源

   

   

8. 配置sidecar的configuration抓取规则 选择collector收集器为filebeat 下方输入filebeat的抓取配置规则

   

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["localhost:5044"]
path:
  data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
  logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
 - windows
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\Users\admin\Desktop\test_log\info/*.log
  multiline.pattern: '^20'                    #多行匹配规则
  multiline.negate: true                    #将不匹配的规则的行合并在一起
  multiline.match: after                #合并到匹配规则的上一行末尾
  tags: ["demo","info"]

- type: log
  enabled: true
  paths:
    - C:\Users\admin\Desktop\test_log\*-error-*.log
  multiline.pattern: '^20'                    #多行匹配规则
  multiline.negate: true                    #将不匹配的规则的行合并在一起
  multiline.match: after                #合并到匹配规则的上一行末尾
  tags: ["demo","error"]
 

9. 下发filebeat的配置文件 Assign Configuration 并且执行 Process

   

10. 选择sidecar Show messages 查看日志

    

    

配置graylog 日志告警

1. 选择顶部导航栏Alerts 选择 Notification 添加通知渠道

   

2. 选择添加一个web hook的通知渠道

   

3. 选择顶部导航栏Alerts 选择Event Definition 创建一个事件定义

   

4. 根据提示步骤 定义事件基础信息 描述 级别

5. 配置filter 核心需要配置Search Query 规则 右侧可以返回Search Query返回的数据

   

6. 配置执行调度时间等规则

7. 自定义字段如有需要的话

   

8. 选择之前配置的通知渠道 选择message backlog 会最终通知到webhook的backlog字段中 包含了消息原文信息 建议配置 grace period 配置安静期避免告警风暴

   

9. 查看配置概述 没问题选择Update event definition

   

10. 发送ERROR日志查看webhook接口的告警

    返回json报文示例

    {
        "event_definition_id": "643775ac247b2e0934262df5",
        "event_definition_type": "aggregation-v1",
        "event_definition_title": "demo-error-log-alert",
        "event_definition_description": "",
        "job_definition_id": "6437759a247b2e0934262dcd",
        "job_trigger_id": "6437bfc8247b2e093426c914",
        "event": {
            "id": "01GXWWA9QSVY0G89SMD94YB0W2",
            "event_definition_type": "aggregation-v1",
            "event_definition_id": "643775ac247b2e0934262df5",
            "origin_context": "urn:graylog:message:es:graylog_0:a5250950-d9d6-11ed-bcc0-0242ac110002",
            "timestamp": "2023-04-13T08:38:54.824Z",
            "timestamp_processing": "2023-04-13T08:39:36.441Z",
            "timerange_start": null,
            "timerange_end": null,
            "streams": [
                
            ],
            "source_streams": [
                "000000000000000000000001"
            ],
            "message": "demo-error-log-alert",
            "source": "1ac9b7a6d52e",
            "key_tuple": [
                
            ],
            "key": "",
            "priority": 3,
            "alert": true,
            "fields": {
                
            },
            "group_by_fields": {
                
            }
        },
        "backlog": [
            {
                "index": "graylog_0",
                "message": "2023-04-13 16:38:54.824 -- [demo1] -- ERROR 14996 --- [http-nio-4100-exec-5] [] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause\n\njava.lang.ArithmeticException: / by zero\n\tat com.corn.controller.LogController.error(LogController.java:26)\n\tat sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:655)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.lang.Thread.run(Thread.java:748)",
                "fields": {
                    "filebeat_log_offset": 186267,
                    "filebeat_agent_name": "LAPTOP-1BQNQ8EO",
                    "gl2_remote_ip": "172.17.0.1",
                    "gl2_remote_port": 59776,
                    "filebeat_agent_hostname": "LAPTOP-1BQNQ8EO",
                    "beats_type": "filebeat",
                    "gl2_source_input": "64377490247b2e0934262b8e",
                    "filebeat_@metadata_beat": "filebeat",
                    "filebeat_@timestamp": "2023-04-13T08:38:54.824Z",
                    "filebeat_agent_type": "filebeat",
                    "filebeat_@metadata_version": "7.11.1",
                    "filebeat_host_name": "LAPTOP-1BQNQ8EO",
                    "gl2_source_node": "95acd4bd-4f23-40de-a422-5274f220a40a",
                    "filebeat_agent_version": "7.11.1",
                    "filebeat_agent_ephemeral_id": "d5392ef5-44f5-4d83-9007-1c947788c0f2",
                    "gl2_accounted_message_size": 5774,
                    "gl2_source_collector": "41e84e5f-67c5-422f-8fb0-a5205146cd4b",
                    "filebeat_input_type": "log",
                    "gl2_message_id": "01GXWW99K54AZYA4JCYQ0CT0ZG",
                    "filebeat_tags": [
                        "windows",
                        "demo",
                        "error"
                    ],
                    "filebeat_ecs_version": "1.6.0",
                    "filebeat_collector_node_id": "LAPTOP-1BQNQ8EO",
                    "filebeat_@metadata_type": "_doc",
                    "filebeat_log_time": "2023-04-13 16:38:54.824",
                    "filebeat_agent_id": "4b8086ce-67f1-4aec-b93b-723744d56ab3",
                    "filebeat_log_file_path": "C:\\Users\\JimWu\\Desktop\\test_log\\demo1-error-2023-04-13-0.log",
                    "filebeat_log_flags": [
                        "multiline"
                    ]
                },
                "id": "a5250950-d9d6-11ed-bcc0-0242ac110002",
                "source": "LAPTOP-1BQNQ8EO",
                "timestamp": "2023-04-13T08:38:54.824Z",
                "stream_ids": [
                    "000000000000000000000001"
                ]
            }
        ]
    }