graylog实现日志监控
graylog
graylog是一个轻量级的日志管理工具,依托elasticsearch作为日志存储中间件,MongoDB作为元数据信息存储中间件.自带WEB-UI界面,LDAP整合各种日志类型.提供了日志收集、日志查询、监控告警等相关功能。提供了graylog sidecar通过sidecar模式可以很方便的收集目标主机、容器的各种日志信息，无缝整合filebeat。搜索语法跟kibana类似,自带简单的监控告警功能提供了webhook email等方式。
部署
官方提供了docker docker-compose 手动部署等模式可供用户自行选择
本次采用docker部署
-
graylog依赖了ES和MongoDB 需要提前将ES 和 MongoDB部署完成并且保持与graylog之间网络互通
-
生成**
password_secret** 和root_password_sha2#password 需要至少64位的随机数即可 如果有多个节点graylog需要保证password_secret是一致的 pwgen -N 1 -s 96 # root_password_sha2 作为WEB-UI 的管理员密码 默认用户名admin echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1 -
采用docker部署
docker run -p 9000:9000 -p 12201:12201 -p 1514:1514 -p 5555:5555 -p 5044:5044 -e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/" -e GRAYLOG_IS_MASTER="true" -e GRAYLOG_PASSWORD_SECRET="填入上面生成的password_secret" -e GRAYLOG_ROOT_PASSWORD_SHA2="填入上面生成的root_password" -e GRAYLOG_ELASTICSEARCH_HOSTS="http://username:[email protected]:9200" -e GRAYLOG_MONGODB_URI="mongodb://10.60.36.230:27017/graylog" -e GRAYLOG_ROOT_TIMEZONE=Asia/Shanghai --name=graylog -d graylog/graylog:5.0 -
访问graylog WEB-UI localhost:9000 默认端口9000 输入用户名admin和上面设置的root_password密码的原文
此上graylog-server部署完成
graylog sidecar
sidecar类似一个agent的模式作为日志的采集者发送数据给graylog-server也是官方比较推荐的一种日志收集方式
-
选择符合graylog-server版本的 sidecar https://go2docs.graylog.org/5-0/getting_in_log_data/graylog_sidecar.html?tocpath=Getting%20in%20Log%20Data%7CGraylog%20Sidecar%7C_____0
-
以windows为例 其他安装参考官方文档
-
通过WEB-UI 生成一个API token
-
根据步骤逐步安装 修改graylog API的URL 和刚刚生成API token 还有实例名称(默认用hostname)
-
如果上述配置错了可以再
C:\\Program Files\\Graylog\\sidecar\\sidecar.yml. 修改其中的配置重启 -
此时可以在WEB-UI看见sidecar
-
添加INPUT BEAT数据源
-
配置sidecar的configuration抓取规则 选择collector收集器为filebeat 下方输入filebeat的抓取配置规则
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["localhost:5044"]
path:
data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
- windows
filebeat.inputs:
- type: log
enabled: true
paths:
- C:\Users\admin\Desktop\test_log\info/*.log
multiline.pattern: '^20' #多行匹配规则
multiline.negate: true #将不匹配的规则的行合并在一起
multiline.match: after #合并到匹配规则的上一行末尾
tags: ["demo","info"]
- type: log
enabled: true
paths:
- C:\Users\admin\Desktop\test_log\*-error-*.log
multiline.pattern: '^20' #多行匹配规则
multiline.negate: true #将不匹配的规则的行合并在一起
multiline.match: after #合并到匹配规则的上一行末尾
tags: ["demo","error"]
-
下发filebeat的配置文件 Assign Configuration 并且执行 Process
-
选择sidecar Show messages 查看日志
配置graylog 日志告警
-
选择顶部导航栏Alerts 选择 Notification 添加通知渠道
-
选择添加一个web hook的通知渠道
-
选择顶部导航栏Alerts 选择Event Definition 创建一个事件定义
-
根据提示步骤 定义事件基础信息 描述 级别
-
配置filter 核心需要配置Search Query 规则 右侧可以返回Search Query返回的数据
-
配置执行调度时间等规则
-
自定义字段如有需要的话
-
选择之前配置的通知渠道 选择message backlog 会最终通知到webhook的backlog字段中 包含了消息原文信息 建议配置 grace period 配置安静期避免告警风暴
-
查看配置概述 没问题选择Update event definition
-
发送ERROR日志查看webhook接口的告警
返回json报文示例
{ "event_definition_id": "643775ac247b2e0934262df5", "event_definition_type": "aggregation-v1", "event_definition_title": "demo-error-log-alert", "event_definition_description": "", "job_definition_id": "6437759a247b2e0934262dcd", "job_trigger_id": "6437bfc8247b2e093426c914", "event": { "id": "01GXWWA9QSVY0G89SMD94YB0W2", "event_definition_type": "aggregation-v1", "event_definition_id": "643775ac247b2e0934262df5", "origin_context": "urn:graylog:message:es:graylog_0:a5250950-d9d6-11ed-bcc0-0242ac110002", "timestamp": "2023-04-13T08:38:54.824Z", "timestamp_processing": "2023-04-13T08:39:36.441Z", "timerange_start": null, "timerange_end": null, "streams": [ ], "source_streams": [ "000000000000000000000001" ], "message": "demo-error-log-alert", "source": "1ac9b7a6d52e", "key_tuple": [ ], "key": "", "priority": 3, "alert": true, "fields": { }, "group_by_fields": { } }, "backlog": [ { "index": "graylog_0", "message": "2023-04-13 16:38:54.824 -- [demo1] -- ERROR 14996 --- [http-nio-4100-exec-5] [] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause\n\njava.lang.ArithmeticException: / by zero\n\tat com.corn.controller.LogController.error(LogController.java:26)\n\tat sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:655)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.lang.Thread.run(Thread.java:748)", "fields": { "filebeat_log_offset": 186267, "filebeat_agent_name": "LAPTOP-1BQNQ8EO", "gl2_remote_ip": "172.17.0.1", "gl2_remote_port": 59776, "filebeat_agent_hostname": "LAPTOP-1BQNQ8EO", "beats_type": "filebeat", "gl2_source_input": "64377490247b2e0934262b8e", "filebeat_@metadata_beat": "filebeat", "filebeat_@timestamp": "2023-04-13T08:38:54.824Z", "filebeat_agent_type": "filebeat", "filebeat_@metadata_version": "7.11.1", "filebeat_host_name": "LAPTOP-1BQNQ8EO", "gl2_source_node": "95acd4bd-4f23-40de-a422-5274f220a40a", "filebeat_agent_version": "7.11.1", "filebeat_agent_ephemeral_id": "d5392ef5-44f5-4d83-9007-1c947788c0f2", "gl2_accounted_message_size": 5774, "gl2_source_collector": "41e84e5f-67c5-422f-8fb0-a5205146cd4b", "filebeat_input_type": "log", "gl2_message_id": "01GXWW99K54AZYA4JCYQ0CT0ZG", "filebeat_tags": [ "windows", "demo", "error" ], "filebeat_ecs_version": "1.6.0", "filebeat_collector_node_id": "LAPTOP-1BQNQ8EO", "filebeat_@metadata_type": "_doc", "filebeat_log_time": "2023-04-13 16:38:54.824", "filebeat_agent_id": "4b8086ce-67f1-4aec-b93b-723744d56ab3", "filebeat_log_file_path": "C:\\Users\\JimWu\\Desktop\\test_log\\demo1-error-2023-04-13-0.log", "filebeat_log_flags": [ "multiline" ] }, "id": "a5250950-d9d6-11ed-bcc0-0242ac110002", "source": "LAPTOP-1BQNQ8EO", "timestamp": "2023-04-13T08:38:54.824Z", "stream_ids": [ "000000000000000000000001" ] } ] }