较详细的filebeat+es+kibana+sentinl，日志采集+mail、钉钉报警
1.总体架构图
2.日志截图，filebeat需要采集json日志行。中文变量是为了可读性。JSON.toJsonstring(obj,true)进行pretty打印JSON
3.filebeat配置，多使用 ./filebeat -e -c filebeat.yml -d '*' 查看采集状况，filebeat自带一些常用模块如mysql慢查询，错误日志。进入filebeat/modules.d/ 去掉要使用的模块.disabled后缀然后修改配置。 ./filebeat setup -e即可将默认模板写入kibana
filebeat.prospectors:
- type: log
enabled: true
paths:
- /home/work/logs/*Log/all_level/all_level.log
#tags: jwanj
include_lines: ['^ +'] #只采集空格开头的行
#exclude_lines: ['ERROR','INFO','DEBUG'] #排除行
multiline.pattern: "^ +{" #匹配空格{开头的json行
multiline.negate: true
multiline.match: after
scan_frequency: 10s
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#============================== Dashboards =====================================
setup.dashboards.enabled: false
#============================== Kibana =====================================
setup.kibana:
host: "4x.xxx.xx.18:8101"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
pipelines:
- pipeline: logaop #指定es ingest节点的日志清洗管道
username: "elastic"
password: "6esDhSTr8lXXXXXXX"4.es ingest 节点创建pipeline
grok抽取匹配的json串，并json格式化，geoip是es的一个插件 ./elasticsearch-plugin install ingest-geoip,不需要安装 GeoLite2-City.mmdb这个城市数据库。on_failure这个配置非常必要，pipeline处理异常的信息会存入failed-{{ _index }}这个索引，如果没有配置这个会导致filebeat不断报错而不能继续处理后面的日志。
PUT /_ingest/pipeline/logaop
{
"description" : "logaop",
"processors" : [
{
"json":{
"field": "message",
"target_field": "content",
"add_to_root": false
},
"geoip":{
"field":"content.请求IP",
"target_field":"geoip"
},
"remove": {
"field": ["message","beat","prospector","geoip.continent_name","geoip.country_iso_code","geoip.region_name"]
}
}
],
"on_failure": [
{
"set": {
"field": "_index",
"value": "failed-{{ _index }}"
}
}
]
}5.监控大盘，步骤4处理出来的geoip.location并不是geo_point类型的，暂未解决，导致请求热力图无法画出
6.报警。最初打算使用elastAlert的，安装特别坑，就换sentinel了,这个两个报警方案都暂时不支持高版本7.xES，sentinl github地址=>https://github.com/sirensolutions/sentinl/releases/ 。安装方法类似kibana安装xpack插件,推荐在线安装./kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.4/sentinl.zip。注意版本一致，这里所有的组件都是08年的6.2.4
要获取payload的列表信息时需要使用transform，里面的script是js，如下即是将循环命中的文档将_source.content字段转json后存入数组，发送邮件就可以{{payload.newlist}}获取
"script": "payload.newlist=[];payload.hits.hits.forEach(function(bucket){ payload.newlist.push(JSON.stringify(bucket._source.content))})"
QQ邮件发送服务器stmp.qq.com的25端口可能被阿里云服务器封了，可以ping通但Telnet测试端口是不通的，当然也可以解决但比较麻烦，推荐考虑钉钉报警或微信，钉钉需要去pc端钉钉获取一个机器人webhook。配置时body配置参考钉钉文档https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq。
{
"actions": {
"email_html_alarm_2b622930-10e5-11ea-8156-a34196b5558f": {
"name": "email html alarm",
"throttle_period": "1m",
"email_html": {
"to": "[email protected];",
"from": "[email protected]",
"stateless": false,
"subject": "{{watcher.title}},异常数{{payload.hits.total}}",
"priority": "medium",
"html": "Hi {{watcher.username}},
\n{{watcher.title}},异常数{{payload.hits.total}},详情如下(应用---接口---耗时毫秒数)：
{{payload.newlist}}
\nClick 点我 查看监控面板.
\n\n
\n 报警触发条件:
\n - {{watcher.condition.script.script}}
\n"
}
},
"Webhook_93b7f699-8bb4-45c3-96b4-48e1fddf2f19": {
"name": "Webhook1",
"throttle_period": "1m",
"webhook": {
"priority": "high",
"stateless": false,
"method": "POST",
"host": "oapi.dingtalk.com",
"port": "443",
"path": "/robot/send?access_token=XXXXXXX",
"body": "{\"msgtype\": \"markdown\",\"at\": {\"isAtAll\": \"True\"},\"markdown\": {\"title\": \"{{watcher.title}},异常数{{payload.hits.total}}\",\"text\": \"**{{watcher.title}},异常数{{payload.hits.total}}。** \n\n #### 详情如下(应用---接口---耗时毫秒数)：\n > {{payload.newlist}}\n > ##### [查看dashbord](http://47.108.28.18:8101) \n \"}}",
"params": {
"watcher": "{{watcher.title}}",
"payload_count": "{{payload.hits.total}}"
},
"headers": {
"Content-Type": "application/json"
},
"message": "测试报警",
"use_https": true
}
}
},
"input": {
"search": {
"request": {
"index": [
"filebeat-*"
],
"body": {
"size": 100,
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s",
"time_zone": "Asia/Shanghai",
"min_doc_count": 1
},
"aggs": {
"3": {
"max": {
"field": "content.请求耗时"
}
}
}
}
},
"version": true,
"_source": {
"excludes": []
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
"@timestamp"
],
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-10m/m",
"lte": "now/m"
}
}
},
{
"range": {
"content.请求耗时": {
"gte": 1000
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
}
}
}
},
"condition": {
"script": {
"script": "payload.hits.total>0"
}
},
"transform": {
"script": {
"script": "payload.newlist=[];payload.hits.hits.forEach(function(bucket){ payload.newlist.push(bucket._source.content.应用名+'-----'+bucket._source.content.请求URI+'-----'+bucket._source.content.请求耗时)+' ' })"
}
},
"trigger": {
"schedule": {
"later": "every 10 minutes"
}
},
"disable": true,
"report": false,
"title": "【测试环境】10分钟一次，接口耗时超过1s报警",
"save_payload": false,
"spy": true,
"impersonate": false,
"dashboard_link": "http://47.108.28.18:8101/app/kibana#/dashboard/5867fa70-fafd-11e9-8f31-bf285ea1d362?_g=(refreshInterval:('$$hashKey':'object:1714',display:'10%20seconds',pause:!f,section:1,value:10000),time:(from:now-15m,mode:quick,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(spy:!n),gridData:(h:3,i:'1',w:6,x:0,y:0),id:da698230-faf5-11e9-8f31-bf285ea1d362,panelIndex:'1',type:visualization,version:'6.2.4'),(embeddableConfig:(spy:(mode:(name:setalarm))),gridData:(h:3,i:'2',w:6,x:6,y:0),id:'2e3f0960-faf7-11e9-8f31-bf285ea1d362',panelIndex:'2',type:visualization,version:'6.2.4'),(embeddableConfig:(mapCenter:!(32.39851580247402,108.19335937500001),mapZoom:4),gridData:(h:3,i:'3',w:6,x:6,y:3),id:b7ee8390-fb09-11e9-8f31-bf285ea1d362,panelIndex:'3',title:%E4%BC%81%E4%B8%9A%E5%9C%B0%E5%9D%80%E5%88%86%E6%AD%A5%E5%9B%BE,type:visualization,version:'6.2.4'),(gridData:(h:3,i:'4',w:6,x:0,y:6),id:'50115a30-fb0a-11e9-8f31-bf285ea1d362',panelIndex:'4',type:visualization,version:'6.2.4'),(gridData:(h:3,i:'5',w:6,x:0,y:3),id:'1af42580-fb0f-11e9-8f31-bf285ea1d362',panelIndex:'5',type:visualization,version:'6.2.4')),query:(language:lucene,query:''),timeRestore:!f,title:%E6%9C%BA%E4%B8%87%E5%AE%B6%E6%97%A5%E5%BF%97%E5%A4%A7%E7%9B%98,viewMode:edit)"
}kibana.yml增加sentinl配置项，sentinl详细配置参考官网https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq。egrep -v "^$|#" kibana.yml 去#注释及空行查看配置如下,password是QQ邮箱-设置-POP3/SMTP的授权码
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'
server.port: 8101
sentinl:
settings:
email:
active: true
user: [email protected]
password: knozhtzhtodscbca
host: smtp.qq.com
ssl: false
report:
active: true
server.host: "1xx.xx.xx.205"
elasticsearch.url: "http://localhost:9200"
elasticsearch.username: "elastic"
elasticsearch.password: "6esDhSTr8lINSgjaliQf"效果图
