ELK - Watcher发送告警邮件和调用接口
相比使用ElastAlert发送告警邮件，ELK提供的Wathcer要简单得多，也可以在发生警报的时候调用Web Service接口。
Configure SMTP
https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html
以上文档提供了多种Email系统的配置方法(elasticsearch.yml)，包括Gmail, Outlook, Microsoft Exchange, Amazon SES。
比如Gmail：
xpack.notification.email.account:
gmail_account:
profile: gmail
smtp:
auth: true
starttls.enable: true
host: smtp.gmail.com
port: 587
user: >
当然还要在elasticsearch-keystore里配置相应的password
bin/elasticsearch-keystore add xpack.notification.email.account.gmail_account.smtp.secure_password
实际上公司一般有内部SMTP，只需授权，而无需用户名和密码。
xpack.notification.email.account:
work:
profile: standard
email_defaults:
from: [email protected]
smtp:
auth: false
starttls.enable: false
host: my.dummy.smtp.host
port: 25
Create Watcher
Create Threshold Alert
Management -> Elasticsearch -> Watcher -> Create threshold alert.
填入Name，Indices，Time field， 则会出现Add action按钮。
Watcher supports the following types of actions: email, webhook, index, logging, slack, and pagerduty.
Threshold Alert的主要作用，是它提供了界面，可以简单测试下配置有没有起效果。
比如Email， 填入邮件地址和内容，点击Send test email。
如果SMTP配置没问题的话，应该可以成功收到邮件。
调用接口则选Webhook，一样可以直接Send request进行测试。
注意到7.3还不支持HTTPS，7.6以后才有此选项。
Advance Watch则没有这个问题。
Create Advance Watch
以下是缺省的模板，30分钟执行一次，查询所有indices，因而一般都能执行。
把时间调小，很快就可以在elasticsearch.log里看到输出的text。
{
"trigger": {
"schedule": {
"interval": "30m"
}
},
"input": {
"search": {
"request": {
"body": {
"size": 0,
"query": {
"match_all": {}
}
},
"indices": [
"*"
]
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 10
}
}
},
"actions": {
"my-logging-action": {
"logging": {
"text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
}
}
}
}
Email Action
发送告警邮件的配置一般长这样。
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"[email protected]"
],
"subject": "ELK Alert - XXX is Down",
"body": {
"html": "{{ctx.payload.hits.total}} XXX is over limit, please take action.Note: Automatic email from ELK, please do not reply."
}
}
}
}
Webhook Action
告警时调用接口配置一般长这样。
可以同时支持多个Action。
"actions": {
"my-logging-action": {
"logging": {
"level": "info",
"text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
}
},
"my-webhook-action": {
"webhook": {
"scheme": "https",
"host": "my.api.dummy.host",
"port": 8443,
"method": "put",
"path": "api/alert",
"params": {},
"headers": {
"Content-type": "application/json"
},
"body": "{status: 1}"
}
}
}