前言

sentinl插件可以实现按某种规则查询数据，当数据达到一定的阈值，可以实现邮件告警

GitHub：https://github.com/sirensolutions/sentinl
download: https://github.com/sirensolutions/sentinl/releases

安装

注意：sentinl.zip的版本和Kibana的版本需要相互对应

方式1：kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-5.5/sentinl-v5.5.3.zip

方式2：./kibana-plugin install file:///sentinl-v5.5.3.zip

配置kibana.yml

在末尾添加

sentinl:
  es:
    host: 192.168.1.1
    port: 9200
    timefield: '@timestamp'
    default_index: watcher
    type: sentinl-watcher
    alarm_index: watcher_alarms
    alarm_type: sentinl-alarm
    script_type: sentinl-script
  sentinl:
    history: 20
    results: 50
    scriptResults: 50
  settings:
    email:
      active: true
      user: "[email protected]"
      password: "xxxxxx"
      host: smtp.mailgun.org
      ssl: true
      timeout: 100000  # mail server connection timeout
    slack:
      active: false
      username: username
      hook: 'https://hooks.slack.com/services/'
      channel: '#channel'
    webhook:
      active: false
      method: POST
      host: host
      port: 9200
      path: ':/{{payload.watcher_id}}'
      body: '{{payload.watcher_id}}{payload.hits.total}}'
    report:
      active: false
      tmp_path: /tmp/
      search_guard: false
      simple_authentication: false
    pushapps:
      active: false
      api_key: ''

重启并观察日志

service kibana stop
service kibana start
tail -f /data/logs/kibana/kibana.log

普通用户授权访问Sentinal：

cat /usr/share/elasticsearch/plugins/search-guard-5/sgconfig/sg_roles.yml
sg_indexinnertest_read:
  indices:
    '?kibana':
      '*':
        - INDICES_ALL
    'watcher*':
      '*':
        - indices:data/read/search
        - UNLIMITED
    'credit_card*':
      '*':
        - indices:data/read/search
        - UNLIMITED

同步权限        
/usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_prod.sh

界面配置告警规则

Sentinal-> new ->+ Watcher

General
Title:
Run every:

input
Title
Body:
Condition
Title
Body:

Transform #不用配置

Title
Body:

Action
email_admin:
Throttle
Priortiy
Title:
TO:
From:
subject:
body:

Raw:

上面配置的综合：

{
  "_index": "watcher",
  "_type": "sentinl-watcher",
  "_id": "balabala-o0q97d9z34-y2n14vaoux",
  "_version": 18,
  "found": true,
  "_source": {
    "title": "test-api-error-alarm",
    "disable": false,
    "report": false,
    "trigger": {
      "schedule": {
        "later": "every 5 mins"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "test-api-debug-app-prod-*"
          ],
          "body": {
            "query": {
              "bool": {
                "must": [
                  {
                    "match": {
                      "level": {
                        "query": "ERROR",
                        "type": "phrase"
                      }
                    }
                  }
                ],
                "filter": [
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-10m",
                        "lt": "now"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 5"   #命中多少条告警
      }
    },
    "actions": {
      "email_app_developer": {
        "throttle_period": "0h15m0s",
        "email": {
          "to": "[email protected]",
          "from": "[email protected]",
          "subject": "ES Alarm test-api",
          "priority": "high",
          "body": "ES Found {{payload.hits.total}} ERROR in logs"
        }
      }
    }
  }
}

问题解决

1、"Failed to install phantomjs: Error: EPERM: operation not permitted
解决办法：
保证插件的目录权限是kibana用户权限：
chown kibana.kibana -R /usr/share/kibana/plugins/sentinl

2、no permissions for indices:data/read/search :: {"path":"/watcher/sentinl-watcher/_count"}
解决办法：
保证kibana访问ES有data/read/search的权限
授予kibana_server 足够的权限
或者换高权限账号
elasticsearch.username: "admin"
elasticsearch.password: "xxxxx"