使用钉钉完成对虚拟机监控告警

本篇文章不讲述如何使用prometheus监控VM,详细来说一下监控VM后续使用钉钉报警的讲解

---

前言：

之前一直使用的是prometheus-webhook-dingtalk0.3.0版本来进行告警的
效果图如下:

微信截图_20191209164428.png

作者关注了很久项目今天迎来了重大更新 Click me迎来了prometheus-webhook-dingtalk1.2.2版本,整体有很大的革新,所以按奈不住躁动的心,前来尝试一番:
效果图如下:

微信截图_20191209165028.png

接下来就来说说是如何实现的

1. 首先下载对应版本的压缩包

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.2.2/prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz
tar xf prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz

2. 添加钉钉机器人的webhook

cd prometheus-webhook-dingtalk-1.2.2.linux-amd64
mv config.example.yml config.yml
vim config.yml
# 我这里使用的是webhook2来进行告警的

## Request timeout
# timeout: 5s

## Customizable templates path
# templates:
#   - contrib/templates/legacy/template.tmpl

## Targets, previously was known as "profiles"
targets:
#  webhook1:
#    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
    # secret for signature
#    secret: SEC000000000000000000000
  webhook2:
    url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook
  webhook_legacy:
    url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook
    # Customize template content
    message:
      # Use legacy template
      title: '{{ template "default.title" . }}'
      text: '{{ template "default.content" . }}'
  webhook_mention_all:
    url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook
    mention:
      all: true
  webhook_mention_users:
    url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook
    mention:
      mobiles: ['15xxxxxxxx0']

3. 配置alertmanager

• 我这里使用的是webhook2来进行告警的
• 详情见

vim /etc/alertmanager/alertmanager.yml

global: #全局设置
  resolve_timeout: 1m #没有报警情况下申明已解决的时间
route:
  group_by: ['alertname'] #分组
  group_wait: 1m #至少等待group_wait时间来初始化
  group_interval: 1m #当第一个报警发送后，等待'group_interval'时间来发送新的一组报警信息
  repeat_interval: 2m #如果一个报警信息已经发送成功了，等待'repeat_interval'时间来重新发送他们
  receiver: 'webhook' #接收器
  routes: #子路由
  - receiver: webhook #接收器
    group_wait: 1m
    match:
      team: node #匹配一个标签
receivers:
- name: 'webhook' #接收器
  webhook_configs:
  - url: http://localhost:8060/dingtalk/webhook2/send #钉钉的发送方式,如果你是webhook1方式启动的 这里要换一下 
    send_resolved: true #解决告警之后回复信息

#重启alertmanager
systemctl restart alertmanager

3. 启动prometheus-webhook-dingtalk

• --web.ui-enabled打开webUI界面
  • http://localhost:8060/ui/
• --config.file指定配置文件启动
• 详情见

nohup ./prometheus-webhook-dingtalk --web.ui-enabled --config.file=config.yml &

接下来就可以看到全新的报警方式了,看起来比之前的0.3.0版本清爽多了

微信截图_20191209170426.png

说明： 上图中 []不是异常也不是模板问题，是prometheus没取到值,然后传给alertmanager的时候那里是空数组,所以就这样显示了。

下面是我用的一些报警规则

groups:
  - name: host_monitoring
    rules:
    - alert: 内存报警
      expr: (1 - (node_memory_MemAvailable_bytes / (node_memory_MemTotal_bytes)))* 100 > 80
      for: 2m
      labels:
        team: node
      annotations:
        Alert_type: 内存报警
        Server: '{{$labels.instance}}'
        description: "内存使用量超过80%，当前使用量为：{{ $value }}%"
    - alert: CPU报警
      expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
      for: 2m
      labels:
        team: node
      annotations:
        Alert_type: CPU报警
        Server: '{{$labels.instance}}'
        description: "CPU使用量超过80%，当前使用量为：{{ $value }}%"
    - alert: 磁盘报警
      expr: max by (instance, namespace, pod, device) ((node_filesystem_size_bytes{fstype=~"ext[234]|btrfs|xfs|zfs"} - node_filesystem_avail_bytes{fstype=~"ext[234]|btrfs|xfs|zfs"}) / node_filesystem_size_bytes{fstype=~"ext[234]|btrfs|xfs|zfs"}) * 100 >80
      for: 2m
      labels:
        team: node
      annotations:
        Alert_type: 磁盘报警
        Server: '{{$labels.instance}}'
        description: "磁盘使用量超过80%，当前使用量为：{{ $value }}%"
    - alert: 服务报警
      expr: up == 0
      for: 2m
      labels:
        team: node
      annotations:
        Alert_type: 服务报警
        Server: '{{$labels.instance}}'
        description: "{{$labels.instance}}服务已关闭"
    - alert: 硬件温度报警
      expr: node_hwmon_temp_celsius >= 63
      for: 2m
      labels:
        team: node
      annotations:
        Alert_type: 硬件温度报警
        Server: '{{$labels.instance}}'
        description: "硬件温度超过55°，当前温度为：{{ $value }}"
  - name: blackbox_network_stats
    rules:
    - alert: tcp端口检测/icmp主机检测
      expr: probe_success == 0
      for: 10s
      labels:
        severity: critical
      annotations:
        summary: "Instance {{ $labels.instance }}  is down"
        description: "This requires immediate action!"

总结:prometheus-webhook-dingtalk1.2.2比prometheus-webhook-dingtalk0.3.0告警信息看的更加清爽了,另外还迎来一个webUI

优化

#!/bin/sh
wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.2.2/prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz
if [ $? -ne 0 ]; then
    while true
    do
        wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.2.2/prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz
        if [ $? -eq 0 ]; then
            break
        fi
    done
else
    tar xf prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz
    cp -R prometheus-webhook-dingtalk-1.2.2.linux-amd64/prometheus-webhook-dingtalk /usr/local/bin/
fi

#echo "---验证版本---"
#node_exporter --version
echo "---剪切并命名---"
mkdir /etc/prometheus-webhook-dingtalk/ -p
mv prometheus-webhook-dingtalk-1.2.2.linux-amd64/config.example.yml /etc/prometheus-webhook-dingtalk/config.yml
#创建工作目录
mkdir -p /var/lib/prometheus-webhook-dingtalk
#创建并编写配置启动项配置文件
cat > /usr/lib/systemd/system/prometheus-webhook-dingtalk.service <