Prometheus 监控之 Blackbox_exporter黑盒监测 [icmp、tcp、http(get\post)、dns、ssl证书过期时间]

Blackbox_exporter 主动监测主机与服务状态

Prometheus 官方提供的 exporter 之一，可以提供 http、dns、tcp、icmp 的监控数据采集
官方github: https://github.com/prometheus/blackbox_exporter

部署Blackbox_exporter

[sss@prometheus01 ]$ cd /usr/local/blackbox_exporter/
[sss@prometheus01 ]$ wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.12.0/blackbox_exporter-0.12.0.linux-amd64.tar.gz 
[sss@prometheus01 ]$ tar zxvf blackbox_exporter-0.12.0.linux-amd64.tar.gz
[sss@prometheus01 blackbox_exporter-0.12.0.linux-amd64]$ cd blackbox_exporter-0.12.0.linux-amd64
[sss@prometheus01 blackbox_exporter-0.12.0.linux-amd64]$ ll
total 15720
-rwxr-xr-x. 1 1000 1000 16074005 Feb 27  2018 blackbox_exporter
-rw-rw-r--. 1 1000 1000      932 Nov 21 16:05 blackbox.yml
-rw-rw-r--. 1 1000 1000    11357 Feb 27  2018 LICENSE
-rw-rw-r--. 1 1000 1000       94 Feb 27  2018 NOTICE
[sss@prometheus01 blackbox_exporter-0.12.0.linux-amd64]$cp -r blackbox_exporter /usr/local/bin
[sss@prometheus01 blackbox_exporter-0.12.0.linux-amd64]$ cat /etc/supervisord.conf|grep blackbox -A 20
[program:blackbox_exporter]
command=/usr/local/bin/blackbox_exporter   --config.file=/usr/local/prometheus/blackbox_exporter/blackbox_exporter-0.12.0.linux-amd64/blackbox.yml 
stdout_logfile=/tmp/prometheus/blackbox_exporter.log
autostart=true
autorestart=true
startsecs=5
priority=1
user=root
stopasgroup=true
killasgroup=true
[sss@prometheus01 blackbox_exporter-0.12.0.linux-amd64]$ supervisorctl  status |grep blackbox
blackbox_exporter                RUNNING   pid 25343, uptime 0:19:25

blackbox.yml 文件

• 通过 blackbox.yml 定义模块详细信息
• 在 Prometheus 配置文件中引用该模块以及配置被监控目标主机

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      preferred_ip_protocol: "ip4" ##如果http监测是使用ipv4 就要写上，目前国内使用ipv6很少。
  http_post_2xx_query: ##用于post请求使用的模块）由于每个接口传参不同 可以定义多个module 用于不同接口（例如此命名为http_post_2xx_query 用于监测query.action接口 
    prober: http
    timeout: 15s
    http:
      preferred_ip_protocol: "ip4" ##使用ipv4
      method: POST
      headers:
        Content-Type: application/json ##header头
      body: '{"hmac":"","params":{"publicFundsKeyWords":"xxx"}}' ##传参
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
    timeout: 5s
    icmp:

Blackbox_exporter 应用场景

• HTTP 测试
  定义 Request Header 信息
  判断 Http status / Http Respones Header / Http Body 内容
• TCP 测试
  业务组件端口状态监听
  应用层协议定义与监听
• ICMP 测试
  主机探活机制
• POST 测试
  接口联通性
• SSL 证书过期时间

HTTP 测试

• 相关代码块添加到 Prometheus 文件内
• 对应 blackbox.yml文件的 http_2xx 模块

- job_name: 'blackbox_http_2xx'
  scrape_interval: 45s
  metrics_path: /probe
  params:
    module: [http_2xx]  # Look for a HTTP 200 response.
  static_configs:
      - targets:
        - https://www.baidu.com/
        - 172.0.0.1:9090
  relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.XXX.XX.XX:9115  # The blackbox exporter's real hostname:port.

http截图

TCP 测试

• 监听 业务端口地址，用来判断服务是否在线，我觉的和telnet 差不多
• 相关代码块添加到 Prometheus 文件内
• 对应 blackbox.yml文件的 tcp_connect 模块

- job_name: "blackbox_telnet_port]"
  scrape_interval: 5s
  metrics_path: /probe
  params:
    module: [tcp_connect]
  static_configs:
      - targets: [ '1x3.x1.xx.xx4:443' ]
        labels:
          group: 'xxxidc机房ip监控'
      - targets: ['10.xx.xx.xxx:443']
        labels:
          group: 'Process status of nginx(main) server'
  relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.xxx.xx.xx:9115

tcp_connect截图

ICMP 测试

• 相关代码块添加到 Prometheus 配置文件内
• 对应 blackbox.yml文件的 icmp 模块

- job_name: 'blackbox00_ping_idc_ip'
  scrape_interval: 10s
  metrics_path: /probe
  params:
    module: [icmp]  #ping
  static_configs:
      - targets: [ '1x.xx.xx.xx' ]
        labels:
          group: 'xxnginx 虚拟IP'
  relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        regex: (.*)
        target_label: ping
        replacement: ${1}
      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: 1x.xxx.xx.xx:9115

icmp截图

POST 测试

• 监听业务接口地址，用来判断接口是否在线
• 相关代码块添加到 Prometheus 文件内
• 对应 blackbox.yml文件的 http_post_2xx_query 模块（监听query.action这个接口）

- job_name: 'blackbox_http_2xx_post'
  scrape_interval: 10s
  metrics_path: /probe
  params:
    module: [http_post_2xx_query]
  static_configs:
      - targets:
        - https://xx.xxx.com/api/xx/xx/fund/query.action
        labels:
          group: 'Interface monitoring'
  relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 1x.xx.xx.xx:9115  # The blackbox exporter's real hostname:port.

POST截图

查看监听过程

类似于

curl http://172.16.10.65:9115/probe?target=prometheus.io&module=http_2xx&debug=true

告警应用测试

icmp、tcp、http、post 监测是否正常可以观察probe_success 这一指标
probe_success == 0 ##联通性异常
probe_success == 1 ##联通性正常
告警也是判断这个指标是否等于0，如等于0 则触发异常报警

[sss@prometheus01 prometheus]$ cat rules/blackbox-alert.rules 
groups:
- name: blackbox_network_stats
  rules:
  - alert: blackbox_network_stats
    expr: probe_success == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }}  is down"
      description: "This requires immediate action!"

参考：https://www.tidb.cc/Monitor/170603-Blackbox_exporter.html#告警测试案例

SSL 证书过期时间监测

cat << 'EOF' > prometheus.yml
rule_files:
  - ssl_expiry.rules
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - example.com  # Target to probe
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter.
        EOF 
cat << 'EOF' > ssl_expiry.rules 
groups: 
  - name: ssl_expiry.rules 
    rules: 
      - alert: SSLCertExpiringSoon 
        expr: probe_ssl_earliest_cert_expiry{job="blackbox"} - time() < 86400 * 30 
        for: 10m
EOF