prometheus 黑盒监控

黑盒监控

“白盒监控” 是需要把对应的Exporter程序安装到被监控的目标主机上，从而实现对主机各种资源以及状态的数据采集工作 ”黑盒监控“ 是不需要把Exporter程序部署到被监控的目标主机上，比如全球的网络质量的稳定性，通常用ping操作，对选取的节点进行icmp测试 Prometheus社区提供黑盒解决方案，用户只需要将其安装在与Prometheus和被监控目标互通的环境中，通过HTTP、HTTPS、DNS、TCP、ICMP,证书的监控等 方式对网络进行探测监控。

blackbox_exporter

Prometheus 官方提供的exporter，可以提供http dns tcp icmp的监控数据采集

创建配置文件

mkdir /data/blackbox_exporter -p
​
cat >/data/blackbox_exporter/config.yml <<'EOF'
modules:
  http_2xx:
    prober: http
    http:
      method: GET
  http_port_2xx:
    prober: http
    http:  
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  grpc:
    prober: grpc
    grpc:
      tls: true
      preferred_ip_protocol: "ip4"
  grpc_plain:
    prober: grpc
      tls: false
      service: "service1"
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
      - send: "SSH-2.0-blackbox-ssh-check"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
  icmp_ttl5:
    prober: icmp
    timeout: 5s
    icmp:
      ttl: 5
EOF

修改后

root@prometheus220:/data/blackbox_exporter# cat /data/blackbox_exporter/config.yml 
modules:
  http_2xx:
    prober: http
    http:
      method: GET
  http_port_2xx:
    prober: http
    http:  
      method: POST
  tcp_connect:
    prober: tcp
  icmp:
    prober: icmp

docker 直接运行

docker run -d --restart=always --name blackbox-exporter -p 9115:9115 \
-v /data/blackbox_exporter:/etc/blackbox_exporter prom/blackbox-exporter:v0.16.0 \
--config.file=/ect/blackbox_exporter/config.yml

docker-compose 方式

cd /data/blackbox_exporter/
cat >docker-compose.yml << 'EOF'
version: '3.3'
services:
  blackbox_exporter:
    image: prom/blackbox-exporter:v0.16.0
    container_name: blackbox_exporter
    restart: always
    volumes:
    - /data/blackbox_exporter:/etc/blackbox_exporter
    ports:
    - 9115:9115
EOF

启动

docker-compose up -d

确认启动

docker ps -a

访问测试

http://10.19.1.220:9115/

添加Prometheus的配置

cd /opt/data/docker-prometheus/prometheus
cat >> prometheus/prometheus.yml<<'EOF'
#http 配置
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params: 
      module: [http_2xx]
    static_configs:
    - targets: 
      - https://www.baidu.com
      - https://www.163.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.19.1.220:9115
​
#tcp 配置
  - job_name: 'blackbox_tcp'
    metrics_path: /probe
    params: 
      module: [tcp_connect]
    static_configs:
    - targets: 
      - 10.19.1.220:22
      - 10.19.1.206:9090
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.19.1.220:9115
​
#icmp 配置 ping
  - job_name: 'blackbox_icmp'
    metrics_path: /probe
    params: 
      module: [icmp]
    static_configs:
    - targets: 
      - 10.19.1.206
      - 10.19.1.220
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.19.1.220:9115
EOF

热启动

curl -X POST http://localhost:9090/-/reload

Prometheus界面已经显示配置成功

监控项：

probe_
​
probe_success  #是否探测成功(取值1、0，分别表示成功、失败)
probe_duration_seconds #探测的耗时
#关于DNS
probe_dns_lookup_time_seconds #DNS解析的耗时
probe_ip_protocol #IP协议取值为4、6
probe_ip_addr_hash #IP地址的哈希值，用于判断IP是否变化
#关于HTTP
probe_http_status_code #HTTP响应的状态码，如果发生重定向，则取决于最后一次响应
probe_http_content_length #HTTP响应的body长度，单位bytes
probe_http_version #HTTP响应的协议版本，比如1.1
probe_http_ssl #HTTP响应是否采用SSL，取值 1、0
probe_ssl_earliest_cert_expiry #SSL证书过期时间，为Unix时间戳
​

触发器

Prometheus 配置

#报警(触发器)配置
rule_files:
  - "alert.yml"
  - "rules/*.yml"

添加blackbox_exporter触发器

cd /opt/data/docker-prometheus/

创建触发器文件

cat > prometheus/rules/blackbox_exporter.yml << 'EOF'
groups:
- name: Blackbox
  rules:
  - alert: 黑盒子探测失败报警
    expr: probe_success == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: '黑盒子探测失败 {{ $labels.instance }}'
      description: "黑盒子探测失败,当前值{{ $value }}"
  - alert: 请求慢告警
    expr: avg_over_time(probe_duration_seconds[1m]) > 1
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: '请求慢 {{ $labels.instance }}'
      description: "请求时间操过1秒，值{{ $value }}"
  - alert: http状态码检测失败
    expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: 'http状态码检测失败{{ $labels.instance }}'
      description: "HTTP状态码非200-399,当前状态码为{{ $value }}"
  - alert: ssl证书即将到期
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 *30
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: '证书即将到期 {{ $labels.instance }}'
      description: "SSL 证书在30天后到期,值{{ $value }}"    
EOF

重新加载配置

curl -X POST http://localhost:9090/-/reload

alert中生效

Grafana增加dashboard

Blackbox Exporter Dashboard 20220412-StarsL.cn | Grafana Labs

Grafana

admin/password