blackbox_exporter 黑盒监控

一、简介

blackbox_exporter是Prometheus 官方提供的 exporter 之一，可以提供 http、dns、tcp、icmp 的监控数据采集，blackbox_exporter 可用于以下监测：
1.HTTP 测试
定义 Request Header 信息
判断 Http status / Http Respones Header / Http Body 内容

2.TCP 测试
业务组件端口状态监听
应用层协议定义与监听

3.ICMP 测试
主机探活机制

4.POST 测试
接口联通性
SSL 证书过期时间

github地址：GitHub - prometheus/blackbox_exporter: Blackbox prober exporterhttps://github.com/prometheus/blackbox_exporter

二、安装

二进制安装

$ wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
$ tar -zxvf blackbox_exporter-0.16.0.linux-amd64.tar.gz -C /data
$ mv /data/blackbox_exporter-0.16.0.linux-amd64 /data/blackbox_exporter
$ cd /data/blackbox_exporter/
$ ./blackbox_exporter --version
$ nohup ./blackbox_exporter &

语法使用：

语法结构：
http://localhost:9115/probe?target=xxx&module=xxx&debug=true

例如：
$ curl "http://localhost:9115/probe?target=10.50.10.7:22022&module=tcp_connect&debug=true"
$ curl "http://localhost:9115/probe?target=10.50.10.7&module=icmp&debug=true"

K8S安装

配置文件

$ cat config.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-exporter
  namespace: kube-mon
data:
  blackbox.yml: |-
    modules:
      http_2xx:
        prober: http
        timeout: 2s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          valid_status_codes: [200,301,302]
          method: GET
          preferred_ip_protocol: "ip4"
      tcp_connect:
        prober: tcp
        timeout: 2s
      http_403:
        prober: http
        timeout: 2s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          valid_status_codes: [403]
          method: GET
          preferred_ip_protocol: "ip4"
      https_403:
        prober: https
        timeout: 2s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          valid_status_codes: [403]
          method: GET
          preferred_ip_protocol: "ip4"
      icmp:
        prober: icmp

deploy部署

$ cat deploy.yml 
kind: Deployment
apiVersion: apps/v1
metadata:
  name: blackbox-exporter
  namespace: kube-mon
  labels:
    app.kubernetes.io/name: blackbox
    app.kubernetes.io/version: 0.16.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: blackbox
  template:
    metadata:
      labels:
        app.kubernetes.io/name: blackbox
    spec:
      volumes:
      - name: config
        configMap:
          name: blackbox-exporter
          defaultMode: 420
      containers:
      - name: blackbox-exporter
        image: prom/blackbox-exporter:v0.16.0
        imagePullPolicy: IfNotPresent
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --log.level=info
        - --web.listen-address=:9115
        ports:
        - name: blackbox-port
          containerPort: 9115
          protocol: TCP
        resources:
          limits:
            cpu: 30m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 50Mi
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3

svc 服务发现

$ cat svc.yml
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: blackbox
  name: blackbox-exporter
  namespace: kube-mon
spec:
  ports:
  - name: balckbox
    port: 9115
    protocol: TCP
    targetPort: 9115
  selector:
    app.kubernetes.io/name: blackbox

创建以上资源：

$ kubectl apply -f .

如更改配置文件需要重新加载配置

$ curl -X POST localhost:31425/-/reload

三、使用于Prometheus中

1、ICMP 测试（主机探活）
可以通过 ping(icmp) 检测服务器的存活,在 blackbox.yml 使用的配置是icmp 模块：

modules:
  icmp:
    prober: icmp

添加Prometheus配置：

   - job_name: "check_hosts"
     metrics_path: /probe
     params:
       modelus: [icmp]
     static_configs:
     - targets:
       - 10.60.38.208
     relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: blackbox-exporter:9115

2、TCP 测试（监控主机端口存活状态）
检测端口是否存活,在blackbox.yml 配置文件中使用的配置是tcp_connect模块：

modules:
  tcp_connect:
    prober: tcp

添加pronetheus配置

   - job_name: "check_ports"
     metrics_path: /probe
     params:
       module: [tcp_connect]
     static_configs:
     - targets:
       - 10.60.38.208:8080
     relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: blackbox-exporter:9115

四、告警规则

端口监测失败 （正则匹配8080端口）
sum by(instance) (probe_success{instance=~".*:8080$",job="check_ports"}) == 0

主机ping不通
sum by (instance)(probe_success{job="check_hosts"}) == 0