blackbox_exporter+grafana+prometheus监控主机存活，端口存活及网站状态

blackbox_exporter是Prometheus 官方提供的 exporter 之一，可以提供 http、dns、tcp、icmp 的监控数据采集。

Blackbox_exporter 应用场景

• HTTP 测试
  定义 Request Header 信息
  判断 Http status / Http Respones Header / Http Body 内容
• TCP 测试
  业务组件端口状态监听
  应用层协议定义与监听
• ICMP 测试
  主机探活机制
• POST 测试
  接口联通性
• SSL 证书过期时间

安装blackbox_exporter

• blackbox_exporter安装
  各个版本的blackbox_exporter https://github.com/prometheus/blackbox_exporter/releases
  以linux系统为例，下载编译好的二进制包，解压使用：

$ wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.14.0/blackbox_exporter-0.14.0.linux-amd64.tar.gz
$ tar -xvf blackbox_exporter-0.14.0.linux-amd64.tar.gz
$ mv blackbox_exporter-0.14.0.linux-amd64 /usr/local/blackbox_exporter

• 验证是否安装成功

$ cd /usr/local/blackbox_exporter
$ ./blackbox_exporter --version
blackbox_exporter, version 0.14.0 (branch: HEAD, revision: bba7ef76193948a333a5868a1ab38b864f7d968a)
  build user:       root@63d11aa5b6c6
  build date:       20190315-13:32:31
  go version:       go1.11.5

• 创建systemd服务

$ vim /lib/systemd/system/blackbox_exporter.service

[Unit]
Description=blackbox_exporter
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/blackbox_exporter/blackbox_exporter --config.file=/usr/local/blackbox_exporter/blackbox.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target

如果以非root用户运行blackbox_exporter，为了使用icmp prober，需要设置CAP_NET_RAW，即对可执行文件blackbox_exporter执行下面的命令：

$ cd /usr/local/blackbox_exporter
$ setcap cap_net_raw+ep blackbox_exporter

• 启动blackbox_exporter

$ systemctl daemon-reload
$ systemctl start blackbox_exporter

• 验证是否启动成功
  默认监听端口为9115

$ systemctl status blackbox_exporter
$ netstat -lnpt|grep 9115

• prometheus.yml中加入blackbox_exporter

1. 监控主机存活状态

$ vim /usr/local/prometheus/prometheus.yml

  - job_name: 'node_status'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['127.0.0.1']
        labels:
          instance: 'node_status'
          group: 'node'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
        #      - source_labels: [__param_target]
        #        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

2. 监控主机端口存活状态

$ vim /usr/local/prometheus/prometheus.yml

  - job_name: 'port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['127.0.0.1:9100','127.0.0.1:9090']
        labels:
          instance: 'port_status'
          group: 'tcp'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
        #      - source_labels: [__param_target]
        #        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

3. 监控网站状态

$ vim /usr/local/prometheus/prometheus.yml

  - job_name: web_status
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://www.baidu.com']
        labels:
          instance: web_status
          group: web
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 127.0.0.1:9115

• 检查配置文件是否书写正确

$ cd /usr/local/prometheus
$ ./promtool check config prometheus.yml

• 重新加载prometheus的配置

$ systemctl reload prometheus
或
$ curl -X POST http://127.0.0.1:9090/-/reload (启用了--web.enable-lifecycle选项)

• 访问web界面
  访问 http://127.0.0.1:9090/targets 查看加入的监控信息。
  

grafana中加入blackbox_exporter监控数据

• 登录grafana
  访问 http://127.0.0.1:3000,用户名密码都为admin。
  
• 导入blackbox_exporter模板
  此模板为9965号模板，数据源选择Prometheus 模板下载地址 https://grafana.com/grafana/dashboards/9965
  
  此模板需要安装饼状图插件 下载地址 https://grafana.com/grafana/plugins/grafana-piechart-panel
  安装插件，重启grafana生效。

$ grafana-cli plugins install grafana-piechart-panel
$ service grafana-server restart

• 访问grafana
  访问 http://127.0.0.1:3000，查看数据。