基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus监控

转发于：基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus监控

https://www.cnblogs.com/huligong1234/p/15142981.html

目录
基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus及Exporter安装
一、概述
二、安装Prometheus
2.1.安装Prometheus
2.2.将Promethues配置成系统服务
2.3.通过systemctl启动prometheus
2.4.prometheus界面
三、监控远程Linux主机
3.1.安装node_exporter
3.2.将node_exporter配置成系统服务
3.3.通过systemctl启动node_exporter
3.4.验证node_exporter
3.5.防火墙配置
3.6.Prometheus侧配置
3.6.1.prometheus.yml配置调整
3.6.2.校验配置正确性
3.6.3.重新加载配置
3.6.4.验证结果
四、监控MySQL
4.1.创建监控用账号
4.2.安装mysqld_exporter
4.4.将mysqld_exporter配置成系统服务
4.5.通过systemctl启动mysqld_exporter
4.6.验证mysqld_exporter
4.7.防火墙配置
4.8.Prometheus侧配置
4.8.1.prometheus.yml配置调整
4.8.2.校验配置正确性
4.8.3.重新加载配置
4.8.4.验证结果
五、监控Redis
5.1.安装redis_exporter
5.2.将redis_exporter配置成系统服务
5.3.通过systemctl启动redis_exporter
5.4.验证redis_exporter
5.5.防火墙配置
5.6.Prometheus侧配置
5.6.1.prometheus.yml配置调整
5.6.2.校验配置正确性
5.6.3.重新加载配置
5.6.4.验证结果
5.6.5 Redis集群监控配置示例
六、监控RabbitMQ
6.1.安装rabbitmq_prometheus
6.2.验证rabbitmq_prometheus
6.3.防火墙配置
6.4.Prometheus侧配置
6.4.1.prometheus.yml配置调整
6.4.2.校验配置正确性
6.4.3.重新加载配置
6.4.4.验证结果
七、监控Elasticsearch
7.1.安装elasticsearch_exporter
7.2.将elasticsearch_exporter配置成系统服务
7.3.通过systemctl启动elasticsearch_exporter
7.4.验证elasticsearch_exporter
7.5.防火墙配置
7.6.Prometheus侧配置
7.6.1.prometheus.yml配置调整
7.6.2.校验配置正确性
7.6.3.重新加载配置
7.6.4.验证结果
八、监控Kafka
8.1.安装kafka_exporter
8.2.将kafka_exporter配置成系统服务
8.3.通过systemctl启动kafka_exporter
8.4.验证kafka_exporter
8.5.防火墙配置
8.6.Prometheus侧配置
8.6.1.prometheus.yml配置调整
8.6.2.校验配置正确性
8.6.3.重新加载配置
8.6.4.验证结果
九、监控Doris
9.1.验证doris_prometheus
9.2.Prometheus侧配置
9.2.1.prometheus.yml配置调整
9.2.2.校验配置正确性
9.2.3.重新加载配置
9.2.4.验证结果
十、监控端口(HTTP/TCP等)
10.1.安装blackbox_exporter
10.2.将blackbox_exporter配置成系统服务
10.3.通过systemctl启动blackbox_exporter
10.4.Prometheus侧配置
10.4.1.prometheus.yml配置调整
10.4.2.校验配置正确性
10.4.3.重新加载配置
10.4.4.验证结果
十一、监控MongoDB
11.1.安装mongodb_exporter
11.2.将mongodb_exporter配置成系统服务
11.3.通过systemctl启动mongodb_exporter
11.4.验证mongodb_exporter
11.5.防火墙配置
11.6.Prometheus侧配置
11.6.1.prometheus.yml配置调整
附:其他参考资料
一、概述
Prometheus官网文档: https://prometheus.io/docs/introduction/overview/

各组件默认端口:
prometheus ：9090
node 默认端口：9091(原默认端口为9100,与elasticsearch端口冲突，故这里改为9091)
mysql_exporter默认端口：9104
redis_exporter 默认端口：9121
rabbitmq_prometheus 默认端口： 15692
elasticsearch_exporter 默认端口： 9114
kafka_exporter 默认端口： 9308
blackbox_exporter默认端口: 9115
mongodb_exporter 端口: 9216
alertmanager默认端口：9093

注意：
1.本文使用的各程序版本是当下最新版本，实际需要到官网去看是否有更新的版本，推荐使用最新版本。
2.本文Prometheus等程序安装及监控目标都为Linux环境(CentOS7)
3.Promethues支持pull和push两种方式，本文描述的是pull方式
4.exporter一般都部署在监控目标服务器上，blackbox_exporter则和prometheus部署在一起

二、安装Prometheus
2.1.安装Prometheus
[root@server ~]# cd /usr/local/src
[root@server src]# wget https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz
[root@server src]# mkdir -p /usr/local/prometheus/
[root@server src]# tar xvf prometheus-2.28.1.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@server src]# mv /usr/local/prometheus/prometheus-2.28.1.linux-amd64/ /usr/local/prometheus/prometheus
2.2.将Promethues配置成系统服务
[root@server src]# vi /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/usr/local/prometheus/prometheus
ExecStart=/usr/local/prometheus/prometheus/prometheus
–config.file=/usr/local/prometheus/prometheus/prometheus.yml
–storage.tsdb.path=/usr/local/prometheus/prometheus/data
–storage.tsdb.retention.time=30d
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -SIGINT $MAINPID
[Install]
WantedBy=multi-user.target
config.file指定配置文件地址
storage.tsdb.path指定数据文件目录
storage.tsdb.retention.time指定数据文件保留时间

Promethues提供了很多的命令行配置选项，可通过./prometheus -h进行查看

2.3.通过systemctl启动prometheus
[root@server src]# systemctl daemon-reload
[root@server src]# systemctl start prometheus
[root@server src]# systemctl status prometheus
[root@server src]# systemctl enable prometheus
[root@server src]# systemctl reload prometheus
2.4.prometheus界面
通过浏览器访问http://服务器IP:9090就可以访问到prometheus的主界面
http://192.168.1.100:9090/targets

三、监控远程Linux主机
https://github.com/prometheus/node_exporter/

3.1.安装node_exporter
[root@dbserver ~]# cd /usr/local/src
[root@dbserver src]# wget https://github.com/prometheus/node_exporter/releases/download/v1.2.0/node_exporter-1.2.0.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf node_exporter-1.2.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/node_exporter-1.2.0.linux-amd64/ /usr/local/prometheus/node_exporter
3.2.将node_exporter配置成系统服务
[root@dbserver src]# vi /etc/systemd/system/node_exporter.service
[Unit]
Description=node_exporter Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/usr/local/prometheus/node_exporter
ExecStart=/usr/local/prometheus/node_exporter/node_exporter
–web.listen-address=0.0.0.0:9101
ExecStop=/bin/kill -SIGINT $MAINPID

[Install]
WantedBy=multi-user.target
3.3.通过systemctl启动node_exporter
[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start node_exporter
[root@dbserver src]# systemctl status node_exporter
[root@dbserver src]# systemctl enable node_exporter
3.4.验证node_exporter
[root@dbserver src]# curl “http://127.0.0.1:9101/metrics”
3.5.防火墙配置
[root@dbserver src]# firewall-cmd --permanent --add-port=9101/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule=“rule family=“ipv4” source address=“192.168.1.100” port protocol=“tcp” port=“9101” accept”

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

3.6.Prometheus侧配置
3.6.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:

The job name is added as a label job= to any timeseries scraped from this config.

• job_name: ‘prometheus’

  metrics_path defaults to ‘/metrics’

  scheme defaults to ‘http’.

  static_configs:

  • targets: [‘127.0.0.1:9090’]

• job_name: ‘node_exporter’
  static_configs:

  • targets: [‘192.168.1.100:9101’,‘192.168.1.101:9101’]
    labels:
    service: 服务A
  • targets: [‘192.168.1.102:9101’,‘192.168.1.103:9101’]
    labels:
    service: 服务B
    3.6.2.校验配置正确性
    [root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml
    3.6.3.重新加载配置
    [root@dbserver src]# systemctl reload prometheus
    3.6.4.验证结果
    访问http://ip:9090/targets查看

四、监控MySQL
https://github.com/prometheus/mysqld_exporter/

4.1.创建监控用账号
CREATE USER ‘exporter’@‘localhost’ IDENTIFIED BY ‘Exporter#123’;
GRANT PROCESS,RELOAD,REPLICATION CLIENT, SELECT ON . TO ‘exporter’@‘localhost’;
FLUSH PRIVILEGES;
4.2.安装mysqld_exporter
[root@dbserver ~]# cd /usr/local/src
[root@dbserver src]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.13.0/mysqld_exporter-0.13.0.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf mysqld_exporter-0.13.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/mysqld_exporter-0.13.0.linux-amd64/ /usr/local/prometheus/mysqld_exporter

### 4.3.增加配置文件
```shell
[root@dbserver src]# vi /usr/local/prometheus/mysqld_exporter/.my.cnf
[client]
user=exporter
password=Exporter#123
socket=/var/lib/mysql/mysql.sock
4.4.将mysqld_exporter配置成系统服务
[root@dbserver src]# vi /etc/systemd/system/mysqld_exporter.service
[Unit]
Description=mysqld_exporter Server
Documentation=https://github.com/prometheus/mysqld_exporter/
After=network.target

[Service]
Environment=DATA_SOURCE_NAME=exporter:Exporter#123@(localhost:3306)/
ExecStart=/usr/local/prometheus/mysqld_exporter/mysqld_exporter \
  --web.listen-address=0.0.0.0:9104 \
  --config.my-cnf=/usr/local/prometheus/mysqld_exporter/.my.cnf \
  --collect.binlog_size \
  --collect.slave_status \
  --collect.slave_hosts \
  --collect.engine_innodb_status \
  --collect.info_schema.processlist \
  --collect.info_schema.innodb_metrics \
  --collect.info_schema.innodb_tablespaces \
  --collect.info_schema.clientstats \
  --collect.perf_schema.tableiowaits \
  --collect.perf_schema.indexiowaits \
  --collect.perf_schema.tablelocks \
  --collect.perf_schema.file_events \
  --collect.perf_schema.eventswaits
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
4.5.通过systemctl启动mysqld_exporter
[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start mysqld_exporter
[root@dbserver src]# systemctl status mysqld_exporter
[root@dbserver src]# systemctl enable mysqld_exporter
4.6.验证mysqld_exporter
[root@dbserver src]# curl "http://127.0.0.1:9104/metrics"
4.7.防火墙配置
[root@dbserver src]# firewall-cmd --permanent --add-port=9104/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9104" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

4.8.Prometheus侧配置
4.8.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'mysql_exporter'
    static_configs:
    - targets: ['192.168.1.100:9104']
      labels:
        service: 服务A-MySQL
    - targets: ['192.168.1.101:9104']
      labels:
        service: 服务B-MySQL
4.8.2.校验配置正确性
[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 
4.8.3.重新加载配置
[root@dbserver src]# systemctl reload prometheus
4.8.4.验证结果
访问http://ip:9090/targets查看

五、监控Redis
https://github.com/oliver006/redis_exporter/

5.1.安装redis_exporter
[root@dbserver ~]# cd /usr/local/src/
[root@dbserver src]# wget https://github.com/oliver006/redis_exporter/releases/download/v1.24.0/redis_exporter-v1.24.0.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf redis_exporter-v1.24.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/redis_exporter-v1.24.0.linux-amd64/ /usr/local/prometheus/redis_exporter
5.2.将redis_exporter配置成系统服务
[root@dbserver src]# vi /etc/systemd/system/redis_exporter.service
[Unit]
Description=redis_exporter Server
Documentation=https://github.com/oliver006/redis_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/redis_exporter/redis_exporter \
  --web.listen-address=0.0.0.0:9121 \
  --redis.addr=127.0.0.1:6379 \
  --redis.password=Test#123
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
5.3.通过systemctl启动redis_exporter
[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start redis_exporter
[root@dbserver src]# systemctl status redis_exporter
[root@dbserver src]# systemctl enable redis_exporter
5.4.验证redis_exporter
[root@dbserver src]# curl "http://127.0.0.1:9121/metrics"
5.5.防火墙配置
[root@dbserver src]# firewall-cmd --permanent --add-port=9121/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9121" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

5.6.Prometheus侧配置
5.6.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'redis_exporter'
    static_configs:
    - targets: ['192.168.1.100:9121']
      labels:
        service: 服务A-Redis
    - targets: ['192.168.1.101:9121']
      labels:
        service: 服务B-Redis
5.6.2.校验配置正确性
[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 
5.6.3.重新加载配置
[root@dbserver src]# systemctl reload prometheus
5.6.4.验证结果
访问http://ip:9090/targets查看

5.6.5 Redis集群监控配置示例
(本人未验证)

  - job_name: 'redis_exporter_targets'
    static_configs:
      - targets:
        - redis://192.168.1.101:7000
        - redis://192.168.1.102:7001
        - redis://192.168.1.103:7002
        - redis://192.168.1.104:7003
        - redis://192.168.1.105:7004
        - redis://192.168.1.106:7005
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.1.100:9121
  ## config for scraping the exporter itself
  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - 192.168.1.100:9121
六、监控RabbitMQ
6.1.安装rabbitmq_prometheus
RabbitMQ 3.8版本以后已内置支持Prometheus插件
https://www.rabbitmq.com/prometheus.html

启用方式：

[root@dbserver src]# rabbitmq-plugins enable rabbitmq_prometheus
默认端口号为15692
启动后可访问 http://ip:15692/metrics

如要修改默认配置，对应配置文件为 /etc/rabbitmq/rabbitmq.conf
prometheus.path = /metrics
prometheus.tcp.port = 15692

也可以使用单独的rabbitmq_exporter来监控，可适应rabbitmq3.8以前版本
（注意 3.6.x与更早版本监控有差别，需要注意)
https://github.com/kbudde/rabbitmq_exporter/

6.2.验证rabbitmq_prometheus
[root@dbserver src]# curl "http://127.0.0.1:15692/metrics"
6.3.防火墙配置
[root@dbserver src]# firewall-cmd --permanent --add-port=15692/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="15692" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

6.4.Prometheus侧配置
6.4.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'rabbitmq_exporter'
    static_configs:
    - targets: ['192.168.1.100:15692']
      labels:
        service: 服务A-RabbitMQ
    - targets: ['192.168.1.101:15692']
      labels:
        service: 服务B-RabbitMQ
6.4.2.校验配置正确性
[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 
6.4.3.重新加载配置
[root@dbserver src]# systemctl reload prometheus
6.4.4.验证结果
访问http://ip:9090/targets查看

七、监控Elasticsearch
Elasticsearch Exporter
https://grafana.com/oss/prometheus/exporters/elasticsearch-exporter/
https://github.com/prometheus-community/elasticsearch_exporter

7.1.安装elasticsearch_exporter
[root@dbserver ~]# cd /usr/local/src
[root@dbserver src]# wget https://github.com/justwatchcom/elasticsearch_exporter/releases/download/v1.2.1/elasticsearch_exporter-1.2.1.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf elasticsearch_exporter-1.2.1.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/elasticsearch_exporter-1.2.1.linux-amd64/ /usr/local/prometheus/elasticsearch_exporter
7.2.将elasticsearch_exporter配置成系统服务
[root@dbserver src]# vi /etc/systemd/system/elasticsearch_exporter.service
[Unit]
Description=elasticsearch_exporter Server
Documentation=https://github.com/prometheus-community/elasticsearch_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/elasticsearch_exporter/elasticsearch_exporter \
  --web.listen-address=0.0.0.0:9114 \
  --es.uri=http://127.0.0.1:9200
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
如果有启用账号密码，则需要携带上 http://admin:pass@localhost:9200

7.3.通过systemctl启动elasticsearch_exporter
[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start elasticsearch_exporter
[root@dbserver src]# systemctl status elasticsearch_exporter
[root@dbserver src]# systemctl enable elasticsearch_exporter
7.4.验证elasticsearch_exporter
[root@dbserver src]# curl "http://127.0.0.1:9114/metrics"
7.5.防火墙配置
[root@dbserver src]# firewall-cmd --permanent --add-port=9114/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9114" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld
7.6.Prometheus侧配置
7.6.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'elasticsearch_exporter'
    static_configs:
    - targets: ['192.168.1.100:9114']
      labels:
        service: 服务A-Elasticsearch
    - targets: ['192.168.1.101:9114']
      labels:
        service: 服务B-Elasticsearch
7.6.2.校验配置正确性
[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 
7.6.3.重新加载配置
[root@dbserver src]# systemctl reload prometheus
7.6.4.验证结果
访问http://ip:9090/targets查看

八、监控Kafka
https://github.com/danielqsj/kafka_exporter/

8.1.安装kafka_exporter
[root@dbserver ~]# cd /usr/local/src/
[root@dbserver src]# wget https://github.com/danielqsj/kafka_exporter/releases/download/v1.3.1/kafka_exporter-1.3.1.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf kafka_exporter-1.3.1.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/kafka_exporter-1.3.1.linux-amd64/ /usr/local/prometheus/kafka_exporter
8.2.将kafka_exporter配置成系统服务
[root@dbserver src]# vi /etc/systemd/system/kafka_exporter.service
[Unit]
Description=kafka_exporter Server
Documentation=https://github.com/danielqsj/kafka_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/kafka_exporter/kafka_exporter \
  --web.listen-address=0.0.0.0:9308 \
  --kafka.server=127.0.0.1:9092
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
8.3.通过systemctl启动kafka_exporter
[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start kafka_exporter
[root@dbserver src]# systemctl status kafka_exporter
[root@dbserver src]# systemctl enable kafka_exporter
8.4.验证kafka_exporter
[root@dbserver src]# curl "http://127.0.0.1:9308/metrics"
8.5.防火墙配置
[root@dbserver src]# firewall-cmd --permanent --add-port=9308/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9308" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

8.6.Prometheus侧配置
8.6.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'elasticsearch_exporter'
    static_configs:
    - targets: ['192.168.1.100:9308']
      labels:
        service: 服务A-Kafka
    - targets: ['192.168.1.101:9308']
      labels:
        service: 服务B-Kafka
8.6.2.校验配置正确性
[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 
8.6.3.重新加载配置
[root@dbserver src]# systemctl reload prometheus
8.6.4.验证结果
访问http://ip:9090/targets查看

九、监控Doris
Doris已内置支持对FE,BE的Promotheus监控支持
无需额外配置，防护墙允许Prometheus访问 8030,8040端口即可

9.1.验证doris_prometheus
#FE
[root@dbserver src]# curl "http://127.0.0.1:8030/metrics"

#BE
[root@dbserver src]# curl "http://127.0.0.1:8040/metrics"
9.2.Prometheus侧配置
9.2.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'doris_cluster'
    static_configs:
    - targets: ['192.168.1.100:8030','192.168.1.101:8030']
      labels:
	    service: 服务A-Doris-FE
        group: fe

    - targets: ['192.168.1.102:8040','192.168.1.103:8040','192.168.1.104:8040']
      labels:
	    service: 服务A-Doris-BE
        group: be

9.2.2.校验配置正确性
[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 
9.2.3.重新加载配置
[root@dbserver src]# systemctl reload prometheus
9.2.4.验证结果
访问http://ip:9090/targets查看

十、监控端口(HTTP/TCP等)
10.1.安装blackbox_exporter
这里使用 blackbox_exporter来监控HTTP/TCP

[root@appserver ~]# cd /usr/local/src/
[root@appserver src]# wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
[root@appserver src]# tar xvf blackbox_exporter-0.19.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@appserver src]# mv /usr/local/prometheus/blackbox_exporter-0.19.0.linux-amd64/ /usr/local/prometheus/blackbox_exporter
10.2.将blackbox_exporter配置成系统服务
编辑启动文件

[root@appserver src]# vi /etc/systemd/system/blackbox_exporter.service
[Unit]
Description=blackbox_exporter Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target
  
[Service]
Restart=on-failure
WorkingDirectory=/usr/local/prometheus/blackbox_exporter
ExecStart=/usr/local/prometheus/blackbox_exporter/blackbox_exporter \
  --web.listen-address ":9115" \
  --config.file=/usr/local/prometheus/blackbox_exporter/blackbox.yml
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -SIGINT $MAINPID

[Install]
WantedBy=multi-user.target
10.3.通过systemctl启动blackbox_exporter
[root@appserver src]# systemctl daemon-reload
[root@appserver src]# systemctl start blackbox_exporter
[root@appserver src]# systemctl status blackbox_exporter
[root@appserver src]# systemctl enable blackbox_exporter
10.4.Prometheus侧配置
10.4.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

# 网站监控
  - job_name: 'http_status'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['http://a1.domain.com:81']
        labels:
          env: PROD
          domain: "a1.domain.com"
          port: "81"
          service: 服务A
          instance: "http://a1.domain.com:81"
          group: web
      - targets: ['http://a1.domain.com:82/abc']
        labels:
          env: PROD
          domain: "a1.domain.com"
          port: "82"
          service: XX API接口
          instance: "http://a1.domain.com:82/abc"
          group: api

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.1.100:9115 
		
# ping 检测
  - job_name: 'ping_status'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['192.168.1.100']
        labels:
          env: PROD
          service: 服务B IP
          instance: "192.168.1.100"
          group: ping

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.1.100:9115 
		
# 端口监控
  - job_name: 'port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['192.168.1.100:84']
        labels:
          env: PROD
          port: "84"
          service: 服务B XX端口
          instance: "192.168.1.100:84"
          group: port

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.1.100:9115 
blackbox_exporter涉及的module配置文件 /usr/local/prometheus/blackbox_exporter/blackbox.yml

网上示例参考：https://github.com/tools-env/blackbox_exporter/blob/master/example.yml

10.4.2.校验配置正确性
[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 
10.4.3.重新加载配置
[root@dbserver src]# systemctl reload prometheus
10.4.4.验证结果
访问http://ip:9090/targets查看

十一、监控MongoDB
https://github.com/percona/mongodb_exporter

11.1.安装mongodb_exporter
[root@dbserver ~]# cd /usr/local/src/
[root@dbserver src]# wget https://github.com/percona/mongodb_exporter/releases/download/v0.20.7/mongodb_exporter-0.20.7.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf mongodb_exporter-0.20.7.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/mongodb_exporter-0.20.7.linux-amd64 /usr/local/prometheus/mongodb_exporter

11.2.将mongodb_exporter配置成系统服务
[root@dbserver src]# vi /etc/systemd/system/mongodb_exporter.service
[Unit]
Description=mongodb_exporter Server
Documentation=https://github.com/percona/mongodb_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/mongodb_exporter/mongodb_exporter \
  --web.listen-address=0.0.0.0:9216 \
  --mongodb.uri=mongodb://username:[email protected]:27017/admin?replicaSet=rs0&&authSource=admin
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
11.3.通过systemctl启动mongodb_exporter
[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start mongodb_exporter
[root@dbserver src]# systemctl status mongodb_exporter
[root@dbserver src]# systemctl enable mongodb_exporter
11.4.验证mongodb_exporter
[root@dbserver src]# curl "http://127.0.0.1:9216/metrics"
11.5.防火墙配置
[root@dbserver src]# firewall-cmd --permanent --add-port=9216/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9216" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

11.6.Prometheus侧配置
11.6.1.prometheus.yml配置调整
[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'mongodb_exporter'
    static_configs:
    - targets: ['192.168.1.100:9216']
      labels:
        service: 服务A-MongoDB
附:其他参考资料
https://www.cnblogs.com/cyleon/category/1756369.html
https://www.cnblogs.com/gaofeng-henu/category/1675861.html
https://www.cnblogs.com/winstom/p/11926140.html
https://www.cnblogs.com/caoweixiong/category/1622389.html
https://www.bookstack.cn/read/prometheus-book/README.md