prometheus+grafana监控系统部署

一、环境说明

通过Prometheus+grafana 监控展示数据库、中间件、微服务项目的相关指标。本系统适用于开发、测试环境使用。

环境规划：

主机IP	角色	备注
10.10.117.95	被监控主机	postgre、redis、kafka、minio、nacos、mongo、微服务项目、node_erporter、redis_exporter、postges_exporter、kafka_exporter、cAdvisor
10.10.239.119	prometheus服务端	prometheus、grafana

一、组件安装

1、prometheus安装

#!/bin/bash
# 下载
download_url="https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz"
if [ ! -e "prometheus-2.28.1.linux-amd64.tar.gz" ];then
        wget ${download_url}
fi
# 解压并安装
mkdir /data 
tar -xvf prometheus-2.28.1.linux-amd64.tar.gz - C /data
ln -s /data/prometheus-2.28.1.linux-amd64 /opt/prometheus
安装服务脚本
ip=`ip a|grep 'inet '|grep -v '127.0.0.1'|awk  '{print $2}'|awk -F '/' '{print $1}'`
cat </usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus server daemon
After=network.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/opt/prometheus/prometheus \
    --config.file=/opt/prometheus/prometheus.yml \
    --storage.tsdb.path="/opt/prometheus/data" \
    --storage.tsdb.retention=30d \
    --web.console.templates="/opt/prometheus/consoles" \
    --web.console.libraries="/opt/prometheus/console_libraries" \
    --web.external-url=http://${ip}:9090  \
    --web.listen-address=0.0.0.0:9090
Restart=on-failure
EOF
systemctl daemon-reload
systemctl enable prometheus.service
systemctl start prometheus.service
systemctl restart prometheus.service
systemctl status prometheus.service

2、grafana安装  

2.1 yum安装

vim /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://mirrors.tuna.tsinghua.edu.cn/grafana/yum/rpm
repo_gpgcheck=0
gpgkey=https://packages.grafana.com/gpg.key
enabled=1
yum –y install grafana
systemctl enable grafana-server --now
systemctl status grafana-server

2.2 二进制包安装

wget https://dl.grafana.com/oss/release/grafana-8.3.7.linux-amd64.tar.gz
tar -xf grafana-8.3.7.linux-amd64.tar.gz -C /data
ln -s /data/grafana-8.3.7 /opt/grafana
cd /opt/grafana/
./bin/grafana-server web &

二、Export 部署

1、安装node_exporter

#！/bin/bash
# 下载安装包
download_url="https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz"
if [ ! -e "node_exporter-1.1.2.linux-amd64.tar.gz" ];then
        wget ${download_url}
fi
# 解压安装
tar -xvf node_exporter-1.1.2.linux-amd64.tar.gz -C /data
ln -s /data/node_exporter-1.1.2.linux-amd64 /opt/node_exporter
# 生成服务器管理脚本
cat </usr/lib/systemd/system/node_exporter.service
[Unit]
Description=node_exporter for linux server
Documentation=https://prometheus.io/
After=network-online.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/opt/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
# 服务管理
systemctl daemon-reload
systemctl enable node_exporter
systemctl start node_exporter
systemctl restart node_exporter

/opt/prometheus/prometheus.yml配置

  - job_name: 'static config'
    static_configs:
    - targets: ['10.10.239.119:9100','10.10.117.95:9100']

2、安装postgre_exporter

2.1 docker-compose 方式部署

version: "3.1"
services:
  postgres-exporter:
    image: wrouesnel/postgres_exporter
    container_name: postgres-exporter
    environment:
      DATA_SOURCE_NAME: "postgresql://user:[email protected]:5432/postgres?sslmode=disable"
    ports:
      - "9187:9187"

2.2 docker方式部署

sudo docker run --name postgres-exporter  -d -e DATA_SOURCE_NAME="postgresql://user:[email protected]:5432/postgres?sslmode=disable" -p 9187:9187 wrouesnel/postgres_exporter

2.3 /opt/prometheus/prometheus.yml配置

  - job_name: 'postgres_exporter'
    static_configs:
      - targets: ['10.10.117.95:9187']

3、安装redis_exporter

3.1 下载并安装redis_exporter

下载链接   https://github.com/oliver006/redis_exporter/releases

tar -xf redis_exporter-v1.37.0.linux-amd64.tar.gz -C /data
ln -s /data/redis_exporter-v1.37.0.linux-amd64 /opt/redis_exporter
cd cd /opt/redis_exporter/
./redis_exporter -redis.addr=10.10.117.95:6379 -redis.password=xxxxxxx &

3.2 /opt/prometheus/prometheus.yml配置

  - job_name: 'redis_exporter'
    scrape_interval: 15s
    static_configs:
      - targets: ['10.10.117.95:9121']

4、安装kafka_exporter

4.1 下载并安装kafka_exporter

下载链接  https://github.com/danielqsj/kafka_exporter/releases/tag/v1.2.0

tar –xf kafka_exporter-1.2.0.linux-amd64.tar.gz -C /data
ln –s /data/kafka_exporter-1.2.0.linux-amd64 /opt/kafka_exporter
#启动
nohup /usr/local/kafka_exporter/kafka_exporter --kafka.server=10.10.117.95:9092 --kafka.version=1.1.0 --log.level=info > kafka_exporter.log  --web.listen-address=0.0.0.0:9308 --sasl.enabled --sasl.username="kafka" --sasl.password"=xxxx" &

4.2 /opt/prometheus/prometheus.yml配置

  - job_name: 'kafkas_exporter'
    scrape_interval: 15s
    static_configs:
      - targets:
        - 10.10.117.95:9308

5、安装cAdvisor

5.1 docker-compose 方式部署

version: "3.1"
services:
  cadvisor:
    image: google/cadvisor:latest
    container_name: monitoring_cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg:/dev/kmsg
    ports:
      - "8085:8080"

5.2 docker 方式部署

docker run   --volume=/:/rootfs:ro   --volume=/var/run:/var/run:rw   --volume=/sys:/sys:ro   --volume=/var/lib/docker/:/var/lib/docker:ro   --volume=/dev/disk/:/dev/disk:ro   --publish=8085:8080   --detach=true   --name=cadvisor   google/cadvisor:latest

5.3 /opt/prometheus/prometheus.yml配置

  - job_name: 'cAdvisor'
    static_configs:
      - targets: ['10.10.117.95:8085']

6、minio nacos 指标收集

/opt/prometheus/prometheus.yml配置

  - job_name: 'minio'
    bearer_token: Boe888888
    metrics_path: /minio/v2/metrics/cluster
    scheme: http
    static_configs:
      - targets: ['10.10.117.95:9000']

  - job_name: 'nacos'
    scrape_interval: 15s
    metrics_path: '/nacos/actuator/prometheus'
    static_configs:
      - targets: ['10.10.117.95:8848']

#nacos由于认证收集可能会有些问题，请参考nacos官方文档，后续解决后更新。

7、收集微服务暴露出来的指标

7.1 /opt/prometheus/prometheus.yml配置

  - job_name:  'application'
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: '/actuator/prometheus'
    file_sd_configs:
      - files:
        - /opt/prometheus/application.yaml
        refresh_interval: 5s

7.2 /opt/prometheus/application.yaml 配置

- targets:
  - '10.10.117.95:10000'
  labels:
    application: 'gateway'

- targets:
  - '10.10.117.95:9899'
  labels:
    application: 'stores'

- targets:
  - '10.10.117.95:9328'
  labels:
    application: 'flowable'

- targets:
  - '10.10.117.95:9228'
  labels:
    application: 'map'

- targets:
  - '10.10.117.95:9325'
  labels:
    application: 'security'

- targets:
  - '10.10.117.95:8088'
  labels:
    application: 'gis-server'

- targets:
  - '10.10.117.95:10002'
  labels:
    application: 'user'

- targets:
  - '10.10.117.95:9625'
  labels:
    application: 'messages'

- targets:
  - '10.10.117.95:8126'
  labels:
    application: 'sys-manager'

- targets:
  - '10.10.117.95:9888'
  labels:
    application: 'portal'

- targets:
  - '10.10.117.95:9002'
  labels:
    application: 'opmaint'

- targets:
  - '10.10.117.95:10001'
  labels:
    application: 'auth'

三、模板导入

登录grafana添加数据源后导入下列模板

spring boot   
https://grafana.com/grafana/dashboards/10280

node_exporter
https://grafana.com/grafana/dashboards/8919

postgre_exporter
https://grafana.com/grafana/dashboards/455

redis_exporter
https://grafana.com/grafana/dashboards/763	

kafka_exporter
https://grafana.com/grafana/dashboards/7589

cAvsior
https://grafana.com/grafana/dashboards/10566