Prometheus+grafana监控系统

Prometheus简介

Prometheus是一套开源的系统监控和报警框架。2012年，SoundCloud的Google前员工创造了Prometheus，并作为社区开源项目进行开发。2015年，该项目正式发布。2016年，Prometheus加入云原生计算基金会（Cloud Native Computing Foundation），成为受欢迎度仅次于Kubernetes的项目。
Prometheus基本原理是通过HTTP协议周期性抓取被监控组件的状态，这样做的好处是任意组件只要提供HTTP接口就可以接入监控系统，不需要任何SDK或者其他的集成过程。这样做非常适合虚拟化环境比如VM或者Docker

Prometheus具有以下特性：

多维的数据模型（基于时间序列的Key、Value键值对）

灵活的查询和聚合语言PromQL

提供本地存储和分布式存储

通过基于HTTP的Pull模型采集时间序列数据

可利用Pushgateway（Prometheus的可选中间件）实现Push模式

可通过动态服务发现或静态配置发现目标机器

支持多种图表和数据大盘

一、安装Prometheus

官网 https://prometheus.io/download/

1. 首先安装go

`yum install -y go`
`go version`

2. 下载prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.21.0-rc.0/prometheus-2.21.0-rc.0.linux-amd64.tar.gz
tar zxvf prometheus-2.21.0-rc.0.linux-amd64.tar.gz
mv prometheus-2.21.0-rc.0.linux-amd64 /data/prometheus

3. 添加自启

vim /usr/lib/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/
After=network.target

[Service]
# Type设置为notify时，服务会不断重启
Type=simple
User=prometheus
# --storage.tsdb.path是可选项，默认数据目录在运行目录的./dada目录中
ExecStart=/data/prometheus/prometheus --config.file=/data/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus/prometheus-data
Restart=on-failure

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start prometheus && systemctl enable prometheus

4.创建用户、授权目录

useradd prometheus
chown prometheus:prometheus /data/prometheus -R

5. 按需修改配置

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
- targets:
  # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ['localhost:9090']

###agent
  - job_name: linux
static_configs:
  - targets: ['192.168.1.153:9100','192.168.1.157:9100']
labels:
  instance: node1

  - job_name: mysql
static_configs:
  - targets: ['192.168.1.157:9104']
    labels:
      instance: devdb

二、exporter（受监控端）

exporter是用于收集监控数据的一个http接口，目前常用的应用组件官方都有提供相应的exporter，如：nginx、redis、mysql以及linux和windows操作系统（cpu、mem、disk、net）等，查看更多

1. 下载node_exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar zxvf node_exporter-1.0.1.linux-amd64.tar.gz
mv node_exporter-1.0.1.linux-amd64 /data1/srv/node_exporter

2. 添加node_exporter自启

vim /usr/lib/systemd/system/node_exporter.service

[Unit]
Description=node_exporter
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=root
ExecStart=/data1/srv/node_exporter/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start node_exporter && systemctl enable node_exporter

3. mysqld_exporter下载

...

4. mysqld_exporter自启脚本

需要创建一个mysql监控用户，将其填写在/data/mysqld_exporter/.my.cnf

vim /data/mysqld_exporter/.my.cnf

[client]
user=mysqluser
password=xxxxxx

vim /usr/lib/systemd/system/mysqld_exporter.service

[Unit]
Description=mysqld_exporter
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=root
ExecStart=/data/mysqld_exporter/mysqld_exporter --config.my-cnf=/data/mysqld_exporter/.my.cnf
Restart=on-failure

[Install]
WantedBy=multi-user.target