prometheus的几个关键组件
prometheus server: 负责定时轮询采集数据、存储、对外提供数据查询、告警规则检测
exporter：以http的方式，暴露收集的metric，然后Prometheus server会定期来拉取数据，可安装在被监控主机
AlertManager：Prometheus server会将通过规则匹配的告警发送到alertmanager，然后alertmanager会对告警发送到邮箱、企业微信等，期间会对告警进行分组、静默或抑制
PushGateway：出于网络或安全问题，有时数据无法直接暴露给prometheus采集，这时需要pushgateway完成中转工作。用户把数据推送到pushgateway，然后由prometheus采用拉取pull的方式采集数据
1.Prometheus安装
1.1 下载解压
[root@prometheus ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.20.1/prometheus-2.20.1.linux-amd64.tar.gz
[root@prometheus ~]# tar xf prometheus-2.20.1.linux-amd64.tar.gz
[root@prometheus ~]# mv prometheus-2.20.1.linux-amd64 /usr/local/prometheus
1.2 查看prometheus参数
[root@prometheus ~]# cd /usr/local/prometheus/
[root@prometheus prometheus]# ./prometheus --help
1.3 启动
指定配置文件和数据存储路径，默认存储在prometheus路径下的data
[root@prometheus prometheus]# ./prometheus --config.file="./prometheus.yml" --storage.tsdb.path="/usr/local/prometheus/data"> /dev/null 2>&1 &
[2] 7558
托管到systemd（添加为服务）
[root@prometheus ~]# cat /usr/lib/systemd/system/prometheus.service
[Unit]
Description=https://prometheus.io
After=network.target
[Service]
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml -- storage.tsdb.path=/usr/local/prometheus/data
[Install]
WantedBy=multi-user.target
[root@prometheus ~]# systemctl daemon-reload
[root@prometheus ~]# systemctl start prometheus
[root@prometheus ~]# systemctl status prometheus
[root@prometheus ~]# systemctl enable prometheus
1.4 浏览器访问,http://192.168.31.207:9090
1.5 配置文件了解
[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
# 全局配置
global:
scrape_interval: 15s # 抓取数据间隔设置为15秒，默认为1分钟
evaluation_interval: 15s # 评估规则默认周期为15秒评估一次，默认1分钟
#scrape_timeout: 1m # 抓取超时时间默认为1分钟
# Alertmanager告警相关配置
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# 加载告警规则，并根据全局定义的评估规则时间定期评估
rule_files:
# - "first_rules.yml" #告警规则文件所在位置
# - "second_rules.yml"
# 收集数据配置列表
scrape_configs:
# 作业命名
- job_name: 'prometheus'
#静态配置目录列表
static_configs:
#静态配置指定目标
- targets: ['localhost:9090']
2.node_exporter安装
在Prometheus中，不仅提供了丰富的exporter，用于监控基础指标、中间件、网络设备等
在这里，node_exporter用于收集机器基础的监控指标，像cpu、内存、磁盘可用空间等，常用exporter如下：
2.1 下载解压
[root@prometheus ~]# wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
[root@prometheus ~]# tar xf node_exporter-1.0.1.linux-amd64.tar.gz
[root@prometheus ~]# mv /root/mv node_exporter-1.0.1.linux-amd64 /usr/local/node_exporter
2.2 托管到systemd
[root@prometheus ~]# vim /usr/lib/systemd/system/node_exporter.service
[Unit]
Description=node-exporter
[Service]
ExecStart=/usr/local/node_exporter/node_exporter
[Install]
WantedBy=multi-user.target
[root@prometheus ~]# systemctl daemon-reload
[root@prometheus ~]# systemctl start node_exporter
[root@prometheus ~]# ss -lnt | grep 9100
LISTEN 0 128 :::9100 :::*
2.3 查看node_exporter暴露的数据
[root@prometheus ~]# curl http://192.168.31.207:9100/metrics
# HELP node_network_iface_link iface_link value of /sys/class/net/.
# TYPE node_network_iface_link gauge
node_network_iface_link{device="ens33"} 2
node_network_iface_link{device="lo"} 1
# HELP node_network_iface_link_mode iface_link_mode value of /sys/class/net/.
# TYPE node_network_iface_link_mode gauge
node_network_iface_link_mode{device="ens33"} 0
node_network_iface_link_mode{device="lo"} 0
解释
HELP：用于解释当前指标的含义
type：数据类型.如counter(计数器)只增不减数据类型;gauge(仪表盘),数据会发生变化;summary,获取x轴坐标和y轴坐标,计算分位数;histogram(直方图),获取x轴坐标和y轴坐标，并显示总和
最后一行：监控指标和对应的值
2.4 接入到prometheus
[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter' #监控其他机器的node_exporter，都可以添加到targets下
static_configs:
- targets:
- '192.168.31.207:9100'
检查配置文件是否有语法错误
[root@prometheus prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
SUCCESS: 0 rule files found
语法没有错误，重启服务
[root@prometheus ~]# systemctl restart prometheus
3.influxdb安装
存储数据到influxdb
3.1 安装influxdb
[root@prometheus ~]# wget https://dl.influxdata.com/influxdb/releases/influxdb-1.8.1.x86_64.rpm
[root@prometheus ~]# rpm -ivh influxdb-1.8.1.x86_64.rpm
[root@prometheus ~]#yum install urw-fonts ##如果报错缺少urw-fonts 可以yum安装
[root@prometheus ~]# systemctl start influxdb
[root@prometheus ~]# systemctl enable influxdb
查看端口是否已开启
[root@prometheus ~]# ss -lntp | grep influxd
LISTEN 0 128 127.0.0.1:8088 *:* users:(("influxd",pid=6392,fd=3))
LISTEN 0 128 :::8086 :::* users:(("influxd",pid=6392,fd=23))
3.2 登录influxdb,创建对应数据库
[root@prometheus ~]# influx
Connected to http://localhost:8086 version 1.8.1
InfluxDB shell version: 1.8.1
> show databases;
name: databases
name
----
_internal
> create database prometheus;
> show databases;
name: databases
name
----
_internal
prometheus
> exit
3.3 修改prometheus配置文件，使数据写入influxdb并从influxbd读取数据
修改prometheus配置文件，添加读取和写入数据的url，如果登录influxdb需要密码，可参考
https://docs.influxdata.com/influxdb/v1.8/supported_protocols/prometheus/
[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
（直接在文件末尾追加内容即可）
remote_write:
- url: "http://192.168.31.207:8086/api/v1/prom/write?db=prometheus"
remote_read:
- url: "http://192.168.31.207:8086/api/v1/prom/read?db=prometheus"
重启prometheus
[root@prometheus ~]# systemctl restart prometheus
登录influxdb，查看采集数据是否已写入
[root@prometheus ~]# influx
Connected to http://localhost:8086 version 1.8.1
InfluxDB shell version: 1.8.1
> use prometheus
Using database prometheus
> show measurements
name: measurements
name
----
go_gc_duration_seconds
go_gc_duration_seconds_count
go_gc_duration_seconds_sum
go_goroutines
go_info
4.安装grafana
接入grafana，对Prometheus获取的数据进行可视化展示，模板分享https://grafana.com/dashboards
4.1 下载安装grafana
[root@prometheus ~]# wget https://dl.grafana.com/oss/release/grafana-7.1.5-1.x86_64.rpm
[root@prometheus ~]# rpm -ivh grafana-7.1.5-1.x86_64.rpm
[root@prometheus ~]# systemctl grafana-server status
[root@prometheus ~]# systemctl start grafana-server
[root@prometheus ~]# systemctl enable grafana-server
4.2 访问登录
192.168.31.207:3000,默认用户名密码:admin/admin,进入系统后会提示重新设置密码。
添加数据源：
选择要添加的数据源类型，这里我们选择prometheus：
配置数据源名称和URL地址：
添加dashboard：
点击dashboard setting添加dashboard：
编辑dashboard：
给dashboard添加panel：
选择dashboard并配置相关项：
配置完成后监控项：