Prometheus实战篇:Prometheus监控redis
准备环境
docker-compose安装redis
docker-compose.yaml
version: '3'
services:
redis:
image:redis:5
container_name: redis
command: redis-server --requirepass 123456 --maxmemory 512mb
restart: always
volumes:
- /data/redis/data: /data
port:
- "6379:6379"
docker-compose up -d
监控redis
docker安装exporter
docker直接运行
docker run -d --name redis_exporter -p 9121:9121 oliver006/redis_exporter --redis.addr redis://localhost:6379 --redis.password '123456'
docker-compose方式
cat >docker-compose.yaml <<FOF
version: '3.3'
services:
redis_exproter:
image:oliver006/redis-exporter
container_name: redis_exporter
restart: always
environment:
REDIS_ADDR:"localhost:6379"
REDIS_PASSWORD: 123456
port:
- "9113:9113"
EOF
启动
docker-compose up -d
检查
查看正在运行的容器
docker ps
或者:
查看redis_exporter容器的运行日志
docker logs -f redis_exporter
参数解释
| Environment variable | 值 | 描述 |
|---|---|---|
| REDIS_ADDR | localhost:6379 | redis服务地址 |
| REDIS_PASSWORD | 123456 | redis服务密码 |
metrics地址
安装好Exporter后会暴露一个/metrics结尾的服务
| 名称 | 地址 |
|---|---|
| redis_exporter | http://localhost:9121/metrics |
Prometheus配置
配置Prometheus去采集(拉取)nginx_exporter的监控样本数据
cd /data/docker-prometheus
# 在scrapc_configs(搜刮配置):下面增加如下配置:
cat >prometheus/prometheus.yml <<FOF
- job_name: 'redis_exporter'
static_configs:
- targets: ['localhost:9121']
labels:
instance: test服务器
EOF
重新加载配置
curl -x POST http://localhost:9090/-/reload
检查
常用的redis监控指标
redis_up 服务器是否在线
redis_uptime_in_seconds 运行时长,单位s
rate(redis_cpu_sys_seconds_total[1m])+rate(redis_cpu_user_seconds_total[1m]) 占用CPU核数
redis_memory_used_bytes 占用内存量
redis_memory_max_bytes 限制的最大内存,如果没限制则为0
delta(redis_net_input_bytes_total[1m]) 网络接受的bytes
delta(redis_net_output_bytes_total[1m]) 网络发送的bytes
redis_connected_clients 客户端连接数
redis_connected_clients / redis_config_maxclients
连接数使用率
redis_rejected_connections_total
拒绝的客户端连接数
redis_connected_slaves
slave连接数
触发器配置
由于之前的触发器是全部写在了一个yml里面就是alert.yam,这样随着后面配置的触发器越来越多最终会变得难以维护.这里我们让它去读rules目录下所有的yml文件即可
Prometheus配置
rule_files:
- "alert.yml"
- "rules/*.yml"
配置redis触发器
因为是单机所以未配置集群的触发器
cat >prometheus/redis.yml <<FOF
groups:
- name: redis
rules:
- alert: RedisDown
expr: redis_up ==0
for: 0m
labels:
severity: critical
annotations:
summary: "redis Down,实例:{{$labels.instance }}"
description: "Redis实例 is down"
- alert: RedisMissingBackup
expr: time() - redis_rdb_last_save_timestamp_seconds > 60 * 60 * 24
for: 0m
labels:
severity: critical
annotations:
summary: "redis 备份丢失,实例:{{$labels.instance }}"
description: "Redis 24小时未备份"
- alert: RedisOutOfConfigredMaxmemory
expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 >90
for: 2m
labels:
severity: warning
annotations:
summary: "redis超出配置的最大内存,实例:{{$labels.instance }}"
description: "Redis内存使用超过配置的最大内存的90%"
- alert: RedisTooManyConnections
expr: redis_connected_clients > 100
for: 2m
labels:
severity: warning
annotations:
summary: "redis连接数过多,实例:{{$labels.instance }}"
description: "Redis当前连接数为:{{ $value }}"
- alert: RedisNotEnoughConnections
expr: redis_connected_clients < 1
for: 2m
labels:
severity: warning
annotations:
summary: "redis没有足够的连接,实例:{{$labels.instance }}"
description: "Redis当前连接数为:{{ $value }}"
- alert: RedisRejectedConnections
expr: increase(redis_rejected_connections_total[1m]) > 0
for: 0m
labels:
severity: critical
annotations:
summary: "redis有拒绝连接,实例:{{$labels.instance }}"
description: "与Redis的某些连接被拒绝:{{ $value }}"
EOF
一定记住这里需要仔细校对yaml语法,最好是能去在线验证yaml语法的网站上看看.yaml语法还是比较严格的一点缩进错误都不能有
检查配置
docker exec -it prometheus promtool check config /etc/prometheus/prometheus.yml
重新加载配置
curl -x POST http://localhost:9090/-/reload
检查
http://localhost:9090/alerts?search=
或:
http://localhost:9090/rules
dashboard
grafana展示Prometheus从redis_exporter收集到的数据
id :11835
