Prometheus, Grafana and Alertmanager 实战配置

Prometheus时序数据库

系统环境

[ccli@pek tool]$ cat /etc/redhat-release

Red Hat Enterprise Linux Server release 7.1 (Maipo)

[ccli@pek tool]$ uname -a

Linux pek 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 6 17:12:08 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

软件下载源

https://github.com/prometheus/prometheus/releases/

https://github.com/prometheus/node_exporter/releases

https://grafana.com/grafana/download?platform=linux

https://prometheus.io/download/

Prometheus 部署

下载/安装

wget https://github.com/prometheus/prometheus/releases/download/v2.7.0/prometheus-2.7.0.linux-amd64.tar.gz

tar zxvf prometheus-2.7.0.linux-amd64.tar.gz

配置

主配置文件ccli.yml

# my global config

global:

scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# scrape_timeout is set to the global default (10s).

# Alertmanager configuration

alerting:

alertmanagers:

- static_configs:

- targets:

- localhost:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

- "node_down.yml"

- "memory_over.yml"

# - "first_rules.yml"

# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

# The job name is added as a label `job=` to any timeseries scraped from this config.

- job_name: 'prometheus'

# metrics_path defaults to '/metrics'

# scheme defaults to 'http'.

static_configs:

- targets: ['localhost:9090']

# The job name is added as a label `job=` to any timeseries scraped from this config.

- job_name: 'testnodeexporter'

# metrics_path defaults to '/metrics'

# scheme defaults to 'http'.

static_configs:

- targets: ['localhost:9100']

Rule Files

node_down.yml

groups:

- name: example

rules:

- alert: InstanceDown

expr: up == 0

for: 1m

labels:

user: caizh

annotations:

summary: "Instance {{ $labels.instance }} down"

description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."

memory_over.yml

groups:

- name: example

rules:

- alert: NodeMemoryUsage

expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes )) / node_memory_MemTotal_bytes * 100 > 80

for: 1m

labels:

user: caizh

annotations:

summary: "{{$labels.instance}}: High Memory usage detected"

description: "{{$labels.instance}}: Memory usage is above 80% (current value is:{{ $value }})"

Node_exporter 部署

下载/安装

wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz

tar zxvf node_exporter-0.17.0.linux-amd64.tar.gz

Grafana 部署

下载/安装

wget https://dl.grafana.com/oss/release/grafana-5.4.3.linux-amd64.tar.gz

tar zxvf grafana-5.4.3.linux-amd64.tar.gz

AlertManagement 部署

下载/安装

wget https://github.com/prometheus/alertmanager/releases/download/v0.16.0/alertmanager-0.16.0.linux-amd64.tar.gz

tar zxvf alertmanager-0.16.0.linux-amd64.tar.gz

配置

global:

smtp_smarthost: 'xxx.email.com:25'

smtp_from: '[email protected]'

smtp_auth_username: '[email protected]'

smtp_auth_password: 'xxxxxx'

smtp_require_tls: false

route:

group_by: ['alertname']

group_interval: 1m

repeat_interval: 1h

receiver: live-monitoring

receivers:

- name: 'live-monitoring'

email_configs:

- to: '[email protected]'