prometheus之docker监控与告警系列（一）

序

本系列主要介绍prometheus+cadvisor+alertmanager打造docker监控，主要监控指定docker容器是否挂掉

本节主要熟悉prometheus的部署和基本使用

一、部署环境

mac

二、下载prometheus的mac版

进入下载页，操作系统选择darwin

https://prometheus.io/download/

download.png

三、解压，进入目录运行

./prometheus --config.file=prometheus.yml

四、浏览监控页面，查看对自身监控的一些信息

图表页： http://localhost:9090/graph
metrics页面： http://localhost:9090/metrics

1）查看控制台日志输出
切换到Console尝试输入表达式，并点击execute，查看结果

prometheus_target_interval_length_seconds

image.png

count(prometheus_target_interval_length_seconds)

image.png

更多表达式参考：expression language documentation

2）查看图表输出

切换到Graph尝试输入表达式，并点击execute，查看结果

rate(prometheus_tsdb_head_chunks_created_total[1m])

image.png

五、做点实验

1）确保已经安装了go的开发环境，并配置了环境变量，golang的protobuf依赖是最新的

# Fetch the client library code and compile example.
git clone https://github.com/prometheus/client_golang.git
cd client_golang/examples/random
go get -d
go build

# Start 3 example targets in separate terminals:
./random -listen-address=:8080
./random -listen-address=:8081
./random -listen-address=:8082

接下来分别打开以下网址查看metrics

http://localhost:8080/metrics
http://localhost:8081/metrics
http://localhost:8082/metrics

2）修改prometheus.yml

scrape_configs:
  - job_name:       'example-random'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']
        labels:
          group: 'production'

      - targets: ['localhost:8082']
        labels:
          group: 'canary'

打开网址 http://localhost:9090/
输入以下条件过滤，查看console

rpc_durations_seconds

image.png

3）根据给定规则输出监控内容
新增prometheus.rules.yml

groups:
- name: example
  rules:
  - record: job_service:rpc_durations_seconds_count:avg_rate5m
    expr: avg(rate(rpc_durations_seconds_count[5m])) by (job, service)

修改prometheus.yml指定prometheus.rules.yml

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

  # Attach these extra labels to all timeseries collected by this Prometheus instance.
  external_labels:
    monitor: 'codelab-monitor'

rule_files:
  - 'prometheus.rules.yml'

scrape_configs:
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

  - job_name:       'example-random'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']
        labels:
          group: 'production'

      - targets: ['localhost:8082']
        labels:
          group: 'canary'

在localhost:9090的console查看新规则的聚合数据

job_service:rpc_durations_seconds_count:avg_rate5m

image.png

参考：https://prometheus.io/docs/prometheus/latest/getting_started/

欢迎继续阅读：

prometheus之docker监控与告警系列（二）

prometheus之docker监控与告警系列（三）