prometheus 常用指标

一，获取存活节点

http://192.168.0.2:9090/api/v1/query?query=up

二，内存

包含 node-exporter  cadvisor 指标

一天平均使用

sum(avg_over_time(container_memory_usage_bytes{container_label_name=~".+"}[1d])) by  (name,instance)

一天最大

sum(max_over_time(container_memory_usage_bytes{instance=~"172.20.70.100.+",container_label_name=~".+"}[1d])) by  (name,instance)

一天最小

sum(min_over_time(container_memory_usage_bytes{instance=~"172.20.70.100.+",container_label_name=~".+"}[1d])) by  (name,instance)

二，cpu

sum(rate(container_cpu_usage_seconds_total{instance=~"172.20.70.2:.+",name=~".+"}[1d])) by (name) 

注意:

在使用 http api 的时候，要对参数进行 urlencode  编码；

def prometheus_query_range(url, sql):

    select_sql = urlencode({"query": sql})

    req_url = "%s/api/v1/query_range?%s&start=%s&end=%s&step=240" % (url, select_sql, before_day, now_time)

    result = request_url(req_url)

    return result

各类指标:

所有磁盘空间

sum(container_fs_limit_bytes{instance=~"$server:.+",device=~".+sd[b-m]"}) by (instance)

已经使用的磁盘空间

sum(container_fs_usage_bytes{instance=~"$server:.+",device=~".+sd[b-m]"}) by (instance)

根目录大小

sum(container_fs_limit_bytes{instance=~"$server:.+",device=~"/dev/mapper/system-root",id="/"}) by (instance)

内存可用大小 包含cache

node_memory_MemAvailable_bytes{instance=~"$server:.+"}

内存总量

node_memory_MemTotal_bytes{instance=~"$server:.+"}

内存空闲 not include cache

node_memory_MemFree_bytes{instance=~"$server:.+"}

cpu util

avg by (instance,mode) (rate(node_cpu_seconds_total{instance=~"$server:.+"}[5m])) * 100

磁盘读速度

rate(node_disk_read_bytes_total{instance=~"$server:.+",device=~"sd[b-m]"}[$interval])

磁盘 read iops

rate(node_disk_read_time_seconds_total{instance=~"$server:.+",device=~"sd[b-m]"}[$interval])

磁盘每秒读占用时间

rate(node_disk_read_time_seconds_total{instance=~"$server:.+",device=~"sd[b-m]"}[$interval])

磁盘 write iops

rate(node_disk_writes_completed_total{instance=~"$server:.+",device=~"sd[b-m]"}[$interval])

磁盘写带宽

rate(node_disk_written_bytes_total{instance=~"$server:.+",device=~"sd[b-m]"}[$interval])

磁盘读每秒占用时间

rate(node_disk_write_time_seconds_total{instance=~"$server:.+",device=~"sd[b-m]"}[$interval])/60

网络

query_result(sum(container_memory_usage_bytes{instance=~"$server:.+",name=~".+"})) by (name)))

sum by (device) (rate(node_network_transmit_bytes_total{instance=~"$server:.+" ,device=~"(bond0)"}[$interval]))

容器层面

容器启动时间

container_start_time_seconds

容器内存限制

sum(container_spec_memory_limit_bytes{instance=~"$server:.+",name=~"$container"}) by (name)

容器cpu限制

 sum(container_spec_cpu_quota{instance=~"$server:.+",name=~"$container"} / container_spec_cpu_period{instance=~"$server:.+",name=~"$container"})  by (name)

容器磁盘读速度

sum(rate(container_fs_reads_bytes_total{instance=~"$server:.+",name="$container"}[$interval])) by (name)

容器写带宽

sum(rate(container_fs_writes_bytes_total{instance=~"$server:.+",name="$container"}[$interval])) by (name)

容器读iops

sum(rate(container_fs_reads_total{instance=~"$server:.+",name=~"$container"}[$interval])) by (name)

容器写iops

sum(rate(container_fs_writes_total{instance=~"$server:.+",name=~"$container"}[$interval])) by (name)

容器占用磁盘空间(不包括挂载)

sum(container_fs_usage_bytes{instance=~"$server:.+",name=~"$container"}) by (name)

容器所有内存使用

sum(container_memory_usage_bytes{instance=~"$server:.+",name=~"$container"}) by (name)

容器内存 rss

sum(container_memory_rss{instance=~"$server:.+",name=~"$container"}) by (name)

容器内存

sum(container_memory_cache{instance=~"$server:.+",name=~"$container"}) by (name)