李远君,Web和云计算开发人员。Java,Python,Golang爱好者。
个人网站:https://tomoncle.com
本文作者已加入Python中文社区专栏作者计划。
Prometheus 是源于 Google Borgmon 的一个开源监控系统,用 Golang 开发。被很多人称为下一代监控系统。 Prometheus 基本原理是通过 HTTP 协议周期性抓取被监控组件的状态,这样做的好处是任意组件只要提供 HTTP 接口就可以接入监控系统,不需要任何 SDK.(注意:prometheus使用UTC世界标准时间进行数据记录)
这几年一直在做云相关的工作,所以一直会有监控的业务场景,而我们常用的监控工具主要就是rometheus这款开源工具,当然Prometheus这个生态圈已经非常成熟,但是对于我们某些特定的场景
比如就获取虚拟机或容器上下线,这些我们一般是在PaaS 容器列表或IaaS 虚拟机列表中查看,他原生的UI还满足不了我们的需求,所以需要我们自己去通过Http接口查询,再举例,假如我们不用grafana这种组件显示,而是自定制自己UI监控页面,http接口查询更是必须实现的,基于此,我才有想写这篇文章。
关于prometheus, 这里我们不做太多的介绍,因为官方文档(https://prometheus.io/)是一个更好的选择,当然在下面我也会分享一些简单的配置。
之前做云监控的时候,因为时间紧任务重,而且当时我们公司的监控引擎已经使用Java书写了一部分,所以我也就是直接在原来的基础上来开发了,但是越到最后,需求越多,越觉得麻烦,因为Java里面实现一个简单的http查询,还要转json(相比于Python实现而言复杂),还要实现对应的http查询接口等等,所以痛定思痛,我就自己闲余时间实现了一款基于Python的Prometheus Http API. 非常简单易用,便于拓展,目前支持 prometheus, node_exporter, mysqld_exporter, consul_exporter, memcached_exporter等等客户端查询,通过Python装饰器实现,如果你对监控没有兴趣,你也可以浏览该项目中装饰器的使用。 官方文档:https://tomoncle.github.io/prometheus-http-client/
prometheus安装:
$ wget https://github.com/prometheus/prometheus/releases/download/v2.8.0-rc.0/prometheus-2.8.0-rc.0.linux-amd64.tar.gz
$ tar zxvf prometheus-2.8.0-rc.0.linux-amd64.tar.gz
默认配置(其他组件配置请看官方文档,安装了其他组件,即可查询对应的客户端):
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - '127.0.0.1:9093'
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first.rules"
# - "second.rules"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
@prom
def go_gc_duration_seconds(self, **kwargs):
pass
@prom
def go_gc_duration_seconds_count(self, **kwargs):
pass
@relabel('100 - (avg by (instance, job) (irate(node_cpu{mode="idle"}[5m])) * 100)')
def node_cpu_rate(self, **kwargs):
pass
@relabel(
'(node_filesystem_size{} - node_filesystem_free{})'
' / (node_filesystem_size{} - node_filesystem_free{} + node_filesystem_avail{})'
' * 100')
def node_disk_rate(self, **kwargs):
pass
参数:
graph
: True/False , 默认 False
start
: timestamp , 假如graph 为 True, 必须提供
end
: timestamp , 假如graph 为 True, 必须提供
step
: int (步长间距) , 默认 None
time
: timestamp , 默认 time.time()
filter
: {'instance':'127.0.0.1', 'label':'go lang', …} , 默认 None
返回json数据:
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from prometheus_http_client import NodeExporter
>>> node_exporter = NodeExporter()
>>> node_exporter.node_cpu_rate(filter={'instance': '127.0.0.1:9100','mode':'user','job':'static_nodes'})
u'{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes"},"value":[1533779495.799,"70.7333333333338"]}]}}'
>>>
>>> node_exporter.node_cpu_rate(filter={'instance': '127.0.0.1:9100','job':'static_nodes'})
u'{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes"},"value":[1533779522.109,"2.2500000000051443"]}]}}'
>>>
从源码安装:
$ git clone https://github.com/tomoncle/prometheus-http-client.git
$ cd prometheus-http-client && sudo python setup.py install
pip安装:
$ pip install prometheus-http-client
默认 : http://localhost:9090
更新 : $ export PROMETHEUS_URL='http://192.168.1.2:9090'
认证 : $ export PROMETHEUS_HEAD='{"Cookie": "123456"}'
参考该目录的其他exporter, 继承 "Exporter" 类即可,不需要任何实现
示例:
prom : 使用该装饰器,自动解析函数名,并根据函数名称自动构建查询表达式,发起HTTP请求,返回json数据
def prom(func):
"""
execute metric request
:param func: exporter method
:return: metric monitor data, json
"""
def wrapper(*args, **kwargs):
graph = kwargs.pop('graph', False)
filters = kwargs.pop('filter', {})
prometheus = Prometheus()
kwargs['metric'] = func.__name__ + _build_params(filters)
function = prometheus.query_rang if graph else prometheus.query
try:
_ignore(args)
return function(**kwargs)
except AssertionError:
raise
except Exception as e:
raise PrometheusError(e)
return wrapper
relabel:该装饰器将显示的使用relabel参数值指定的表达式,这个是基于prom装饰器的灵活实现,供用户来自定义复杂表达式
def relabel(name):
"""
rename metric name
:param name:
:return: metric monitor data, json
"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
func.__name__ = name
filters = kwargs.pop('filter', {})
if '{mode="idle"}' in name and 'mode' not in filters.keys():
filters['mode'] = 'idle'
query = _build_params(filters)
if query:
func.__name__ = name.replace('{mode="idle"}', query).replace('{}', query)
return prom(func)(*args, **kwargs)
return wrapper
return decorator
check_params: 检测函数参数值
def check_params(params):
def decorator(func):
def wrapper(*args, **kwargs):
if isinstance(params, str):
if not kwargs.get(params, None):
raise AssertionError('parameter "{}" is necessary.'.format(params))
elif isinstance(params, list):
for param in params:
if not kwargs.get(param, None):
raise AssertionError('parameter "{}" is necessary.'.format(params))
else:
raise AssertionError('parameter "{}" is necessary.'.format(params))
return func(*args, **kwargs)
return wrapper
return decorator
Prometheus 客户端
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from prometheus_http_client import Prometheus
>>> prometheus = Prometheus()
>>> prometheus.query(metric='irate(node_cpu{job="static_nodes"}[5m])')
u'{"status":"success","data":{"resultType":"vector","result":[{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"idle"},"value":[1533779660.16,"0.9340000000001358"]},{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"iowait"},"value":[1533779660.16,"0.003333333333334091"]},{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"irq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"nice"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"softirq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"steal"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"system"},"value":[1533779660.16,"0.016666666666666666"]},{"metric":{"cpu":"cpu0","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"user"},"value":[1533779660.16,"0.025333333333340608"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"idle"},"value":[1533779660.16,"0.9373333333333297"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"iowait"},"value":[1533779660.16,"0.025333333333333503"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"irq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"nice"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"softirq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"steal"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"system"},"value":[1533779660.16,"0.0073333333333304536"]},{"metric":{"cpu":"cpu1","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"user"},"value":[1533779660.16,"0.02333333333332727"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"idle"},"value":[1533779660.16,"0.9486666666666679"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"iowait"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"irq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"nice"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"softirq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"steal"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"system"},"value":[1533779660.16,"0.009333333333332423"]},{"metric":{"cpu":"cpu2","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"user"},"value":[1533779660.16,"0.0319999999999709"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"idle"},"value":[1533779660.16,"0.9540000000000267"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"iowait"},"value":[1533779660.16,"0.0006666666666670077"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"irq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"nice"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"softirq"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"steal"},"value":[1533779660.16,"0"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"system"},"value":[1533779660.16,"0.008666666666666363"]},{"metric":{"cpu":"cpu3","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","mode":"user"},"value":[1533779660.16,"0.03266666666665211"]}]}}'
>>>
>>> prometheus.query(metric='sum by (mode, instance,job) (irate(node_cpu{job="static_nodes"}[5m]))')
u'{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"system"},"value":[1533779830.339,"0.038000000000010914"]},{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"user"},"value":[1533779830.339,"0.09133333333332606"]},{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"idle"},"value":[1533779830.339,"3.8"]},{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"iowait"},"value":[1533779830.339,"0.035333333333332696"]},{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"irq"},"value":[1533779830.339,"0"]},{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"nice"},"value":[1533779830.339,"0"]},{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"softirq"},"value":[1533779830.339,"0.0006666666666666524"]},{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes","mode":"steal"},"value":[1533779830.339,"0"]}]}}'
>>>
公共的Exporter(https://prometheus.io/)标签
code :
from prometheus_http_client import Exporter
print(Exporter().go_gc_duration_seconds())
data :
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"go_gc_duration_seconds","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","quantile":"0"},"value":[1533734495.25,"0.00002624"]},{"metric":{"__name__":"go_gc_duration_seconds","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","quantile":"0.25"},"value":[1533734495.25,"0.000048674"]},{"metric":{"__name__":"go_gc_duration_seconds","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","quantile":"0.5"},"value":[1533734495.25,"0.000064599"]},{"metric":{"__name__":"go_gc_duration_seconds","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","quantile":"0.75"},"value":[1533734495.25,"0.000084196"]},{"metric":{"__name__":"go_gc_duration_seconds","device_ID":"static_node","instance":"127.0.0.1:9100","job":"static_nodes","quantile":"1"},"value":[1533734495.25,"0.002391278"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9104","job":"mysql","mysql_version":"5.7","quantile":"0"},"value":[1533734495.25,"0.000028267"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9104","job":"mysql","mysql_version":"5.7","quantile":"0.25"},"value":[1533734495.25,"0.000040906"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9104","job":"mysql","mysql_version":"5.7","quantile":"0.5"},"value":[1533734495.25,"0.00005304"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9104","job":"mysql","mysql_version":"5.7","quantile":"0.75"},"value":[1533734495.25,"0.000093971"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9104","job":"mysql","mysql_version":"5.7","quantile":"1"},"value":[1533734495.25,"0.004254732"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9107","job":"consul","mysql_version":"5.7","quantile":"0"},"value":[1533734495.25,"0.00006202700000000001"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9107","job":"consul","mysql_version":"5.7","quantile":"0.25"},"value":[1533734495.25,"0.000198266"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9107","job":"consul","mysql_version":"5.7","quantile":"0.5"},"value":[1533734495.25,"0.00020824000000000003"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9107","job":"consul","mysql_version":"5.7","quantile":"0.75"},"value":[1533734495.25,"0.000271729"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9107","job":"consul","mysql_version":"5.7","quantile":"1"},"value":[1533734495.25,"0.001790822"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","quantile":"0"},"value":[1533734495.25,"0.000025387"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","quantile":"0.25"},"value":[1533734495.25,"0.000037497"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","quantile":"0.5"},"value":[1533734495.25,"0.000060197"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","quantile":"0.75"},"value":[1533734495.25,"0.000111523"]},{"metric":{"__name__":"go_gc_duration_seconds","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","quantile":"1"},"value":[1533734495.25,"0.000544621"]}]}}
node_exporter
code :
from prometheus_http_client import NodeExporter
node_exporter = NodeExporter()
# print(node_exporter.node_uname_info())
print(node_exporter.node_cpu_rate(filter={'instance': '127.0.0.1:9100'}))
data :
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"127.0.0.1:9100","job":"static_nodes"},"value":[1533734848.099,"11.816666666676014"]}]}}
mysqld_exporter
code :
from prometheus_http_client import MysqlExporter
print(MysqlExporter().mysql_exporter_scrapes_total())
data :
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"mysql_exporter_scrapes_total","instance":"127.0.0.1:9104","job":"mysql","mysql_version":"5.7"},"value":[1533735043.46,"345"]}]}}
consul_exporter
code :
from prometheus_http_client import ConsulExporter
print(ConsulExporter().consul_catalog_services())
data :
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"consul_catalog_services","instance":"127.0.0.1:9107","job":"consul","mysql_version":"5.7"},"value":[1533735232.039,"1"]}]}}
memcached_exporter
code :
from prometheus_http_client import MemcachedExporter
print(MemcachedExporter().memcached_commands_total())
data :
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"memcached_commands_total","command":"cas","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"badval"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"cas","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"cas","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"miss"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"decr","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"decr","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"miss"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"delete","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"delete","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"miss"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"flush","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"get","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"get","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"miss"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"incr","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"incr","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"miss"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"set","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"touch","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"hit"},"value":[1533735286.809,"0"]},{"metric":{"__name__":"memcached_commands_total","command":"touch","instance":"127.0.0.1:9150","job":"memcached","mysql_version":"5.7","status":"miss"},"value":[1533735286.809,"0"]}]}}
zhi
支
chi
持
zuo
作
zhe
者
长按扫码鼓励作者
本文作者已加入Python中文社区专栏作者计划
▼ 点击下方阅读原文
免费成为社区注册会员,会员可以享受更多权益