Prometheus 实战于源码分析之API与联邦
在进行源码讲解关于prometheus还有一些配置和使用，需要解释一下。首先是API的使用，prometheus提供了一套HTTP的接口
curl http://localhost:9090/api/v1/query?query=go_goroutines|python -m json.tool
{
"data": {
"result": [
{
"metric": {
"__name__": "go_goroutines",
"instance": "localhost:9090",
"job": "prometheus"
},
"value": [
1493347106.901,
"119"
]
},
{
"metric": {
"__name__": "go_goroutines",
"instance": "10.39.0.45:9100",
"job": "node"
},
"value": [
1493347106.901,
"13"
]
},
{
"metric": {
"__name__": "go_goroutines",
"instance": "10.39.0.53:9100",
"job": "node"
},
"value": [
1493347106.901,
"11"
]
}
],
"resultType": "vector"
},
"status": "success"
}上面演示一个查询go_goroutines这一个监控指标的数据。让然也可以基于开始时间和截止时间查询，但更强大的功能应该是支持OR查询
[root@slave3 ~]# curl -g 'http://localhost:9090/api/v1/series?match[]=up&match[]=process_start_time_seconds{job="prometheus"}'|python -m json.tool
{
"data": [
{
"__name__": "up",
"instance": "10.39.0.53:9100",
"job": "node"
},
{
"__name__": "up",
"instance": "localhost:9090",
"job": "prometheus"
},
{
"__name__": "up",
"instance": "10.39.0.45:9100",
"job": "node"
},
{
"__name__": "process_start_time_seconds",
"instance": "localhost:9090",
"job": "prometheus"
}
],
"status": "success"
}查询一个系列的数据，当然还可以通过DELETE去删除系列。还记得上一篇说的设置job和targets了吗？也可以通过API查询
curl http://localhost:9090/api/v1/label/job/values
{"status":"success","data":["node","prometheus"]}当然有哪些监控对象也可以查询
curl http://localhost:9090/api/v1/targets|python -m json.tool
{
"data": {
"activeTargets": [
{
"discoveredLabels": {
"__address__": "10.39.0.53:9100",
"__metrics_path__": "/metrics",
"__scheme__": "http",
"job": "node"
},
"health": "up",
"labels": {
"instance": "10.39.0.53:9100",
"job": "node"
},
"lastError": "",
"lastScrape": "2017-04-28T02:47:40.871586825Z",
"scrapeUrl": "http://10.39.0.53:9100/metrics"
},
{
"discoveredLabels": {
"__address__": "10.39.0.45:9100",
"__metrics_path__": "/metrics",
"__scheme__": "http",
"job": "node"
},
"health": "up",
"labels": {
"instance": "10.39.0.45:9100",
"job": "node"
},
"lastError": "",
"lastScrape": "2017-04-28T02:47:45.144032466Z",
"scrapeUrl": "http://10.39.0.45:9100/metrics"
},
{
"discoveredLabels": {
"__address__": "localhost:9090",
"__metrics_path__": "/metrics",
"__scheme__": "http",
"job": "prometheus"
},
"health": "up",
"labels": {
"instance": "localhost:9090",
"job": "prometheus"
},
"lastError": "",
"lastScrape": "2017-04-28T02:47:44.079111193Z",
"scrapeUrl": "http://localhost:9090/metrics"
}
]
},
"status": "success"
}查询这些target。alertmanagers也是通过/api/v1/alertmanagers可以查询的。对应prometheus的本地存储还有一些关键的配置需要注意：
prometheus_local_storage_memory_series：当前的系列数量在内存中保存。
prometheus_local_storage_open_head_chunks：打开头块的数量。
prometheus_local_storage_chunks_to_persist：仍然需要将其持续到磁盘的内存块数。
prometheus_local_storage_memory_chunks：目前在记忆中的块数。如果减去前两个，则可以得到持久化块的数量（如果查询当前没有使用，则它们是可驱动的）。
prometheus_local_storage_series_chunks_persisted：每个批次持续存在块数的直方图。
prometheus_local_storage_rushed_mode如果prometheus斯处于“冲动模式”，则为1，否则为0。可用于计算prometheus处于冲动模式的时间百分比。
prometheus_local_storage_checkpoint_last_duration_seconds：最后一个检查点需要多长时间
prometheus_local_storage_checkpoint_last_size_bytes：最后一个检查点的大小（以字节为单位）。
prometheus_local_storage_checkpointing是1，而prometheus是检查点，否则为0。可以用来计算普罗米修斯检查点的时间百分比。
prometheus_local_storage_inconsistencies_total：找到存储不一致的计数器。如果大于0，请重新启动服务器进行恢复。
prometheus_local_storage_persist_errors_total：反对持续错误。
prometheus_local_storage_memory_dirty_series：当前脏系列数量。
process_resident_memory_bytes广义地说，prometheus进程所占据的物理内存。
go_memstats_alloc_bytes：去堆大小（分配的对象在使用中加分配对象不再使用，但尚未被垃圾回收）。
prometheus还另一个高级应用就是集群联邦，通过定义slave，这样就可以在每个数据中心部署一个，然后通过联邦汇聚。
- scrape_config:
- job_name: dc_prometheus
honor_labels: true
metrics_path: /federate
params:
match[]:
- '{__name__=~"^job:.*"}' # Request all job-level time series
static_configs:
- targets:
- dc1-prometheus:9090
- dc2-prometheus:9090当然如果存储量不够还可以通过分片去采集，
global:
external_labels:
slave: 1 # This is the 2nd slave. This prevents clashes between slaves.
scrape_configs:
- job_name: some_job
# Add usual service discovery here, such as static_configs
relabel_configs:
- source_labels: [__address__]
modulus: 4 # 4 slaves
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^1$ # This is the 2nd slave
action: keep上面定义hash的方式去决定每个prometheus负责的targe他。
- scrape_config:
- job_name: slaves
honor_labels: true
metrics_path: /federate
params:
match[]:
- '{__name__=~"^slave:.*"}' # Request all slave-level time series
static_configs:
- targets:
- slave0:9090
- slave1:9090
- slave3:9090
- slave4:9090下面定义了多个slave。这样数据就可以分片存储了。