prometheus数据持久化设计

一、prometheus数据持久化设计

1、需求

1分钟级别的数据保留 10天,10分钟级别的数据保留20天,1小时级别的数据保留30天,1天级别的保留3个月

2、方案

1、创建4个数据库,分别存储不同频率的数据,prometheus:原始数据 、prometheus_10m:按10分钟频率的数据、prometheus_1h:按1小时频率的数据、prometheus_1d:按天频率的数据

2、创建CQ,统计上级频率数据并写入相应频率的数据库中;10m由原始数据统计,1h由10m数据统计,1d数据由1h数据统计

3、开发influxdb remote read adapter,根据查询的开始时间来确定从哪个数据库查数据

3、db设计

prometheus , 数据保留:15d,分区:1d

prometheus_10m , 数据保留:20d,分区:1d

prometheus_1h , 数据保留:30d,分区:1d

prometheus_1d , 数据保留:730d,分区:7d

4、相应的sql

CREATE DATABASE "prometheus" WITH DURATION 15d SHARD DURATION 1d NAME "prometheus"
CREATE DATABASE "prometheus_10m" WITH DURATION 20d SHARD DURATION 1d NAME "prometheus_10m"
CREATE DATABASE "prometheus_1h" WITH DURATION 30d SHARD DURATION 1d NAME "prometheus_1h"
CREATE DATABASE "prometheus_1d" WITH DURATION 730d SHARD DURATION 7d NAME "prometheus_1d"

 

5、CQ

 

CREATE CONTINUOUS QUERY "cq_prometheus_10m" ON "prometheus" BEGIN SELECT mean(value) as "value" INTO "prometheus_10m"..:MEASUREMENT FROM prometheus../.*/ GROUP BY time(10m),* END 

 

CREATE CONTINUOUS QUERY "cq_prometheus_1h" ON "prometheus_10m" BEGIN SELECT mean(value) as "value" INTO "prometheus_1h"..:MEASUREMENT FROM prometheus_10m../.*/ GROUP BY time(1h),* END 

 

CREATE CONTINUOUS QUERY "cq_prometheus_1d" ON "prometheus_1h" BEGIN SELECT mean(value) as "value" INTO "prometheus_1d"..:MEASUREMENT FROM prometheus_1h../.*/ GROUP BY time(1d),* END 

adapter

主要是做http请求转发,基于prometheus remote read的请求中Query的StartTimestampMs来确定/api/v1/prom/read?db=%s最终db调用哪一个

StartTimestampMs在7天内转发到/api/v1/prom/read?db=prometheus

StartTimestampMs在7-15天内转发到/api/v1/prom/read?db=prometheus_10m

StartTimestampMs在15-30天内转发到/api/v1/prom/read?db=prometheus_1h

StartTimestampMs在30天以上转发到/api/v1/prom/read?db=prometheus_1d

 

你可能感兴趣的:(prometheus数据持久化设计)