thanos 官方主页：https://thanos.io/
thanos 逻辑架构图
thanos 主要组件
- sidecar: 和 prometheus 部署在同一节点，收集 prometheus 的数据，并持久化到对象存储（比如阿里云OSS），同时提供查询接口给 query 查询数据
- query: 查询 sidecar 的数据（一般是两小时内的数据，可配置）和 store gateway 的数据（一般是超过两小时的持久化数据，可配置）
- store gateway: 提供查询接口给 query 来查询在对象存储（比如阿里云OSS）的持久化数据
thanos 系统部署架构图
示例环境
- 节点1（CentOS 7.x / 172.18.255.129 / 119.23.40.213）: promethues + thanos sidecar + thanos store gateway + thanos query
- 节点2（CentOS 7.x / 172.18.255.130 / 120.79.71.229）: promethues + thanos sidecar + thanos store gateway + thanos query
部署过程
1、节点1（172.18.255.129/119.23.40.213）部署 prometheus
# cd /usr/local/src/
# wget https://github.com/prometheus/prometheus/releases/download/v2.28.0/prometheus-2.28.0.linux-amd64.tar.gz
# tar xvf prometheus-2.28.0.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s prometheus-2.28.0.linux-amd64/ prometheus
# cd prometheus
// 这里需要声明 external_labels，给 prometheus 打上标签
// prometheus 高可用集群的第 1 个节点，region 设置为 cn-shenzhen，replica 设置为 1
# vim prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
external_labels:
region: cn-shenzhen
monitor: infrastructure
replica: 1
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
// --web.external-url= 修改为对应节点的对外访问 IP
# vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/usr/local/prometheus/data/ --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h --storage.tsdb.retention.time=2h --storage.tsdb.wal-compression --web.enable-lifecycle --log.level=info --web.listen-address=0.0.0.0:9090 --web.external-url=http://119.23.40.213:9090
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable prometheus && systemctl start prometheus && systemctl status prometheus
2、节点2（172.18.255.130/120.79.71.229）部署 prometheus
# cd /usr/local/src/
# wget https://github.com/prometheus/prometheus/releases/download/v2.28.0/prometheus-2.28.0.linux-amd64.tar.gz
# tar xvf prometheus-2.28.0.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s prometheus-2.28.0.linux-amd64/ prometheus
# cd prometheus
// 这里需要声明 external_labels，给 prometheus 打上标签
// prometheus 高可用集群的第 2 个节点，region 设置为 cn-shanghai，replica 设置为 2。
# vim prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
external_labels:
region: cn-shanghai
monitor: infrastructure
replica: 2
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
// --web.external-url= 修改为对应节点的对外访问 IP
# vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/usr/local/prometheus/data/ --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h --storage.tsdb.retention.time=2h --storage.tsdb.wal-compression --web.enable-lifecycle --log.level=info --web.listen-address=0.0.0.0:9090 --web.external-url=http://120.79.71.229:9090
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable prometheus && systemctl start prometheus && systemctl status prometheus
3、节点1（172.18.255.129/119.23.40.213）部署 thanos，并配置组件 thanos sidecar 和 thanos store gateway
# cd /usr/local/src/
# wget https://github.com/thanos-io/thanos/releases/download/v0.21.1/thanos-0.21.1.linux-amd64.tar.gz
# tar xvf thanos-0.21.1.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s thanos-0.21.1.linux-amd64/ thanos
# cd thanos
// 配置 thanos sidecar，实现收集 promethues 高可用集群的第 1 个节点的监控数据，并持久化到阿里云 oss
// 阿里云 oss bucket 请自行开通并配置好 access_key_id 和 access_key_secret，后面的步骤都是使用这个 bucket
# vim aliyun-oss.yaml
type: ALIYUNOSS
config:
endpoint: "oss-cn-shenzhen-internal.aliyuncs.com"
bucket: "prometheus-persistent-data"
access_key_id: "XXX"
access_key_secret: "XXX"
# vim /etc/systemd/system/thanos-sidecar.service
[Unit]
Description=Thanos Sidecar
After=network-online.target
[Service]
Restart=on-failure
# --prometheus.url=http://localhost:9090 --> 指定本节点的 prometheus 地址
# --tsdb.path=/usr/local/prometheus/data/ --> 指定本节点的 prometheus 数据目录
ExecStart=/usr/local/thanos/thanos sidecar --prometheus.url=http://localhost:9090 --tsdb.path=/usr/local/prometheus/data/ --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --grpc-address=0.0.0.0:19090 --http-address=0.0.0.0:19091
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable thanos-sidecar && systemctl start thanos-sidecar && systemctl status thanos-sidecar
// 配置 thanos store gateway，实现读取对象存储（比如阿里云OSS）的数据，并提供给 thanos query 查询
# vim /etc/systemd/system/thanos-store-gateway.service
[Unit]
Description=Thanos Store Gateway
After=network-online.target
[Service]
Restart=on-failure
ExecStart=/usr/local/thanos/thanos store --data-dir=/usr/local/thanos/data --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --http-address=0.0.0.0:10902 --grpc-address=0.0.0.0:10901
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable thanos-store-gateway && systemctl start thanos-store-gateway && systemctl status thanos-store-gateway
4、节点2（172.18.255.130/120.79.71.229）部署 thanos，并配置组件 thanos sidecar 和 thanos store gateway
// 安装 thanos
# cd /usr/local/src/
# wget https://github.com/thanos-io/thanos/releases/download/v0.21.1/thanos-0.21.1.linux-amd64.tar.gz
# tar xvf thanos-0.21.1.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s thanos-0.21.1.linux-amd64/ thanos
# cd thanos
// 配置 thanos sidecar，实现收集 promethues 高可用集群的第 2 个节点的监控数据，并持久化到阿里云 oss
# vim aliyun-oss.yaml
type: ALIYUNOSS
config:
endpoint: "oss-cn-shenzhen-internal.aliyuncs.com"
bucket: "prometheus-persistent-data"
access_key_id: "XXX"
access_key_secret: "XXX"
# vim /etc/systemd/system/thanos-sidecar.service
[Unit]
Description=Thanos Sidecar
After=network-online.target
[Service]
Restart=on-failure
# --prometheus.url=http://localhost:9090 --> 指定本节点的 prometheus 地址
# --tsdb.path=/usr/local/prometheus/data/ --> 指定本节点的 prometheus 数据目录
ExecStart=/usr/local/thanos/thanos sidecar --prometheus.url=http://localhost:9090 --tsdb.path=/usr/local/prometheus/data/ --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --grpc-address=0.0.0.0:19090 --http-address=0.0.0.0:19091
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable thanos-sidecar && systemctl start thanos-sidecar && systemctl status thanos-sidecar
// 配置 thanos store gateway，实现读取对象存储（比如阿里云OSS）的数据，并提供给 thanos query 查询
# vim /etc/systemd/system/thanos-store-gateway.service
[Unit]
Description=Thanos Store Gateway
After=network-online.target
[Service]
Restart=on-failure
ExecStart=/usr/local/thanos/thanos store --data-dir=/usr/local/thanos/data --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --http-address=0.0.0.0:10902 --grpc-address=0.0.0.0:10901
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable thanos-store-gateway && systemctl start thanos-store-gateway && systemctl status thanos-store-gateway
5、节点1（172.18.255.129/119.23.40.213）配置组件 thanos-query，查询节点 1 和节点 2 的 sidecar、store gateway 数据
# vim /etc/systemd/system/thanos-query.service
[Unit]
Description=Thanos Query
After=network-online.target
[Service]
Restart=on-failure
# --store=172.18.255.129:19090 --> 请求当前节点 thanos sidecar 数据
# --store=172.18.255.130:19090 --> 请求另外一个节点 thanos sidecar 数据
# --store=172.18.255.129:10901 --> 请求当前节点 thanos store gateway 数据
# --store=172.18.255.130:10901 --> 请求另外一个节点 thanos store gateway 数据
# --query.replica-label "replica" --query.replica-label "region" --> 加上后，thanos query 查询同一节点的数据时，会自动去重
ExecStart=/usr/local/thanos/thanos query --http-address=0.0.0.0:19192 --grpc-address 0.0.0.0:19092 --store=172.18.255.129:19090 --store=172.18.255.130:19090 --store=172.18.255.129:10901 --store=172.18.255.130:10901 --query.replica-label "replica" --query.replica-label "region"
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable thanos-query && systemctl start thanos-query && systemctl status thanos-query
实现的效果：
6、节点2（172.18.255.130/120.79.71.229）配置组件 thanos-query，查询节点 1 和节点 2 的 sidecar、store gateway 数据
# vim /etc/systemd/system/thanos-query.service
[Unit]
Description=Thanos Query
After=network-online.target
[Service]
Restart=on-failure
# --store=172.18.255.130:19090 --> 请求当前节点 thanos sidecar 数据
# --store=172.18.255.129:19090 --> 请求另外一个节点 thanos sidecar 数据
# --store=172.18.255.130:10901 --> 请求当前节点 thanos store gateway 数据
# --store=172.18.255.129:10901 --> 请求另外一个节点 thanos store gateway 数据
# --query.replica-label "replica" --query.replica-label "region" --> 加上后，thanos query 查询同一节点的数据时，会自动去重
ExecStart=/usr/local/thanos/thanos query --http-address=0.0.0.0:19192 --grpc-address 0.0.0.0:19092 --store=172.18.255.130:19090 --store=172.18.255.129:19090 --store=172.18.255.130:10901 --store=172.18.255.129:10901 --query.replica-label "replica" --query.replica-label "region"
[Install]
WantedBy=multi-user.target
# systemctl daemon-reload && systemctl enable thanos-query && systemctl start thanos-query && systemctl status thanos-query
实现的效果：
7、在负载均衡服务（比如 HAProxy、阿里云 SLB 等）配置两个节点的 thanos query ，就可以实现 thanso + prometheus 的高可用了，后续的查询操作都通过 thanos query 进行
其他
1、prometheus 的配置参数 external_labels 和 thanos query 启动参数 --query.replica-label "replica" --query.replica-label "region" 的作用
2、thanos 只会保留其中一个节点上传的持久化监控数据（根据配置，要 2 小时后才会上传到阿里云 OSS）
3、除了实现 prometheus 高可用场景，thanos 也可以用作多个 prometheus 的汇总查询入口，只需部署 thanos sidecar 和 thanos store gateway 到相应的 prometheus 节点，然后 thanos query 通过参数 --store=ip:port 连接 thanos sidecar 和 thanos store gateway 即可