使用 Prometheus Agent 模式写入远端 Prometheus

相关介绍

Prometheus Agent 是 Prometheus 在 2.32.0 版本推出的实验性功能，当启用此功能后，将不会在本地文件系统上生成块，并且无法在本地查询。如果网络出现异常状态无法连接至远程端点，数据将暂存在本地磁盘，但是仅限于两个小时的缓冲。
remote-write-receiver 是 Prometheus 在 2.25 版本推出的实验性功能，当启用此功能后，prometheus 可以作为另一个 prometheus 的远程存储。

本文使用 docker-compose 部署相关服务

版本信息：

docker: 20.10.2
docker-compose: 1.29.2
OS: centos 7.9.2009
kernel: 4.18.0-305.10.2.el7.aarch64

部署流程

创建 docker-compose 启动文件

$ cat prometheus-agent-test.yaml
version: "3.9"
services:
  prometheus:
    image: prom/prometheus:v2.32.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'    # 指定配置文件存储路径
      - '--storage.tsdb.path=/prometheus'     # 指定 tsdb 存储路径
      - '--web.enable-lifecycle'    # 允许通过发送 HTTP POST 请求的方式热加载配置文件 `curl -X POST [http://IP/-/reload](http://ip/-/reload)`
      - '--enable-feature=remote-write-receiver'    # 启用试验性功能 remote-write-receiver
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'    # 指定页面组件存储路径
      - '--web.console.templates=/usr/share/prometheus/consoles'    # 指定页面存储路径
    ports:
      - 23335:9090
    volumes:
      - ./prometheus/data:/prometheus
      - ./prometheus/conf:/etc/prometheus
    networks:
      - test

  prometheus-agent:
    image: prom/prometheus:v2.32.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-lifecycle'
      - '--enable-feature=agent'    # 启用试验性功能 agent
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    volumes:
      - ./prometheus-agent/data:/prometheus
      - ./prometheus-agent/conf:/etc/prometheus
    networks:
      - test

networks:
  test:
    external: true

Prometheus 配置文件

$ mkdir -p ./prometheus/{conf,data}
$ cd ./prometheus/conf
$ cat prometheus.yml 
global:                           
  scrape_interval: 15s            
  evaluation_interval: 15s        
  scrape_timeout: 10s             
  external_labels:
    environment: test

Prometheus Agent 配置文件

$ cd ../../
$ mkdir -p ./prometheus-agent/{conf,data}
$ cd ./prometheus-agent/conf
$ cat prometheus.yml 
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    environment: test

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

remote_write:
- url: 'http://prometheus:9090/api/v1/write'

启动

$ cd ../../
$ chmod 777 -R prometheus/
$ chmod 777 -R prometheus-agent/
$ docker-compose -f prometheus-agent-test.yaml up -d

查看

$ docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED         STATUS         PORTS                                      NAMES
a645439377d0   prom/prometheus:v2.32.1   "/bin/prometheus --c…"   4 seconds ago   Up 2 seconds   9090/tcp                                   prometheus_prometheus-agent_1
250ef708325c   prom/prometheus:v2.32.1   "/bin/prometheus --c…"   4 seconds ago   Up 2 seconds   0.0.0.0:23335->9090/tcp                    prometheus_prometheus_1

# Prometheus Agent 日志
$ docker logs -f a645439377d0
ts=2021-12-27T09:25:39.060Z caller=main.go:195 level=info msg="Experimental agent mode enabled."
ts=2021-12-27T09:25:39.060Z caller=main.go:515 level=info msg="Starting Prometheus" version="(version=2.32.1, branch=HEAD, revision=41f1a8125e664985dd30674e5bdf6b683eff5d32)"
ts=2021-12-27T09:25:39.060Z caller=main.go:520 level=info build_context="(go=go1.17.5, user=root@54b6dbd48b97, date=20211217-22:08:06)"
ts=2021-12-27T09:25:39.060Z caller=main.go:521 level=info host_details="(Linux 4.18.0-305.10.2.el7.aarch64 #1 SMP Fri Jul 23 21:19:40 UTC 2021 aarch64 920dcdb94365 (none))"
ts=2021-12-27T09:25:39.061Z caller=main.go:522 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2021-12-27T09:25:39.061Z caller=main.go:523 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2021-12-27T09:25:39.075Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2021-12-27T09:25:39.076Z caller=main.go:980 level=info msg="Starting WAL storage ..."
ts=2021-12-27T09:25:39.120Z caller=db.go:332 level=info msg="replaying WAL, this may take a while" dir=data-agent/wal
ts=2021-12-27T09:25:39.120Z caller=db.go:383 level=info msg="WAL segment loaded" segment=0 maxSegment=0
ts=2021-12-27T09:25:39.120Z caller=main.go:1001 level=info fs_type=EXT4_SUPER_MAGIC
ts=2021-12-27T09:25:39.121Z caller=main.go:1004 level=info msg="Agent WAL storage started"
ts=2021-12-27T09:25:39.121Z caller=main.go:1129 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2021-12-27T09:25:39.122Z caller=dedupe.go:112 component=remote level=info remote_name=dc2753 url=http://prometheus:9090/api/v1/write msg="Starting WAL watcher" queue=dc2753
ts=2021-12-27T09:25:39.122Z caller=dedupe.go:112 component=remote level=info remote_name=dc2753 url=http://prometheus:9090/api/v1/write msg="Starting scraped metadata watcher"
ts=2021-12-27T09:25:39.122Z caller=main.go:1166 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.680778ms db_storage=1.081µs remote_storage=828.171µs web_handler=1.163µs query_engine=650ns scrape=323.645µs scrape_sd=42.589µs notify=1.7µs notify_sd=2.639µs rules=400ns
ts=2021-12-27T09:25:39.122Z caller=main.go:897 level=info msg="Server is ready to receive web requests."
ts=2021-12-27T09:25:39.124Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2021-12-27T09:25:39.124Z caller=dedupe.go:112 component=remote level=info remote_name=dc2753 url=http://prometheus:9090/api/v1/write msg="Replaying WAL" queue=dc2753
ts=2021-12-27T09:25:57.516Z caller=dedupe.go:112 component=remote level=info remote_name=dc2753 url=http://prometheus:9090/api/v1/write msg="Done replaying WAL" duration=18.391770521s

# Prometheus 日志
$ docker logs -f 250ef708325c
ts=2021-12-27T09:25:39.149Z caller=main.go:177 level=info msg="Experimental remote-write-receiver enabled"
ts=2021-12-27T09:25:39.149Z caller=main.go:478 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2021-12-27T09:25:39.149Z caller=main.go:515 level=info msg="Starting Prometheus" version="(version=2.32.1, branch=HEAD, revision=41f1a8125e664985dd30674e5bdf6b683eff5d32)"
ts=2021-12-27T09:25:39.150Z caller=main.go:520 level=info build_context="(go=go1.17.5, user=root@54b6dbd48b97, date=20211217-22:08:06)"
ts=2021-12-27T09:25:39.150Z caller=main.go:521 level=info host_details="(Linux 4.18.0-305.10.2.el7.aarch64 #1 SMP Fri Jul 23 21:19:40 UTC 2021 aarch64 920dcdb94365 (none))"
ts=2021-12-27T09:25:39.150Z caller=main.go:522 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2021-12-27T09:25:39.150Z caller=main.go:523 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2021-12-27T09:25:39.152Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2021-12-27T09:25:39.153Z caller=main.go:924 level=info msg="Starting TSDB ..."
ts=2021-12-27T09:25:39.168Z caller=head.go:488 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2021-12-27T09:25:39.168Z caller=head.go:522 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=8.439µs
ts=2021-12-27T09:25:39.168Z caller=head.go:528 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2021-12-27T09:25:39.169Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2021-12-27T09:25:39.169Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
ts=2021-12-27T09:25:39.169Z caller=head.go:605 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=74.352µs wal_replay_duration=1.21529ms total_replay_duration=1.314955ms
ts=2021-12-27T09:25:39.171Z caller=main.go:945 level=info fs_type=EXT4_SUPER_MAGIC
ts=2021-12-27T09:25:39.171Z caller=main.go:948 level=info msg="TSDB started"
ts=2021-12-27T09:25:39.171Z caller=main.go:1129 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2021-12-27T09:25:39.172Z caller=main.go:1166 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=691.44µs db_storage=1.512µs remote_storage=1.844µs web_handler=463ns query_engine=1.376µs scrape=414.021µs scrape_sd=10.649µs notify=1.256µs notify_sd=2.381µs rules=3.884µs
ts=2021-12-27T09:25:39.172Z caller=main.go:897 level=info msg="Server is ready to receive web requests."

打开 web 查看

image.png

其它

在实际生产中，需要按照具体情况调节参数
例如

remote_write:
- url: 'http://prometheus:9090/api/v1/write'
  remote_timeout: 30s
  follow_redirects: true
  queue_config:
    capacity: 2500
    max_shards: 200
    min_shards: 1
    max_samples_per_send: 500
    batch_send_deadline: 5s
    min_backoff: 30ms
    max_backoff: 5s
  metadata_config:
    send: true
    send_interval: 1m
    max_samples_per_send: 500

---

转载请注明出处