grafana tempo 分布式追踪框架应用

环境准备（基于docker-compose 运行）

docker-compose 文件

version: "3"
services:
  tempo:
    image: grafana/tempo:latest
    command: 
      - "-storage.trace.backend=local"# tell tempo where to permanently put traces
      - "-storage.trace.local.path=/tmp/tempo/traces"
      - "-storage.trace.wal.path=/tmp/tempo/wal"# tell tempo where to store the wal
      - "-auth.enabled=false"# disables the requirement for the X-Scope-OrgID header
      - "-server.http-listen-port=3100"
    volumes:
      - ./example-data/tempo:/tmp/tempo
    ports:
      - "14268"# jaeger ingest
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  tempo-query:
    image: grafana/tempo-query:latest
    command: ["--grpc-storage-plugin.configuration-file=/etc/tempo-query.yaml"]
    volumes:
      - ./etc/tempo-query.yaml:/etc/tempo-query.yaml
    ports:
      - "16686:16686"# jaeger-ui
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  minio:
    image: minio/minio
    environment:
      - MINIO_ACCESS_KEY=tempo
      - MINIO_SECRET_KEY=supersecret
    command:  minio server /data
    ports:
      - "9000:9000"
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./etc/prometheus.yaml:/etc/prometheus.yaml
    entrypoint:
      - /bin/prometheus
      - --config.file=/etc/prometheus.yaml
    ports:
      - "9090:9090"
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  synthetic-load-generator:
    image: omnition/synthetic-load-generator:1.0.25
    volumes:
      - ./etc/load-generator.json:/etc/load-generator.json
    environment:
      - TOPOLOGY_FILE=/etc/load-generator.json
      - JAEGER_COLLECTOR_URL=http://tempo:14268
  grafana:
    image: grafana/grafana:7.3.6
    volumes:
      - ./example-data/datasources:/etc/grafana/provisioning/datasources
      - ./example-data/dashboards-provisioning:/etc/grafana/provisioning/dashboard
    ports:
      - "3000:3000"
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  loki:
    image: grafana/loki:2.1.0
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"
    environment:
      - JAEGER_AGENT_HOST=tempo
      - JAEGER_ENDPOINT=http://tempo:14268/api/traces # send traces to Tempo
      - JAEGER_SAMPLER_TYPE=const
      - JAEGER_SAMPLER_PARAM=1
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'

• 依赖的组件说明
  依赖了loki，grafana，prometheus，synthetic-load-generator（生成metrics）
tempo 包含的组件（engine，以及query），容器的日志基于了loki plugin,相关prometheus
以及grafana datasource 还有dashboard 配置，参考github 配置

运行

首先安装docker loki log plugin

docker plugin install \
grafana/loki-docker-driver:latest \
--alias loki \
--grant-all-permissions

运行

docker-compose up -

效果

注意需要输入一个traceid，可以同loki 查找:

说明

grafana 的tempo是一个很不错的工具，至少目前来看比使用jaeger的技术原生方案至少会简化好多，当然tempo 也是在巨人的肩膀上成长的,grafana 的agent 也是一个很不错的工具，集成了log，metrics，trace 好处是保证了各种元数据的一致。