Promtail+Loki+Grafana搭建轻量级日志管理平台

1. 简介

  现在几乎所有的服务都是分布式部署,在定位环境问题时,往往需要在不同的主机上查看日志,并且对于某些行业来说,客户不能接受通过远程登录服务器查看日志。因此,需要一个统一的日志管理平台,可以集中查看日志,并对日子进行过滤、检索、定期清除等。

2. 技术选型

  集成日志采集、存储、展示功能的管理平台有很多成熟的解决方案,例如:Logstash + Elasticsearch + KibanaFileBeat/Fluentd + Elasticsearch + KibanaPromtail + Loki + Grafana
  如果需求复杂,服务器资源不受限制,推荐使用ELK(Logstash + Elasticsearch + Kibana)方案;如果需求仅是将不同服务器上的日志采集上来集中展示,且需要一个轻量级的框架,那使用PLG(Promtail + Loki + Grafana)最合适不过了。
  各组件官网:

  • ELK/EFK:https://www.elastic.co/cn/elastic-stack/
  • Fluentd::Fluentd | Open Source Data Collector | Unified Logging Layer
  • PLG:https://grafana.com/logs/

3. 相关博客

SpringBoot 2.x + Prometheus + Grafana 实现应用监控

4. 安装

  本博客使用的PromtailLoki版本为:2.6.1,Grafana版本为:8.0.7。安装环境为:Linux x86_64。

  • 下载介质和配置文件
组件 下载地址 配置文件地址
Promtail https://github.com/grafana/loki/releases/download/v2.6.1/promtail-linux-arm64.zip loki/promtail-local-config.yaml at v2.6.1 · grafana/loki · GitHub
Loki https://github.com/grafana/loki/releases/download/v2.6.1/loki-linux-arm64.zip loki/loki-local-config.yaml at v2.6.1 · grafana/loki · GitHub
Grafana https://dl.grafana.com/enterprise/release/grafana-enterprise-8.0.7.linux-amd64.tar.gz
  • 创建安装目录,并将配置文件和介质上传到对应目录
mkdir -p /usr/local/log-manager/promtail
mkdir -p /usr/local/log-manager/loki
mkdir -p /usr/local/log-manager/grafana
  • 修改Promtail配置文件
      配置参考官网文档:https://grafana.com/docs/loki/v2.6.x/clients/promtail/configuration/
server:
  http_listen_port: 9080
  grpc_listen_port: 0

# 保存采集日志的位置信息
positions:
  filename: /home/caolei/log-manager/promtail/tmp/positions.yaml

# Loki接收日志接口
clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: tomcat
    pipeline_stages:
	# 合并行配置
    - multiline:
        firstline: ^(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})
        max_wait_time: 3s
        max_lines: 500
    - match:
        selector: '{job="tomcat"}'
        stages:
		# 日志过滤,删除DEBUG级别日志
        - drop:
            expression: ".*DEBUG.*"
    static_configs:
    - targets:
        - localhost
      labels:
        job: tomcat   # 给日志富化的标签
        node: node1   # 给日志富化的标签
		# 采集的日志路径
        __path__: /usr/local/tomcat/logs/catalina.out
		# 排除的日志路径
		__path_exclude__: /usr/local/tomcat/logs/{localhost,catalina}.*.log
  • 修改Loki配置文件
      配置参考官网文档:https://grafana.com/docs/loki/v2.6.x/configuration/
# 需要运行的组件,默认all。测试中发现,定期删除日志记录需要额外指定:table-manager
target: "all,table-manager"

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

# 配置Loki存储数据路径
common:
  path_prefix: /usr/local/loki/data
  storage:
    filesystem:
      chunks_directory: /usr/local/loki/data/chunks
      rules_directory: /usr/local/loki/data/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
	  # 每24小时存储一个索引目录
      index:
        prefix: index_
        period: 24h
      chunks:
        prefix: chunk_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /usr/local/loki/data/boltdb/boltdb-shipper-active
    cache_location: /usr/local/loki/data/boltdb/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
 
compactor:
  working_directory: /usr/local/loki/data/boltdb/boltdb-shipper-compactor
  shared_store: filesystem

# 限制配置
limits_config:
  # 拒绝历史数据开关
  reject_old_samples: true
  # 拒绝历史数据最大时长
  reject_old_samples_max_age: 168h

# 表管理配置
table_manager:
  # 删除历史数据开关
  retention_deletes_enabled: true
  # 删除历史数据时长,不能大于:limits_config.reject_old_samples_max_age
  # 实际上保留时长:(table_manager.retention_period + schema_config.config.chunk.period) <= 保留时长 < (table_manager.retention_period + schema_config.config.chunk.period + 12h)
  retention_period: 168h
# alert-manager组件,不涉及
ruler:
  alertmanager_url: http://localhost:9093
  • 对安装目录授权
chmod -R 755 /usr/local/log-manager/*
  • 启动LokiPromtail一定先启动Loki再启动Promtail
nohup /usr/local/log-manager/loki/loki-linux-amd64 -config.file=/usr/local/log-manager/loki/loki-local-config.yaml >/dev/null 2>/usr/local/log-manager/loki/logs/loki.log 2>&1 &

nohup /usr/local/log-manager/promtail/promtail-linux-amd64 -config.file=/usr/local/log-manager/promtail/promtail-local-config.yaml > /usr/local/log-manager/promtail/logs/promtail.log 2>&1 &
  • 启动Grafana
cd /usr/local/log-manager/grafana && tar -zxvf grafana-enterprise-8.0.7.linux-amd64.tar.gz

nohup /usr/local/log-manager/grafana/grafana-8.0.7/bin/grafana-server > /dev/null 2>&1 &
  • 停止各个组件如果重启了Loki,必须同时重启Promtail。如果仅重启Promtail,可以不重启Loki
# 停止Promtail
ps -ef | grep promtail | grep promtail-local-config.yaml | grep -v grep | awk '{print $2}'

kill -9 xxx

# 停止Loki
ps -ef | grep loki | grep loki-local-config.yaml | grep -v grep | awk '{print $2}'

kill -9 xxx

# 停止Grafana
ps -ef | grep grafana-server

kill -9 xxx

5. 配置Grafana数据源

  • 浏览器访问:http://192.168.0.100:3000

  • 登录:admin/admin

  • Grafana配置数据源

Promtail+Loki+Grafana搭建轻量级日志管理平台_第1张图片

Promtail+Loki+Grafana搭建轻量级日志管理平台_第2张图片

  • 根据富化的标签,检索日志

Promtail+Loki+Grafana搭建轻量级日志管理平台_第3张图片​​​​​​​

  • 查询语法使用的是LogQL语言,学习文档:https://grafana.com/docs/loki/latest/logql/

你可能感兴趣的:(grafana,服务器,linux,elasticsearch,运维)