plg（Loki+Promtail+Grafana）监控nginx日志、messages日志监控平台

Loki 是一个水平可扩展，高可用性，多租户的日志聚合系统 。它的设计非常经济高效且易于操作，因为它不会为日志内容编制索引，而是为每个日志流编制一组标签。

Loki 组成：

    loki : 主服务器，负责存储日志和处理查询

    promtail : 代理，负责收集日志并将其发送给 loki

    Grafana : Go语言开发的开源数据可视化工具，可以做数据监控和数据统计，带有告警功能

登录官网：loki官网Like Prometheus, but for logs. Contribute to grafana/loki development by creating an account on GitHub.https://github.com/grafana/loki/releases/

一、loki安装配置

1.1 环境准备

----root用户操作
###创建用户
useradd loki
passwd loki

###创建安装目录
mkdir /opt/loki


###授权
chown -R loki:loki /opt/loki
chmod -R 755 /opt/loki

----loki用户操作
###下载二进制包
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.4/loki-linux-amd64.zip"

###解压二进制包
$ unzip "loki-linux-amd64.zip"

### make sure it is executable
$ chmod a+x "loki-linux-amd64"

1.2 编写配置文件

$ vim loki-local-config.yml

更多配置请参考：https://grafana.com/docs/loki/latest/configuration/

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 10m
  chunk_retain_period: 30s
schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 168h
storage_config:
  boltdb:
    directory: /opt/loki/index
  filesystem:
    directory: /opt/loki/chunks  # 块存储路径

limits_config:
  enforce_metric_name: false
  reject_old_samples: true          # 是否拒绝老样本
  reject_old_samples_max_age: 168h  # 168小时之前的样本将会被删除
  ingestion_rate_mb: 200
  ingestion_burst_size_mb: 300
  per_stream_rate_limit: 1000MB
  max_entries_limit_per_query: 10000
chunk_store_config:
  max_look_back_period: 168h        # 为避免查询超过保留期的数据，必须小于或等于下方的时间值
table_manager:
  retention_deletes_enabled: true   # 保留删除开启
  retention_period: 168h            # 超过168h的块数据将被删除

ruler:
  storage:
    type: local
    local:
      directory: /opt/loki/rules
  rule_path: /opt/loki/rules-temp
  alertmanager_url: http://192.168.0.1:9093    # alertmanager地址
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  enable_alertmanager_v2: true

1.3 编写启动脚本

$ vim restart-loki.sh

#!/bin/bash
echo "stop loki"
ps -ef | grep loki-linux-amd64 | grep -v grep | awk '{print $2}'| xargs kill -9 

echo "Begin start loki"
sleep 1
str=$"\n"
nohup ./loki-linux-amd64 --config.file=loki-local-config.yml &
sstr=$(echo -e $str)
echo $sstr

1.4 增加执行权限

$chmod +x restart-loki.sh

1.5 loki的启动

cd /opt/loki
./restart-loki.sh

1.6 loki日志查看

[root@localhost loki]# tail -10f nohup.out 
level=info ts=2023-03-26T08:40:48.324631438Z caller=checkpoint.go:614 msg="starting checkpoint"
level=info ts=2023-03-26T08:40:48.324805699Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=wal/checkpoint.000022
level=info ts=2023-03-26T08:45:18.327347566Z caller=checkpoint.go:501 msg="atomic checkpoint finished" old=wal/checkpoint.000022.tmp new=wal/checkpoint.000022
level=info ts=2023-03-26T08:45:18.32797424Z caller=checkpoint.go:572 msg="checkpoint done" time=4m30.002891425s
level=info ts=2023-03-26T08:45:48.325049331Z caller=checkpoint.go:614 msg="starting checkpoint"
level=info ts=2023-03-26T08:45:48.32524561Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=wal/checkpoint.000023
level=info ts=2023-03-26T08:50:18.327325742Z caller=checkpoint.go:501 msg="atomic checkpoint finished" old=wal/checkpoint.000023.tmp new=wal/checkpoint.000023
level=info ts=2023-03-26T08:50:18.327880793Z caller=checkpoint.go:572 msg="checkpoint done" time=4m30.002463306s
level=info ts=2023-03-26T08:50:48.325234397Z caller=checkpoint.go:614 msg="starting checkpoint"
level=info ts=2023-03-26T08:50:48.325402194Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=wal/checkpoint.000024

二、日志代理Promtail安装和配置

负责收集日志并将其发送给 loki，在需要采集日志的主机上都安装Promtail

2.1 Promtail环境准备

mkdir /opt/promtail
chown -R loki:loki /opt/promtail
chmod -R loki:loki /opt/promtail

###下载
curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.4/promtail-linux-amd64.zip"

###解压
unzip loki-linux-amd64.zip # 解压

2.2 编写Promtail配置文件

vim promtail-local-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /opt/promtail/positions.yaml

clients:
  - url: http://10.X.X.125:3100/loki/api/v1/push # 填写好Loki地址

scrape_configs:
- job_name: messagelog
  static_configs:
  - targets:
      - localhost
    labels:
      #job: messagelog
      host: namenode01
      __path__: /var/log/messages

- job_name: agentlog
  static_configs:
  - targets:
      - localhost
    labels:
      #job: agentlog
      host: namenode01
      __path__: /ssd/ssd0/agentlog/agent.log

- job_name: datanodelog
  static_configs:
  - targets:
      - localhost
    labels:
#      #job: datanodelog
      host: namenode01
      __path__: /ssd/ssd0/datanodelog/*datanode*.log

2.3 编写promtail启动脚本

vi restart-promtail.sh

#!/bin/bash
echo "Begin stop promtail"
ps -ef | grep promtail-linux-amd64 | grep -v grep | awk '{print $2}' | xargs kill -9

echo "Begin start promtail...."
nohup ./promtail-linux-amd64 --config.file=promtail-local-config.yml > ./promtail-9080.log 2>&1 &

2.4 增加执行权限

$chmod +x restart-promtail.sh

2.5 启动promtail服务

[root@localhost promtail]# cd /opt/promtail

[root@localhost promtail]# ./restart-promtail.sh 
Begin stop promtail
Begin start promtail....
[root@localhost promtail]# 

三、 Grafana可视化安装和配置

    Grafana : Go语言开发的开源数据可视化工具，可以做数据监控和数据统计，带有告警功能

3.1 Grafana安装配置

Download Grafana | Grafana LabsOverview of how to download and install different versions of Grafana on different operating systems.https://grafana.com/grafana/download

###CentOS系统通过rpm方式安装

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.4.3-1.x86_64.rpm

sudo yum install grafana-enterprise-9.4.3-1.x86_64.rpm -y


[root@localhost grafana]# rpm -qa|grep grafana
grafana-enterprise-9.4.3-1.x86_64
[root@localhost grafana]# 

3.2 启动grafana

grafana会占用服务器3000端口

##启动
systemctl start grafana-server.service

##停止
systemctl stop grafana-server.service

##重启
systemctl restart grafana-server.service

##设置开机自启动
systemctl enable grafana-server.service

##状态查看
systemctl status grafana-server.service

3.3 grafana导入nginx dashboard

Grafana Loki Dashboard for NGINX Service Mesh | Grafana LabsLoki v2+ showcase using JSON NGINX access logs.https://grafana.com/grafana/dashboards/12559-grafana-loki-dashboard-for-nginx-service-mesh/

3.4 grafana配置告警邮箱

编辑Grafana配置文件，设置发件人
docker安装的Grafana：vim /home/monitor/grafana/config/grafana.ini
yum安装：vim /etc/grafana/grafana.ini

在[smtp] 标签下 修改配置

# 配置邮件服务器
[smtp]
enabled = true

# 发件服务器
host = smtp.mail.139.com:465

# smtp 密码
password = PAASmail2019

# smtp账号
user = [email protected]

# 发信邮箱
from_address = [email protected]

# 发信人
from_name = Grafana

保存好配置文件之后，重启Grafana

##重启
systemctl restart grafana-server.service

3.5 测试grafana告警发送

点击页面左边的铃铛，并选择 Contact points

然后选择 Add contact point 创建一个 contact point,填写 Name 和收件人邮箱地址

然后点击Test ，提示成功后，刚填写的邮箱应该就可以收到如下内容的测试邮件 

查看邮箱，成功收到邮件

 

参考：简易日志系统LPG生产环境实践指南 | 坑我已经帮你们踩好了

参考：PLG日志平台搭建: Promtail + Loki + Grafana 全步骤_promtail loki_dxccccccccccc的博客-CSDN博客

参考：Promtail + Loki + Grafana 构建日志监控告警系统（一）_promtail+loki_5D金豆豆的博客-CSDN博客

参考:

PLG日志平台搭建: Promtail + Loki + Grafana 全步骤_promtail loki_dxccccccccccc的博客-CSDN博客

参考：

搭建Loki、Promtail、Grafana轻量级日志系统（centos7）_loki promtail_成钰的博客-CSDN博客

参考：

轻量日志收集系统loki_loki日志收集_wyl9527的博客-CSDN博客

参考：

轻量级日志可视化平台Grafana Loki接入nginx访问日志-阿里云开发者社区

参考：

PLG 云原生日志系统在压测中的落地实践 · TesterHome

参考：

快速部署Grafana日志监控+Nginx封禁IP-阿里云开发者社区

参考：

使用使用grafana中文插件-阿里云开发者社区