Prometheus 监控Nginx 日志 - Kubernetes版本

 

背景

公司有个项目通过Nginx做反向代理，每个服务分配了一个URL(映射关系如下）,公司想统计每个URL的访问总量。

    location /automation-api/ {
        proxy_pass http://serverstatistics-automation-api-dev:9980/;
    }
    location /utilization/ {
        proxy_pass http://serverstatistics-utilization-dev:7000/;
    }
    location /shouquan/ {
        proxy_pass http://serverstatistics-shouquan-dev:5000/;
    }
    location /health_check {
      return 200;
    }

实现思路

1. 通过nginx-log-exporter 把nginx的日志转换为Prometheus可以监控的数据

2. 通过Prometheus采集metrics

3. 在Grafana中查看统计数据

实现步骤

1. 制作nginx-log-exporter的镜像

git clone https://gitee.com/kevinliu_CQ/nginx-log-exporter.git
cd nginx-log-exporter
docker build -t harbor.****.work/serverstatistics/nginx-log-exporter:dev .
docker push harbor.****.work/serverstatistics/nginx-log-exporter:dev

2. 在kubernetes中部署容器，

kubectl apply -f nginx-config.yaml -f deploy.yaml -f service.yaml

3. 在流量器中访问一下运行在nginx中的业务系统，让access log中有数据

4. 查看nginx-log-exporter暴露出来的metrics

[root@server nginx-log]# kubectl get svc -n serverstatistics-dev 
NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
serverstatistics-nginx-dev            NodePort    172.20.125.194           80:38617/TCP,6666:36666/TCP   22d
   
[root@server nginx-log]#curl 10.128.142.43:36666/metrics
nginx_http_upstream_time_seconds_histogram_sum{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/2.ad8fe80d.chunk.css.map HTTP/1.1",request_method="GET",server_port="",status="200",upstream_addr="172.20.87.247:80"} 0.04600000008940697
nginx_http_upstream_time_seconds_histogram_count{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/2.ad8fe80d.chunk.css.map HTTP/1.1",request_method="GET",server_port="",status="200",upstream_addr="172.20.87.247:80"} 2
nginx_http_upstream_time_seconds_histogram_bucket{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/2.ad8fe80d.chunk.css.map HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80",le="50"} 1
nginx_http_upstream_time_seconds_histogram_bucket{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/2.ad8fe80d.chunk.css.map HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80",le="+Inf"} 1
nginx_http_upstream_time_seconds_histogram_sum{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/2.ad8fe80d.chunk.css.map HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80"} 0.0020000000949949026
nginx_http_upstream_time_seconds_histogram_count{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/2.ad8fe80d.chunk.css.map HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80"} 1
nginx_http_upstream_time_seconds_histogram_bucket{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="200",upstream_addr="172.20.87.247:80",le="50"} 5
nginx_http_upstream_time_seconds_histogram_bucket{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="200",upstream_addr="172.20.87.247:80",le="+Inf"} 5
nginx_http_upstream_time_seconds_histogram_sum{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="200",upstream_addr="172.20.87.247:80"} 0.011000000406056643
nginx_http_upstream_time_seconds_histogram_count{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="200",upstream_addr="172.20.87.247:80"} 5
nginx_http_upstream_time_seconds_histogram_bucket{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80",le="50"} 6
nginx_http_upstream_time_seconds_histogram_bucket{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80",le="+Inf"} 6
nginx_http_upstream_time_seconds_histogram_sum{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80"} 0.010000000358559191
nginx_http_upstream_time_seconds_histogram_count{POD="serverstatistics-nginx-dev-557954d47d-t88vc",http_host="",request="GET /static/css/main.b7c7b33e.chunk.css HTTP/1.1",request_method="GET",server_port="",status="304",upstream_addr="172.20.87.247:80"} 6

5.配置prometheus取来这些metrics

[root@server nginx-log]#vi /data/prometheus/prometheus.yml
#在监控target加入一个static的监控目标
- job_name: 'nginx-log'
  static_configs:
  - targets: ['10.128.142.42:36666']
  scrape_interval: 3s


[root@server nginx-log]#curl -X POST 127.0.0.1:9090/-/reload

 

 数据就收录到prometheus中了

 

 5.配置grafana展示数据(grafana的使用就不详细将了，使用的查询语句如下)

sum(max(nginx_http_request_counter_total{instance="10.128.142.43",tag="nginx-log",request!~"GET /health_check.*"}) by (POD,request)) by (request)

 6.如果只是需要查看某一个URL的统计量，可以修改查询语句完成，收工。