Prometheus监控nginx

配置nginx-module-vts模块

收集nginx指标的较多采用的有nginx-vts模块，prometheus-lua两种方式进行采集，本文采用nginx-vts模块方式进行数据收集。

nginx配置

查看是否安装了nginx-vts模块

$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.0
...
configure arguments: --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module

安装nginx-vts模块

$ cd /root/
$ git clone https://github.com/vozlt/nginx-module-vts 

$ cd /usr/local/src/nginx-1.10.0
$ ./configure --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
$ make 

# 使用新编译的nginx执行文件替换掉旧的
$ mv /usr/local/nginx/sbin/nginx  /usr/local/nginx/sbin/nginx.bak.20191211
$ cp objs/nginx  /usr/local/nginx/sbin/

# 生成新的nginx.pid文件
$ kill -USR2 `cat  /usr/local/nginx/logs/nginx.pid`

# 退出旧版本低nginx
$ /usr/local/nginx/sbin/nginx.bak.20191211 -s stop

# 查看最新的编译参数
$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.0
...
configure arguments: --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module

添加nginx配置文件,暴露监控指标

$ /usr/local/nginx/conf/nginx.conf
http {
    ...
    vhost_traffic_status_zone;
    ...
}

$ vim /usr/local/nginx/conf/vhost/nginx-vts-status.conf
server {        
    listen 8088;
    
    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
    }
    
}

#需要重启模块才能生效
$ /usr/local/nginx/sbin/nginx -s stop
$ /usr/local/nginx/sbin/nginx

然后访问ip:8088/status,可以看到下面的数据

image

指标的数据类型转换

要将获取到的数据接入prometheus还需要将数据转为metrics类型的数据，vts模块中直接提供了/status/format/prometheus 接口，访问IP:8088/status/format/prometheus 即可看到转换后的数据。

image

部署nginx-vts-exporter

默认端口为 :9913/metrics

$ wget http://download.baiyongjie.com/linux/prometheus/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
$ tar zxvf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz 
$ cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter  /usr/local/node_exporter/
$ mv nginx-vts-exporter-0.10.3.linux-amd64/ /usr/local/src/
$ cd /tmp/; nohup /usr/local/node_exporter/nginx-vts-exporter -nginx.scrape_uri=http://127.0.0.1:8088/status/format/json &> /dev/null &

$ netstat -nptl|grep 9913
tcp6    0    0 :::9913    :::*    LISTEN    27435/nginx-vts-exp 

$ curl -s 127.0.0.1:9913/metrics|tail
nginx_upstream_responseMsec{backend="100.115.173.4:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.5:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.6:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.7:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.8:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.9:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="127.0.0.1:9000",upstream="::nogroups"} 0
# HELP nginx_vts_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which nginx_vts_exporter was built.
# TYPE nginx_vts_exporter_build_info gauge
nginx_vts_exporter_build_info{branch="HEAD",goversion="go1.10",revision="8aa2881c7050d9b28f2312d7ce99d93458611d04",version="0.10.3"} 1

配置prometheus

$ vim /usr/local/prometheus/prometheus.yml 
...
  - job_name: 'nginx-vts'
    static_configs:
    - targets:
      - 172.31.217.169:9913
      
$ kill -hup `ps -ef |grep prometheus|grep -v grep|awk '{print $2}'`

查看Prometheus是否采集到的数据

image

配置grafana

• 导入模板: 2949

image

安装GeoIP模块

$ yum -y install epel-release geoip-devel

$ cd /usr/local/src/nginx-1.10.0
$ ./configure --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_geoip_module
$ make 

...平滑升级参考上面安装nginx-vts模块

# 查看最新的编译参数
$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.0
...
configure arguments: --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_geoip_module

$ ldd /usr/local/nginx/sbin/nginx |grep libGeoIP
libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f49d503d000)

# 引用地址库
$ vim /usr/local/nginx/conf/nginx.conf
http {

    ...

    vhost_traffic_status_zone;
    geoip_country /usr/share/GeoIP/GeoIP.dat;

    ...
}    

# 重启nginx
$ /usr/local/nginx/sbin/nginx -s stop
$ /usr/local/nginx/sbin/nginx

通过VTS自定义key

通过以上部署只能拿到默认的指标，生产中可能还会需要监控uri的请求量，监控IP访问情况（同一个IP出现大量访问时可能被攻击），获取不同agent请求量用于分析等，通过vts模块的vhost_traffic_status_filter_by_set_key功能可以自定义需要获取的指标。此处的指标需要加到对应的server配置中

# 添加自定义配置
$ vim /usr/local/nginx/conf/vhost/baiyongjie.com.conf
server {
    listen      443 ssl;
    server_name baiyongjie.com  blog.baiyongjie.com;
    
    ...
    
    vhost_traffic_status_filter_by_set_key $uri uri::$server_name;     #每个uri访问量
    vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;     #不同国家/区域请求量
    vhost_traffic_status_filter_by_set_key $status $server_name;     #http code统计
    vhost_traffic_status_filter_by_set_key $upstream_addr upstream::backend;     #后端转发统计
    vhost_traffic_status_filter_by_set_key $remote_port client::ports::$server_name;     #请求端口统计
    vhost_traffic_status_filter_by_set_key $remote_addr client::addr::$server_name;     #请求IP统计

    location ~ ^/storage/(.+)/.*$ {
        set $volume $1;
        vhost_traffic_status_filter_by_set_key $volume storage::$server_name;     #请求路径统计
    }
    
    ...
    
}    

再次访问status接口, 可以看到多了很多自定义的监控项

image

到Prometheus查询,可以看到新的字段已经可以查询到了

image

参考:

• https://juejin.im/post/5cde1c436fb9a037d1393d18
• https://vircloud.net/build/nginx-geoip.html#selection-141.11-141.34