elk 搭建nginx 日志监控

一 配置nginx日志字段收集源

• 在nginx_home/nginx.conf的http模块里面加上以下配置

    log_format  main  '$http_host '
                      '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent "$request_body" '
                      '"$http_referer" "$http_user_agent" "$proxy_add_x_forwarded_for" '
                      '$request_time '
                      '$upstream_response_time';

1.1、参数介绍

• 参数解释
• $proxy_add_x_forwarded_for 参数详解

二 配置logstash数据采集

logstash.nginx.conf

index 的名字 必须是 logstash开头

## nginx log format config
#    log_format  main  '$http_host '
#                      '$remote_addr - $remote_user [$time_local] '
#                      '"$request" $status $body_bytes_sent "$request_body" '
#                      '"$http_referer" "$http_user_agent" "$proxy_add_x_forwarded_for" '
#                      '$request_time '
#                      '$upstream_response_time';

filter {
    grok {
        match => { "message" => "%{NGINXACCESS}" }
    }

    if [http_user_agent] =~ "inf-ssl-duty-scan" {
        drop { }
    }

    date {
        match => [ "time_local" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }

    geoip {
        source => "http_x_forwarded_for"
    }
    kv {
        source => "request"
        field_split => "&?"
        value_split => "="
        include_keys => [ "network", "country", "language", "deviceId" ]
    }

    urldecode {
        all_fields => true
    }

     mutate
     {
         replace => {"host" => "10.26.127.163"}
     }


}

output {
    elasticsearch {

        host => "10.169.97.191"
        port => 9200
        protocol => "http"
        index => "logstash-nginx-prd-%{+YYYY.MM.dd}"
  }
#  stdout { codec => rubydebug }
}

pattens/nginx

• pattens 的配置请看2.2的配置介绍

NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} %{QS:request_body} %{QS:http_referer} %{QS:http_user_agent} "%{IPORHOST:http_x_forwarded_for}" %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float}

启动脚本

• 脚本可以启动多个logstash实例，分别加载不同的配置文件即可
• agent-nginx.sh

#!/bin/sh
status()
{
        info=`ps -elf | grep logstash|grep -v "grep"`
        if [ -n "$info" ];then
                echo "logstash is running."
        else
                echo "logstash stopped."
        fi
}

stop()
{
        pid=`ps -elf | grep logstash | grep -v "grep" | awk '{print $4}'`
        kill -9 $pid
}
case $1 in
        start)
        nohup ./logstash -f ./conf/logstash.nginx.conf -w 3 > nohup.out 2>&1 &
        status
        ;;
        stop)
        stop
        status
        ;;
        status)
        status
        ;;
        *)
        echo "Usage:{start|stop|status}"
        ;;
esac

2.2、配置介绍

• 利用 ELK系统分析Nginx日志并对数据进行可视化展示
  • 重点：
    • 第2点的 patterns的配置方法 下面这个配置对应 ==一== 的 收集

      NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} %{QS:request_body} %{QS:http_referer} %{QS:http_user_agent} "%{IPORHOST:http_x_forwarded_for}" %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float}