nginx 日志配置详解

有一天突发奇想，nginx反向代理能不能记录后端接口的处理时间呢？如果能记录那么就可用定位分析接口的处理异常了。

配置

如果需要nginx记录请求时间，那么就需要配置nginx的log_fromat

具体修改nginx.conf配置文件

http {
    include /etc/nginx/conf/mime.types;
    default_type application/octet-stream;

log_format  main '$remote_addr $remote_user [$time_local] "$request" '
	'$status $body_bytes_sent "$http_referer" '
	'$http_user_agent $http_x_forwarded_for $request_time $upstream_response_time $upstream_addr $upstream_status';

log_format apm '[$time_local]\tclient=$remote_addr\t'
   'request="$request"\t request_length=$request_length\t'
   'http_referer="$http_referer"\t'
   'bytes_sent=$bytes_sent\t'
   'body_bytes_sent=$body_bytes_sent\t'
   'user_agent="$http_user_agent"\t'
   'upstream_addr=$upstream_addr\t'
   'upstream_status=$upstream_status\t'
   'cookie="$http_cookie"\t'
   'request_body="$request_body"\t'
   'document_root="$document_root"\t'
   'fastcgi_script_name="$fastcgi_script_name"\t'
   'request_filename="$request_filename"\t'
   'request_time=$request_time\t'
   'upstream_response_time=$upstream_response_time\t'
   'upstream_connect_time=$upstream_connect_time\t'
   'upstream_header_time=$upstream_header_time\t';

# 为什么有的变量有双引号，有的没有呢？
# 如果变量结果是数字的，就不需要双引号，变量是字符的，就需要双引号

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log crit;

上面配置了两个log_format，设置两个log_format的目的是为了让不同的后端的服务，使用不用日志格式。
如果后端的server没有配置，就默认使用main 标记的日志，

    access_log /var/log/nginx/access.log main;    # 这里配置的

如果需要apm格式，只需要在access_log配置apm即可

server {
  listen 8000;
  server_name www.test.com;
  location / {
    proxy_pass http://192.168.1.80:8080;
    # 这里设置apm即可使用apm格式的log_format
    access_log /var/log/nginx/getip-access.log apm;   
    include /conf.d/proxy-params.conf;
  }
}

结果验证

请求 www.test.com

输出日志结果为：

[11/May/2020:11:22:20 +0800]    
client=192.168.1.100
request="GET / HTTP/1.1"         
request_length=452     
http_referer="-"        
bytes_sent=170  
body_bytes_sent=14      
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36" 
upstream_addr=192.168.1.80:8080      
upstream_status=200     
cookie="-"      
request_body="-"        
document_root="/etc/nginx/html" 
fastcgi_script_name="/" 
request_filename="/etc/nginx/html/"      
request_time=5.006      
upstream_response_time=5.006    
upstream_connect_time=0.000     
upstream_header_time=5.006

下面为proxy_pass http://192.168.1.80:8080; 接口处理时间，

upstream_response_time=5.006    
upstream_connect_time=0.000     
upstream_header_time=5.006

因为我在后端服务器配置了sleep 5秒，因此处理时间为5秒多，这个时间就反应了
用户发送一次请求，到nginx代理后端服务器，再从后端服务器处理完毕，反馈给用户的时间
记录了一次完整的请求时间

后续可以从日志中提取时间，分析网页中哪个接口请求比较慢

后端服务www.test.com的代码

#!/bin/python
# -*- coding: UTF-8 -*-

from flask import Flask, render_template, request
import time
# Initialize the Flask application
app = Flask(__name__)
@app.route('/')
def index():
    user_ip = None
    if 'X-Real-Ip' in request.headers.keys():
        user_ip = request.headers['X-Real-Ip']
    else:
        user_ip = request.remote_addr
    time.sleep(5)
    return user_ip

if __name__ == '__main__':
    app.run(host="0.0.0.0",port=int("5000"))

后续

这个访问日志格式还可以优化，还可以定制不同的参数，设置为json格式等等。

log_format  main  '{
"remote_addr":"$remote_addr",
"remote_user":"$remote_user",
"time_local":"$time_local",
"request":"$request",
"status":"$status",
"request_time":"$request_time",
"upstream_response_time":"$upstream_response_time",
"request_length":"$request_length",
"bytes_sent":"$bytes_sent",
"body_bytes_sent":"$body_bytes_sent",
"gzip_ratio":"$gzip_ratio",
"connection_requests":"$connection_requests",
"http_referer":"$http_referer",
"http_user_agent":"$http_user_agent",
"http_x_forwarded_for":"$http_x_forwarded_for"
}';

log_format  main  '{
	"remote_addr":"$remote_addr",
	"upstream_addr":"$upstream_addr",
	"status":"$status",
	"upstream_status":"$upstream_status",
	"http_referer":"$http_referer",
	"http_user_agent":"$http_user_agent",
	"http_x_forwarded_for":"$http_x_forwarded_for"
	"remote_user":"$remote_user",
	"request_body":"$request_body",
	"document_root":"$document_root",
	"fastcgi_script_name":"$fastcgi_script_name",
	"request_filename":"$request_filename",
	"request":"$request",
	"request_length":"$request_length",
	"bytes_sent":"$bytes_sent",
	"body_bytes_sent":"$body_bytes_sent",
	"gzip_ratio":"$gzip_ratio",
	"connection_requests":"$connection_requests",
	"time_local":"$time_local",
	"request_time":"$request_time",
	"upstream_response_time":"$upstream_response_time",
	"upstream_connect_time":"$upstream_connect_time",
	"upstream_header_time":"$upstream_header_time",
	}';

format变量说明
$remote_addr 	#发起请求的客户端所在ip地址
$remote_user	#发起请求的客户端用户名称，获取不到则显示为 -
$time_local		#用来记录访问时间与时区(依赖nginx服务器本地时间)，形如 20/Aug/2017:21:15:19 +0800，获取不到则显示为 -
$time_iso8601	#类似$time_local，不同的是这里采用ISO 8601标准格式 
$request		#记录发起的请求，形如 POST /zentaopms/www/index.php?m=user&f=login&referer=L3plbnRhb3Btcy93d3cvaW5kZXgucGhw HTTP/1.1
$status			#记录响应状态，比如 200
$request_time	
#记录请求处理时间（以秒为单位，携带毫秒的解决方案），
#从读取客户端第一个字节开始算起，到发送最后一个字节给客户端的时间间隔
#（原文：request processing time in seconds with a milliseconds resolution; time elapsed between the first bytes were read from the client and the log write after the last bytes were sent to the client）

$upstream_response_time		#记录nginx从后端服务器(upstream server)获取响应的时间（以秒为单位，携带毫秒的解决方案），多个请求的时间以逗号分隔

$request_length		#记录请求长度(包括请求行，请求头，请求体)
$gzip_ratio		#记录nginx gzip压缩比例，获取不到则显示为 -
$bytes_sent		#发送给客户端的字节数
$body_bytes_sent		#发送给客户端的响应体字节数
$connection_requests		#单个连接的并发请求数（the current number of requests made through a connection (1.1.18)
$http_referer		#记录请求引用页面地址
$http_user_agent		#记录用户代理信息（通常是浏览器信息
$http_x_forwarded_for	
#当为了承受更大的负载使用反向代理时，web服务器不能获取真实的客户端IP，
#$remote_addr获取到的是反向代理服务器的ip，
#这种情况下，代理服务器通常会增加一个叫做x_forwarded_for的信息头，
#把连接它的真实客户端IP加到这个信息头里，这样就能保证网站的web服务器能获取到真实IP，获取不到则显示为 -

$connection		#连接序列号
$msec		#写入日志的时间（以秒为单位，携带毫秒的解决方案）(原文：time in seconds with a milliseconds resolution at the time of the log write)
$pipe		#如果为管道请求则显示为p，否则显示为 .  

参考链接：
# http://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_response_time

附上 nginx可用的参数大全
https://nginx.org/en/docs/http/ngx_http_core_module.html#var_status

参考链接：

https://www.cnblogs.com/kevingrace/p/5893499.html
https://www.cnblogs.com/shouke/p/10157556.html
http://www.ttlsa.com/linux/the-nginx-log-configuration/
https://www.cnblogs.com/taoshihan/p/11978842.html
https://blog.csdn.net/ronon77/article/details/84732095

https://www.baidu.com/s?ie=UTF-8&wd=ngx_http_log_request_speed