ngx_http_upstream_keepalive_module原理分析

我们知道针对后端的keepalive是通过nginx.conf配置文件来指定的，例如

 

    upstream resins{
        server 61.135.250.217:6800;
        keepalive 1024;
    }

 

nginx在读取配置文件的时候，就会执行指令相应的函数，查看ngx_http_upstream_keepalive_module源代码，

我们发现keepalive指令对应ngx_http_upstream_keepalive函数

 

     67 static ngx_command_t  ngx_http_upstream_keepalive_commands[] = {
     68
     69     { ngx_string("keepalive"),
     70       NGX_HTTP_UPS_CONF|NGX_CONF_TAKE12,
     71       ngx_http_upstream_keepalive, //这里会指明遇到keepalive指令会执行这个函数，这个函数来改变一切
     72       0,
     73       0,
     74       NULL },
     75
     76       ngx_null_command
     77 };

 

读取配置的时候就会执行下面的函数，执行堆栈如下：

 

    #0  ngx_http_upstream_keepalive (cf=0xbffe8ec0, cmd=0x80e8180, conf=0x0)
        at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:438
    #1  0x0805d892 in ngx_conf_handler (cf=0xbffe8ec0, last=0) at src/core/ngx_conf_file.c:393
    #2  0x0805d50b in ngx_conf_parse (cf=0xbffe8ec0, filename=0x0) at src/core/ngx_conf_file.c:243
    #3  0x0809449d in ngx_http_upstream (cf=0xbffe8ec0, cmd=0x80e4740, dummy=0x9725bfc) at src/http/ngx_http_upstream.c:3911
    #4  0x0805d892 in ngx_conf_handler (cf=0xbffe8ec0, last=1) at src/core/ngx_conf_file.c:393
    #5  0x0805d50b in ngx_conf_parse (cf=0xbffe8ec0, filename=0x0) at src/core/ngx_conf_file.c:243
    #6  0x080727dc in ngx_http_block (cf=0xbffe8ec0, cmd=0x80e1640, conf=0x97252a8) at src/http/ngx_http.c:241
    #7  0x0805d892 in ngx_conf_handler (cf=0xbffe8ec0, last=1) at src/core/ngx_conf_file.c:393
    #8  0x0805d50b in ngx_conf_parse (cf=0xbffe8ec0, filename=0x9724c00) at src/core/ngx_conf_file.c:243
    #9  0x0805a80f in ngx_init_cycle (old_cycle=0xbffe8f60) at src/core/ngx_cycle.c:263
    #10 0x0804a8fd in main (argc=1, argv=0xbffe90e4) at src/core/nginx.c:324

 

 

我们可以看到执行ngx_http_upstream_keepalive的时候，已经改变了upstream的相关内容

 

ngx_http_upstream_keepalive(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)

 

    446     kcf->original_init_upstream = uscf->peer.init_upstream
    447                                   ? uscf->peer.init_upstream
    448                                   : ngx_http_upstream_init_round_robin;//保留旧的函数指针
    449
    450     uscf->peer.init_upstream = ngx_http_upstream_init_keepalive;  //设置新的初始化upstream函数指针

改变ngx_http_upstream_srv_conf_t的init_upstream为ngx_http_upstream_init_keepalive，

kcf->original_init_upstream保存了uscf->peer.init_upstream的指针这样就可以截获upstream，让其先到ngx_http_upstream_init_keepalive这儿来

 

 

在ngx_http_upstream_init_keepalive中
    先利用原来保存的 kcf->original_init_upstream执行upstream的初始化，然后再执行keepalive方面的初始化
    128     if (kcf->original_init_upstream(cf, us) != NGX_OK) {
    129         return NGX_ERROR;
    130     }

    131
    132     kcf->original_init_peer = us->peer.init;//保存原始的init
    133                                                                                                                                
    134     us->peer.init = ngx_http_upstream_init_keepalive_peer;//设置先到这个函数中来

 

ngx_http_upstream_init_keepalive的执行堆栈如下：

   #0  ngx_http_upstream_init_keepalive (cf=0xbfe493e0, us=0x8656694)
        at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:122
    #1  0x08090d1c in ngx_http_upstream_init_main_conf (cf=0xbfe493e0, conf=0x864afa0) at src/http/ngx_http_upstream.c:4394
    #2  0x0806fadd in ngx_http_block (cf=0xbfe493e0, cmd=0x80ca620, conf=0x864a784) at src/http/ngx_http.c:260
    #3  0x0805d023 in ngx_conf_handler (cf=0xbfe493e0, last=1) at src/core/ngx_conf_file.c:393
    #4  0x0805cc9f in ngx_conf_parse (cf=0xbfe493e0, filename=0x864a0d0) at src/core/ngx_conf_file.c:243
    #5  0x0805a1c7 in ngx_init_cycle (old_cycle=0xbfe49480) at src/core/ngx_cycle.c:263
    #6  0x0804a59b in main (argc=1, argv=0xbfe49604) at src/core/nginx.c:327

 

上面的函数执行是为了预先设置好http请求过来时的执行顺序，让其先执行ngx_http_upstream_init_keepalive_peer，而不是原来默认的函数

 

当一个请求过来时，会执行一系列操作，如果需要访问后端，就会执行ngx_http_upstream_init_request，

而ngx_http_upstream_init_request就会执行uscf->peer.init(r, uscf)，上面已经改变了upstream默认的函数指针

，已经指向了ngx_http_upstream_init_keepalive_peer。

 

 

在ngx_http_upstream_init_keepalive_peer函数中
    174     if (kcf->original_init_peer(r, us) != NGX_OK) {//先执行原始upstream peer的初始化
    175         return NGX_ERROR;
    176     }

 

    178     kp->conf = kcf;
    179     kp->upstream = r->upstream;//这两句是为了辅助目地设置的
   
    180     kp->data = r->upstream->peer.data;
    181     kp->original_get_peer = r->upstream->peer.get;
    182     kp->original_free_peer = r->upstream->peer.free; //这上面3句就是保留原始peer的初始化信息
    183
    184     r->upstream->peer.data = kp;//这句是为了ngx_http_upstream_get_keepalive_peer参数中的void* data,否则就是原始的r->upstream->peer.data
    185     r->upstream->peer.get = ngx_http_upstream_get_keepalive_peer;
    186     r->upstream->peer.free = ngx_http_upstream_free_keepalive_peer;//进一步截获，让系统先到这儿来

 

在ngx_http_upstream_init_keepalive_peer函数中，改变了peer默认的函数指针，使其先到keepalive模块来

 

真正需要连接的时候，nginx会执行u->peer.get，于是就执行了上面设置的ngx_http_upstream_get_keepalive_peer

 

部分执行堆栈如下：

   #0  ngx_http_upstream_get_keepalive_peer (pc=0x84860dc, data=0x8486400)
        at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:260
    #1  0x08067cdf in ngx_event_connect_peer (pc=0x84860dc) at src/event/ngx_event_connect.c:24
    #2  0x080919ef in ngx_http_upstream_connect (r=0x8481328, u=0x84860d4) at src/http/ngx_http_upstream.c:1084

 

 

在ngx_http_upstream_get_keepalive_peer函数中
    230     rc = kp->original_get_peer(pc, kp->data);//先获取原始的信息，比如访问ngx_http_upstream_get_round_robin_peer函数，这里还 没有赋connection
                    (gdb) p *pc
                    $2 = {connection = 0x0, sockaddr = 0x8491dc4, socklen = 16, name = 0x8493a9c,

tries = 1, get = 0x80cbc73 , free = 0x80cbf04 ,
                      data = 0x8486400, local = 0x0, rcvbuf = 0, log = 0x8481258, cached = 0, log_error = 1}

    。。。
    259             pc->connection = c;//从cache中赋值给pc
    260             pc->cached = 1;
   
  这里举例的就是cache中获取connection，赋值给pc->connection

   
    返回ngx_event_connect_peer 函数时，代码如下：
     24     rc = pc->get(pc, pc->data);
     25     if (rc != NGX_OK) {
     26         return rc;
     27     }

     29     s = ngx_socket(pc->sockaddr->sa_family, SOCK_STREAM, 0);

 

      这里需要注意的是如果是第一次访问，cache没有现成的连接，ngx_http_upstream_get_keepalive_peer会返回OK，会执行29行代码及其后面的连接操作；

     如果有现成的连接，ngx_http_upstream_get_keepalive_peer会返回NGX_DONE,所以会跳过后面29行的代码，就不会进入创建连接的过程。

 

     当请求处理完的时候，nginx会执行u->peer.free，也就是先到ngx_http_upstream_free_keepalive_peer函数中来。

 

    ngx_http_upstream_free_keepalive_peer执行堆栈如下：
        #0  ngx_http_upstream_free_keepalive_peer (pc=0x84860e8, data=0x848640c, state=0)
            at /home/wangbin/work/memcached/keepalive/ngx_http_upstream_keepalive_module.c:368
        #1  0x08094fbc in ngx_http_upstream_finalize_request (r=0x84966b0, u=0x84860e0, rc=0) at src/http/ngx_http_upstream.c:2952
        #2  0x08094841 in ngx_http_upstream_process_request (r=0x84966b0) at src/http/ngx_http_upstream.c:2679
        #3  0x080945ba in ngx_http_upstream_process_upstream (r=0x84966b0, u=0x84860e0) at src/http/ngx_http_upstream.c:2606
        #4  0x08093d1c in ngx_http_upstream_send_response (r=0x84966b0, u=0x84860e0) at src/http/ngx_http_upstream.c:2295
        #5  0x08092722 in ngx_http_upstream_process_header (r=0x84966b0, u=0x84860e0) at src/http/ngx_http_upstream.c:1592
        #6  0x080915ad in ngx_http_upstream_handler (ev=0xb7df30d8) at src/http/ngx_http_upstream.c:898
        #7  0x08071c2e in ngx_epoll_process_events (cycle=0x8481b98, timer=56857, flags=1) at src/event/modules/ngx_epoll_module.c:642
        #8  0x08064d87 in ngx_process_events_and_timers (cycle=0x8481b98) at src/event/ngx_event.c:245
        #9  0x0806ddf0 in ngx_single_process_cycle (cycle=0x8481b98) at src/os/unix/ngx_process_cycle.c:306
        #10 0x0804a969 in main (argc=1, argv=0xbfff98f4) at src/core/nginx.c:398

    需要注意的是，在函数ngx_http_upstream_free_keepalive_peer中，都会执行

       return   kp->original_free_peer(pc, kp->data, state);

    这个函数并没有做关闭socket的操作。

 

    真正改变tcp连接不关闭的原因如下：
    由于在ngx_http_upstream_free_keepalive_peer函数执行了下面这句
    346             pc->connection = NULL;
   
            (gdb) p pc
            $18 = (ngx_peer_connection_t *) 0x84860e8
           
    导致了nginx没有关闭对后端连接的关闭操作，由于是否关闭连接操作的控制函数在ngx_http_upstream_finalize_request，部分其函数代码如下：
   
        2955        if (u->peer.connection) {
        2956
        2875         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
        2876                        "close http upstream connection: %d",
        2877                        u->peer.connection->fd);
        2878
        2879         ngx_close_connection(u->peer.connection);
        2880     }

    由于pc->connection = NULL，也就是u->peer.connection为null，所以连接保存了下来（保存在 ngx_http_upstream_keepalive_module 的cache中），没有被关闭，供下次调用时继续使用，省去了tcp连接建立和关闭的过程。