【深度】nginx strace和gstack源码调试

一、前提

1 nginx编译安装

假设已经安装好了nginx服务器，切能正常启动访问，个人版本为 nginx version: nginx/1.14.0，使用strace/pstack进行调试（或者gdb，没调试过）。strace 和ltrace 可以查看一个应用程序在运行过程中所发起的系统调用

2 strace命令

strace - trace system calls and signals 跟踪系统调用和信号，最重要的参数 -p，

strace -p pid

Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). -p "pidof PROG" syntax is supported

详细参数，自行查看 man strace（yum install strace）

3 nginx配置

nginx采用master-worker模式，且只有一个worker进程，方便调试

worker_processes  1;
events {
    worker_connections  1024;
}

二、gstack 调试

ps aux | grep nginx | grep -v grep

www      13366  0.0  0.1  21784  2164 ?        S    Mar23  0:00 nginx: worker process

root    15221  0.0  0.1  21784  2132 ?        ts  Mar13  0:00 nginx: master process /usr/local/nginx/sbin/nginx

[@bx16-77-246 /usr/local/nginx]# gstack 13366

#0  0x00007fe98da830f3 in __epoll_wait_nocancel () from /lib64/libc.so.6

#1  0x00000000004320b8 in ngx_epoll_process_events (cycle=0xd7e1b0, timer=18446744073709551615, flags=1) at src/event/modules/ngx_epoll_module.c:800

#2  0x0000000000429c48 in ngx_process_events_and_timers (cycle=cycle@entry=0xd7e1b0) at src/event/ngx_event.c:242

#3  0x0000000000430644 in ngx_worker_process_cycle (cycle=0xd7e1b0, data=) at src/os/unix/ngx_process_cycle.c:750

#4  0x000000000042ed72 in ngx_spawn_process (cycle=cycle@entry=0xd7e1b0, proc=proc@entry=0x4305d3 , data=data@entry=0x0, name=name@entry=0x480ead "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:199

#5  0x000000000042f90b in ngx_start_worker_processes (cycle=cycle@entry=0xd7e1b0, n=1, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:359

#6  0x0000000000431358 in ngx_master_process_cycle (cycle=cycle@entry=0xd7e1b0) at src/os/unix/ngx_process_cycle.c:244

#7  0x000000000040c84b in main (argc=, argv=) at src/core/nginx.c:382

从main()函数到epoll_wait()函数的调用关系一目了然，说明：（调用栈从下自上）

1 入口是nginx.c文件里的main函数，解析命令行参数，初始化

2 根据ngx_process变量，进入分支ngx_master_process_cycle（另一个分支是ngx_single_process_cycle）

3 进入master后，就需要启动worker进程，调用ngx_start_worker_processes

4 调用ngx_spawn_process启动worker进行，并执行回调ngx_worker_process_cycle

5 进入worker主循环，其中最重要的逻辑是ngx_process_events_and_timers

6 调用ngx_worker_process_init，执行各个模块的init钩子，并设置action为ngx_epoll_process_events

7 epoll开始进入事件等待 __epoll_wait_nocancel

三、strace 调试

#参数1：9是epoll_create创建文件描述符，等待事件触发，1代表rdlist就绪队列有事件要处理

epoll_wait(9, [{EPOLLIN, {u32=14344048, u64=14344048}}], 512, -1) = 1

#接收client请求，3是socket的文件描述符

accept4(17, {sa_family=AF_INET, sin_port=htons(57695), sin_addr=inet_addr("10.2.82.135")}, [16], SOCK_NONBLOCK) = 3

#将3 （accept4 产生）加入到epoll的事件监听机制

epoll_ctl(9, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0

#等待就绪事件，1表示有事件要处理

epoll_wait(9, [{EPOLLIN, {u32=14344720, u64=14344720}}], 512, 60000) = 1

#从client接收数据，长度484字节

recvfrom(3, "GET /ae/test HTTP/1.1\r\nHost: tem"..., 16384, 0, NULL, NULL) = 484

#判断_not_exists_是否存在，-1表示不存在

stat("/usr/local/nginx/html/_not_exists_", 0x7fff9e42c1c0) = -1 ENOENT (No such file or directory)

#更新事件注册

epoll_ctl(9, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0

#初始化新的socket 4

socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4

#对设备的I/O通道进行管理，0表示成功

ioctl(4, FIONBIO, [1])                  = 0

#将socket 4 加入epoll的事件监听机制

epoll_ctl(9, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344944, u64=14344944}}) = 0

# socket 4 建立连接

connect(4, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)

#rdlist就绪队列里有两个事件待处理

epoll_wait(9, [{EPOLLOUT, {u32=14344720, u64=14344720}}, {EPOLLOUT, {u32=14344944, u64=14344944}}], 512, 60000) = 2

#获取套接字的选项信息

# 第一个参数：socket套接字

# 第二个参数：套接字层次

# 第三个参数：要获取的信息项

getsockopt(4, SOL_SOCKET, SO_ERROR, [0], [4]) = 0

#响应头509字节内容写入socket4缓冲区

writev(4, [{"GET /ae/test HTTP/1.0\r\nX-Forward"..., 509}], 1) = 509

#rdlist就绪队列里有一个事件待处理，即socket 3

epoll_wait(9, [{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=14344944, u64=14344944}}], 512, 60000) = 1

#从socket 4 读取响应头及内容

recvfrom(4, "HTTP/1.0 200 OK\r\nContent-Length:"..., 4096, 0, NULL, NULL) = 4096

readv(4, [{"m' => 'input',\n  "..., 4096}], 1) = 4096

readv(4, [{"  'file_size' => 0,\n "..., 4096}], 1) = 4096

readv(4, [{"put',\r\n      'rule' "..., 4096}], 1) = 4096

readv(4, [{"#39;2' => array(\r\n      "..., 4096}], 1) = 4096

readv(4, [{"  'file_size' => arr"..., 4096}], 1) = 4096

readv(4, [{"\r\n        'file_size' =&"..., 4096}], 1) = 4096

readv(4, [{"            'predownload'"..., 4096}], 1) = 4096

readv(4, [{"y (\n      'label' => "..., 4096}], 1) = 4096

#向socket 3 写入数据

writev(3, [{"HTTP/1.1 200 OK\r\nServer: nginx/1"..., 165}, {"\n\n  \n"..., 3949}, {"m' => 'input',\n  "..., 4096}], 3) = 8210

writev(3, [{"  'file_size' => 0,\n "..., 4096}, {"put',\r\n      'rule' "..., 4096}], 2) = 8192

writev(3, [{"#39;2' => array(\r\n      "..., 4096}, {"  'file_size' => arr"..., 4096}], 2) = 8192

writev(3, [{"\r\n        'file_size' =&"..., 4096}, {"            'predownload'"..., 4096}], 2) = 8192

writev(3, [{"y (\n      'label' => "..., 4096}], 1) = 4096

readv(4, [{"  array(\n                    "..., 4096}, {"9;file_ext' => 'mp4,g"..., 4096}, {"\r\n    
99

整个过程，你应该清晰的知道epoll对象上的rdlist就绪队列和rbtree的变化情况

延伸阅读

nginx 源码注释项目

参考文档
1 epoll_create
2 strace工具
3 nginx 源码注释