# ps aux | grep nginx | grep -v "grep"
# yum -y install ps_mem
# ps_mem -p 444575
测试环境,nginx 代理websocket连接不多,就只有一个 worker进程参与这类连接的维护。
如果有大量的websocket 长连接,使得 fork出来的 16个worker 进程都参与了
reload一次,会导致 新增 16个 “worker process is shutting down”。
每个 " worker process is shutting down" ,占用的内存从 几十M到几百M不等,短时间内多次reload,会使得nginx进程内存占用激增。
不管nginx是否开启长连接,nginx在reload过程中,nginx对客户端和反向代理的后端在TCP代理,websocket代理和upstream反向代理的情况下均没有影响,nginx会在reload时把正常处理连接的worker设置shutting down状态,不接受新的请求,然后新启动一个worker进程接收处理新的请求,shutting down的worker直至处理完当前连接之后优雅退出。
将nginx reload,websocket连接未断开,当前已经维持的连接应该会一直在老进程持续下去,直到自己的生命周期结束(主动或被动),新晋连接会使用新的进程处理。
HTTP keep-alive的场景也是类似的。
在实际使用过程中,由于nginx可能会多次reload,这样会导致shutting down进程会持续增加,最终会导致Nginx OOM,会影响业务。
解决方法:
###################################
worker_shutdown_timeout 240s;
####################################
参考
谈谈 nginx 信号集
http://io.upyun.com/2017/08/19/nginx-signals
nginx: worker process is shutting down
https://www.cnblogs.com/felixzh/p/8670770.html
http://siwei.me/blog/posts/nginx-worker-is-shutting-down
worker_shutdown_timeout
https://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout
是什么让你的 nginx 服务退出这么慢?
https://zhuanlan.zhihu.com/p/34792840
从websocket服务的nginx配置说起
https://echizen.github.io/tech/2018/10-21-nginx-websocket
我眼里的 nginx
https://www.zhihu.com/column/nginxio
如何优雅地关闭worker进程?
https://blog.csdn.net/qq_27276045/article/details/103645876
https://blog.csdn.net/error311/article/details/104712237
https://cloud.tencent.com/developer/article/1558426
一次百万长连接压测Nginx内存溢出问题
https://www.51cto.com/article/612789.html
DevOps nginx: worker process is shutting down 原因解析
https://www.dazhuanlan.com/freelock/topics/1500064
nginx处理websocket连接
https://blog.csdn.net/weixin_43931358/article/details/111039750
NGINX as a WebSocket Proxy
https://www.nginx.com/blog/websocket-nginx
Nginx does not shut down when websocket connections are still open
https://github.com/phusion/passenger/issues/2211
nginx的graceful shutdown和worker shutdown timeout
https://cloud.tencent.com/developer/article/1162090
worker_shutdown_timeout 指令
https://www.hxstrive.com/subject/nginx.htm?id=785&p=757
关于k8s下使用Ingress保持长连接的异常情况排查
https://blog.jobshen.com/posts/eee2bd13.html
nginx热加载时已建立的websocket连接是否中断?
https://mp.weixin.qq.com/s/Nn44qLLQmJnzH-JLu_I-VA
Nginx reload了,连接怎么办?
https://mp.weixin.qq.com/s/i7DBCKF-ZouBihFEZKB6YQ
worker_shutdown_timeout" directive did not work websocket
https://github.com/Miss-you/apisix-book/issues/57
worker_shutdown_timeout(seconds) - Set the timeout for worker shutdown
https://docs.contrastsecurity.com/en/configure-with-puma.html