Nginx单机上万并发优化

1、使用httpd的ab压力测试

centos7 自带 httpd 

查看httpd状态

$ systemctl status httpd.service 

显示如下则表示服务未启动

● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:httpd(8)
           man:apachectl(8)


5月 09 12:17:45 cdh-dev4 systemd[1]: Starting The Apache HTTP Server...
5月 09 12:17:45 cdh-dev4 httpd[1859]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.32.136. Set the 'ServerName' directive globally to suppress this message
5月 09 12:17:45 cdh-dev4 systemd[1]: Started The Apache HTTP Server.
5月 09 12:22:53 cdh-dev4 systemd[1]: Stopping The Apache HTTP Server...
5月 09 12:22:54 cdh-dev4 systemd[1]: Stopped The Apache HTTP Server.

启动httpd

$ systemctl start httpd.service

查看httpd状态

$ systemctl status httpd.service 

显示如下 则表示httpd启动成功

● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2018-05-09 12:24:44 CST; 2s ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 1962 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─1962 /usr/sbin/httpd -DFOREGROUND
           ├─1965 /usr/sbin/httpd -DFOREGROUND
           ├─1966 /usr/sbin/httpd -DFOREGROUND
           ├─1967 /usr/sbin/httpd -DFOREGROUND
           ├─1968 /usr/sbin/httpd -DFOREGROUND
           └─1969 /usr/sbin/httpd -DFOREGROUND


5月 09 12:24:44 cdh-dev4 systemd[1]: Starting The Apache HTTP Server...
5月 09 12:24:44 cdh-dev4 httpd[1962]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.32.136. Set the 'ServerName' directive globally to suppress this message
5月 09 12:24:44 cdh-dev4 systemd[1]: Started The Apache HTTP Server.

###ab命令 -c 指定并发数  -n 指定请求数  对138上的nginx进行压力测试

$ ab -c 2000 -n 40000  http://192.168.32.138/

2000并发下40000次请求测试结果如下，nginx服务器硬件配置4g内存，单核双线程，centos7.

This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/


Benchmarking 192.168.32.138 (be patient)
Completed 4000 requests
Completed 8000 requests
Completed 12000 requests
Completed 16000 requests
Completed 20000 requests
Completed 24000 requests
Completed 28000 requests
Completed 32000 requests
Completed 36000 requests
Completed 40000 requests
Finished 40000 requests




Server Software:        nginx/1.12.2
Server Hostname:        192.168.32.138
Server Port:            80


Document Path:          /
Document Length:        612 bytes


Concurrency Level:      2000
Time taken for tests:   4.623 seconds
Complete requests:      40000
Failed requests:        24640      
   (Connect: 0, Receive: 0, Length: 12448, Exceptions: 12192)
Write errors:           0
Total transferred:      23281440 bytes
HTML transferred:       16861824 bytes
Requests per second:    8652.27 [#/sec] (mean)
Time per request:       231.153 [ms] (mean)
Time per request:       0.116 [ms] (mean, across all concurrent requests)
Transfer rate:          4917.90 [Kbytes/sec] received


Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  104  64.5    104    1121
Processing:    25  122  24.1    122     240
Waiting:        0   67  47.1     89     148
Total:        130  227  65.8    223    1226


Percentage of the requests served within a certain time (ms)
  50%    223
  66%    233
  75%    237
  80%    239
  90%    246
  95%    251
  98%    271
  99%    282
 100%   1226 (longest request)

可以看到失败的请求次数是

Failed requests:        24640      

2、高并发nginx优化

在nginx服务器上执行以下命令

系统内核层面:

# 允许等待中的监听

echo 50000 >/proc/sys/net/core/somaxconn  

#tcp连接快速回收
echo 1 >/proc/sys/net/ipv4/tcp_tw_recycle  

# tcp连接重用 
echo 1 >/proc/sys/net/ipv4/tcp_tw_reuse   

#不抵御洪水攻击
echo 0 >/proc/sys/net/ipv4/tcp_syncookies   

查看每个用户最大允许打开文件数量

$ ulimit -a

1024

这个1024太小，需要改大。

永久设置方法

 $ vi /etc/security/limits.conf  

    

在文件最后加入  

  

 * soft nofile 65535  
 * hard nofile 65535  

Nginx层面:

对nginx.conf 进行修改优化。

worker_processes默认情况下为1，一般情况下不用修改，但考虑到实际情况，可以修改这个数值，以提高性能;
官方的建议是修改成CPU的内核数，这里引用一段翻译过的文章： 
worker_processes指明了nginx要开启的进程数，据官方说法，一般开一个就够了，多开几个，可以减少机器io带来的影响。
据实践表明，nginx的这个参数在一般情况下开4个或8个就可以了，再往上开的话优化不太大。
据另一种说法是，nginx开启太多的进程，会影响主进程调度，所以占用的cpu会增高。使用lscpu确定可用的核心数。

worker_processes  4; #指定开启4个worker进程
keepalive_timeout 0;   #高并发下设为0  上传文件等长连接要保持连接

worker_rlimit_nofile ,将此值增加到大于worker_processes * worker_connections的值。 应该是增加当前worker运行用户的最大文件打开数值。
NGINX提供了worker_rlimit_nofile指令，这是除了ulimit的一种设置可用的描述符的方式。 该指令与使用ulimit对用户的设置是同样的效果。此指令的值将覆盖ulimit的值.

worker_rlimit_nofile 102400;

events {
    worker_connections  20480;  #设置worker进程最大打开的连接数:
}

以下是138 nginx.conf配置

#user  nobody;
worker_processes  4;
worker_rlimit_nofile 204800;
error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;


pid        /usr/local/nginx/nginx.pid;




events {
    worker_connections  20480;
}




http {
    include       mime.types;
    default_type  application/octet-stream;


    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';


    #access_log  logs/access.log  main;


    sendfile        on;
    #tcp_nopush     on;


    #高并发下设为0
    keepalive_timeout  0;
    #keepalive_timeout  65;


    #gzip  on;


    server {
        listen       80;
        server_name  localhost;


        #charset koi8-r;


        #access_log  logs/host.access.log  main;


        location / {
            root   html;
            index  index.html index.htm;
        }


        #error_page  404              /404.html;


        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }


        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}


        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}


        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }




    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;


    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}




    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;


    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;


    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;


    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;


    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


}


stream{
  log_format basic '$remote_addr [$time_local] ' '$protocol $status $bytes_sent $bytes_received' '$session_time';


  upstream impala { #impala daemon
    least_conn; #least_conn
    server 192.168.32.134:21000;
    server 192.168.32.138:21000;
  }
  upstream impalajdbc { 
    least_conn;
    server 192.168.32.134:21050;
    server 192.168.32.138:21050;
  }
  server{ #impala
    listen 21001;
    proxy_pass impala;
  }
  server{ #impalajdbc
    listen 21051;
    proxy_pass impalajdbc;
  }
}

3、优化后的测试结果

$ ab -c 20000 -n 400000 http://192.168.32.138/

20000并发400000 次请求结果如下

This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.32.138 (be patient)
Completed 40000 requests
Completed 80000 requests
Completed 120000 requests
Completed 160000 requests
Completed 200000 requests
Completed 240000 requests
Completed 280000 requests
Completed 320000 requests
Completed 360000 requests
Completed 400000 requests
Finished 400000 requests


Server Software:        nginx/1.12.2
Server Hostname:        192.168.32.138
Server Port:            80

Document Path:          /
Document Length:        612 bytes

Concurrency Level:      20000
Time taken for tests:   25.185 seconds
Complete requests:      400000
Failed requests:        0
Write errors:           0
Total transferred:      338000000 bytes
HTML transferred:       244800000 bytes
Requests per second:    15882.53 [#/sec] (mean)
Time per request:       1259.245 [ms] (mean)
Time per request:       0.063 [ms] (mean, across all concurrent requests)
Transfer rate:          13106.19 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  683 1282.5    302   15428
Processing:   122  437 159.4    435    5071
Waiting:        0  348 155.7    353    5071
Total:        220 1120 1294.8    768   16006

Percentage of the requests served within a certain time (ms)
  50%    768
  66%    851
  75%    926
  80%   1299
  90%   1765
  95%   3622
  98%   3982
  99%   7728
 100%  16006 (longest request)

可以看到请求失败的次数

Failed requests: 0

就是时间长了些，这也与机器配置有关。