(HY000/2003): Can't connect to MySQL server on '192.168.2.98' (99)

线上lnmp环境，高并发(qps几十k)下报，php连mysql异常，问题分析，解决。

一.问题描述

PHP Warning:  mysqli::real_connect(): (HY000/2003): Can't connect to MySQL server on '192.168.2.98' (99) in

二.问题分析

php连mysql失败，看到99，99是 os的错误，对应错误信息用perror看看。

perror 99

OS error code  99:  Cannot assign requested address

给出，Cannot assign requested address

出现这个错误信息，说明client，没有可以使用的连接，无法创建tcp连接到 mysql server，说明达到了client的最大可用连接限制。

知道这个，就需要调优，调大限制client连接的对应参数。

三.问题解决

1.修改local 可用端口数.local端口，一个ip作为tcp源ip，最多使用65535个端口，如果想变多，需要配置多个网卡ip。做成cluster。

net.ipv4.ip_local_port_range = 10000    65535

2.系统的TIME-WAIT sockets进行快速回收

net.ipv4.tcp_tw_recycle = 1 

3.重用TIME_WAIT状态的端口

net.ipv4.tcp_tw_reuse = 1

4.调大相关内核限制

# Controls IP packet forwarding
net.ipv4.ip_forward = 0


# Controls source route verification
net.ipv4.conf.default.rp_filter = 1


# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0


# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0


# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1


# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1


# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0


# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536


# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536


# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736


# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296


net.ipv4.tcp_fin_timeout = 20


net.ipv4.ip_local_port_range = 10000 65000


net.ipv4.tcp_tw_recycle = 1


net.ipv4.tcp_tw_reuse = 1


net.ipv4.tcp_timestamps = 0


fs.aio-max-nr = 1048576


kernel.sem = 250 32000 100 128


net.core.rmem_default = 31457280
 
net.core.rmem_max = 12582912
 
net.core.wmem_default = 31457280
 
net.core.wmem_max = 12582912
 
fs.file-max = 6815744


net.ipv4.tcp_max_syn_backlog = 20480


net.core.somaxconn = 65536


# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65536


# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824
 
vm.swappiness = 10


vm.dirty_ratio = 60


# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144


# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384


# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.udp_wmem_min = 16384

5.调大/etc/security/limits.conf相关限制(这个主要对server端有限制,client不影响)

* soft nofile 65535
* hard nofile 65535
* hard nproc  102400
* soft nproc  102400
mysql soft nproc 1024000
mysql hard nproc 1024000
mysql soft nofile 1024000
mysql hard nofile 1024000
www soft nproc 1024000
www hard nproc 1024000
www soft nofile 1024000
www hard nofile 1024000
root soft nproc 1024000
root hard nproc 1024000
root soft nofile 1024000
root hard nofile 1024000

6.总体上，把client端，server端涉及到的影响并发处理能力的，参数都调大，做压测，能达到C100K级别。就说明系统限制没太大问题。剩余就是应用代码程序方面的优化了。