8000 nodes continously disconnect in large network environment
根本原因是socket资源不足，导致连接失败， 下面是对exception的分析：
1. java.net.BindException: Address already in use.
It may be caused by the port resources are not released quickly after we call socket.close(), but be in TIME_WAIT state for some time. Below is the result.
380g7x09:/opt # netstat -na|grep TIME_WAIT|wc -l
28851
380g7x09:/opt # netstat -na|grep TIME_WAIT|grep 8080|wc -l
22326
The solution is maybe to set following parameter to 1. Or do we have other solutions?
380g7x09:/opt # sysctl -a|grep net.ipv4.tcp_tw
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
2. java.net.SocketException: Too many open files
Could we enlarge the "open files" ulimit? Currently it is 1024. FYI.
380g7x09:/opt # ulimit -a
core file size (blocks, -c) 1
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127424
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 13873868
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 127424
virtual memory (kbytes, -v) 26479520
file locks (-x) unlimited
/etc/security/limits.conf
例如，把用户nmcman的limit设成8192:
nmcman soft nofile 8192
从代码角度，可以设置SO_REUSEADDR这个Socket Option:
socket.setReuseAddress(true);
使用lsof可以找出所有打开的FD， 发现很多Socket显示"can't identify protocol
"。 这个通常是由于socket用完后没有close导致的。Code改动如下：
finally{
if(socket != null)
try {
socket.close();
} catch (IOException e) {
}
}
Wed Jun 20 17:02:03 CEST 2012
->4205<-
=== begin lsof -p 22160 ===
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
。。。
java 22160 nmcman 712u sock 0,7 0t0 882473171 can't identify protocol
java 22160 nmcman 713u sock 0,7 0t0 882472188 can't identify protocol
java 22160 nmcman 714u sock 0,7 0t0 882467758 can't identify protocol
java 22160 nmcman 715u sock 0,7 0t0 882472189 can't identify protocol
java 22160 nmcman 716u sock 0,7 0t0 882470568 can't identify protocol
java 22160 nmcman 717u sock 0,7 0t0 882473172 can't identify protocol
java 22160 nmcman 718u sock 0,7 0t0 882468704 can't identify protocol
java 22160 nmcman 719u sock 0,7 0t0 882470569 can't identify protocol
java 22160 nmcman 720u sock 0,7 0t0 882467759 can't identify protocol
java 22160 nmcman 721u sock 0,7 0t0 882472190 can't identify protocol
java 22160 nmcman 722u sock 0,7 0t0 882474125 can't identify protocol
java 22160 nmcman 723u sock 0,7 0t0 882473173 can't identify protocol
java 22160 nmcman 724u sock 0,7 0t0 882468705 can't identify protocol
java 22160 nmcman 725u sock 0,7 0t0 882469535 can't identify protocol
java 22160 nmcman 726u sock 0,7 0t0 882472191 can't identify protocol
java 22160 nmcman 727u sock 0,7 0t0 882473174 can't identify protocol
java 22160 nmcman 728u sock 0,7 0t0 882470570 can't identify protocol
java 22160 nmcman 729u sock 0,7 0t0 882474126 can't identify protocol
java 22160 nmcman 730u sock 0,7 0t0 882469536 can't identify protocol
java 22160 nmcman 731u sock 0,7 0t0 882468706 can't identify protocol
java 22160 nmcman 732u sock 0,7 0t0 882467760 can't identify protocol
java 22160 nmcman 733u sock 0,7 0t0 882473175 can't identify protocol
java 22160 nmcman 734u sock 0,7 0t0 882472192 can't identify protocol
java 22160 nmcman 735u sock 0,7 0t0 882469537 can't identify protocol
java 22160 nmcman 736u sock 0,7 0t0 882474127 can't identify protocol
java 22160 nmcman 737u sock 0,7 0t0 882470571 can't identify protocol
java 22160 nmcman 738u sock 0,7 0t0 882473176 can't identify protocol
java 22160 nmcman 739u sock 0,7 0t0 882467761 can't identify protocol
java 22160 nmcman 740u sock 0,7 0t0 882468707 can't identify protocol
java 22160 nmcman 741u sock 0,7 0t0 882472193 can't identify protocol
java 22160 nmcman 742u sock 0,7 0t0 882469538 can't identify protocol
java 22160 nmcman 743u sock 0,7 0t0 882470572 can't identify protocol
java 22160 nmcman 744u sock 0,7 0t0 882474128 can't identify protocol
java 22160 nmcman 745u sock 0,7 0t0 882472194 can't identify protocol
java 22160 nmcman 746u sock 0,7 0t0 882467762 can't identify protocol
java 22160 nmcman 747u sock 0,7 0t0 882468708 can't identify protocol
java 22160 nmcman 748u sock 0,7 0t0 882470573 can't identify protocol
java 22160 nmcman 749u sock 0,7 0t0 882467763 can't identify protocol
java 22160 nmcman 750u sock 0,7 0t0 882469539 can't identify protocol
java 22160 nmcman 751u sock 0,7 0t0 882468709 can't identify protocol
java 22160 nmcman 752u sock 0,7 0t0 882472195 can't identify protocol
java 22160 nmcman 753u sock 0,7 0t0 882474129 can't identify protocol
java 22160 nmcman 754u sock 0,7 0t0 882470574 can't identify protocol
java 22160 nmcman 755u sock 0,7 0t0 882467764 can't identify protocol
java 22160 nmcman 756u sock 0,7 0t0 882469540 can't identify protocol
java 22160 nmcman 757u sock 0,7 0t0 882474130 can't identify protocol
java 22160 nmcman 758u sock 0,7 0t0 882470575 can't identify protocol
java 22160 nmcman 759u sock 0,7 0t0 882467765 can't identify protocol
java 22160 nmcman 760u sock 0,7 0t0 882469541 can't identify protocol
java 22160 nmcman 761u sock 0,7 0t0 882474131 can't identify protocol
java 22160 nmcman 762u sock 0,7 0t0 882469451 can't identify protocol
java 22160 nmcman 763u sock 0,7 0t0 882470576 can't identify protocol
java 22160 nmcman 764u sock 0,7 0t0 882469542 can't identify protocol
java 22160 nmcman 765u sock 0,7 0t0 882474132 can't identify protocol
java 22160 nmcman 766u sock 0,7 0t0 882470577 can't identify protocol
java 22160 nmcman 767u sock 0,7 0t0 882469543 can't identify protocol
java 22160 nmcman 768u sock 0,7 0t0 882468710 can't identify protocol
java 22160 nmcman 769u sock 0,7 0t0 882467766 can't identify protocol
java 22160 nmcman 770u sock 0,7 0t0 882474133 can't identify protocol
java 22160 nmcman 771u sock 0,7 0t0 882469544 can't identify protocol
java 22160 nmcman 772u sock 0,7 0t0 882468711 can't identify protocol
java 22160 nmcman 773u sock 0,7 0t0 882467767 can't identify protocol
java 22160 nmcman 774u sock 0,7 0t0 882474134 can't identify protocol
java 22160 nmcman 775u sock 0,7 0t0 882469545 can't identify protocol
java 22160 nmcman 776u sock 0,7 0t0 882472196 can't identify protocol
java 22160 nmcman 777u sock 0,7 0t0 882468712 can't identify protocol
java 22160 nmcman 778u sock 0,7 0t0 882470578 can't identify protocol
java 22160 nmcman 779u sock 0,7 0t0 882467768 can't identify protocol
java 22160 nmcman 780u sock 0,7 0t0 882469546 can't identify protocol
java 22160 nmcman 781u sock 0,7 0t0 882472197 can't identify protocol
java 22160 nmcman 782u sock 0,7 0t0 882474135 can't identify protocol
java 22160 nmcman 783u sock 0,7 0t0 882468713 can't identify protocol
java 22160 nmcman 784u sock 0,7 0t0 882470579 can't identify protocol
java 22160 nmcman 785u sock 0,7 0t0 882467769 can't identify protocol
java 22160 nmcman 786u sock 0,7 0t0 882472198 can't identify protocol
java 22160 nmcman 787u sock 0,7 0t0 882474136 can't identify protocol
java 22160 nmcman 788u sock 0,7 0t0 882469547 can't identify protocol
java 22160 nmcman 789u sock 0,7 0t0 882470580 can't identify protocol
java 22160 nmcman 790u sock 0,7 0t0 882467770 can't identify protocol
java 22160 nmcman 791u sock 0,7 0t0 882467771 can't identify protocol
java 22160 nmcman 792u sock 0,7 0t0 882472199 can't identify protocol
java 22160 nmcman 793u sock 0,7 0t0 882474137 can't identify protocol
java 22160 nmcman 794u sock 0,7 0t0 882469548 can't identify protocol
java 22160 nmcman 795u sock 0,7 0t0 882470581 can't identify protocol
java 22160 nmcman 796u sock 0,7 0t0 882474138 can't identify protocol
java 22160 nmcman 797u sock 0,7 0t0 882469549 can't identify protocol
java 22160 nmcman 798u sock 0,7 0t0 882472200 can't identify protocol
java 22160 nmcman 799u sock 0,7 0t0 882467772 can't identify protocol
java 22160 nmcman 800u sock 0,7 0t0 882469550 can't identify protocol
java 22160 nmcman 801u sock 0,7 0t0 882472201 can't identify protocol
java 22160 nmcman 802u sock 0,7 0t0 882474139 can't identify protocol
java 22160 nmcman 803u sock 0,7 0t0 882467773 can't identify protocol
java 22160 nmcman 804u sock 0,7 0t0 882469551 can't identify protocol
java 22160 nmcman 805u sock 0,7 0t0 882468715 can't identify protocol
java 22160 nmcman 806u sock 0,7 0t0 882474140 can't identify protocol
java 22160 nmcman 807u sock 0,7 0t0 882472202 can't identify protocol
java 22160 nmcman 808u sock 0,7 0t0 882469552 can't identify protocol
java 22160 nmcman 809u sock 0,7 0t0 882468716 can't identify protocol
java 22160 nmcman 810u sock 0,7 0t0 882474141 can't identify protocol
java 22160 nmcman 811u sock 0,7 0t0 882467774 can't identify protocol
java 22160 nmcman 812u sock 0,7 0t0 882472203 can't identify protocol
用strace -p <pid>能查看系统调用， 下面看到的应该是某些socket没有close完成。
[pid 22910] close(1022 <unfinished ...>
1. java.net.BindException: Address already in use.
It may be caused by the port resources are not released quickly after we call socket.close(), but be in TIME_WAIT state for some time. Below is the result.
380g7x09:/opt # netstat -na|grep TIME_WAIT|wc -l
28851
380g7x09:/opt # netstat -na|grep TIME_WAIT|grep 8080|wc -l
22326
The solution is maybe to set following parameter to 1. Or do we have other solutions?
380g7x09:/opt # sysctl -a|grep net.ipv4.tcp_tw
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
2. java.net.SocketException: Too many open files
Could we enlarge the "open files" ulimit? Currently it is 1024. FYI.
380g7x09:/opt # ulimit -a
core file size (blocks, -c) 1
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127424
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 13873868
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 127424
virtual memory (kbytes, -v) 26479520
file locks (-x) unlimited
/etc/security/limits.conf
例如，把用户nmcman的limit设成8192:
nmcman soft nofile 8192
从代码角度，可以设置SO_REUSEADDR这个Socket Option:
socket.setReuseAddress(true);
使用lsof可以找出所有打开的FD， 发现很多Socket显示"can't identify protocol
"。 这个通常是由于socket用完后没有close导致的。Code改动如下：
finally{
if(socket != null)
try {
socket.close();
} catch (IOException e) {
}
}
Wed Jun 20 17:02:03 CEST 2012
->4205<-
=== begin lsof -p 22160 ===
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
。。。
java 22160 nmcman 712u sock 0,7 0t0 882473171 can't identify protocol
java 22160 nmcman 713u sock 0,7 0t0 882472188 can't identify protocol
java 22160 nmcman 714u sock 0,7 0t0 882467758 can't identify protocol
java 22160 nmcman 715u sock 0,7 0t0 882472189 can't identify protocol
java 22160 nmcman 716u sock 0,7 0t0 882470568 can't identify protocol
java 22160 nmcman 717u sock 0,7 0t0 882473172 can't identify protocol
java 22160 nmcman 718u sock 0,7 0t0 882468704 can't identify protocol
java 22160 nmcman 719u sock 0,7 0t0 882470569 can't identify protocol
java 22160 nmcman 720u sock 0,7 0t0 882467759 can't identify protocol
java 22160 nmcman 721u sock 0,7 0t0 882472190 can't identify protocol
java 22160 nmcman 722u sock 0,7 0t0 882474125 can't identify protocol
java 22160 nmcman 723u sock 0,7 0t0 882473173 can't identify protocol
java 22160 nmcman 724u sock 0,7 0t0 882468705 can't identify protocol
java 22160 nmcman 725u sock 0,7 0t0 882469535 can't identify protocol
java 22160 nmcman 726u sock 0,7 0t0 882472191 can't identify protocol
java 22160 nmcman 727u sock 0,7 0t0 882473174 can't identify protocol
java 22160 nmcman 728u sock 0,7 0t0 882470570 can't identify protocol
java 22160 nmcman 729u sock 0,7 0t0 882474126 can't identify protocol
java 22160 nmcman 730u sock 0,7 0t0 882469536 can't identify protocol
java 22160 nmcman 731u sock 0,7 0t0 882468706 can't identify protocol
java 22160 nmcman 732u sock 0,7 0t0 882467760 can't identify protocol
java 22160 nmcman 733u sock 0,7 0t0 882473175 can't identify protocol
java 22160 nmcman 734u sock 0,7 0t0 882472192 can't identify protocol
java 22160 nmcman 735u sock 0,7 0t0 882469537 can't identify protocol
java 22160 nmcman 736u sock 0,7 0t0 882474127 can't identify protocol
java 22160 nmcman 737u sock 0,7 0t0 882470571 can't identify protocol
java 22160 nmcman 738u sock 0,7 0t0 882473176 can't identify protocol
java 22160 nmcman 739u sock 0,7 0t0 882467761 can't identify protocol
java 22160 nmcman 740u sock 0,7 0t0 882468707 can't identify protocol
java 22160 nmcman 741u sock 0,7 0t0 882472193 can't identify protocol
java 22160 nmcman 742u sock 0,7 0t0 882469538 can't identify protocol
java 22160 nmcman 743u sock 0,7 0t0 882470572 can't identify protocol
java 22160 nmcman 744u sock 0,7 0t0 882474128 can't identify protocol
java 22160 nmcman 745u sock 0,7 0t0 882472194 can't identify protocol
java 22160 nmcman 746u sock 0,7 0t0 882467762 can't identify protocol
java 22160 nmcman 747u sock 0,7 0t0 882468708 can't identify protocol
java 22160 nmcman 748u sock 0,7 0t0 882470573 can't identify protocol
java 22160 nmcman 749u sock 0,7 0t0 882467763 can't identify protocol
java 22160 nmcman 750u sock 0,7 0t0 882469539 can't identify protocol
java 22160 nmcman 751u sock 0,7 0t0 882468709 can't identify protocol
java 22160 nmcman 752u sock 0,7 0t0 882472195 can't identify protocol
java 22160 nmcman 753u sock 0,7 0t0 882474129 can't identify protocol
java 22160 nmcman 754u sock 0,7 0t0 882470574 can't identify protocol
java 22160 nmcman 755u sock 0,7 0t0 882467764 can't identify protocol
java 22160 nmcman 756u sock 0,7 0t0 882469540 can't identify protocol
java 22160 nmcman 757u sock 0,7 0t0 882474130 can't identify protocol
java 22160 nmcman 758u sock 0,7 0t0 882470575 can't identify protocol
java 22160 nmcman 759u sock 0,7 0t0 882467765 can't identify protocol
java 22160 nmcman 760u sock 0,7 0t0 882469541 can't identify protocol
java 22160 nmcman 761u sock 0,7 0t0 882474131 can't identify protocol
java 22160 nmcman 762u sock 0,7 0t0 882469451 can't identify protocol
java 22160 nmcman 763u sock 0,7 0t0 882470576 can't identify protocol
java 22160 nmcman 764u sock 0,7 0t0 882469542 can't identify protocol
java 22160 nmcman 765u sock 0,7 0t0 882474132 can't identify protocol
java 22160 nmcman 766u sock 0,7 0t0 882470577 can't identify protocol
java 22160 nmcman 767u sock 0,7 0t0 882469543 can't identify protocol
java 22160 nmcman 768u sock 0,7 0t0 882468710 can't identify protocol
java 22160 nmcman 769u sock 0,7 0t0 882467766 can't identify protocol
java 22160 nmcman 770u sock 0,7 0t0 882474133 can't identify protocol
java 22160 nmcman 771u sock 0,7 0t0 882469544 can't identify protocol
java 22160 nmcman 772u sock 0,7 0t0 882468711 can't identify protocol
java 22160 nmcman 773u sock 0,7 0t0 882467767 can't identify protocol
java 22160 nmcman 774u sock 0,7 0t0 882474134 can't identify protocol
java 22160 nmcman 775u sock 0,7 0t0 882469545 can't identify protocol
java 22160 nmcman 776u sock 0,7 0t0 882472196 can't identify protocol
java 22160 nmcman 777u sock 0,7 0t0 882468712 can't identify protocol
java 22160 nmcman 778u sock 0,7 0t0 882470578 can't identify protocol
java 22160 nmcman 779u sock 0,7 0t0 882467768 can't identify protocol
java 22160 nmcman 780u sock 0,7 0t0 882469546 can't identify protocol
java 22160 nmcman 781u sock 0,7 0t0 882472197 can't identify protocol
java 22160 nmcman 782u sock 0,7 0t0 882474135 can't identify protocol
java 22160 nmcman 783u sock 0,7 0t0 882468713 can't identify protocol
java 22160 nmcman 784u sock 0,7 0t0 882470579 can't identify protocol
java 22160 nmcman 785u sock 0,7 0t0 882467769 can't identify protocol
java 22160 nmcman 786u sock 0,7 0t0 882472198 can't identify protocol
java 22160 nmcman 787u sock 0,7 0t0 882474136 can't identify protocol
java 22160 nmcman 788u sock 0,7 0t0 882469547 can't identify protocol
java 22160 nmcman 789u sock 0,7 0t0 882470580 can't identify protocol
java 22160 nmcman 790u sock 0,7 0t0 882467770 can't identify protocol
java 22160 nmcman 791u sock 0,7 0t0 882467771 can't identify protocol
java 22160 nmcman 792u sock 0,7 0t0 882472199 can't identify protocol
java 22160 nmcman 793u sock 0,7 0t0 882474137 can't identify protocol
java 22160 nmcman 794u sock 0,7 0t0 882469548 can't identify protocol
java 22160 nmcman 795u sock 0,7 0t0 882470581 can't identify protocol
java 22160 nmcman 796u sock 0,7 0t0 882474138 can't identify protocol
java 22160 nmcman 797u sock 0,7 0t0 882469549 can't identify protocol
java 22160 nmcman 798u sock 0,7 0t0 882472200 can't identify protocol
java 22160 nmcman 799u sock 0,7 0t0 882467772 can't identify protocol
java 22160 nmcman 800u sock 0,7 0t0 882469550 can't identify protocol
java 22160 nmcman 801u sock 0,7 0t0 882472201 can't identify protocol
java 22160 nmcman 802u sock 0,7 0t0 882474139 can't identify protocol
java 22160 nmcman 803u sock 0,7 0t0 882467773 can't identify protocol
java 22160 nmcman 804u sock 0,7 0t0 882469551 can't identify protocol
java 22160 nmcman 805u sock 0,7 0t0 882468715 can't identify protocol
java 22160 nmcman 806u sock 0,7 0t0 882474140 can't identify protocol
java 22160 nmcman 807u sock 0,7 0t0 882472202 can't identify protocol
java 22160 nmcman 808u sock 0,7 0t0 882469552 can't identify protocol
java 22160 nmcman 809u sock 0,7 0t0 882468716 can't identify protocol
java 22160 nmcman 810u sock 0,7 0t0 882474141 can't identify protocol
java 22160 nmcman 811u sock 0,7 0t0 882467774 can't identify protocol
java 22160 nmcman 812u sock 0,7 0t0 882472203 can't identify protocol
用strace -p <pid>能查看系统调用， 下面看到的应该是某些socket没有close完成。
[pid 22910] close(1022 <unfinished ...>