生产发生java.io.IOException: Too many open files问题解决

生产环境某个服务报Too many open files问题
1、首先确认允许打开的最大文件数是多少

[root@101 ~]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15215
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

看open files参数，结果是65535，挺大的了
2、查看系统现在打开的文件数

[root@101 ~]# lsof |wc -l
66801

3、查看java进程打开的文件数

[root@101 ~]# lsof -p 12025|wc -l
65613

12025是进程id，65613超过了系统设置的65535了
4、查看进程打开了哪些文件

[root@101 ~]# lsof -p 12025
java      12025 tomcat   92r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat   93r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat   94r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat   95r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat   96r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat   97r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat   98r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat   99r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  100r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  101r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  102r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  103r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  104r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  105r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  106r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  107r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  108r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  109r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  110r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  111r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  112r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  113r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  114r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  115r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  116r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  117r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  118r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  119r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  120r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  121r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  122r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  123r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  124r   REG              253,0  9281131     655486 /data/java/qqwry.dat
java      12025 tomcat  125r   REG              253,0  9281131     655486 /data/java/qqwry.dat

发现大量打开qqwry.dat这个文件。
5、计算一下打开qqwry.dat的次数

[root@101 ~]# lsof -p 12025|grep "qqwry.dat" |wc -l
65212

打开65212次，让开发的同学去查看与这个文件相关的代码逻辑，发现这个类的逻辑里，只管打开文件，并没有关闭文件的代码逻辑，也就是打开了，并不关闭，而且每一次请求过来，就new一个这样的对象出来，很明显这样的逻辑不妥。让其不要每次都new一个对象了，建议改成单例模式。
上线后，再来看结果

[root@101 ~]# lsof -u tomcat |grep "qqwry.dat" |wc -l
1

哈哈，这个数量变成1了，爽歪歪:)
另外，发现一个博主写的关于qqwry.dat的封装类，在这里https://blog.51cto.com/binghe001/2924976，有需要的小伙伴可以参考