诡异的Linux磁盘空间100%

用linux的人都频繁碰到过一个问题：Linux磁盘满了，好那就开始排查问题吧！
使用df -h命令

[work@jx-qa-test06 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        40G   25G   13G  68% /
tmpfs            63G   51M   63G   1% /dev/shm
/dev/sda1       190M   62M  118M  35% /boot
/dev/sda5       1.3T  276M  1.2T  100% /home

确实磁盘是满了，于是习惯性的用du -sh * 看下是每个文件占用的空间

[work@jx-qa-test06 ~]$ du -sh *
4.0K    bin
1.8G    cidir
4.7G    cprd
24K data
12K download
12K dubbo-registry.properties
0   dubbo-registry.properties.lock
3.8G    fnrd
2.7G    iprd
17G jenkins_home
du: 无法读取目录"local/nginx/proxy_temp": 权限不够
du: 无法读取目录"local/nginx/uwsgi_temp": 权限不够
du: 无法读取目录"local/nginx/fastcgi_temp": 权限不够
du: 无法读取目录"local/nginx/scgi_temp": 权限不够
du: 无法读取目录"local/nginx/client_body_temp": 权限不够
222G    local
4.0K    nohup.out
317M    nts
3.2M    opbin
73M personneldata
540M    qa
84K tmp
351M    var
12K www
12K zk-manage
0   zk-manage.lock
16K zk-new
0   zk-new.lock

哎呀我去，好奇怪有木有，这些文件大小加起来都没有1.3T，啥情况啊！

这就到问题的根本了！好多人认为磁盘满了只是因为磁盘上面的内容多了，其实不是，磁盘满了还有一个原因就是－－“句柄”。了解计算机结构或者JVM内存模型的人应该知道“句柄”是个啥东西，此处就不讲解了哈！
直接上干货如何查询机器上那些没有用的句柄呢！

lsof -n | grep deleted

命令运行结果

[work@jx-qa-test06 ~]$ lsof -n | grep deleted
java      139634       work   39r      REG                8,5      624231   5249294 /home/work/local1/jdk1.8.0_51/jre/lib/jsse.jar (deleted)
java      139634       work   40r      REG                8,5      114687   5249298 /home/work/local1/jdk1.8.0_51/jre/lib/jce.jar (deleted)
java      139634       work   41r      REG                8,5      249403   5249282 /home/work/local1/jdk1.8.0_51/jre/lib/ext/sunpkcs11.jar (deleted)
java      139634       work   42r      REG                8,5       39773   5249285 /home/work/local1/jdk1.8.0_51/jre/lib/ext/sunec.jar (deleted)
java      139634       work   49r      REG                8,5      278433   5249288 /home/work/local1/jdk1.8.0_51/jre/lib/ext/sunjce_provider.jar (deleted)
java      139634       work   56w      REG                8,5        3662  88080391 /home/work/var/log/uc-personal/personal.error.log (deleted)
java      139634       work  177r      REG                8,5     3500533   5249389 /home/work/local1/jdk1.8.0_51/jre/lib/resources.jar (deleted)
java      139634       work  180w      REG                8,5           0  88098635 /home/work/var/log/uc-personal/personal.debug.log (deleted)
java      139634       work  188r      REG                8,5     3131363   5249479 /home/work/local1/jdk1.8.0_51/jre/lib/charsets.jar (deleted)
php-fpm   172821       work    3u      REG                8,2           0   1573219 /tmp/.ZendSem.0lHtLT (deleted)
mysqld_sa 181657       work    1u      CHR             136,22         0t0        25 /dev/pts/22 (deleted)
mysqld_sa 181657       work    2u      CHR             136,22         0t0        25 /dev/pts/22 (deleted)
mysqld    182527       work    4u      REG                8,2           0   1572873 /tmp/ib8tUCSu (deleted)
mysqld    182527       work    5u      REG                8,2           0   1572935 /tmp/ibA3SF6e (deleted)
mysqld    182527       work    6u      REG                8,2           0   1572936 /tmp/ibpti4kZ (deleted)
mysqld    182527       work    7u      REG                8,2           0   1573063 /tmp/ib37bgQK (deleted)
mysqld    182527       work   20u      REG                8,2           0   1573075 /tmp/ibrLfoH5 (deleted)

所以删掉没用的“句柄”磁盘空间就会释放好多了！使用kill -9 命令删掉占用句柄的进程就可以了！最后看一下磁盘，简直是完美。

[work@jx-qa-test06 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        40G   25G   13G  68% /
tmpfs            63G   51M   63G   1% /dev/shm
/dev/sda1       190M   62M  118M  35% /boot
/dev/sda5       1.3T  278G  981G  23% /home