linux的clientmqueue问题和No space left on device
今日打开程序发现无法访问，一看后台日志提示
2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Complete Status: Managed Threads: 3 Active Threads: 3 Active Tasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3f0b70 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1) com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1080f38 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2) com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f5438 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0) Pending Tasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@d26103 com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1df3255 Pool thread stack traces: Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main] java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(Unknown Source) oracle.net.ns.Packet.receive(Unknown Source) oracle.net.ns.NSProtocol.connect(Unknown Source) oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858) oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268) oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441) oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135) com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129) com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862) com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33) com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527) com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main] java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(Unknown Source) oracle.net.ns.Packet.receive(Unknown Source) oracle.net.ns.NSProtocol.connect(Unknown Source) oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858) oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268) oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441) oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135) com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129) com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862) com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33) com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527) com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main] java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(Unknown Source) oracle.net.ns.Packet.receive(Unknown Source) oracle.net.ns.NSProtocol.connect(Unknown Source) oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858) oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268) oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441) oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135) com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129) com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862) com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33) com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527) com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
然后连接oracle的工具也打不开，感觉需要重启oracle服务，但是去服务器采用Linu命令重启后就出现【No space left on device 】
[oracle@SERVER233 bin]$ lsnrctl start LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 29-NOV-2012 10:30:23 Copyright (c) 1991, 2005, Oracle. All rights reserved. Starting /oracle10g/app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait... TNSLSNR for Linux: Version 10.2.0.1.0 - Production System parameter file is /oracle10g/app/oracle/product/10.2.0/db_1/network/admin/listener.ora Log messages written to /oracle10g/app/oracle/product/10.2.0/db_1/network/log/listener.log Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1))) TNS-12549: TNS:operating system resource quota exceeded TNS-12560: TNS:protocol adapter error TNS-00519: Operating system resource quota exceeded Linux Error: 28: No space left on device
df -i
出现
[oracle@SERVER233 bin]$ df -i Filesystem Inode (I)已用 (I)可用 (I)已用% 挂载点 /dev/sda11 25608192 191904 25416288 1% / /dev/sda1 6406144 44 6406100 1% /boot none 214052 1 214051 1% /dev/shm /dev/sda10 6406144 212 6405932 1% /home /dev/sda5 25608192 19 25608173 1% /opt /dev/sda2 29949952 12 29949940 1% /sowft /dev/sda9 2562240 60 2562180 1% /tmp /dev/sda8 12812288 305651 12506637 3% /usr /dev/sda6 2562240 991 2561249 1% /usr/local /dev/sda7 6406144 6406144 0 100% /var [oracle@SERVER233 bin]$ df -h Filesystem 容量 已用 可用 已用% 挂载点 /dev/sda11 193G 17G 167G 9% / /dev/sda1 49G 91M 46G 1% /boot none 3.0G 0 3.0G 0% /dev/shm /dev/sda10 49G 135M 46G 1% /home /dev/sda5 193G 93M 183G 1% /opt /dev/sda2 225G 731M 213G 1% /sowft /dev/sda9 20G 77M 19G 1% /tmp /dev/sda8 97G 6.4G 85G 7% /usr /dev/sda6 20G 177M 19G 1% /usr/local /dev/sda7 49G 26G 21G 55% /var觉得是var目录占用空间比较大 然后进入 clientmqueue 发现占用比较大
日志是定期分析处理的，怎么会占这么大的空间？
查看日志文件大小
# du -sh /var/log
31M /var/log
好小耶，那到底是什么文件占用了大量空间了？
继续找。
du -sh /var/spool/ clientmqueue/
889M /var/spool/ clientmqueue
找到 clientmqueue 目录占了大量空间。
这个目录底下的文件到底是干嘛的？？
分析：
系统中有用户开启了cron，而cron中执行的程序有输出内容，输出内容会以邮件形式发给cron的用户，而sendmail没有启动所以就产生了这些文件；
解决办法:
将crontab里面的命令后面加上> /dev/null 2>&1
注：2>&1：把错误重定向到输出要送到的地方。即把上述命令的执行结果重定向到/dev/null，即抛弃，同时，把产生的错误也抛弃。
rm -rf /var/spool/ clientmqueue 删除这个目录。
重新修改所有的cron
这样的话就解决了clinetmqueue占用空间的问题。
参考网址：http://baozoutianshi.blog.sohu.com/203000776.html