服务器磁盘爆满引发的dubbo请求超时问题
今天遇到了运用dubbo开发过程中经常遇到的问题，请求超时
Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method subscribe in the service com.alibaba.dubbo.registry.RegistryService. Tried 3 times of the providers [192.168.146.125:6060] (1/1) from the registry 192.168.146.125:6060 on the consumer 10.14.137.59 using the dubbo version 2.4.5. Last error is: Invoke remote method timeout. method: subscribe, provider: dubbo://192.168.146.125:6060/com.alibaba.dubbo.registry.RegistryService?application=im-tracker&callbacks=10000&check=false&connect.timeout=10000&dubbo=2.4.5&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&methods=register,subscribe,unregister,unsubscribe,lookup&organization=chat.jd.com&owner=lvsheng&pid=6820&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1455521782295&unsubscribe.1.callback=false, cause: Waiting server-side response timeout. start time: 2016-02-15 15:36:42.643, end time: 2016-02-15 15:36:52.644, client elapsed: 0 ms, server elapsed: 10001 ms, timeout: 10000 ms, request: Request [id=2, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=subscribe, parameterTypes=[class com.alibaba.dubbo.common.URL, interface com.alibaba.dubbo.registry.NotifyListener], arguments=[consumer://10.14.137.59/com.alibaba.dubbo.registry.RegistryService?application=im-tracker&callbacks=10000&connect.timeout=10000&dubbo=2.4.5&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&methods=register,subscribe,unregister,unsubscribe,lookup&organization=chat.jd.com&owner=lvsheng&pid=6820&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1455521782295&unsubscribe.1.callback=false, com.alibaba.dubbo.registry.integration.RegistryDirectory@c4d412], attachments={sys_callback_arg-1=12899346, path=com.alibaba.dubbo.registry.RegistryService, interface=com.alibaba.dubbo.registry.RegistryService, timeout=10000, version=0.0.0}]], channel: /10.14.137.59:56595 -> /192.168.146.125:6060
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy0.subscribe(proxy0.java)
at com.alibaba.dubbo.registry.dubbo.DubboRegistry.doSubscribe(DubboRegistry.java:138)
at com.alibaba.dubbo.registry.support.FailbackRegistry.subscribe(FailbackRegistry.java:189)
... 45 more
Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side response timeout. start time: 2016-02-15 15:36:42.643, end time: 2016-02-15 15:36:52.644, client elapsed: 0 ms, server elapsed: 10001 ms, timeout: 10000 ms, request: Request [id=2, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=subscribe, parameterTypes=[class com.alibaba.dubbo.common.URL, interface com.alibaba.dubbo.registry.NotifyListener], arguments=[consumer://10.14.137.59/com.alibaba.dubbo.registry.RegistryService?application=im-tracker&callbacks=10000&connect.timeout=10000&dubbo=2.4.5&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&methods=register,subscribe,unregister,unsubscribe,lookup&organization=chat.jd.com&owner=lvsheng&pid=6820&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1455521782295&unsubscribe.1.callback=false, com.alibaba.dubbo.registry.integration.RegistryDirectory@c4d412], attachments={sys_callback_arg-1=12899346, path=com.alibaba.dubbo.registry.RegistryService, interface=com.alibaba.dubbo.registry.RegistryService, timeout=10000, version=0.0.0}]], channel: /10.14.137.59:56595 -> /192.168.146.125:6060
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:107)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:84)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96)
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)
at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
... 51 more
正常思路是，服务方压力过大，导致请求阻塞，长时间没有返回。但是照这条线下去排查根本不行。
这个时候用df命令查机器磁盘
[root@localhost /]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 20G 20G 0 100% / tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 9.9G 172M 9.2G 2% /boot /dev/sda6 71G 503M 67G 1% /data /dev/sda2 40G 282M 38G 1% /usr/local
根目录所在磁盘满了，把磁盘清理了下，问题得以修复。
一个合理的解释是，当我发出请求后，服务提供方进入服务流程，但是需要向OS申请磁盘资源。可惜OS没有，于是服务提供方进入阻塞状态。