dubbo报Run shutdown hook now，tomcat不定时宕机

日志报了很多but has failed to stop it. This is very likely to create a memory leak.一开始分析都以为是tomcat内存不够，加大内存还是一样！也调整过dubbo线程数，问题还是没有解决，百度、google参考的资料都很少
后面分析zk日志，有出现error:keeperErrorCode=nodeExists！节点重复，zkClient登录zookeeper，然后rmr /节点进行删除节点!重启启动服务，到目前为止运行正常
以下为错误日志

[INFO][2019-01-22 21:29:04,690][com.alibaba.dubbo.config.AbstractConfig] [DUBBO] Run shutdown hook now., dubbo version: 1.0.1, current host: 192.168.1.216
[INFO][2019-01-22 21:29:04,691][com.alibaba.dubbo.registry.support.AbstractRegistryFactory] [DUBBO] Close all registries [zookeeper://192.168.1.153:2181/sh_xm/com.alibaba.dubbo.registry.RegistryServiceapplication=XM-C&client=curator&dubbo=1.0.1&group=sh_xm&interface=com.alibaba.dubbo.registry.RegistryService&pid=64730×tamp=1548146393648], dubbo version: 1.0.1, current host: 192.168.1.216
Jan 22, 2019 9:29:04 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler [“http-bio-8080”]
[INFO][2019-01-22 21:29:04,691][com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry] [DUBBO] Destroy registry:zookeeper://192.168.1.153:2181/sh_xm/com.alibaba.dubbo.registry.RegistryServiceapplication=XM-C&client=curator&dubbo=1.0.1&group=sh_xm&interface=com.alibaba.dubbo.registry.RegistryService&pid=64730×tamp=1548146393648, dubbo version: 1.0.1, current host: 192.168.1.216
[INFO][2019-01-22 21:29:04,692][com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry] [DUBBO] Unregister: consumer://192.168.1.216/default/com.pactera.sys.codeType.service.SysCodeTypeServiceIapplication=XM-C&category=consumers&check=false&default.check=false&dubbo=1.0.1&interface=com.pactera.sys.codeType.service.SysCodeTypeServiceI&methods=redisGetListValue,get,saveCodeType,redisSaveListAllRight,getListByWhereSql,redisGetListSize,listCodeType,redisDelete,redisGetListByLeftPop,redisUpdateList,redisDeleteListByValue,redisSave,redisHasKey,redisGetValue,redisGetExpire&pid=64730&side=consumer×tamp=1548146394521, dubbo version: 1.0.1, current host: 192.168.1.216
Jan 22, 2019 9:29:04 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler [“ajp-bio-8009”]
Jan 22, 2019 9:29:04 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
[INFO][2019-01-22 21:29:04,703][org.springframework.web.context.support.XmlWebApplicationContext]Closing WebApplicationContext for namespace ‘springMvc-servlet’: startup date [Tue Jan 22 16:39:57 CST 2019]; parent: Root WebApplicationContext
[INFO][2019-01-22 21:29:04,705][org.springframework.web.context.support.XmlWebApplicationContext]Closing Root WebApplicationContext: startup date [Tue Jan 22 16:39:51 CST 2019]; root of context hierarchy

Jan 22, 2019 9:29:04 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/XM-C] created a ThreadLocal with key of type [com.alibaba.dubbo.rpc.RpcContext$1] (value [com.alibaba.dubbo.rpc.RpcContext$1@5063af73]) and a value of type [com.alibaba.dubbo.rpc.RpcContext] (value [com.alibaba.dubbo.rpc.RpcContext@284d3adf]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 22, 2019 9:29:04 PM org.apache.catalina.loader.WebappClassLoaderBase loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.apache.log4j.spi.ThrowableInformation. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1813)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:165)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at com.alibaba.dubbo.common.logger.log4j.Log4jLogger.warn(Log4jLogger.java:78)
at com.alibaba.dubbo.common.logger.support.FailsafeLogger.warn(FailsafeLogger.java:100)
at com.alibaba.dubbo.remoting.zookeeper.support.AbstractZookeeperClient.close(AbstractZookeeperClient.java:98)
at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.destroy(ZookeeperRegistry.java:90)
at com.alibaba.dubbo.registry.support.AbstractRegistryFactory.destroyAll(AbstractRegistryFactory.java:70)
at com.alibaba.dubbo.config.ProtocolConfig.destroyAll(ProtocolConfig.java:465)
at com.alibaba.dubbo.config.AbstractConfig$1.run(AbstractConfig.java:452)
at java.lang.Thread.run(Thread.java:745)

[INFO][2019-01-22 21:29:04,905][com.alibaba.dubbo.config.AbstractConfig] [DUBBO] Protocol destroy in 10000ms …, dubbo version: 1.0.1, current host: 192.168.1.216
Jan 22, 2019 9:29:04 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler [“http-bio-8080”]
Jan 22, 2019 9:29:04 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler [“ajp-bio-8009”]
Jan 22, 2019 9:29:04 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler [“http-bio-8080”]
Jan 22, 2019 9:29:04 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler [“ajp-bio-8009”]
[INFO][2019-01-22 21:29:14,850][com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol] [DUBBO] disconected from /192.168.1.218:20880,url:dubbo://192.168.1.218:20880/default/com.pactera.demo.leave.service.DemoLeaveServiceIanyhost=true&application=XM-C&check=false&codec=dubbo&default.check=false&default.retries=0&default.timeout=100000&dispatcher=message&dubbo=1.0.1&generic=false&heartbeat=60000&interface=com.pactera.demo.leave.service.DemoLeaveServiceI&methods=redisGetListValue,saveLeaveApprove,get,redisSaveListAllRight,getListByWhereSql,redisGetListSize,redisDelete,redisGetListByLeftPop,redisUpdateList,redisDeleteListByValue,getCurrentUserUnDoStep,getWfParamName,redisSave,redisHasKey,getWfParamValue,redisGetValue,saveLeaveStart,redisGetExpire&pid=64730&side=consumer×tamp=1548146393614, dubbo version: 1.0.1, current host: 192.168.1.216
[INFO][2019-01-22 21:29:14,858][com.alibaba.dubbo.remoting.transport.netty.NettyChannel] [DUBBO] Close netty channel [id: 0xca2a818b, /192.168.1.216:33638 => /192.168.1.220:20880], dubbo version: 1.0.1, current host: 192.168.1.216
Exception in thread “pool-100-thread-1” java.lang.NoClassDefFoundError: com/alibaba/dubbo/rpc/protocol/dubbo/LazyConnectExchangeClient
at com.alibaba.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.replaceWithLazyClient(ReferenceCountExchangeClient.java:159)
at com.alibaba.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.close(ReferenceCountExchangeClient.java:141)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker 1. r u n ( D u b b o I n v o k e r . j a v a : 147 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r ( T h r e a d P o o l E x e c u t o r . j a v a : 1145 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r 1.run(DubboInvoker.java:147) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor 1.run(DubboInvoker.java:147)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)atjava.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.alibaba.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
… 6 more
[INFO][2019-01-22 21:29:14,859][com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol] [DUBBO] disconected from /192.168.1.220:20880,url:dubbo://192.168.1.220:20880/default/com.pactera.demo.leave.service.DemoLeaveServiceIanyhost=true&application=XM-C&check=false&codec=dubbo&default.check=false&default.retries=0&default.timeout=100000&dispatcher=message&dubbo=1.0.1&generic=false&heartbeat=60000&interface=com.pactera.demo.leave.service.DemoLeaveServiceI&methods=redisGetListValue,saveLeaveApprove,get,redisSaveListAllRight,getListByWhereSql,redisGetListSize,redisDelete,redisGetListByLeftPop,redisUpdateList,redisDeleteListByValue,getCurrentUserUnDoStep,getWfParamName,redisSave,redisHasKey,getWfParamValue,redisGetValue,saveLeaveStart,redisGetExpire&pid=64730&side=consumer×tamp=1548146393614, dubbo version: 1.0.1, current host: 192.168.1.216
Exception in thread “pool-101-thread-1” java.lang.NoClassDefFoundError: com/alibaba/dubbo/rpc/protocol/dubbo/LazyConnectExchangeClient
at com.alibaba.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.replaceWithLazyClient(ReferenceCountExchangeClient.java:159)
at com.alibaba.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.close(ReferenceCountExchangeClient.java:141)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker 1. r u n ( D u b b o I n v o k e r . j a v a : 147 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r ( T h r e a d P o o l E x e c u t o r . j a v a : 1145 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r 1.run(DubboInvoker.java:147) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor 1.run(DubboInvoker.java:147)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)atjava.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

[INFO][2019-01-22 21:29:14,908][com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol] [DUBBO] Close dubbo connect: 192.168.1.216:0–>/192.168.1.220:20880, dubbo version: 1.0.1, current host: 192.168.1.216
[INFO][2019-01-22 21:29:14,909][com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol] [DUBBO] Close dubbo connect: 192.168.1.216:0–>/192.168.1.218:20880, dubbo version: 1.0.1, current host: 192.168.1.216

踩了无数坑之后，今天终于确定了问题！因为我们之前上了一个自动化部署脚本，有个检查是否启动成功！我这个检查在shell里面使用tail -f catalina.out直接打印启动日志，就是因为这个导致tomcat时不时宕机，一开始以为是节点冲突，zookeeper也有报节点冲突