(实战篇)Java服务线上死锁日志怎么看?
前言
别嫌弃日志篇幅长哈, 只要是想让看到的同学可以完整的感受到死锁的排查过程.
一、死锁栈日志信息
Found one Java-level deadlock:
=============================
"pool-1-thread-5":
waiting to lock monitor 0x00007efa20010788 (object 0x00000005f7800000, a com.alibaba.fastjson.util.ASMClassLoader),
which is held by "catalina-exec-345"
"catalina-exec-345":
waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
which is held by "catalina-exec-311"
"catalina-exec-311":
waiting for ownable synchronizer 0x00000005ed54a980, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "catalina-exec-57"
"catalina-exec-57":
waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
which is held by "catalina-exec-311"
Java stack information for the threads listed above:
===================================================
"catalina-exec-311":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005ed54a980> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:70)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.(InstMethodsInter.java:56)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhanceInstance(ClassEnhancePluginDefine.java:162)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhance(ClassEnhancePluginDefine.java:74)
at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.define(AbstractClassEnhancePluginDefine.java:75)
at org.apache.skywalking.apm.agent.SkyWalkingAgent$Transformer.transform(SkyWalkingAgent.java:145)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Transformer$Compound.transform(AgentBuilder.java:2335)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:8902)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:9306)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9269)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1300(AgentBuilder.java:9047)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9625)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9575)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9194)
at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:3148)
- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:1388)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1876)
- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1750)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:250)
at org.springframework.http.client.HttpComponentsClientHttpRequestFactory.(HttpComponentsClientHttpRequestFactory.java:67)
at com.lianjia.mls.data.processor.model.spring.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:33)
at com.lianjia.mls.data.processor.model.spring.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:15)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
- locked <0x00000005f8602888> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1081)
at com.lianjia.mls.data.processor.api.business.utils.SearchSpringUtils.getBean(SearchSpringUtils.java:25)
at com.lianjia.mls.data.processor.api.business.utils.SearchApiUtils.search(SearchApiUtils.java:43)
at com.lianjia.mls.data.processor.api.business.utils.SearchApiUtils.mapSearchResult(SearchApiUtils.java:128)
at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl.getBubbleData(DpMapSearchServiceImpl.java:194)
at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl$$FastClassBySpringCGLIB$$2ccc99c4.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl$$EnhancerBySpringCGLIB$$5d130a26.getBubbleData()
at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch$original$DZdTpsHH(AppMapSearchController.java:223)
at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch$original$DZdTpsHH$accessor$JqYP12Bi(AppMapSearchController.java)
at com.alliance.house.ljweb.controller.app.search.AppMapSearchController$auxiliary$YhLPQIyA.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch(AppMapSearchController.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3(InvocableHandlerMethod.java:133)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3$accessor$5EHWRHFG(InvocableHandlerMethod.java)
at org.springframework.web.method.support.InvocableHandlerMethod$auxiliary$cHOWfir9.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service$dkimi$impl$1005(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ke.iprd.log.filter.HttpRequestFilter.doFilter(HttpRequestFilter.java:55)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:335)
at com.dianping.cat.servlet.CatFilter$CatHandler$2.handle(CatFilter.java:122)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter$CatHandler$3.handle(CatFilter.java:139)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter$CatHandler$4.handle(CatFilter.java:269)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter$CatHandler$1.handle(CatFilter.java:75)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter.doFilter(CatFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.alliance.house.ljweb.filter.AgainstCrawlerFilter.doFilter(AgainstCrawlerFilter.java:186)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.alliance.house.ljweb.filter.AccessTokenFilter.doFilter(AccessTokenFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.alliance.house.ljweb.filter.ApiTokenFilter.doFilter(ApiTokenFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.alliance.house.ljweb.filter.WebContextFilter.doFilter(WebContextFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3(StandardHostValve.java:169)
at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3$accessor$sKW1Xg7z(StandardHostValve.java)
at org.apache.catalina.core.StandardHostValve$auxiliary$n9pFJr5w.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
- locked <0x000000076e63c630> (a org.apache.tomcat.util.net.NioChannel)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"catalina-exec-57":
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1784)
- waiting to lock <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000007a5a655e0> (a java.lang.Object)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
==========================================
at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:70)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.(InstMethodsInter.java:56)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhanceInstance(ClassEnhancePluginDefine.java:162)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhance(ClassEnhancePluginDefine.java:74)
at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.define(AbstractClassEnhancePluginDefine.java:75)
at org.apache.skywalking.apm.agent.SkyWalkingAgent$Transformer.transform(SkyWalkingAgent.java:145)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Transformer$Compound.transform(AgentBuilder.java:2335)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:8902)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:9306)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9269)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1300(AgentBuilder.java:9047)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9625)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9575)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9194)
at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:77)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:73)
at com.lianjia.common.rpc.client.transport.http.HttpClientUtils.post(HttpClientUtils.java)
at com.lianjia.common.rpc.client.transport.http.HttpTransporter.doTransport(HttpTransporter.java:36)
at com.lianjia.common.rpc.client.transport.AbstractTransporter.transport(AbstractTransporter.java:55)
at com.lianjia.common.rpc.client.impl.DefaultStatement.call(DefaultStatement.java:62)
at com.lianjia.hdic.api.client.http.template.TcpClientTemplate.service(TcpClientTemplate.java:122)
at com.lianjia.hdic.api.client.http.template.TcpClientTemplate.callService(TcpClientTemplate.java:94)
at com.lianjia.hdic.api.client.http.TcpProvider.getCitys(TcpProvider.java:51)
at com.alliance.house.service.impl.HdicServiceImpl.getCityByOfficeAddress(HdicServiceImpl.java:173)
at com.alliance.house.service.impl.HdicServiceImpl$$FastClassBySpringCGLIB$$b7650762.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.alliance.house.service.impl.CacheableHdicService$$EnhancerBySpringCGLIB$$f45f621a.getCityByOfficeAddress()
at com.alliance.house.ljweb.controller.app.AppUserController.getWorkCities$original$xLcCm5Qp(AppUserController.java:69)
at com.alliance.house.ljweb.controller.app.AppUserController.getWorkCities$original$xLcCm5Qp$accessor$pUlQ77mc(AppUserController.java)
at com.alliance.house.ljweb.controller.app.AppUserController$auxiliary$T2QOK0NL.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
at com.alliance.house.ljweb.controller.app.AppUserController.getWorkCities(AppUserController.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3(InvocableHandlerMethod.java:133)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest$original$WbsKYmi3$accessor$5EHWRHFG(InvocableHandlerMethod.java)
at org.springframework.web.method.support.InvocableHandlerMethod$auxiliary$cHOWfir9.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service$dkimi$impl$1005(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ke.iprd.log.filter.HttpRequestFilter.doFilter(HttpRequestFilter.java:55)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:335)
at com.dianping.cat.servlet.CatFilter$CatHandler$2.handle(CatFilter.java:122)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter$CatHandler$3.handle(CatFilter.java:139)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter$CatHandler$4.handle(CatFilter.java:269)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter$CatHandler$1.handle(CatFilter.java:75)
at com.dianping.cat.servlet.CatFilter$Context.handle(CatFilter.java:333)
at com.dianping.cat.servlet.CatFilter.doFilter(CatFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.alliance.house.ljweb.filter.WebContextFilter.doFilter(WebContextFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3(StandardHostValve.java:169)
at org.apache.catalina.core.StandardHostValve.invoke$original$P961naY3$accessor$sKW1Xg7z(StandardHostValve.java)
at org.apache.catalina.core.StandardHostValve$auxiliary$n9pFJr5w.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
- locked <0x000000076de69c18> (a org.apache.tomcat.util.net.NioChannel)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
二、出问题的代码
skywalking中的代码, 感兴趣可以自己load下来去看.
public class InterceptorInstanceLoader {
private static ConcurrentHashMap<String, Object> INSTANCE_CACHE = new ConcurrentHashMap<String, Object>();
private static ReentrantLock INSTANCE_LOAD_LOCK = new ReentrantLock();
private static Map<ClassLoader, ClassLoader> EXTEND_PLUGIN_CLASSLOADERS = new HashMap<ClassLoader, ClassLoader>();
/**
* Load an instance of interceptor, and keep it singleton.
* Create {@link AgentClassLoader} for each targetClassLoader, as an extend classloader.
* It can load interceptor classes from plugins, activations folders.
*
* @param className the interceptor class, which is expected to be found
* @param targetClassLoader the class loader for current application context
* @param expected type
* @return the type reference.
* @throws IllegalAccessException
* @throws InstantiationException
* @throws ClassNotFoundException
* @throws AgentPackageNotFoundException
*/
public static <T> T load(String className, ClassLoader targetClassLoader)
throws IllegalAccessException, InstantiationException, ClassNotFoundException, AgentPackageNotFoundException {
if (targetClassLoader == null) {
targetClassLoader = InterceptorInstanceLoader.class.getClassLoader();
}
String instanceKey = className + "_OF_" + targetClassLoader.getClass().getName() + "@" + Integer.toHexString(targetClassLoader.hashCode());
Object inst = INSTANCE_CACHE.get(instanceKey);
if (inst == null) {
INSTANCE_LOAD_LOCK.lock(); // 锁 一
try {
ClassLoader pluginLoader = EXTEND_PLUGIN_CLASSLOADERS.get(targetClassLoader);
if (pluginLoader == null) {
pluginLoader = new AgentClassLoader(targetClassLoader);
EXTEND_PLUGIN_CLASSLOADERS.put(targetClassLoader, pluginLoader);
}
inst = Class.forName(className, true, pluginLoader).newInstance(); // 锁 二
} finally {
INSTANCE_LOAD_LOCK.unlock();
}
if (inst != null) {
INSTANCE_CACHE.put(instanceKey, inst);
}
}
return (T) inst;
}
}
三、分析
Found one Java-level deadlock:
=============================
"pool-1-thread-5":
waiting to lock monitor 0x00007efa20010788 (object 0x00000005f7800000, a com.alibaba.fastjson.util.ASMClassLoader),
which is held by "catalina-exec-345"
"catalina-exec-345":
waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
which is held by "catalina-exec-311"
"catalina-exec-311":
waiting for ownable synchronizer 0x00000005ed54a980, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "catalina-exec-57"
"catalina-exec-57":
waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
which is held by "catalina-exec-311"
Java stack information for the threads listed above:
===================================================
从上面死锁信息可以看出，线程311和线程57在互相等待，引起死锁。
1）311 持有 监视器锁，在等待可重入锁。
2）57 持有可重入锁，在等待监视器锁。
具体看代码，实际上加锁顺序是固定的，看不出来会有死锁的场景，正当我死活看不出死锁原因的时候，同事救了我。
仔细看311线程的线程栈信息，可以看到，线程栈在获取 可重入锁 之前已经获取到 监视器锁 , 如下：
- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
终于明白了，在问题代码段之前有个地方已经锁住了 WebappClassLoader 对象了。
四、反思
- 线程栈日志会将加锁的过程体现出来 ！
- 看日志还是要仔细 ！