又一次Java线程卡死的调试经历(转载)

我的java代码，每天凌晨1点都会执行一个定时任务，定时任务中有循环执行的任务，每次循环都有开始和结束日志，某一天，我忽然发现，日志中前一天的循环只执行了几次就停止了，根据日志来看，执行到第4次循环的时候，只有开始的日志，没有结束的日志，那肯定是在里面卡死了

我首先执行

# ps -ef | grep tomcat

获取到了线程的pid

然后执行jstack

# jstack -l  10115 >> dumpFile.o

获取到线程的dump日志

 

1.  
   "scheduler-10" prio=10 tid=0x00007f6bf4004000 nid=0x7ece waiting on condition [0x00007f6be6ceb000]
2.  
   java.lang.Thread.State: WAITING (parking)
3.  
   at sun.misc.Unsafe.park(Native Method)
4.  
   - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
5.  
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
6.  
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
7.  
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
8.  
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
9.  
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
10.  
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
11.  
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
12.  
    at java.lang.Thread.run(Thread.java:744)
13.  
     
14.  
    Locked ownable synchronizers:
15.  
    - None
16.  
     
17.  
    "DubboResponseTimeoutScanTimer" daemon prio=10 tid=0x00007f6c500b2800 nid=0x7ebd sleeping[0x00007f6ce953c000]
18.  
    java.lang.Thread.State: TIMED_WAITING (sleeping)
19.  
    at java.lang.Thread.sleep(Native Method)
20.  
    at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture$RemotingInvocationTimeoutScan.run(DefaultFuture.java:300)
21.  
    at java.lang.Thread.run(Thread.java:744)
22.  
     
23.  
    Locked ownable synchronizers:
24.  
    - None
25.  
     
26.  
    "scheduler-9" prio=10 tid=0x00007f6bec135800 nid=0x7e1e waiting on condition [0x00007f6be5ddb000]
27.  
    java.lang.Thread.State: TIMED_WAITING (parking)
28.  
    at sun.misc.Unsafe.park(Native Method)
29.  
    - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
30.  
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
31.  
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
32.  
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
33.  
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
34.  
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
35.  
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
36.  
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
37.  
    at java.lang.Thread.run(Thread.java:744)
38.  
     
39.  
    Locked ownable synchronizers:
40.  
    - None
41.  
     
42.  
    "scheduler-8" prio=10 tid=0x00007f6be8011000 nid=0x7e1d waiting on condition [0x00007f6be5edc000]
43.  
    java.lang.Thread.State: WAITING (parking)
44.  
    at sun.misc.Unsafe.park(Native Method)
45.  
    - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
46.  
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
47.  
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
48.  
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
49.  
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
50.  
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
51.  
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
52.  
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
53.  
    at java.lang.Thread.run(Thread.java:744)
54.  
     
55.  
    Locked ownable synchronizers:
56.  
    - None
57.  
     
58.  
    "http-nio-10.137.12.23-9114-Acceptor-1" daemon prio=10 tid=0x00007f6dd4a02800 nid=0x7e1a runnable [0x00007f6be60df000]
59.  
    java.lang.Thread.State: RUNNABLE
60.  
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
61.  
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
62.  
    - locked <0x0000000779128e28> (a java.lang.Object)
63.  
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:809)
64.  
    at java.lang.Thread.run(Thread.java:744)
65.  
     
66.  
    Locked ownable synchronizers:
67.  
    - None
68.  
     
69.  
    "http-nio-10.137.12.23-9114-Acceptor-0" daemon prio=10 tid=0x00007f6dd4a00800 nid=0x7e19 waiting for monitor entry [0x00007f6be61e0000]
70.  
    java.lang.Thread.State: BLOCKED (on object monitor)
71.  
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
72.  
    - waiting to lock <0x0000000779128e28> (a java.lang.Object)
73.  
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:809)
74.  
    at java.lang.Thread.run(Thread.java:744)
75.  
     
76.  
    Locked ownable synchronizers:
77.  
    - None
78.  
     
79.  
    "http-nio-10.137.12.23-9114-ClientPoller-1" daemon prio=10 tid=0x00007f6dd49fe000 nid=0x7e18 runnable [0x00007f6be62e1000]
80.  
    java.lang.Thread.State: RUNNABLE
81.  
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
82.  
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
83.  
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
84.  
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
85.  
    - locked <0x00000007b5404c68> (a sun.nio.ch.Util$2)
86.  
    - locked <0x00000007b5404c78> (a java.util.Collections$UnmodifiableSet)
87.  
    - locked <0x00000007b5404c20> (a sun.nio.ch.EPollSelectorImpl)
88.  
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
89.  
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1182)
90.  
    at java.lang.Thread.run(Thread.java:744)
91.  
     
92.  
    Locked ownable synchronizers:
93.  
    - None
94.  
     
95.  
    "http-nio-10.137.12.23-9114-ClientPoller-0" daemon prio=10 tid=0x00007f6dd49fc800 nid=0x7e17 runnable [0x00007f6be63e2000]
96.  
    java.lang.Thread.State: RUNNABLE
97.  
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
98.  
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
99.  
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
100.  
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
101.  
     - locked <0x00000007b5700198> (a sun.nio.ch.Util$2)
102.  
     - locked <0x00000007b57001a8> (a java.util.Collections$UnmodifiableSet)
103.  
     - locked <0x00000007b5700150> (a sun.nio.ch.EPollSelectorImpl)
104.  
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
105.  
     at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1182)
106.  
     at java.lang.Thread.run(Thread.java:744)
107.  
      
108.  
     Locked ownable synchronizers:
109.  
     - None
110.  
      
111.  
     "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007f6dd49fb800 nid=0x7e16 waiting on condition [0x00007f6d50e4e000]
112.  
     java.lang.Thread.State: TIMED_WAITING (sleeping)
113.  
     at java.lang.Thread.sleep(Native Method)
114.  
     at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1513)
115.  
     at java.lang.Thread.run(Thread.java:744)
116.  
      
117.  
     Locked ownable synchronizers:
118.  
     - None
119.  
      
120.  
     "scheduler-7" prio=10 tid=0x00007f6bf0001800 nid=0x7e15 waiting on condition [0x00007f6be64e3000]
121.  
     java.lang.Thread.State: WAITING (parking)
122.  
     at sun.misc.Unsafe.park(Native Method)
123.  
     - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
124.  
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
125.  
     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
126.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
127.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
128.  
     at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
129.  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
130.  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
131.  
     at java.lang.Thread.run(Thread.java:744)
132.  
      
133.  
     Locked ownable synchronizers:
134.  
     - None
135.  
      
136.  
     "scheduler-6" prio=10 tid=0x00007f6cf64de800 nid=0x7e14 waiting on condition [0x00007f6be65e4000]
137.  
     java.lang.Thread.State: WAITING (parking)
138.  
     at sun.misc.Unsafe.park(Native Method)
139.  
     - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
140.  
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
141.  
     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
142.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
143.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
144.  
     at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
145.  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
146.  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
147.  
     at java.lang.Thread.run(Thread.java:744)
148.  
      
149.  
     Locked ownable synchronizers:
150.  
     - None
151.  
      
152.  
     "scheduler-5" prio=10 tid=0x00007f6bf4001800 nid=0x7e13 waiting on condition [0x00007f6be66e5000]
153.  
     java.lang.Thread.State: WAITING (parking)
154.  
     at sun.misc.Unsafe.park(Native Method)
155.  
     - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
156.  
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
157.  
     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
158.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
159.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
160.  
     at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
161.  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
162.  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
163.  
     at java.lang.Thread.run(Thread.java:744)
164.  
      
165.  
     Locked ownable synchronizers:
166.  
     - None
167.  
      
168.  
     "scheduler-4" prio=10 tid=0x00007f6cf64dc800 nid=0x7e12 waiting on condition [0x00007f6be67e6000]
169.  
     java.lang.Thread.State: WAITING (parking)
170.  
     at sun.misc.Unsafe.park(Native Method)
171.  
     - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
172.  
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
173.  
     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
174.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
175.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
176.  
     at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
177.  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
178.  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
179.  
     at java.lang.Thread.run(Thread.java:744)
180.  
      
181.  
     Locked ownable synchronizers:
182.  
     - None
183.  
      
184.  
     "scheduler-3" prio=10 tid=0x00007f6cf64da800 nid=0x7e11 waiting on condition [0x00007f6be68e7000]
185.  
     java.lang.Thread.State: WAITING (parking)
186.  
     at sun.misc.Unsafe.park(Native Method)
187.  
     - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
188.  
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
189.  
     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
190.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
191.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
192.  
     at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
193.  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
194.  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
195.  
     at java.lang.Thread.run(Thread.java:744)
196.  
      
197.  
     Locked ownable synchronizers:
198.  
     - None
199.  
      
200.  
     "scheduler-2" prio=10 tid=0x00007f6cf64d8000 nid=0x7e10 runnable [0x00007f6be69e7000]
201.  
     java.lang.Thread.State: RUNNABLE
202.  
     at java.net.PlainSocketImpl.socketConnect(Native Method)
203.  
     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
204.  
     - locked <0x00000007c4ed9b40> (a java.net.SocksSocketImpl)
205.  
     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
206.  
     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
207.  
     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
208.  
     at java.net.Socket.connect(Socket.java:579)
209.  
     at org.apache.thrift.transport.TSocket.open(TSocket.java:178)
210.  
     at com.sogou.baike.admin.service.MatchBaikeService.getMatchResult(MatchBaikeService.java:27)
211.  
     at com.sogou.baike.admin.service.LemmaCompareService.markText(LemmaCompareService.java:617)
212.  
     at com.sogou.baike.admin.service.LemmaCompareService.compareAbstract(LemmaCompareService.java:550)
213.  
     at com.sogou.baike.admin.service.LemmaCompareService.compareLemma(LemmaCompareService.java:63)
214.  
     at com.sogou.baike.admin.schedule.ParseBaiduHtmlSchedule.compareAndInsert(ParseBaiduHtmlSchedule.java:303)
215.  
     at com.sogou.baike.admin.schedule.ParseBaiduHtmlSchedule.findAndCompare(ParseBaiduHtmlSchedule.java:228)
216.  
     at com.sogou.baike.admin.schedule.ParseBaiduHtmlSchedule.startParseToLemma(ParseBaiduHtmlSchedule.java:162)
217.  
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
218.  
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
219.  
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
220.  
     at java.lang.reflect.Method.invoke(Method.java:606)
221.  
     at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
222.  
     at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
223.  
     at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
224.  
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
225.  
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
226.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
227.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
228.  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
229.  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
230.  
     at java.lang.Thread.run(Thread.java:744)
231.  
      
232.  
     Locked ownable synchronizers:
233.  
     - <0x00000007b5405620> (a java.util.concurrent.ThreadPoolExecutor$Worker)
234.  
      
235.  
     "scheduler-1" prio=10 tid=0x00007f6cf64d6800 nid=0x7e0e waiting on condition [0x00007f6be6ae9000]
236.  
     java.lang.Thread.State: WAITING (parking)
237.  
     at sun.misc.Unsafe.park(Native Method)
238.  
     - parking to wait for <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
239.  
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
240.  
     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
241.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
242.  
     at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
243.  
     at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
244.  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
245.  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
246.  
     at java.lang.Thread.run(Thread.java:744)
247.  
      
248.  
     Locked ownable synchronizers:
249.  
     - None

scheduler-* 应该是我的定时任务的名称，可以看到他们都在等待一个条件的发生，当时我以为我发现了问题所在，我以为卡死的原因就是这个，于是各种查这个被等待的资源是什么，我想到了是不是线程池满了，而线程池中的任务一直占着资源不退出？导致新的任务卡死无法继续执行？是不是因为发生了deadlock死锁？可是分析了半天这个日志，也没有看出来哪里有死锁。

 

最后继续查了很多资料，说这个应该是定时任务线程池在等待新的定时任务来执行而已，那既然这样，那这个等待日志就是正常的了。

好吧，最后我采用了最原始的debug的方式，看究竟是卡死到哪里去了。

我一步步debug，最终定位到了一段调用thrift接口的代码中

原来那个接口提供方所在的机器已经被下掉了，而我这里又没有设置超时时间，所以一直连不上，就卡死在那了。

 

原文地址：https://blog.csdn.net/u011734144/article/details/62216288