查找线程死锁或挂起的原因

分享一个之前整理的查找线程死锁或挂起的原因；
注：服务器环境 linux ,用于C/C++编写的进程，JAVA原理类似。

常见由线程挂起导致的现象
程序处理速度由慢到严重超时，最后全部超时，重启程序会循环这一现象，那90%是线程被挂起了。
常见的线程挂起或死锁有
线程锁里面出现死循环，锁不能被释放，导致其它线程一直等待；
锁里加锁，即双重锁；
多线程编程里，共享资源没有加线程锁，造成多线程共同强夺资源而挂起。

判断进程是否挂起
使用pstree命令查看某进程的线程数：pstree -p |grep [进程名]。
例如下：

yuejctest:[/yuejc]pstree -p |grep Ywdeal
        |-Ywdeal(8969)-+-{Ywdeal}(9013) [线程1]
        |             `-{Ywdeal}(9016) [线程2]

如果在次执行此函数，发现线程数一直在增加（程序中有限制，达到限制时不在增加也不减）说明线程无法释放，可能被挂起。
什么是pstack
此命令可显示每个进程的栈跟踪，使用 pstack 来确定进程挂起的位置。此命令的唯一选项是‘要检查进程的 PID’。
pstack pid,你会得到很多信息:
例如下：

yuejctest:[/yuejc]pstack 8969
Thread 3 (Thread 0x42e88940 (LWP 9013)):
#0  0x00000039e329a0b1 in nanosleep () from /lib64/libc.so.6
#1  0x00000039e3299f99 in sleep () from /lib64/libc.so.6
#2  0x0000000000406dc9 in pthread_mdb_keepconnect ()
#3  0x00000039e3e064a7 in start_thread () from /lib64/libpthread.so.0
#4  0x00000039e32d3c2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x44e89940 (LWP 9016)):
#0  0x00000039e329a0b1 in nanosleep () from /lib64/libc.so.6
#1  0x00000039e3299f99 in sleep () from /lib64/libc.so.6
#2  0x0000000000406b49 in pthread_db_keepconnect ()
#3  0x00000039e3e064a7 in start_thread () from /lib64/libpthread.so.0
#4  0x00000039e32d3c2d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b2518adcdd0 (LWP 8969)):
#0  0x00000039e32d4f52 in msgrcv () from /lib64/libc.so.6
#1  0x000000000045eab4 in msgRcv ()
#2  0x00000000004056d0 in main ()

现实中遇到的问题，帮你如何从pstack信息中找到挂起原因：

yuejcapp2:[/yuejc]pstack 23677
Thread 12 (Thread 0x43d8b940 (LWP 23686)):
#0  0x00000032ec00d91b in read () from /lib64/libpthread.so.0
#1  0x00000000004a3735 in _NetReadSocket ()
#2  0x00000000004a3d1e in _dci_recv_msg ()
#3  0x0000000000493ca9 in _dci_query_buf ()
#4  0x0000000000494365 in _dci_send_query ()
#5  0x0000000000494daf in si_dci_query_p ()
#6  0x000000000049388a in dci_query_p ()
#7  0x000000000041df0d in mdb_stream::excuteSql() ()
#8  0x000000000041f283 in mdb_stream::open(mdb_connect&, char const*, int, int) ()
#9  0x0000000000406bfa in pthread_mdb_keepconnect(void*) ()
#10 0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0
#11 0x00000032eb4d3d1d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x45d8c940 (LWP 23687)):
#0  0x00000032eb49a1a1 in nanosleep () from /lib64/libc.so.6
#1  0x00000032eb49a089 in sleep () from /lib64/libc.so.6
#2  0x0000000000406b3d in pthread_db_keepconnect(void*) ()
#3  0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0
#4  0x00000032eb4d3d1d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x47d8d940 (LWP 19145)):
#0  0x00000032ec00d91b in read () from /lib64/libpthread.so.0
#1  0x00000000004a3735 in _NetReadSocket ()
#2  0x00000000004a3d1e in _dci_recv_msg ()
#3  0x0000000000493ca9 in _dci_query_buf ()
#4  0x0000000000494365 in _dci_send_query ()
#5  0x0000000000494daf in si_dci_query_p ()
#6  0x000000000049388a in dci_query_p ()
#7  0x000000000041df0d in mdb_stream::excuteSql() ()
#8  0x000000000048ee6b in mdb_stream::operator<<(int const&) ()
#9  0x000000000046c345 in mdb_select_userinfo_W ()
#10 0x0000000000426d2c in GetUserInfo(char*, _USER_INFO*) ()
#11 0x0000000000443005 in UserAbilityDeal(int&) ()
#12 0x000000000044f48f in ServiceOPenNewAdd(_SERVICE_OPEN_REQ&) ()
#13 0x0000000000408160 in pthread_service_open(void*) ()
#14 0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0
#15 0x00000032eb4d3d1d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x49d8e940 (LWP 19375)):
#0  0x00000032ec00d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000032ec008e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000032ec008cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004304e9 in GetOrderInfo(char*, int, _ORDER_QUERY_INFO*) ()
#4  0x0000000000441007 in GetServiceOpenMoreInfo(_SERVICE_OPEN_REQ&) ()
#5  0x0000000000407ce6 in pthread_service_open(void*) ()
#6  0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0
#7  0x00000032eb4d3d1d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x4dd90940 (LWP 19717)):
#0  0x00000032ec00d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000032ec008e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000032ec008cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004463ed in ServiceOpenChangePayType(_SERVICE_OPEN_REQ&) ()
#4  0x0000000000408444 in pthread_service_open(void*) ()
#5  0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0
#6  0x00000032eb4d3d1d in clone () from /lib64/libc.so.6
yuejcapp2:[/yuejc/log]

对以上线程信息的分析，#0表示最底层的那个函数正在处理：
<1>.线程 Thread 12 正在read ()资源，
线程 Thread 11 在nanosleep ()暂停某个线程，
线程 Thread 10 正在read ()资源，
线程 Thread 9 在__lll_lock_wait ()对资源加锁等待，
线程 Thread 8 在__lll_lock_wait ()对资源加锁等待，
<2>.根据对以上线程的分析结果，检查Thread 11 是守护进程，人为正常暂停，且此线程锁正常。
而Thread 9和Thread 8等待加锁锁定资源，是正常的等待。是什么原因让这两个线程一直等待呢？
在看一下Thread 12和Thread 10两个线程同时在read资源，造成了资源强夺现象而被挂起。
<3>.根据以上分析，检查Thread 12和Thread 10信息中的pthread_mdb_keepconnect()函数中的mdb_stream::open()函数和mdb_select_userinfo_W()函数。
发现线程Thread 12提示的mdb_stream::open()函数，在代码中没有加线程锁，增加线程锁后，程序运行正常，挂起现象解决。