一次调试Linux线程死锁经历

程序毫无反应了，ps查看仍然在运行，猜测是出现了死锁，使用gdb进行调试正在运行的程序

 

1、ps命令查看进程id

2、gdb attach 进程id

调试信息及过程如下：

(gdb) info threads
  Id   Target Id         Frame 
  41   Thread 0xb629a450 (LWP 346) "sys_main" 0xb6f7a29c in __pthread_cond_timedwait (cond=0x13c2ae0 , mutex=, abstime=0xb6299e18)
    at pthread_cond_timedwait.c:199
* 40   Thread 0xb5a9a450 (LWP 347) "sys_main" 0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
  39   Thread 0xb4a9a450 (LWP 349) "sys_main" 0xb63646f4 in select () at ../sysdeps/unix/syscall-template.S:81
  38   Thread 0xb429a450 (LWP 350) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  37   Thread 0xb3a9a450 (LWP 351) "sys_main" 0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
  36   Thread 0xb30ff450 (LWP 514) "HI_HDMI_Event" 0xb6363d5c in ioctl () at ../sysdeps/unix/syscall-template.S:81
  35   Thread 0xb2346450 (LWP 521) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  34   Thread 0xb0b72450 (LWP 526) "sys_main" 0xb637825c in __lll_lock_wait_private (futex=futex@entry=0xb0200010) at ./lowlevellock.c:30
  33   Thread 0xb01ff450 (LWP 1020) "hi_Aenc_Get" 0xb6363d58 in ioctl () at ../sysdeps/unix/syscall-template.S:81
  32   Thread 0xaf86f450 (LWP 1021) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x168ede0, mutex=) at pthread_cond_wait.c:188
  31   Thread 0xaf06f450 (LWP 1022) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x168ee38, mutex=) at pthread_cond_wait.c:188
  30   Thread 0xae86f450 (LWP 1023) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1700400, mutex=) at pthread_cond_wait.c:188
  29   Thread 0xae06f450 (LWP 1024) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1700458, mutex=) at pthread_cond_wait.c:188
  28   Thread 0xa886f450 (LWP 1035) "sys_main" 0xb6378224 in __lll_lock_wait_private (futex=futex@entry=0xb0200010) at ./lowlevellock.c:33
  27   Thread 0xa806f450 (LWP 1036) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  26   Thread 0xa786f450 (LWP 1037) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x18596f0, mutex=) at pthread_cond_wait.c:188
  25   Thread 0xa706f450 (LWP 1038) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1859748, mutex=) at pthread_cond_wait.c:188
  24   Thread 0xac06f450 (LWP 1039) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x17551b0, mutex=) at pthread_cond_wait.c:188
  23   Thread 0xac86f450 (LWP 1040) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1755208, mutex=) at pthread_cond_wait.c:188
  22   Thread 0xab06f450 (LWP 1041) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1755ac0, mutex=) at pthread_cond_wait.c:188
  21   Thread 0xab86f450 (LWP 1042) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1755b18, mutex=) at pthread_cond_wait.c:188
  20   Thread 0xaa06f450 (LWP 1043) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1793260, mutex=) at pthread_cond_wait.c:188
  19   Thread 0xaa86f450 (LWP 1044) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x17932b8, mutex=) at pthread_cond_wait.c:188
  18   Thread 0xad06f450 (LWP 1047) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x1823880, mutex=) at pthread_cond_wait.c:188
  17   Thread 0xad86f450 (LWP 1048) "sys_main" 0xb6f79edc in __pthread_cond_wait (cond=0x18238d8, mutex=) at pthread_cond_wait.c:188
  16   Thread 0xa986f450 (LWP 1049) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  15   Thread 0xa906f450 (LWP 1050) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  14   Thread 0xa686f450 (LWP 1051) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  13   Thread 0xa606f450 (LWP 1052) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  12   Thread 0xa5079450 (LWP 1053) "sys_main" 0xb636b674 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
  11   Thread 0xa4879450 (LWP 1054) "sys_main" 0xb636b674 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
  10   Thread 0xa4079450 (LWP 1055) "sys_main" 0xb636b674 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
  9    Thread 0xa3879450 (LWP 1056) "sys_main" 0xb636b674 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
  8    Thread 0xa2077450 (LWP 1100) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  7    Thread 0xa1877450 (LWP 1101) "sys_main" 0xb636b674 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
  6    Thread 0xa1077450 (LWP 1102) "sys_main" 0xb6f7a29c in __pthread_cond_timedwait (cond=0x13b2858 , mutex=, 
    abstime=0xa1076c00) at pthread_cond_timedwait.c:199
  5    Thread 0x9fe72450 (LWP 2939) "sys_main" 0xb6337690 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  4    Thread 0xb529a450 (LWP 18083) "sys_main" 0xb637825c in __lll_lock_wait_private (futex=futex@entry=0xb0200010) at ./lowlevellock.c:30
  3    Thread 0x9f471450 (LWP 26446) "sys_main" 0xb6f7cd3c in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:46
  2    Thread 0x9ec71450 (LWP 9317) "sys_main" 0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
  1    Thread 0xb6fbb000 (LWP 289) "sys_main" 0xb6f751b0 in pthread_join (threadid=, thread_return=0xbed70b10) at pthread_join.c:92
(gdb) thread 2
[Switching to thread 2 (Thread 0x9ec71450 (LWP 9317))]
#0  0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
43      in lowlevellock.c
(gdb) bt
#0  0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
#1  0xb6f767f0 in __GI___pthread_mutex_lock (mutex=0x394748 ) at pthread_mutex_lock.c:80
#2  0x000b4b20 in RTMP_Stream_Init () at rtmp_stream.c:336
#3  0x000b03cc in yb_stream_media_set_param (stream_chn=0 '\000', cmd=4, param=0x9ec70b64) at yb_stream_media.c:241
#4  0x0002b8a4 in server_msg_start_yb_rtmppush (arg=0x2c6638 ) at sys_msg_handle.c:250
#5  0xb6f73da4 in start_thread (arg=0x9ec71450) at pthread_create.c:312
#6  0xb636aff0 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /lib/a9_vfpv3_neon/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) thread 3
[Switching to thread 3 (Thread 0x9f471450 (LWP 26446))]
#0  0xb6f7cd3c in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:46
46      in lowlevellock.c
(gdb) bt
#0  0xb6f7cd3c in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:46
#1  0xb6f767f0 in __GI___pthread_mutex_lock (mutex=0x394748 ) at pthread_mutex_lock.c:80
#2  0x000b4b20 in RTMP_Stream_Init () at rtmp_stream.c:336
#3  0x000b03cc in yb_stream_media_set_param (stream_chn=0 '\000', cmd=4, param=0x9f470b64) at yb_stream_media.c:241
#4  0x0002b8a4 in server_msg_start_yb_rtmppush (arg=0x2c6638 ) at sys_msg_handle.c:250
#5  0xb6f73da4 in start_thread (arg=0x9f471450) at pthread_create.c:312
#6  0xb636aff0 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /lib/a9_vfpv3_neon/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) thread 4
[Switching to thread 4 (Thread 0xb529a450 (LWP 18083))]
#0  0xb637825c in __lll_lock_wait_private (futex=futex@entry=0xb0200010) at ./lowlevellock.c:30
30      ./lowlevellock.c: No such file or directory.
(gdb) bt
#0  0xb637825c in __lll_lock_wait_private (futex=futex@entry=0xb0200010) at ./lowlevellock.c:30
#1  0xb630c140 in get_free_list () at arena.c:783
#2  arena_get2 (size=size@entry=17176, avoid_arena=avoid_arena@entry=0x0, a_tsd=) at arena.c:837
#3  0xb630d474 in __libc_calloc (n=, elem_size=) at malloc.c:3186
#4  0xb69f55ac in RTMP_Alloc () at rtmp.c:541
#5  0x000b4be0 in RTMP_Stream_Init () at rtmp_stream.c:348
#6  0x000b03cc in yb_stream_media_set_param (stream_chn=0 '\000', cmd=4, param=0xb5299b64) at yb_stream_media.c:241
#7  0x0002b8a4 in server_msg_start_yb_rtmppush (arg=0x2c6638 ) at sys_msg_handle.c:250
#8  0xb6f73da4 in start_thread (arg=0xb529a450) at pthread_create.c:312
#9  0xb636aff0 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /lib/a9_vfpv3_neon/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) thread 37
[Switching to thread 37 (Thread 0xb3a9a450 (LWP 351))]
#0  0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
43      lowlevellock.c: No such file or directory.
(gdb) bt
#0  0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
#1  0xb6f767f0 in __GI___pthread_mutex_lock (mutex=0x394748 ) at pthread_mutex_lock.c:80
#2  0x000b3e88 in check_channel (chn=0) at rtmp_stream.c:86
#3  0x000b71e4 in RTMP_Stream_GetPushInfo (chn=0, push_info=0xb3a9960c) at rtmp_stream.c:946
#4  0x000b0b1c in yb_stream_media_set_param (stream_chn=0 '\000', cmd=10, param=0xb3a9960c) at yb_stream_media.c:342
#5  0x0002dacc in sys_msg_cmd_cb (msg_source=0, cmd=53, arg=0xb3a9972c) at sys_msg_handle.c:1271
#6  0x00039870 in yb_device_web_pushstatus (ly_rtmp_flag=0, ly_online_flag=1) at yb_server_msg.c:4619
#7  0x0003c2ac in yb_push_status_to_ybserver (adata=0x0) at yb_server_msg.c:5811
#8  0xb6f73da4 in start_thread (arg=0xb3a9a450) at pthread_create.c:312
#9  0xb636aff0 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /lib/a9_vfpv3_neon/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) thread 40
[Switching to thread 40 (Thread 0xb5a9a450 (LWP 347))]
#0  0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
43      in lowlevellock.c
(gdb) bt
#0  0xb6f7cd74 in __lll_lock_wait (futex=futex@entry=0x394748 , private=0) at lowlevellock.c:43
#1  0xb6f767f0 in __GI___pthread_mutex_lock (mutex=0x394748 ) at pthread_mutex_lock.c:80
#2  0x000b3e88 in check_channel (chn=0) at rtmp_stream.c:86
#3  0x000b4cec in RTMP_Stream_Exit (chn=0) at rtmp_stream.c:367
#4  0x000b0528 in yb_stream_media_set_param (stream_chn=0 '\000', cmd=6, param=0x0) at yb_stream_media.c:265
#5  0x0002bcb8 in stop_rtmp_pushing () at sys_msg_handle.c:349
#6  0x0002db48 in sys_msg_cmd_cb (msg_source=1, cmd=52, arg=0x0) at sys_msg_handle.c:1285
#7  0x00026f04 in media_dealcmd_chnop (recv_msg=0xb5a99a0c, module=1 '\001') at sys_main.c:1639
#8  0x00029278 in sys_main_msg_handle (arg=0x0) at sys_main.c:2256
#9  0xb6f73da4 in start_thread (arg=0xb5a9a450) at pthread_create.c:312
#10 0xb636aff0 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /lib/a9_vfpv3_neon/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) p handle_mutex
$4 = {__data = {__lock = 2, __count = 0, __owner = 18083, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, 
  __size = "\002\000\000\000\000\000\000\000\243F\000\000\000\000\000\000\001\000\000\000\000\000\000", __align = 2}
(gdb) p stream_mutex[0]
$5 = {__data = {__lock = 1, __count = 0, __owner = 18083, __kind = 0, __nusers = 0, {__spins = 0, __list = {__next = 0x0}}}, 
  __size = "\001\000\000\000\000\000\000\000\243F", '\000' , __align = 1}
(gdb) p stream_mutex[1]
$6 = {__data = {__lock = 1, __count = 0, __owner = 14992, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, 
  __size = "\001\000\000\000\000\000\000\000\220:\000\000\000\000\000\000\001\000\000\000\000\000\000", __align = 1}
(gdb) p stream_mutex[2]
$7 = {__data = {__lock = 0, __count = 0, __owner = 0, __kind = 0, __nusers = 0, {__spins = 0, __list = {__next = 0x0}}}, __size = '\000' , 
  __align = 0}
(gdb) 

线程2、3、37和40在等到handle_mutex锁，但是handle_mutex被线程4拥有，查看堆栈发现是到alloc之后出不来。认真检查代码后发现有一处锁没有对应上，但是为何在alloc的时候出不来呢？有些疑惑