无core文件根据系统日志查找 程序core信息
由于系统没有设置core文件大小
[828][@zw_52_72 iproxy]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1056768
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1000000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16384
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
所以 服务core掉没有产生core文件
查看系统日志如下
Aug 11 18:50:06 zw_52_72 kernel: iproxy[18518]: segfault at 00002aad40a28000 rip 00000000004113a0 rsp 0000000043152000 error 4
Aug 11 18:50:08 zw_52_72 kernel: Trying to vfree() nonexistent vm area (ffff8101e16fc000)
Aug 11 18:50:08 zw_52_72 kernel: WARNING: at mm/vmalloc.c:329 __vunmap()
Aug 11 18:50:08 zw_52_72 kernel:
Aug 11 18:50:08 zw_52_72 kernel: Call Trace:
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff800454e4>] __free_fdtable+0x10/0x30
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff800efd9a>] free_fdtable_work+0x36/0x43
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff8004d311>] run_workqueue+0x9e/0xfb
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff80049b20>] worker_thread+0x0/0x122
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff80049c10>] worker_thread+0xf0/0x122
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff8008e857>] default_wake_function+0x0/0xe
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff80032722>] kthread+0xfe/0x132
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff80032624>] kthread+0x0/0x132
Aug 11 18:50:08 zw_52_72 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
使用objdump 反编译iproxy （ I disassembled it.）
objdump -d ./iproxy > disassembled_log
在disassembled_log中查找 segfault at 00002aad40a28000 rip 00000000004113a0 rsp 0000000043152000 error 4
标红的栈地址， 如果找不到 可以找接近于这个地址的， 看看这附近的汇编码属于那个函数
比如本次地址 属于 _ZN13IProxyProcess7IfoxA2tEPK4_A2T 调用， 就找到了服务core掉的地方
4111c4: e8 a7 99 ff ff callq 40ab70 <_Unwind_Resume@plt>
4111c9: 8b 85 a0 f2 ff ff mov 0xfffffffffffff2a0(%rbp),%eax
4111cf: 48 81 c4 78 0d 00 00 add $0xd78,%rsp
4111d6: 5b pop %rbx
4111d7: c9 leaveq
4111d8: c3 retq
4111d9: 90 nop
00000000004111da <_ZN13IProxyProcess7IfoxA2tEPK4_A2T>:
4111da: 55 push %rbp
4111db: 48 89 e5 mov %rsp,%rbp
4111de: 48 81 ec f0 0b 00 00 sub $0xbf0,%rsp
4111e5: 48 89 bd 28 f4 ff ff mov %rdi,0xfffffffffffff428(%rbp)
4111ec: 48 89 b5 20 f4 ff ff mov %rsi,0xfffffffffffff420(%rbp)
4111f3: 48 8b 85 28 f4 ff ff mov 0xfffffffffffff428(%rbp),%rax
4111fa: 48 8b 80 90 00 00 00 mov 0x90(%rax),%rax
411201: 48 85 c0 test %rax,%rax
411204: 75 38 jne 41123e <_ZN13IProxyProcess7IfoxA2tEPK4_A2T+0x64>
411206: 4c 8d 05 6b 02 0c 00 lea 787051(%rip),%r8 # 4d1478 <_ZZN11ProxyServerI12IProxyWorkerE5StartEiPPcE8__func__+0x141>
41120d: 48 8d 0d fc 06 0c 00 lea 788220(%rip),%rcx # 4d1910 <_ZZN13IProxyProcess7IfoxA2tEPK4_A2TE8__func__>
411214: ba cf 02 00 00 mov $0x2cf,%edx
411219: 48 8d 35 30 02 0c 00 lea 786992(%rip),%rsi # 4d1450 <_ZZN11ProxyServerI12IProxyWorkerE5StartEiPPcE8__func__+0x119>
411220: bf 02 00 00 00 mov $0x2,%edi
411225: b8 00 00 00 00 mov $0x0,%eax
41122a: e8 ab 63 02 00 callq 4375da <_ZN6Logger3LogEiPKcjS1_S1_z>
41131c: 66 89 50 24 mov %dx,0x24(%rax)
411320: 48 8b 85 20 f4 ff ff mov 0xfffffffffffff420(%rbp),%rax
411327: 8b 50 1e mov 0x1e(%rax),%edx
41132a: 48 8b 45 f8 mov 0xfffffffffffffff8(%rbp),%rax
41132e: 89 50 26 mov %edx,0x26(%rax)
411331: 48 8b 85 20 f4 ff ff mov 0xfffffffffffff420(%rbp),%rax
411338: 0f b7 50 22 movzwl 0x22(%rax),%edx
41133c: 48 8b 45 f8 mov 0xfffffffffffffff8(%rbp),%rax
411340: 66 89 50 2a mov %dx,0x2a(%rax)
411344: 48 8b 85 28 f4 ff ff mov 0xfffffffffffff428(%rbp),%rax
41134b: 48 8b 80 90 00 00 00 mov 0x90(%rax),%rax
411352: 8b 90 2c 01 00 00 mov 0x12c(%rax),%edx
411358: 48 8b 45 f8 mov 0xfffffffffffffff8(%rbp),%rax
41135c: 89 50 2c mov %edx,0x2c(%rax)
41135f: 48 8b 85 28 f4 ff ff mov 0xfffffffffffff428(%rbp),%rax
411366: 48 8b 80 90 00 00 00 mov 0x90(%rax),%rax
41136d: 0f b6 90 30 01 00 00 movzbl 0x130(%rax),%edx
411374: 48 8b 45 f8 mov 0xfffffffffffffff8(%rbp),%rax
411378: 88 50 30 mov %dl,0x30(%rax)
41137b: 48 8b 45 f0 mov 0xfffffffffffffff0(%rbp),%rax
41137f: 66 c7 00 39 00 movw $0x39,(%rax)
411384: 48 8b 45 f0 mov 0xfffffffffffffff0(%rbp),%rax
411388: 0f b7 00 movzwl (%rax),%eax
41138b: 0f b7 c8 movzwl %ax,%ecx
41138e: 4c 8d 8d 30 f4 ff ff lea 0xfffffffffffff430(%rbp),%r9
411395: 48 8b 55 e8 mov 0xffffffffffffffe8(%rbp),%rdx
411399: 48 8b 85 20 f4 ff ff mov 0xfffffffffffff420(%rbp),%rax
4113a0: 44 8b 54 d0 26 mov 0x26(%rax,%rdx,8),%r10d
4113a5: 48 8b 55 e8 mov 0xffffffffffffffe8(%rbp),%rdx
4113a9: 48 8b 85 20 f4 ff ff mov 0xfffffffffffff420(%rbp),%rax
4113b0: 8b 74 d0 2a mov 0x2a(%rax,%rdx,8),%esi
4113b4: 48 8b bd 28 f4 ff ff mov 0xfffffffffffff428(%rbp),%rdi
4113bb: 41 89 c8 mov %ecx,%r8d
4113be: 4c 89 c9 mov %r9,%rcx
4113c1: 44 89 d2 mov %r10d,%edx
4113c4: e8 71 8e 05 00 callq 46a23a <_ZN12ProxyProcess11AcrossProxyEjjPKhj>
4113c9: 48 83 45 e8 01 addq $0x1,0xffffffffffffffe8(%rbp)
4113ce: 48 8b 85 20 f4 ff ff mov 0xfffffffffffff420(%rbp),%rax
4113d5: 0f b7 40 24 movzwl 0x24(%rax),%eax