有一个docker容器,在高并发时会崩溃,进入容器没找到Java临死前的dump文件,只有一个core文件.
摸索了一下如何通过core文件定位容器中的问题,在此进行记录.
首先core文件可以通过很多工具进行分析,这里选择常用的gdb.
因为容器内没有gdb环境,我把core文件拷贝到了宿主机上了,执行命令:
gdb java core.1
结果如下:
[New LWP 140]
[New LWP 33]
warning: Corrupted shared library list: 0x0 != 0x3100000000
warning: Corrupted shared library list: 0x0 != 0x8cf00
Core was generated by `java -jar localdeployment-pd-0.0.1-SNAPSHOT.jar'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f4c65beed2f in ?? ()
Missing separate debuginfos, use: debuginfo-install java-1.8.0-openjdk-headless-1.8.0.282.b08-1.el7_9.x86_64
从这能得到core文件是由localdeployment-pd-0.0.1-SNAPSHOT.jar生成的,但是更详细的信息没找不到了
使用bt命令得不到任何信息.
(gdb) bt
#0 0x00007f4c65beed2f in ?? ()
Cannot access memory at address 0x7f4f3c00ffd0
此时意识到宿主机的jvm和容器内的不一样,遂回到容器内,安装gdb进行调试
# gdb java core.1
[New LWP 120]
[New LWP 134]
[New LWP 126]
[New LWP 143]
[New LWP 140]
[New LWP 33]
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
warning: File "/usr/glibc-compat/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /usr/glibc-compat/lib/libthread_db.so.1
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: File "/usr/glibc-compat/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `java -jar localdeployment-pd-0.0.1-SNAPSHOT.jar'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f4c65beed2f in WordSegmentorBySpace::w_handleFragment(char*, char*) () from /pd/lib/wordSegmentation/libPuncBase.so
[Current thread is 1 (LWP 146)]
此时已经能看到出问题的lib是哪个了,使用bt打印更详细的堆栈信息:
# (gdb) bt
#0 0x00007f4c65beed2f in WordSegmentorBySpace::w_handleFragment(char*, char*) () from /pd/lib/wordSegmentation/libPuncBase.so
#1 0x00007f4c65befbbb in WordSegmentorBySpace::w_segment(char*) () from /pd/lib/wordSegmentation/libPuncBase.so
#2 0x00007f4c65bf60ec in Java_com_niutrans_localdeploymentpd_jni_WordSegmentationPuncBase_DoJob () from /pd/lib/wordSegmentation/libPuncBase.so
#3 0x00007f4f459a36a2 in ?? ()
#4 0x00000007fc2f0eb8 in ?? ()
#5 0x000000000160af00 in ?? ()
#6 0x000000000160af00 in ?? ()
#7 0x00007f4f3c074ab0 in ?? ()
#8 0x00000001c05391d8 in ?? ()
#9 0x00000001c144b670 in ?? ()
#10 0x00000001c0000000 in ?? ()
#11 0x00000000005391d8 in ?? ()
#12 0x0000000000000000 in ?? ()
至此完成定位.