一、breakpad简介
breakpad 而是一个全平台的C/C++程序的崩溃日志收集工具，适配了Windows/MacOX/Linux，当然也支持了Android。
breadpad的工作原理如下图所示:
image.png
breadpad 主要包含三个模块：
client
编译进入项目中，随项目一起编译发布，发布出去的so是strip掉debug信息的。当在用户手机上崩溃的时候，client就收集信息，写入特定格式的崩溃文件。文件最后被收集到服务端。Breakpad symbol dumper （dump_syms工具）
当你在编译so的时候，除了编译strip后的so，还得保留strip前的so。dump_syms 就是用来从strip前的so 提取符号信息.sym文件
如何获得保留strip信息的so?
假如你是用gradle编译的话，strip前的so和strip后的so都是存在的，可以在特定目录查找,如下：
└── transforms
├── mergeJniLibs
│ └── debug
│ ├── 0
│ │ └── lib
│ │ └── arm64-v8a
│ │ └── libbreakpad-core.so
│ └── __content__.json
└── stripDebugSymbol
└── debug
├── 0
│ └── lib
│ └── arm64-v8a
│ └── libbreakpad-core.so
└── __content__.json
app/build/intermediates/transforms/ 目录下，mergeJniLibs中的so是strip debug信息之前的so;stripDebugSymbol中的so是strip掉debug信息之后的so
找到未strip的so之后，可以用以下命令提取出符号信息
./dump_syms libbreakpad-core.so > libbreakpad-core.so.sy
- minidump processer（minidump_stackwalk工具）
sym符号文件和.dmp minidump文件发送给Server端后，通过minidump_stackwalk指令，从.sym符号文件和包含崩溃信息的.dmp文件中提取出完整的奔溃时的堆栈信息。
二、下载breakpad源码
可以从breadpad-github，然后编译。
很人反映从官网下载的breadpad源码编译会报错，需要补充缺失的文件。
所以也可以直接
feifei-123的github直接下载可以编译通过的版本。
执行命令:
git clone https://github.com/feifei-123/breakpad
三、编译breadpad
breakpad 是跨平台的，支持linux、window和Mac os系统，不同平台上的编译配置也是不同的。
- linux 平台编译出来的dump_syms 仅能再linux上运行，来解析linux上运行的so的符号信息
- macOS 平台编译出来的dump_syms 仅能再mac OS 上运行，来解析mac 上运行的so的符号信息。
- window 平台编译出来的dump_syms，仅能在Window上运行，并解析window上运行的dll的符号信息。
因为我最终的应用场景时 捕捉Android上的natvie crash，然后解析奔溃时的堆栈，而Android 是属于Linux系统的。
所以编译breadpad 也必须在Linux系统上进行。否则编译出来的dump_syms 无法解析android的minidump信息的。
编译过程:
清空上次build的临时文件
make clean
运行配置信息
./configure
parallels@parallels-vm:~/Desktop/Parallels Shared Folders/Home/Desktop/TM/Github/orginal_breakpad/breakpad$ ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '1000' is supported by ustar format... yes
checking whether GID '1000' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether to enable maintainer-specific portions of Makefiles... no
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for ar... ar
checking the archiver (ar) interface... ar
checking dependency style of gcc... gcc3
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking how to run the C preprocessor... gcc -E
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ranlib... ranlib
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking a.out.h usability... yes
checking a.out.h presence... yes
checking for a.out.h... yes
checking sys/random.h usability... no
checking sys/random.h presence... no
checking for sys/random.h... no
checking for arc4random... no
checking for getrandom... no
checking whether g++ supports C++11 features by default... no
checking whether g++ supports C++11 features with -std=c++11... yes
checking whether C++ compiler accepts -Werror=unknown-warning-option... no
checking whether C++ compiler accepts -Wmissing-braces... yes
checking whether C++ compiler accepts -Wnon-virtual-dtor... yes
checking whether C++ compiler accepts -Woverloaded-virtual... yes
checking whether C++ compiler accepts -Wreorder... yes
checking whether C++ compiler accepts -Wsign-compare... yes
checking whether C++ compiler accepts -Wunused-local-typedefs... yes
checking whether C++ compiler accepts -Wunused-variable... yes
checking whether C++ compiler accepts -Wvla... yes
checking for O_CLOEXEC defined in fcntl.h... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating breakpad.pc
config.status: creating breakpad-client.pc
config.status: creating Makefile
config.status: creating src/config.h
config.status: src/config.h is unchanged
config.status: executing depfiles commands
注意 checking build system type... x86_64-pc-linux-gnu 表明我当前的编译环境是linux
make 执行编译
make
编译完成后
libbreakpad_client.a 位于/src/client/linux 目录下，此文件可以编译进android 的app,完成native crash的捕捉和生成minidump文件
dump_syms 文件位于 /src/tools/linux/dump_syms 目录，用于提取so库的sym符号文件
parallels@parallels-vm:~/Desktop/Parallels Shared Folders/Home/Desktop/TM/Github/orginal_breakpad/breakpad/src/tools/linux/dump_syms$ ls
dump_syms dump_syms.cc src_tools_linux_dump_syms_dump_syms-dump_syms.o
- minidump_stackwalk 位于/src/processor目录下，用于将.dmp minudump文件和.sym文件合成可读的堆栈信息
parallels@parallels-vm:~/Desktop/Parallels Shared Folders/Home/Desktop/TM/Github/orginal_breakpad/breakpad/src/processor$ ls | grep minidump_stackwalk
minidump_stackwalk
minidump_stackwalk.cc
minidump_stackwalk_machine_readable_test
minidump_stackwalk.o
minidump_stackwalk_test
三、解析minudump文件
1、 dump_syms 提取特定so库的符号信息
以libbreakpad-core.so为例：
./dump_syms libbreakpad-core.so > libbreakpad-core.so.sym
2、根据1中生成的libbreakpad-core.so.sym生成特定的目录结构：
├── symbol
│ └── libbreakpad-core.so
│ └── 57399AA1EE2607A34686D5DED7D43C310
│ └── libbreakpad-core.so.sym
命令如下:
head -n1 libbreakpad-core.so.sym
MODULE Linux arm64 57399AA1EE2607A34686D5DED7D43C310 libbreakpad-core.so
mkdir -p ./symbol/libbreakpad-core.so/57399AA1EE2607A34686D5DED7D43C310
mv libbreakpad-core.so.sym ./symbol/libbreakpad-core.so/57399AA1EE2607A34686D5DED7D43C310/
3、调用minidump_stackwalk命令，将dmp文件和sym文件合成可读的crashinfo.txt
./minidump_stackwalk 8439c979-cecf-41a6-25eaf89c-dbf9c03b.dmp ./symbol > crashinfo.txt
crashinfo.txt 部分内容如下:
Operating system: Android
0.0.0 Linux 4.4.83 #2 SMP PREEMPT Sun Jan 12 10:48:20 CST 2020 aarch64
CPU: arm64
4 CPUs
GPU: UNKNOWN
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x0
Process uptime: not available
Thread 32 (crashed)
0 libbreakpad-core.so!testCrash1() + 0x14
x0 = 0x0000000000000027 x1 = 0x00000078e60f5f30
x2 = 0x0000000000000005 x3 = 0x0000000000000003
x4 = 0x0000000040100401 x5 = 0xa800000040404000
x6 = 0x0000000000000000 x7 = 0x7f7f7f7f7f7f7f7f
x8 = 0x0000000000000000 x9 = 0x0000000000000001
x10 = 0x00000078e60f60c0 x11 = 0x0000000000000018
x12 = 0x000000000000000b x13 = 0xffffffffffffffff
x14 = 0xff00000000000000 x15 = 0xffffffffffffffff
x16 = 0x00000078e82b7608 x17 = 0x00000078e823b454
x18 = 0x0000000000000008 x19 = 0x00000078e7e2fe00
x20 = 0x0000007902ae7f20 x21 = 0x00000078e7e2fe00
x22 = 0x00000078e60f693c x23 = 0x00000078e8f1bc31
x24 = 0x0000000000000000 x25 = 0x00000078e60f7588
x26 = 0x00000078e7e2fea0 x27 = 0x0000000000000000
x28 = 0x0000000000000000 fp = 0x00000078e60f6650
lr = 0x00000078e823b4c8 sp = 0x00000078e60f6620
pc = 0x00000078e823b468
Found by: given as instruction pointer in context
1 libbreakpad-core.so!testCoffeCacher() + 0x4c
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6650 sp = 0x00000078e60f6630
pc = 0x00000078e823b4c8
Found by: call frame info
2 libbreakpad-core.so!call_dangerous_function() + 0x14
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6670 sp = 0x00000078e60f6660
pc = 0x00000078e823b508
Found by: call frame info
3 libbreakpad-core.so!Java_com_sogou_translate_jni_BreakpadInit_go2crash + 0x14
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6690 sp = 0x00000078e60f6680
pc = 0x00000078e823b534
Found by: call frame info
4 base.odex + 0x111c0
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6768 sp = 0x00000078e60f66a0
pc = 0x00000078e8e261c4
Found by: call frame info
四、一路踩的坑
因为我自己的笔记本是mac pro,最初编译breakpad是在mac上编译的。
执行./configure & make 成功编译之后
./configure & make
feifeideMacBook-Pro:src feifei$ ./configure & make
[1] 45277
make: *** No targets specified and no makefile found. Stop.
feifeideMacBook-Pro:src feifei$ checking build system type... x86_64-apple-darwin18.5.0
checking host system type... x86_64-apple-darwin18.5.0
在/src/client/linux中未生成 dump_syms文件,
feifeideMacBook-Pro:dump_syms feifei$ pwd
/Users/feifei/Desktop/tmp/breakpad/src/src/tools/linux/dump_syms
feifeideMacBook-Pro:dump_syms feifei$ ls
dump_syms.cc
进而去/src/client/mac中 存在一个dump_syms.xcodeproj文件。
/Users/feifei/Desktop/tmp/breakpad/src/src/tools/mac/dump_syms
feifeideMacBook-Pro:dump_syms feifei$ ls
dump_syms.xcodeproj dump_syms_tool.cc macho_dump.cc
利用xcode 打开dump_syms.xcodeproj编译通过
image.png
得到dump_syms,利用该dump_syms 解析Android 工程中的 libbreakpad-core.so 失败。
feifeideMacBook-Pro:try feifei$ mac/dump_syms libbreakpad-core.so > test.txt
libbreakpad-core.so: file is neither a fat binary file nor a Mach-O object file
但是src/processor下的minidump_stackwalk 可以直接处理android 项目生成的dmp文件的.
feifeideMacBook-Pro:processor feifei$ pwd
/Users/feifei/Desktop/tmp/breakpad/src/processor
feifeideMacBook-Pro:processor feifei$ ls | grep minidump_stackwalk
minidump_stackwalk
minidump_stackwalk.cc
minidump_stackwalk.o
minidump_stackwalk_machine_readable_test
minidump_stackwalk_test
/Users/feifei/Desktop/tmp/breakpad/src/processor/minidump_stackwalk 094495a4-c61d-473e-8505f986-72daf425.dmp > crash.info
最终解决办法:安装了一个ubuntn 虚拟机，在虚拟机中对breadpad 进行编译，搞定~
五、breadpad 解析奔溃堆栈的另一种玩法
1、minidump_stackwalk 直接将.dmp 文件解析成可读的信息，但是缺少堆栈的解析(奔溃点的函数符号和行号)
Android studo安装目录自带了一份minidump_stackwalk自带的minidump_stackwalk可以在mac上直接运行 解析dmp文件
查找minidump_stackwalk位置
find / -name minidump_stackwalk
我mac上的目录为
/Applications/Android Studio.app/Contents/bin/lldb/bin/minidump_stackwalk
也可以直接使用“四”步中mac 环境编译生成的 minidump_stackwalk,位置如下:
breakpad/src/src/processor/minidump_stackwalk
minidump_stackwalk 解析minidump文件
/Users/feifei/Library/Android/sdk/lldb/3.1/bin/minidump_stackwalk 094495a4-c61d-473e-8505f986-72daf425.dmp > crash.info
crash.info 内容:
Operating system: Android
0.0.0 Linux 4.4.83 #2 SMP PREEMPT Sun Jan 12 10:48:20 CST 2020 aarch64
CPU: arm64
4 CPUs
GPU: UNKNOWN
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x0
Process uptime: not available
Thread 32 (crashed)
0 libbreakpad-core.so + 0x28468
x0 = 0x0000000000000027 x1 = 0x00000078e6b82f30
x2 = 0x0000000000000005 x3 = 0x0000000000000003
x4 = 0x0000000040100401 x5 = 0xa800000040404000
x6 = 0x0000000000000000 x7 = 0x7f7f7f7f7f7f7f7f
x8 = 0x0000000000000000 x9 = 0x0000000000000001
x10 = 0x00000078e6b830c0 x11 = 0x0000000000000018
x12 = 0x000000000000000b x13 = 0xffffffffffffffff
x14 = 0xff00000000000000 x15 = 0xffffffffffffffff
x16 = 0x00000078e86ec608 x17 = 0x00000078e8670454
x18 = 0x00000078e6b8166c x19 = 0x00000078fa1e8800
x20 = 0x0000007902ae7f20 x21 = 0x00000078fa1e8800
x22 = 0x00000078e6b8393c x23 = 0x00000078e9018c31
x24 = 0x0000000000000000 x25 = 0x00000078e6b84588
x26 = 0x00000078fa1e88a0 x27 = 0x0000000000000000
x28 = 0x0000000000000000 fp = 0x00000078e6b83650
lr = 0x00000078e86704c8 sp = 0x00000078e6b83620
pc = 0x00000078e8670468
Found by: given as instruction pointer in context
1 libbreakpad-core.so + 0x284c4
fp = 0x00000078e6b83670 lr = 0x00000078e8670508
sp = 0x00000078e6b83660 pc = 0x00000078e86704c8
Found by: previous frame's frame pointer
2 libbreakpad-core.so + 0x28504
fp = 0x00000078e6b83690 lr = 0x00000078e8670534
sp = 0x00000078e6b83680 pc = 0x00000078e8670508
Found by: previous frame's frame pointer
3 libbreakpad-core.so + 0x28530
由此可知程序 奔溃在了libbreakpad-core.so 的相对偏移位置 0x28468的地址
调用的关系如下：
libbreakpad-core.so + 0x28468
libbreakpad-core.so + 0x284c4
libbreakpad-core.so + 0x28504
libbreakpad-core.so + 0x28530
2、利用addr2line 根据发生crash的so文件，以及偏移地址，得出产生carsh的方法、行数和调用堆栈关系。
aarch64-linux-android-addr2line 工具也是在android sdk 安装目录下自带的，可以自行查找。
feifeideMacBook-Pro:~ feifei$ find / -name aarch64-linux-android-addr2line
我mac上aarch64-linux-android-addr2line的目录为
/Users/feifei/Library/Android/sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/bin/aarch64-linux-android-addr2line
针对上面得出的调用堆栈，解析libbreakpad-core.so 的 0x28468、0x284c4、0x28504、0x28530分别对应哪个函数符号
libbreakpad-core.so + 0x28468
libbreakpad-core.so + 0x284c4
libbreakpad-core.so + 0x28504
libbreakpad-core.so + 0x28530
feifeideMacBook-Pro:try feifei$ /Users/feifei/Library/Android/sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/bin/aarch64-linux-android-addr2line -f -C -e libbreakpad-core.so 0x28468 0x284c4 0x28504 0x28530
testCrash1()
??:?
testCoffeCacher()
??:?
call_dangerous_function()
??:?
Java_com_sogou_translate_jni_BreakpadInit_go2crash
??:?
可以得出实际的方法调用栈为:
testCrash1()
testCoffeCacher()
call_dangerous_function()
Java_com_sogou_translate_jni_BreakpadInit_go2crash
arm-linux-androideabi-addr2line 使用方法介绍:
arm-linux-androideabi-addr2line -C -f -e ${SOPATH} ${Address}
-C -f //打印错误行数所在的函数名称
-e //打印错误地址的对应路径及行数
${SOPATH} //so库路径
${Address} //需要转换的堆栈错误信息地址，可以添加多个，但是中间要用空格隔开
六、参考文献：
https://www.jianshu.com/p/d91c14a67072
https://www.jianshu.com/p/295ebf42b05b