Crash分析gpu非法访问地址问题
Crash分析gpu非法访问地址问题
1. 问题描述
在我司产品monkey老化过程中，极低概率出现gpu驱动访问非法地址导致kernel panic问题，在kernel panic后，主动触发ramdump机制，抓到相关的ramdump文件，利用crash工具进行离线分析。
2. crash分析ramdump
2.1 获取log信息
在crash工具中，通过dmesg或者log命令获取kernel panic时的logbuf的信息，提取关键信息如下：
[24145.263446] aliperm##error[sess_perm:1692]can't find session with id 651
[24145.291018] Unhandled fault: level 3 address size fault (0x96000043) at 0xffff00001fbaf000
[24145.291021] Mem abort info:
[24145.291024] Exception class = DABT (current EL), IL = 32 bits
[24145.291025] SET = 0, FnV = 0
[24145.291026] EA = 0, S1PTW = 0
[24145.291027] Data abort info:
[24145.291028] ISV = 0, ISS = 0x00000043
[24145.291029] CM = 0, WnR = 1
[24145.291033] Internal error: : 96000043 [#1] PREEMPT SMP
[24145.291039] Modules linked in: 8021q garp mrp bridge stp llc fts_ts veth himax_mmi xrp_hw_semidrive aes_neon_bs crc32_ce aes_neon_blk x9_ref_mach_ak7738 dwmac_dwc_qos_eth alitks_mod(PO) bcmdhd snd_sdrv_i2s_sc aliperm_mod(P) alisec_mod(P) alipatch_mod(P) alintgr_mod(P) alimbedtls_mod(P)
[24145.291075] CPU: 5 PID: 1803 Comm: pvr_defer_free Tainted: P W O 4.14.61+ #1
[24145.291076] Hardware name: Semidrive kunlun x9 REF Board (DT)
[24145.291079] task: ffff8001794fab80 task.stack: ffff00000e7d8000
[24145.291088] PC is at DeviceMemSet+0x8c/0xc8
[24145.291094] LR is at _ZeroPageArray+0x78/0x118
[24145.291096] pc : [] lr : [] pstate: 20c00145
[24145.291097] sp : ffff00000e7dbd00
[24145.291098] x29: ffff00000e7dbd00 x28: ffff80015bc9fb00
[24145.291102] x27: 00000000000186a0 x26: 0000000000000000
[24145.291105] x25: 00e8000000000f0f x24: 0000000000000080
[24145.291108] x23: ffff80012d0db000 x22: ffff00001fb7c000
[24145.291111] x21: 0000000000000000 x20: 0000000000080000
[24145.291114] x19: ffff00001fbaf000 x18: 0000ffffac000bcc
[24145.291117] x17: 0000ffff97c22ec8 x16: ffff00000818ded0
[24145.291121] x15: 0000ffffac000bc8 x14: 0140000000000000
[24145.291124] x13: ffff00001fbfc000 x12: 0000000000000000
[24145.291127] x11: 0000000000000000 x10: 0000000000000040
[24145.291130] x9 : 0040000000000041 x8 : 0040000000000001
[24145.291132] x7 : 0000000000000001 x6 : 000000017fffd7e8
[24145.291135] x5 : ffff8001398feb98 x4 : ffff8001398feb98
[24145.291138] x3 : ffff00001fbfbfff x2 : 0000000000000000
[24145.291140] x1 : 0000000000000000 x0 : ffff00001fbfc000
[24145.291146]
X4: 0xffff8001398feb18:
[24145.291147] eb18 398feb10 ffff8001 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291155] eb38 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291163] eb58 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291170] eb78 00000000 00007e28 1fac5000 ffff0000 1fb7c000 ffff0000 00000000 00000000
[24145.291178] eb98 33317c19 ffff8000 1e1c9018 ffff8000 1e1e2618 ffff8000 1e1c9030 ffff8000
[24145.291186] ebb8 1e1e2630 ffff8000 398fe5c0 ffff8001 00000000 00000000 00000000 00000000
[24145.291193] ebd8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291201] ebf8 00000000 00000000 39a12880 ffff8001 00000000 00000000 00000020 00000000
[24145.291210]
X5: 0xffff8001398feb18:
[24145.291210] eb18 398feb10 ffff8001 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291218] eb38 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291225] eb58 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291233] eb78 00000000 00007e28 1fac5000 ffff0000 1fb7c000 ffff0000 00000000 00000000
[24145.291240] eb98 33317c19 ffff8000 1e1c9018 ffff8000 1e1e2618 ffff8000 1e1c9030 ffff8000
[24145.291248] ebb8 1e1e2630 ffff8000 398fe5c0 ffff8001 00000000 00000000 00000000 00000000
[24145.291256] ebd8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291263] ebf8 00000000 00000000 39a12880 ffff8001 00000000 00000000 00000020 00000000
[24145.291276]
X23: 0xffff80012d0daf80:
[24145.291277] af80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291284] afa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291292] afc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291300] afe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291307] b000 04fcd9c0 ffff7e00 04fcd980 ffff7e00 04fce8c0 ffff7e00 04fce880 ffff7e00
[24145.291315] b020 04fcebc0 ffff7e00 04fceb80 ffff7e00 04fcf0c0 ffff7e00 04fcf080 ffff7e00
[24145.291323] b040 04fcf2c0 ffff7e00 04fcf280 ffff7e00 04fcfac0 ffff7e00 04fcfa80 ffff7e00
[24145.291330] b060 04fcffc0 ffff7e00 04fcff80 ffff7e00 0433aec0 ffff7e00 0433ae80 ffff7e00
[24145.291340]
X28: 0xffff80015bc9fa80:
[24145.291341] fa80 786f7270 5f632e79 31393533 ffff8000 5bc9fa88 ffff8001 718dbc00 ffff8001
[24145.291349] faa0 6596aa00 ffff8001 00000000 00000000 2cf5a295 00000003 00000002 00000000
[24145.291356] fac0 0000001d 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291364] fae0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291371] fb00 00000000 00000000 00000000 00000000 08760a60 ffff0000 5bc9fb00 ffff8001
[24145.291379] fb20 00000000 00000000 000000c8 00000000 00000000 00000000 eb2d59e0 ffff8000
[24145.291386] fb40 0000003c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[24145.291394] fb60 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00005a47
[24145.291402]
[24145.291404] Process pvr_defer_free (pid: 1803, stack limit = 0xffff00000e7d8000)
[24145.291405] Call trace:
[24145.291408] Exception stack(0xffff00000e7dbbc0 to 0xffff00000e7dbd00)
[24145.291411] bbc0: ffff00001fbfc000 0000000000000000 0000000000000000 ffff00001fbfbfff
[24145.291414] bbe0: ffff8001398feb98 ffff8001398feb98 000000017fffd7e8 0000000000000001
[24145.291417] bc00: 0040000000000001 0040000000000041 0000000000000040 0000000000000000
[24145.291420] bc20: 0000000000000000 ffff00001fbfc000 0140000000000000 0000ffffac000bc8
[24145.291422] bc40: ffff00000818ded0 0000ffff97c22ec8 0000ffffac000bcc ffff00001fbaf000
[24145.291425] bc60: 0000000000080000 0000000000000000 ffff00001fb7c000 ffff80012d0db000
[24145.291428] bc80: 0000000000000080 00e8000000000f0f 0000000000000000 00000000000186a0
[24145.291431] bca0: ffff80015bc9fb00 ffff00000e7dbd00 ffff0000087609c0 ffff00000e7dbd00
[24145.291434] bcc0: ffff00000876d444 0000000020c00145 ffff00000e7dbd30 ffff0000087609ac
[24145.291436] bce0: ffffffffffffffff ffff00000877e72c ffff00000e7dbd00 ffff00000876d444
[24145.291440] [] DeviceMemSet+0x8c/0xc8
[24145.291444] [] _ZeroPageArray+0x78/0x118
[24145.291447] [] _CleanupThread_CleanPages+0xfc/0x2b8
[24145.291453] [] CleanupThread+0x148/0x3a0
[24145.291455] [] OSThreadRun+0x28/0x60
[24145.291461] [] kthread+0x138/0x140
[24145.291465] [] ret_from_fork+0x10/0x1c
[24145.291469] Code: 927cec00 91004000 8b000260 b3607c42 (a9000a62)
[24145.291483] SMP: stopping secondary CPUs
[24145.291503] ---[ end trace 2abaad5900994446 ]---
[24145.335190] Kernel panic - not syncing: Fatal exception
[24145.335308] Kernel Offset: disabled
[24145.335312] CPU features: 0x0802210
[24145.335313] Memory Limit: none
[24146.006485] Rebooting in 1 seconds..
[24147.010058] flush all cache
2.2 结合log信息分析
从log看，是gpu驱动在进行memset时，访问非法地址了，初步判断，可能是buffer越界访问导致的，从PC指针获取现场信息
crash> dis DeviceMemSet+0x8c -l
/drivers/gpu/rogue_km/services/shared/common/mem_utils.c: 335
0xffff00000876d444 : stp x2, x2, [x19]
对应源码如下：
同过crash工具获取异常时现场寄存器的上下文信息如下：
crash> bt -e
PID: 1803 TASK: ffff8001794fab80 CPU: 5 COMMAND: "pvr_defer_free"
KERNEL-MODE EXCEPTION FRAME AT: ffff00000e7dbbc0
PC: ffff00000876d444 [DeviceMemSet+140]
LR: ffff0000087609c0 [_ZeroPageArray+120]
SP: ffff00000e7dbd00 PSTATE: 20c00145
X29: ffff00000e7dbd00 X28: ffff80015bc9fb00 X27: 00000000000186a0
X26: 0000000000000000 X25: 00e8000000000f0f X24: 0000000000000080
X23: ffff80012d0db000 X22: ffff00001fb7c000 X21: 0000000000000000
X20: 0000000000080000 X19: ffff00001fbaf000 X18: 0000ffffac000bcc
X17: 0000ffff97c22ec8 X16: ffff00000818ded0 X15: 0000ffffac000bc8
X14: 0140000000000000 X13: ffff00001fbfc000 X12: 0000000000000000
X11: 0000000000000000 X10: 0000000000000040 X9: 0040000000000041
X8: 0040000000000001 X7: 0000000000000001 X6: 000000017fffd7e8
X5: ffff8001398feb98 X4: ffff8001398feb98 X3: ffff00001fbfbfff
X2: 0000000000000000 X1: 0000000000000000 X0: ffff00001fbfc000
从现场寄存器信息来看，X19=ffff00001fbaf000，与panic时log提示的现场一致
反编译整个DeviceMemSet函数:
crash> dis DeviceMemSet
0xffff00000876d3b8 : stp x29, x30, [sp, #-48]!
0xffff00000876d3bc : mov x29, sp
0xffff00000876d3c0 : stp x19, x20, [sp, #16]
0xffff00000876d3c4 : str x21, [sp, #32]
0xffff00000876d3c8 : mov x19, x0
0xffff00000876d3cc : and w21, w1, #0xff
0xffff00000876d3d0 : mov x20, x2
0xffff00000876d3d4 : mov x0, x30
0xffff00000876d3d8 : nop
0xffff00000876d3dc : ands x1, x19, #0xf
0xffff00000876d3e0 : b.eq 0xffff00000876d418 // b.none
0xffff00000876d3e4 : mov x0, #0x10 // #16
0xffff00000876d3e8 : sub x0, x0, x1
0xffff00000876d3ec : cmp x0, x20
0xffff00000876d3f0 : csel x0, x0, x20, ls // ls = plast
0xffff00000876d3f4 : sub x20, x20, x0
0xffff00000876d3f8 : cbz x0, 0xffff00000876d418
0xffff00000876d3fc : add x0, x19, x0
0xffff00000876d400 : mov x1, x19
0xffff00000876d404 : strb w21, [x1]
0xffff00000876d408 : add x19, x19, #0x1
0xffff00000876d40c : mov x1, x19
0xffff00000876d410 : cmp x0, x19
0xffff00000876d414 : b.ne 0xffff00000876d404 // b.any
0xffff00000876d418 : cmp x20, #0xf
0xffff00000876d41c : b.ls 0xffff00000876d458 // b.plast
0xffff00000876d420 : lsl w2, w21, #16
0xffff00000876d424 : orr w1, w21, w21, lsl #8
0xffff00000876d428 : orr w2, w2, w21, lsl #24
0xffff00000876d42c : sub x0, x20, #0x10
0xffff00000876d430 : orr w2, w2, w1
0xffff00000876d434 : and x0, x0, #0xfffffffffffffff0
0xffff00000876d438 : add x0, x0, #0x10
0xffff00000876d43c : add x0, x19, x0
0xffff00000876d440 : bfi x2, x2, #32, #32
0xffff00000876d444 : stp x2, x2, [x19] //panic
...
结合源码和汇编信息分析，应该大概率是pDst这个指针越界了，分析上下文，获取pDst指针指向的buffer以及size如下：
从源码可以看出，PDst访问的buffer和size，是由该函数的参数决定的，因此pDst指针范围，不能超出函数参数给出的限制。因为函数调用时被调用函数会在自己的栈帧中保存即将被修改到的寄存器，因此可以从函数栈帧获取DeviceMemSet()函数的信息如下：
crash> bt -f
PID: 1803 TASK: ffff8001794fab80 CPU: 5 COMMAND: "pvr_defer_free"
#0 [ffff00000e7dbd00] DeviceMemSet at ffff00000876d440
ffff00000e7dbd00: ffff00000e7dbd30 ffff0000087609c0
ffff00000e7dbd10: 0000000000000080 0000000000000080
ffff00000e7dbd20: 0000000000000080 ffff00000877fcb4
#1 [ffff00000e7dbd30] _ZeroPageArray at ffff0000087609bc
ffff00000e7dbd30: ffff00000e7dbd80 ffff000008760b5c
ffff00000e7dbd40: ffff80015bc9fb00 ffff8000eb2d59e0
ffff00000e7dbd50: ffff000008760a60 ffff80015bc9f800
ffff00000e7dbd60: 0000000000000000 0000000000000000
ffff00000e7dbd70: ffff000009334000 ffff00000877283c
#2 [ffff00000e7dbd80] _CleanupThread_CleanPages at ffff000008760b58
ffff00000e7dbd80: ffff00000e7dbdc0 ffff000008781be0
ffff00000e7dbd90: ffff80017910b200 ffff80017910b290
ffff00000e7dbda0: ffff000008760a60 ffff000008781bd4
ffff00000e7dbdb0: ffff80017910b200 ffff80017910b290
#3 [ffff00000e7dbdc0] CleanupThread at ffff000008781bdc
ffff00000e7dbdc0: ffff00000e7dbe50 ffff00000875c258
ffff00000e7dbdd0: ffff8001783be180 ffff80017918d080
ffff00000e7dbde0: ffff8001794fab80 ffff000009b8cf70
ffff00000e7dbdf0: ffff00000a563be8 ffff8001783be180
ffff00000e7dbe00: ffff00000875c230 ffff000009313368
ffff00000e7dbe10: 0000000000000000 0000000000000000
ffff00000e7dbe20: ffff000009391ad8 ffff000009334968
ffff00000e7dbe30: ffff8001783be180 ffff80017918d900
ffff00000e7dbe40: ffff80017918d380 168cfdb4a63a5800
#4 [ffff00000e7dbe50] OSThreadRun at ffff00000875c254
ffff00000e7dbe50: ffff00000e7dbe70 ffff000008108c70
ffff00000e7dbe60: ffff8001783be980 168cfdb4a63a5800
#5 [ffff00000e7dbe70] kthread at ffff000008108c6c
在被调用函数DeviceMemSet()在栈帧中保存了以下参数：
crash> dis DeviceMemSet
0xffff00000876d3b8 : stp x29, x30, [sp, #-48]!
0xffff00000876d3bc : mov x29, sp
0xffff00000876d3c0 : stp x19, x20, [sp, #16] //保存X19,X20到栈中
0xffff00000876d3c4 : str x21, [sp, #32] //保存X21到栈中
而DeviceMemSet()函数的SP为0xffff00000e7dbd00
再看看调用DeviceMemSet()函数的父函数:
crash> dis _ZeroPageArray
0xffff000008760948 <_ZeroPageArray>: stp x29, x30, [sp, #-80]!
0xffff00000876094c <_ZeroPageArray+4>: mov x29, sp
0xffff000008760950 <_ZeroPageArray+8>: stp x20, x21, [sp, #24]
0xffff000008760954 <_ZeroPageArray+12>: str x23, [sp, #48]
0xffff000008760958 <_ZeroPageArray+16>: str x25, [sp, #64]
0xffff00000876095c <_ZeroPageArray+20>: mov w20, w0
0xffff000008760960 <_ZeroPageArray+24>: mov x23, x1
0xffff000008760964 <_ZeroPageArray+28>: mov x25, x2
0xffff000008760968 <_ZeroPageArray+32>: mov x0, x30
0xffff00000876096c <_ZeroPageArray+36>: nop
0xffff000008760970 <_ZeroPageArray+40>: mov w21, #0x400 // #1024
0xffff000008760974 <_ZeroPageArray+44>: cmp w20, w21
0xffff000008760978 <_ZeroPageArray+48>: csel w21, w20, w21, ls // ls = plast
0xffff00000876097c <_ZeroPageArray+52>: cbz w20, 0xffff0000087609e4 <_ZeroPageArray+156>
0xffff000008760980 <_ZeroPageArray+56>: str x19, [x29, #16]
0xffff000008760984 <_ZeroPageArray+60>: str x22, [x29, #40]
0xffff000008760988 <_ZeroPageArray+64>: str x24, [x29, #56]
0xffff00000876098c <_ZeroPageArray+68>: cmp w20, w21
0xffff000008760990 <_ZeroPageArray+72>: mov x3, x25
0xffff000008760994 <_ZeroPageArray+76>: csel w19, w20, w21, ls // ls = plast
0xffff000008760998 <_ZeroPageArray+80>: mov w2, #0xffffffff // #-1
0xffff00000876099c <_ZeroPageArray+84>: mov w1, w19
0xffff0000087609a0 <_ZeroPageArray+88>: mov x0, x23
0xffff0000087609a4 <_ZeroPageArray+92>: mov w24, w19
0xffff0000087609a8 <_ZeroPageArray+96>: bl 0xffff000008267958
0xffff0000087609ac <_ZeroPageArray+100>: mov x22, x0
0xffff0000087609b0 <_ZeroPageArray+104>: cbz x0, 0xffff000008760a00 <_ZeroPageArray+184>
0xffff0000087609b4 <_ZeroPageArray+108>: lsl x2, x24, #12
0xffff0000087609b8 <_ZeroPageArray+112>: mov w1, #0x0 // #0
0xffff0000087609bc <_ZeroPageArray+116>: bl 0xffff00000876d3b8
...
DeviceMemSet()函数如下方式被调用：
0xffff0000087609a4 <_ZeroPageArray+92>: mov w24, w19
0xffff0000087609a8 <_ZeroPageArray+96>: bl 0xffff000008267958
0xffff0000087609ac <_ZeroPageArray+100>: mov x22, x0
0xffff0000087609b0 <_ZeroPageArray+104>: cbz x0, 0xffff000008760a00 <_ZeroPageArray+184>
0xffff0000087609b4 <_ZeroPageArray+108>: lsl x2, x24, #12
0xffff0000087609b8 <_ZeroPageArray+112>: mov w1, #0x0 // #0
0xffff0000087609bc <_ZeroPageArray+116>: bl 0xffff00000876d3b8
可以看到，传个DeviceMemSet()的参数X1 = 0, X0 = X22, X2 = X24 << 12, X24 = X19, 而X19保存到DeviceMemSet()函数的SP + 16的地址，SP = 0xffff00000e7dbd00，所以X19寄存器的值从栈中可以获取到为0x80，因此这个buffer的size = 0x80 * 4096Byte = 0x80000，因为在DeviceMemSet()中没有使用到X22，因此出现异常时，X22还保存了跟父函数_ZeroPageArray()一样的值，为0xffff00001fb7c000
因此，这样要memset的buffer起始地址和size都获取到了，所以要memset的范围是：0xffff00001fb7c000 ~ 0xffff00001fb7c000 + 0x80000 - 1，即0xffff00001fb7c000 ~ 0xffff00001fbfc000 - 1
因为出错指令为stp指令：
stp x2, x2, [x19]
这条指令，因此当X19 = 0xffff00001fbfbff0时，就可以把该片memory清0。而0xffff00001fbfbff0 + 0x10 = 0xffff00001fbfc000，刚好是越界访问的地址，且该地址是没有映射的，所有会提示找不到相关转换的页表。
至此，锅已经找到了，就差负责炖的人了。。。