Linux kernel slub debug
在device driver开发的过程中，有时候会碰到memory相关的问题，比如memory leak，use after free，out of boundary，double free等等。因为device driver运行在kernel环境，kernel是一个global的runtime，driver分配或者释放的memory都是由kernel负责管理，而且所有的device driver，kernel sub system共享这些memory，一旦device driver使用memory不当，就会导致整个kernel的memory管理出现问题。举个简单的例子，比如device driver写buffer发生了越界，那就可能把别人的memory写坏，如果正巧kernel或者别的driver正在使用这块memory，那就立刻会发生oops，或者kernel panic；如果不够幸运，被写坏的memory目前没人在用，而是将来的某个时刻才触发oops或者kernel panic，那就非常麻烦了，因为stack是当前正在运行的上下文的stack，真正的锅应该由写坏别人buffer的driver来背。
kernel中提供了这样一种机制，当访问memory出现问题时，通过记录的track，找到始作俑者，这个机制就是slub debug。
碰巧，最近测试unbind driver的时候，碰到了几个memory相关的问题，比如use after free，double free，甚至是free了栈里的memory ：（，unbind driver是要把driver之前分配的所有resource全部释放，中间就会涉及到大量的memory free的操作，自然容易出现问题。
slub debug的实现原理很简单，就是在要分的memory周围放上围栏，正常的memory访问都只会集中在memory object里面，不可能在外面，这样只要监视memory object周围的读写，就能发现不正常的memory访问，尤其是oob，也就是out of boundary，越界访问，这个围栏就是red zone。此外，为了检查use after free，kernel把当前未被使用的object里的内容全部填成特殊值，一旦发现有人在没有分配的情况下，写了object的值，就会报错，从而发现肇事者。
下面介绍slub debug的使用方法，因为debug功能默认是关闭的，所以使用slub debug之前要build kernel。
1. 先下载kernel的source code，可以直接下载kernel official的code
2. 修改config，把自己机器里的config copy到kernel code根目录下，重命名为.config，然后make menuconfig
Kernel hacking -> Memory debugging -> Slub debugging on by default
General setup -> Enable SLUB debugging support
3. build kernel，推荐使用如下命令：
export DEB_BUILD_OPTIONS='nostrip noopt debug'
make -j8 bindeb-pkg LOCALVERSION=-debug
这样可以build出来linux headers，Linux image和linux kenrel symbol，这些debug用都是需要的。
4，安装刚刚build 出来的kernel，然后就可以安装driver运行了。
slub可以检测到double free，out of boundary等，但是并不是实时的，最开始的时候并不知道，所以当看到slub的call stack时一脸懵逼，这stack压根就不是现在跑的啊。没办法，slub的debug就是有它先天的滞后性，因为它是在free memory的时候去检查，而且不是free完马上去检查，而是等到后面有人再去使用的时候才检查，这就导致看不到出问题的现场。针对这个问题，kernel提供了自己的解决办法-slabinfo，这是kernel自己提供的工具，code在tools/vm下，通过slabinfo -v，可以马上对memory进行检查，一旦发现问题，立即打印，这就要求driver developer知道memory是在哪个阶段出现问题的，才能有针对性的使用slub debug。
所以，我一般在怀疑有可能出现问题的时候，手动运行一次slabinfo -v检查当前所有的slab object，下面看几个例子。
#include
#include
#include
#include
#include
static int oob(void)
{
char *buf = kmalloc(8, GFP_KERNEL);
printk(KERN_INFO "test oob.\n");
//这里写越界，分了8个byte，写了第9个byte
buf[8] = 'a';
kfree(buf);
return 0;
}
static int __init hello_init(void)
{
printk(KERN_INFO "module init.\n");
oob();
return 0;
}
static void hello_exit(void)
{
printk(KERN_INFO "module exit.\n");
}
module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
在insmod这个module之后，手动执行slabinfo -v，在dmesg中就可以看到log了：
[ 204.459596] module init.
[ 204.462197] test oob.
[ 204.464700] =============================================================================
[ 204.473065] BUG kmalloc-8 (Tainted: P OE ): Redzone overwritten
[ 204.480077] -----------------------------------------------------------------------------
[ 204.489899] INFO: 0x00000000e7a736f4-0x00000000e7a736f4. First byte 0x61 instead of 0xcc
[ 204.498151] INFO: Allocated in hello_init+0x2c/0x1000 [hello] age=0 cpu=3 pid=2120
[ 204.505695] __slab_alloc+0x20/0x40
[ 204.509308] kmem_cache_alloc_trace+0x238/0x270
[ 204.513898] hello_init+0x2c/0x1000 [hello]
[ 204.518199] do_one_initcall+0x52/0x1d7
[ 204.522081] do_init_module+0x5f/0x221
[ 204.525799] load_module+0x2697/0x2b30
[ 204.529524] __do_sys_finit_module+0xe5/0x120
[ 204.534063] __x64_sys_finit_module+0x1a/0x20
[ 204.538428] do_syscall_64+0x5a/0x110
[ 204.542140] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 204.547264] INFO: Slab 0x000000003ae1cdd1 objects=23 used=2 fp=0x000000001974d745 flags=0x17ffffc0008101
[ 204.556900] INFO: Object 0x000000003fa16067 @offset=1384 fp=0x (null)
[ 204.565545] Redzone 00000000c5553bcc: cc cc cc cc cc cc cc cc ........
[ 204.574360] Object 000000003fa16067: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk.
[ 204.583092] Redzone 00000000e7a736f4: 61 cc cc cc cc cc cc cc a.......
[ 204.591952] Padding 0000000020f050b3: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[ 204.600599] CPU: 3 PID: 2120 Comm: insmod Tainted: P B OE 4.19.65-devin #8
[ 204.608638] Hardware name: Shanghai Semiconductor Co., Ltd. HX002EA/HX002EA, BIOS HX002EA0_03_R440_R_190912 09/12/2019
[ 204.620206] Call Trace:
[ 204.622688] dump_stack+0x63/0x85
[ 204.626078] print_trailer+0x149/0x210
[ 204.629978] check_bytes_and_report+0xc2/0xf0
[ 204.634343] check_object+0x1ea/0x290
[ 204.638080] free_debug_processing+0x14e/0x330
[ 204.642666] ? hello_init+0x47/0x1000 [hello]
[ 204.647031] __slab_free+0x226/0x370
[ 204.650753] ? vprintk_default+0x29/0x50
[ 204.654664] ? vprintk_func+0x44/0xe0
[ 204.658344] ? hello_init+0x47/0x1000 [hello]
[ 204.662838] kfree+0x1c6/0x1d0
[ 204.665880] ? kfree+0x1c6/0x1d0
[ 204.669089] hello_init+0x47/0x1000 [hello]
[ 204.673267] ? 0xffffffffc051d000
[ 204.676729] do_one_initcall+0x52/0x1d7
[ 204.680556] ? kmem_cache_alloc_trace+0x16d/0x270
[ 204.685266] ? kfree+0x1c6/0x1d0
[ 204.688641] ? do_init_module+0x27/0x221
[ 204.692556] ? do_init_module+0x27/0x221
[ 204.696494] do_init_module+0x5f/0x221
[ 204.700385] load_module+0x2697/0x2b30
[ 204.704134] __do_sys_finit_module+0xe5/0x120
[ 204.708493] ? __do_sys_finit_module+0xe5/0x120
[ 204.713161] __x64_sys_finit_module+0x1a/0x20
[ 204.717506] do_syscall_64+0x5a/0x110
[ 204.721181] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 204.726405] RIP: 0033:0x7fa3ea6564d9
[ 204.729994] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8f 29 2c 00 f7 d8 64 89 01 48
[ 204.748982] RSP: 002b:00007ffd57199718 EFLAGS: 00000206 ORIG_RAX: 0000000000000139
[ 204.756508] RAX: ffffffffffffffda RBX: 000055a41418b250 RCX: 00007fa3ea6564d9
[ 204.763777] RDX: 0000000000000000 RSI: 000055a41364126b RDI: 0000000000000003
[ 204.771087] RBP: 000055a41364126b R08: 0000000000000000 R09: 00007fa3ea91bea0
[ 204.778186] R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000000000
[ 204.785409] R13: 000055a41418b210 R14: 0000000000000000 R15: 0000000000000000
[ 204.792514] FIX kmalloc-8: Restoring 0x00000000e7a736f4-0x00000000e7a736f4=0xcc
[ 204.801545] FIX kmalloc-8: Object at 0x000000003fa16067 not freed
首先，log里可以看到
[ 204.473065] BUG kmalloc-8 (Tainted: P OE ): Redzone overwritten这个log表明slab object里的red zone内容被篡改，red zone位于slab object的前后位置，因此red zone被篡改意味着有人写越界。red zone和slab object的位置关系如下所示：
red zone | slab object | red zone
另外，根据这个log
[ 204.565545] Redzone 00000000c5553bcc: cc cc cc cc cc cc cc cc ........
[ 204.574360] Object 000000003fa16067: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk.
[ 204.583092] Redzone 00000000e7a736f4: 61 cc cc cc cc cc cc cc a.......可以确定，写越界的位置是slab object的尾巴上，因为red zone的值都是0xcc，而尾巴上的red zone第一个byte被写成了0x61，也就是字符‘a’。但是这个stack并不能告诉你是哪里写坏的，需要developer自己查看source code去定位。
再看一个例子：
static int dbf(void)
{
char *buf = kmalloc(8, GFP_KERNEL);
printk(KERN_INFO "test dbf.\n");
kfree(buf);
kfree(buf);
return 0;
}
这是一个double free的例子，同一个memory释放了两次，就会触发double free。
[ 110.716501] module init.
[ 110.719094] test dbf.
[ 110.721669] =============================================================================
[ 110.729839] BUG kmalloc-8 (Tainted: P OE ): Object already free
[ 110.736824] -----------------------------------------------------------------------------
[ 110.746395] INFO: Allocated in hello_init+0x2c/0x1000 [hello] age=1 cpu=1 pid=2106
[ 110.753898] __slab_alloc+0x20/0x40
[ 110.757355] kmem_cache_alloc_trace+0x238/0x270
[ 110.761845] hello_init+0x2c/0x1000 [hello]
[ 110.765991] do_one_initcall+0x52/0x1d7
[ 110.769794] do_init_module+0x5f/0x221
[ 110.773511] load_module+0x2697/0x2b30
[ 110.777227] __do_sys_finit_module+0xe5/0x120
[ 110.781545] __x64_sys_finit_module+0x1a/0x20
[ 110.785863] do_syscall_64+0x5a/0x110
[ 110.789494] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 110.794500] INFO: Freed in hello_init+0x43/0x1000 [hello] age=0 cpu=1 pid=2106
[ 110.801656] kfree+0x1c6/0x1d0
[ 110.804684] hello_init+0x43/0x1000 [hello]
[ 110.808828] do_one_initcall+0x52/0x1d7
[ 110.812628] do_init_module+0x5f/0x221
[ 110.816343] load_module+0x2697/0x2b30
[ 110.820059] __do_sys_finit_module+0xe5/0x120
[ 110.824377] __x64_sys_finit_module+0x1a/0x20
[ 110.828694] do_syscall_64+0x5a/0x110
[ 110.832323] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 110.837330] INFO: Slab 0x000000003dff6e72 objects=23 used=0 fp=0x0000000033c912e4 flags=0x17ffffc0008101
[ 110.846722] INFO: Object 0x0000000033c912e4 @offset=4824 fp=0x (null)
[ 110.855341] Redzone 000000004818bf4f: bb bb bb bb bb bb bb bb ........
[ 110.863959] Object 0000000033c912e4: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk.
[ 110.872490] Redzone 0000000033f6ea75: bb bb bb bb bb bb bb bb ........
[ 110.881108] Padding 0000000065f1700a: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[ 110.889727] CPU: 1 PID: 2106 Comm: insmod Tainted: P B OE 4.19.65-devin #8
[ 110.908854] Call Trace:
[ 110.911282] dump_stack+0x63/0x85
[ 110.914568] print_trailer+0x149/0x210
[ 110.918285] free_debug_processing+0x217/0x330
[ 110.922690] ? hello_init+0x4b/0x1000 [hello]
[ 110.927007] __slab_free+0x226/0x370
[ 110.930552] ? vprintk_default+0x29/0x50
[ 110.934440] ? hello_init+0x4b/0x1000 [hello]
[ 110.938757] kfree+0x1c6/0x1d0
[ 110.941785] ? kfree+0x1c6/0x1d0
[ 110.944985] hello_init+0x4b/0x1000 [hello]
[ 110.949129] ? 0xffffffffc0741000
[ 110.952415] do_one_initcall+0x52/0x1d7
[ 110.956216] ? kmem_cache_alloc_trace+0x16d/0x270
[ 110.960877] ? kfree+0x1c6/0x1d0
[ 110.964076] ? do_init_module+0x27/0x221
[ 110.967962] ? do_init_module+0x27/0x221
[ 110.971851] do_init_module+0x5f/0x221
[ 110.975568] load_module+0x2697/0x2b30
[ 110.979287] __do_sys_finit_module+0xe5/0x120
[ 110.983605] ? __do_sys_finit_module+0xe5/0x120
[ 110.988095] __x64_sys_finit_module+0x1a/0x20
[ 110.992413] do_syscall_64+0x5a/0x110
[ 110.996043] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 111.001049] RIP: 0033:0x7efc3708d4d9
[ 111.004593] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8f 29 2c 00 f7 d8 64 89 01 48
[ 111.023190] RSP: 002b:00007ffdbd1ee6a8 EFLAGS: 00000202 ORIG_RAX: 0000000000000139
[ 111.030691] RAX: ffffffffffffffda RBX: 00005650a5851250 RCX: 00007efc3708d4d9
[ 111.037761] RDX: 0000000000000000 RSI: 00005650a459926b RDI: 0000000000000003
[ 111.044832] RBP: 00005650a459926b R08: 0000000000000000 R09: 00007efc37352ea0
[ 111.051900] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000000000
[ 111.058969] R13: 00005650a5851210 R14: 0000000000000000 R15: 0000000000000000
[ 111.066073] FIX kmalloc-8: Object at 0x0000000033c912e4 not freed
[ 113.001551] =============================================================================
[ 113.012683] BUG kmalloc-8 (Tainted: P B OE ): Redzone overwritten
[ 113.022832] -----------------------------------------------------------------------------
[ 113.036703] INFO: 0x0000000049e00626-0x000000003eda1d4c. First byte 0xbb instead of 0xcc
[ 113.048396] INFO: Slab 0x000000003dff6e72 objects=23 used=22 fp=0x0000000033c912e4 flags=0x17ffffc0008101
[ 113.062247] INFO: Object 0x000000003760475c @offset=8 fp=0x00000000dd759e2d
[ 113.074231] Redzone 0000000049e00626: bb bb bb bb bb bb bb bb ........
[ 113.086515] Object 000000003760475c: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk.
[ 113.098681] Redzone 0000000099655122: bb bb bb bb bb bb bb bb ........
[ 113.111126] Padding 00000000e6ca9c29: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[ 113.121204] CPU: 2 PID: 2109 Comm: slabinfo Tainted: P B OE 4.19.65-devin #8
[ 113.140680] Call Trace:
[ 113.143114] dump_stack+0x63/0x85
[ 113.146402] print_trailer+0x149/0x210
[ 113.150128] check_bytes_and_report+0xc2/0xf0
[ 113.154448] check_object+0x1be/0x290
[ 113.158088] validate_slab+0x18b/0x220
[ 113.161807] validate_store+0xdd/0x210
[ 113.165532] slab_attr_store+0x3e/0xf0
[ 113.169250] ? kernfs_fop_write+0xcb/0x1a0
[ 113.173359] sysfs_kf_write+0x3c/0x50
[ 113.176991] kernfs_fop_write+0x125/0x1a0
[ 113.181018] __vfs_write+0x3a/0x190
[ 113.184480] ? apparmor_file_permission+0x1a/0x20
[ 113.189210] ? security_file_permission+0x3b/0xc0
[ 113.193876] ? _cond_resched+0x1a/0x50
[ 113.197602] vfs_write+0xb8/0x1b0
[ 113.200891] ksys_write+0x5c/0xe0
[ 113.204223] __x64_sys_write+0x1a/0x20
[ 113.208011] do_syscall_64+0x5a/0x110
[ 113.211653] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 113.216661] RIP: 0033:0x7f4fa33ff2c0
[ 113.220211] Code: 73 01 c3 48 8b 0d d8 cb 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d 89 24 2d 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 fe dd 01 00 48 89 04 24
[ 113.238812] RSP: 002b:00007ffc67b99078 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 113.246319] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4fa33ff2c0
[ 113.253394] RDX: 0000000000000002 RSI: 0000000002017ae0 RDI: 0000000000000003
[ 113.260497] RBP: 0000000002017ae0 R08: 00007f4fa38df700 R09: 0000000000000002
[ 113.267636] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
[ 113.274706] R13: 0000000000000001 R14: 00000000020178b0 R15: 0000000000000000
[ 113.281836] FIX kmalloc-8: Restoring 0x0000000049e00626-0x000000003eda1d4c=0xcc
可以通过callstack以及对应的function + offset定位问题发生的位置。
另外，slub debug还可以检测use after free：
static int uaf(void)
{
char *buf = kmalloc(8, GFP_KERNEL);
printk(KERN_INFO "test oob.\n");
kfree(buf);
buf[0] = 'a';
return 0;
}
如上面的code所示，先释放了这块memory，然后又使用，就会出发use after free的问题，log如下：
[ 345.581447] module init.
[ 345.584037] test oob.
[ 345.586659] =============================================================================
[ 345.594994] BUG kmalloc-8 (Tainted: P B OE ): Poison overwritten
[ 345.601979] -----------------------------------------------------------------------------
[ 345.611784] INFO: 0x0000000035afc9b5-0x0000000035afc9b5. First byte 0x61 instead of 0x6b
[ 345.619945] INFO: Allocated in __this_module+0x302c/0x4000 [hello] age=9 cpu=3 pid=2607
[ 345.628007] __slab_alloc+0x20/0x40
[ 345.631523] kmem_cache_alloc_trace+0x238/0x270
[ 345.636139] __this_module+0x302c/0x4000 [hello]
[ 345.640777] do_one_initcall+0x52/0x1d7
[ 345.644660] do_init_module+0x5f/0x221
[ 345.648465] load_module+0x2697/0x2b30
[ 345.652237] __do_sys_finit_module+0xe5/0x120
[ 345.656618] __x64_sys_finit_module+0x1a/0x20
[ 345.661014] do_syscall_64+0x5a/0x110
[ 345.664703] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 345.669785] INFO: Freed in __this_module+0x3043/0x4000 [hello] age=8 cpu=3 pid=2607
[ 345.677521] kfree+0x1c6/0x1d0
[ 345.680626] __this_module+0x3043/0x4000 [hello]
[ 345.685439] do_one_initcall+0x52/0x1d7
[ 345.689250] do_init_module+0x5f/0x221
[ 345.693000] load_module+0x2697/0x2b30
[ 345.696782] __do_sys_finit_module+0xe5/0x120
[ 345.701362] __x64_sys_finit_module+0x1a/0x20
[ 345.705690] do_syscall_64+0x5a/0x110
[ 345.709343] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 345.714475] INFO: Slab 0x00000000919adf01 objects=23 used=22 fp=0x000000007356586e flags=0x17ffffc0008100
[ 345.724035] INFO: Object 0x0000000035afc9b5 @offset=7576 fp=0x00000000b83a42bb
[ 345.732980] Redzone 000000008f0db005: bb bb bb bb bb bb bb bb ........
[ 345.741685] Object 0000000035afc9b5: 61 6b 6b 6b 6b 6b 6b a5 akkkkkk.
[ 345.750357] Redzone 000000001f820f81: bb bb bb bb bb bb bb bb ........
[ 345.759057] Padding 000000003202397c: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[ 345.767751] CPU: 3 PID: 2607 Comm: insmod Tainted: P B OE 4.19.65-devin #8
[ 345.787084] Call Trace:
[ 345.789518] dump_stack+0x63/0x85
[ 345.792867] print_trailer+0x149/0x210
[ 345.796847] check_bytes_and_report+0xc2/0xf0
[ 345.801176] check_object+0x24d/0x290
[ 345.804828] ? alloc_cpumask_var_node+0x1f/0x30
[ 345.809392] alloc_debug_processing+0x102/0x1a0
[ 345.813932] ___slab_alloc+0x4e6/0x540
[ 345.817730] ? is_bpf_text_address+0xe/0x20
[ 345.821933] ? alloc_cpumask_var_node+0x1f/0x30
[ 345.826575] ? __module_text_address+0x12/0x60
[ 345.831250] ? update_stack_state+0x141/0x190
[ 345.835623] ? __module_text_address+0x12/0x60
[ 345.840098] __slab_alloc+0x20/0x40
[ 345.843606] ? __slab_alloc+0x20/0x40
[ 345.847320] __kmalloc_node+0xe5/0x360
[ 345.851095] ? alloc_cpumask_var_node+0x1f/0x30
[ 345.855663] ? load_new_mm_cr3+0xf0/0xf0
[ 345.859608] alloc_cpumask_var_node+0x1f/0x30
[ 345.864013] alloc_cpumask_var+0xe/0x10
[ 345.867879] native_send_call_func_ipi+0x2e/0x130
[ 345.872605] smp_call_function_many+0x1c0/0x260
[ 345.877122] ? load_new_mm_cr3+0xf0/0xf0
[ 345.881045] on_each_cpu+0x2d/0x60
[ 345.884439] flush_tlb_kernel_range+0x4b/0x80
[ 345.888781] ? purge_fragmented_blocks_allcpus+0x49/0x1f0
[ 345.894162] __purge_vmap_area_lazy+0x50/0xc0
[ 345.898540] vm_unmap_aliases+0xfa/0x130
[ 345.902450] change_page_attr_set_clr+0xd9/0x380
[ 345.907055] ? 0xffffffffc0741000
[ 345.910357] set_memory_ro+0x29/0x30
[ 345.913911] ? 0xffffffffc0741000
[ 345.917209] frob_text.isra.35+0x23/0x30
[ 345.921125] module_enable_ro.part.56+0x3a/0xd0
[ 345.925909] do_init_module+0x119/0x221
[ 345.929713] load_module+0x2697/0x2b30
[ 345.933691] __do_sys_finit_module+0xe5/0x120
[ 345.938021] ? __do_sys_finit_module+0xe5/0x120
[ 345.942544] __x64_sys_finit_module+0x1a/0x20
[ 345.946883] do_syscall_64+0x5a/0x110
[ 345.950547] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 345.955568] RIP: 0033:0x7fd0c473a4d9
[ 345.959142] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8f 29 2c 00 f7 d8 64 89 01 48
[ 345.977892] RSP: 002b:00007ffdb3f78df8 EFLAGS: 00000202 ORIG_RAX: 0000000000000139
[ 345.985462] RAX: ffffffffffffffda RBX: 00005623afb5f250 RCX: 00007fd0c473a4d9
[ 345.992610] RDX: 0000000000000000 RSI: 00005623adcea26b RDI: 0000000000000003
[ 345.999838] RBP: 00005623adcea26b R08: 0000000000000000 R09: 00007fd0c49ffea0
[ 346.006970] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000000000
[ 346.014061] R13: 00005623afb5f210 R14: 0000000000000000 R15: 0000000000000000
[ 346.021205] FIX kmalloc-8: Restoring 0x0000000035afc9b5-0x0000000035afc9b5=0x6b
[ 346.030166] FIX kmalloc-8: Marking all objects used
[ 347.816612] =============================================================================
[ 347.824718] BUG kmalloc-8 (Tainted: P B OE ): Redzone overwritten
[ 347.831703] -----------------------------------------------------------------------------
[ 347.841269] INFO: 0x0000000099655122-0x000000002843384c. First byte 0xbb instead of 0xcc
[ 347.849286] INFO: Slab 0x000000003dff6e72 objects=23 used=22 fp=0x0000000033c912e4 flags=0x17ffffc0008101
[ 347.858764] INFO: Object 0x000000003760475c @offset=8 fp=0x00000000dd759e2d
[ 347.867126] Redzone 0000000049e00626: cc cc cc cc cc cc cc cc ........
[ 347.875744] Object 000000003760475c: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk.
[ 347.884276] Redzone 0000000099655122: bb bb bb bb bb bb bb bb ........
[ 347.892899] Padding 00000000e6ca9c29: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[ 347.901522] CPU: 0 PID: 2610 Comm: slabinfo Tainted: P B OE 4.19.65-devin #8
[ 347.920878] Call Trace:
[ 347.923313] dump_stack+0x63/0x85
[ 347.926601] print_trailer+0x149/0x210
[ 347.930318] check_bytes_and_report+0xc2/0xf0
[ 347.934635] check_object+0x1ea/0x290
[ 347.938265] validate_slab+0x18b/0x220
[ 347.941982] validate_store+0xdd/0x210
[ 347.945698] slab_attr_store+0x3e/0xf0
[ 347.949414] ? kernfs_fop_write+0xcb/0x1a0
[ 347.953474] sysfs_kf_write+0x3c/0x50
[ 347.957104] kernfs_fop_write+0x125/0x1a0
[ 347.961078] __vfs_write+0x3a/0x190
[ 347.964537] ? apparmor_file_permission+0x1a/0x20
[ 347.969200] ? security_file_permission+0x3b/0xc0
[ 347.973863] ? _cond_resched+0x1a/0x50
[ 347.977578] vfs_write+0xb8/0x1b0
[ 347.980864] ksys_write+0x5c/0xe0
[ 347.984150] __x64_sys_write+0x1a/0x20
[ 347.987867] do_syscall_64+0x5a/0x110
[ 347.991626] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 347.996635] RIP: 0033:0x7fcadb1742c0
[ 348.000179] Code: 73 01 c3 48 8b 0d d8 cb 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d 89 24 2d 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 fe dd 01 00 48 89 04 24
[ 348.018774] RSP: 002b:00007ffcc51f2548 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 348.026275] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fcadb1742c0
[ 348.033344] RDX: 0000000000000002 RSI: 0000000000c18ae0 RDI: 0000000000000003
[ 348.040413] RBP: 0000000000c18ae0 R08: 00007fcadb654700 R09: 0000000000000002
[ 348.047483] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
[ 348.054551] R13: 0000000000000001 R14: 0000000000c188b0 R15: 0000000000000000
[ 348.061624] FIX kmalloc-8: Restoring 0x0000000099655122-0x000000002843384c=0xcc
[ 348.075571] SLUB: kmalloc-8 471 slabs counted but counter=472
从log里看到是Poison overwritten，因为被free的slab object会被填充poison value，也就是0x6b，在检查的时候，如果发现slab object里面的值不是0x6b，就知道memory在free之后又被人写了。首先根据stack确认是哪个memory，然后一般是结构体，通过查看被写的位置，可以算出是结构体的哪个变量，再在source code里查找就行了。