Linux Kdump分析宕机问题案例
文章目录
- 1. 查看问题原因
- 2. 找出哪一行代码触发的宕机
- 3. 查看宕机函数传入参数的值
-
- 方法一：使用struct查看结构体的值
- 方法二：使用rd命令查看对应内存的值
1. 查看问题原因
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: kernel relocated [680MB]: patching 85605 gdb minimal_symbol values
please wait... (patching 85605 gdb minimal_symbol values)
KERNEL: /usr/lib/debug/lib/modules/3.10.0-957.el7.x86_64/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 2
DATE: Fri Jan 5 17:59:15 2024
UPTIME: 00:03:25
LOAD AVERAGE: 0.44, 0.67, 0.31
TASKS: 490
NODENAME: localhost.localdomain
RELEASE: 3.10.0-957.el7.x86_64
VERSION: #1 SMP Thu Nov 8 23:39:32 UTC 2018
MACHINE: x86_64 (3407 Mhz)
MEMORY: 3 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000050"
PID: 3943
COMMAND: "insmod"
TASK: ffff8ebeece52080 [THREAD_INFO: ffff8ebedfa48000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
其中PANIC直接指出了触发宕机的原因。
上述案例中发生崩溃的原因是"BUG: unable to handle kernel NULL pointer dereference at 0000000000000050"，即内核发生了不能处理的空指针引用。
2. 找出哪一行代码触发的宕机
- 查看堆栈信息
crash> bt
PID: 3943 TASK: ffff8ebeece52080 CPU: 0 COMMAND: "insmod"
#0 [ffff8ebedfa4b930] machine_kexec at ffffffffab863674
#1 [ffff8ebedfa4b990] __crash_kexec at ffffffffab91ce12
#2 [ffff8ebedfa4ba60] crash_kexec at ffffffffab91cf00
#3 [ffff8ebedfa4ba78] oops_end at ffffffffabf6c758
#4 [ffff8ebedfa4baa0] no_context at ffffffffabf5aa7e
#5 [ffff8ebedfa4baf0] __bad_area_nosemaphore at ffffffffabf5ab15
#6 [ffff8ebedfa4bb40] bad_area_nosemaphore at ffffffffabf5ac86
#7 [ffff8ebedfa4bb50] __do_page_fault at ffffffffabf6f6b0
#8 [ffff8ebedfa4bbc0] do_page_fault at ffffffffabf6f915
#9 [ffff8ebedfa4bbf0] page_fault at ffffffffabf6b758
[exception RIP: create_oops+25]
RIP: ffffffffc0854019 RSP: ffff8ebedfa4bca0 RFLAGS: 00010286
RAX: 0000000000000000 RBX: ffffffffac418020 RCX: 0000000000036cdf
RDX: ffff8ebedfa4bce4 RSI: ffff8ebedfa4bce4 RDI: 0000000000000000
RBP: ffff8ebedfa4bcb8 R8: 68737568736e6562 R9: 007568737568736e
R10: ffff8ebefa61f120 R11: fffff9f20263a900 R12: ffff8ebee8ef7120
R13: ffffffffc0859000 R14: 0000000000000000 R15: ffffffffc0856000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff8ebedfa4bcc0] init_module at ffffffffc085909a [oops]
#11 [ffff8ebedfa4bd38] do_one_initcall at ffffffffab80210a
#12 [ffff8ebedfa4bd68] load_module at ffffffffab918f8c
#13 [ffff8ebedfa4beb8] sys_finit_module at ffffffffab919606
#14 [ffff8ebedfa4bf50] system_call_fastpath at ffffffffabf74ddb
RIP: 00007fae7c472e29 RSP: 00007fff1b996bd8 RFLAGS: 00010207
RAX: 0000000000000139 RBX: 0000000002239240 RCX: 00007fae7c4e4f90
RDX: 0000000000000000 RSI: 000000000041a96e RDI: 0000000000000003
RBP: 000000000041a96e R8: 0000000000000000 R9: 00007fff1b996de8
R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000002239220 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 0000000000000139 CS: 0033 SS: 002b
- RIP: 指向造成崩溃的指令地址
- RDI: 存储函数第一个参数的地址
- RSI: 存储函数第二个参数的地址
- 加载内核模块调试信息
crash> mod -s oops /root/rlk_lab/oops.ko
MODULE NAME SIZE OBJECT FILE
ffffffffc0856000 oops 12741 /root/rlk_lab/oops.ko
- 查看出现问题的代码行号
ffffffffc0854019为堆栈信息中RIP所指地址。在输出信息中我们可以看到触发宕机的代码在oops_test.c文件的第16行。
crash> sym ffffffffc0854019
ffffffffc0854019 (T) create_oops+25 [oops] /root/rlk_lab/oops_test.c: 16
3. 查看宕机函数传入参数的值
堆栈中RDI，RSI分别指向了传入的第一个，第二个参数。
方法一：使用struct查看结构体的值
- 使用dis -l命令找到宕机触发点的代码位置
crash> dis -l kfree+316 10
crash> dis -l ffffffffc0748019 10
/root/rlk_lab/oops_test.c: 16
0xffffffffc0748019 <create_oops+25>: mov 0x50(%rax),%rax
0xffffffffc074801d <create_oops+29>: mov %rax,-0x8(%rbp)
- 在源码中找到对应文件，查看传入的结构体类型
结合代码我们可以看到，传入的两个参数类型分别是struct vm_area_struct 和 struct mydev_priv。
int create_oops(struct vm_area_struct *vma, struct mydev_priv *priv)
{
unsigned long flags;
flags = vma->vm_flags;
printk("flags=0x%lx, name=%s\n", flags, priv->name);
return 0;
}
- 使用struct命令结合结构体位置，查看传入的结构体数据
crash> struct mydev_priv ffff95f18619bce4
struct mydev_priv {
name = "benshushu\000\377\377\000\275\031\206\361\225\377\377j\342\303'\000\000\000\000\240\360\001\000\000\000\000\000\200a\350\224\361\225\377\377\000\320t\300\377\377\377\377 \200\241\263\377\377\377\377\300\036`\212",
i = 10
}
参数的地址由堆栈中的RSI和RDI指出。
方法二：使用rd命令查看对应内存的值
crash> rd ffff95f18619bce4
ffff95f18619bce4: 68737568736e6562 benshush