Linux Kdump分析宕机问题案例

1. 查看问题原因

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [680MB]: patching 85605 gdb minimal_symbol values

please wait... (patching 85605 gdb minimal_symbol values)
      KERNEL: /usr/lib/debug/lib/modules/3.10.0-957.el7.x86_64/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 2
        DATE: Fri Jan  5 17:59:15 2024
      UPTIME: 00:03:25
LOAD AVERAGE: 0.44, 0.67, 0.31
       TASKS: 490
    NODENAME: localhost.localdomain
     RELEASE: 3.10.0-957.el7.x86_64
     VERSION: #1 SMP Thu Nov 8 23:39:32 UTC 2018
     MACHINE: x86_64  (3407 Mhz)
      MEMORY: 3 GB
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000050"
         PID: 3943
     COMMAND: "insmod"
        TASK: ffff8ebeece52080  [THREAD_INFO: ffff8ebedfa48000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

其中PANIC直接指出了触发宕机的原因。
上述案例中发生崩溃的原因是"BUG: unable to handle kernel NULL pointer dereference at 0000000000000050"，即内核发生了不能处理的空指针引用。

2. 找出哪一行代码触发的宕机

1. 查看堆栈信息

crash> bt
PID: 3943   TASK: ffff8ebeece52080  CPU: 0   COMMAND: "insmod"
 #0 [ffff8ebedfa4b930] machine_kexec at ffffffffab863674
 #1 [ffff8ebedfa4b990] __crash_kexec at ffffffffab91ce12
 #2 [ffff8ebedfa4ba60] crash_kexec at ffffffffab91cf00
 #3 [ffff8ebedfa4ba78] oops_end at ffffffffabf6c758
 #4 [ffff8ebedfa4baa0] no_context at ffffffffabf5aa7e
 #5 [ffff8ebedfa4baf0] __bad_area_nosemaphore at ffffffffabf5ab15
 #6 [ffff8ebedfa4bb40] bad_area_nosemaphore at ffffffffabf5ac86
 #7 [ffff8ebedfa4bb50] __do_page_fault at ffffffffabf6f6b0
 #8 [ffff8ebedfa4bbc0] do_page_fault at ffffffffabf6f915
 #9 [ffff8ebedfa4bbf0] page_fault at ffffffffabf6b758
    [exception RIP: create_oops+25]
    RIP: ffffffffc0854019  RSP: ffff8ebedfa4bca0  RFLAGS: 00010286
    RAX: 0000000000000000  RBX: ffffffffac418020  RCX: 0000000000036cdf
    RDX: ffff8ebedfa4bce4  RSI: ffff8ebedfa4bce4  RDI: 0000000000000000
    RBP: ffff8ebedfa4bcb8   R8: 68737568736e6562   R9: 007568737568736e
    R10: ffff8ebefa61f120  R11: fffff9f20263a900  R12: ffff8ebee8ef7120
    R13: ffffffffc0859000  R14: 0000000000000000  R15: ffffffffc0856000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff8ebedfa4bcc0] init_module at ffffffffc085909a [oops]
#11 [ffff8ebedfa4bd38] do_one_initcall at ffffffffab80210a
#12 [ffff8ebedfa4bd68] load_module at ffffffffab918f8c
#13 [ffff8ebedfa4beb8] sys_finit_module at ffffffffab919606
#14 [ffff8ebedfa4bf50] system_call_fastpath at ffffffffabf74ddb
    RIP: 00007fae7c472e29  RSP: 00007fff1b996bd8  RFLAGS: 00010207
    RAX: 0000000000000139  RBX: 0000000002239240  RCX: 00007fae7c4e4f90
    RDX: 0000000000000000  RSI: 000000000041a96e  RDI: 0000000000000003
    RBP: 000000000041a96e   R8: 0000000000000000   R9: 00007fff1b996de8
    R10: 0000000000000003  R11: 0000000000000206  R12: 0000000000000000
    R13: 0000000002239220  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 0000000000000139  CS: 0033  SS: 002b

• RIP: 指向造成崩溃的指令地址
• RDI: 存储函数第一个参数的地址
• RSI: 存储函数第二个参数的地址

2. 加载内核模块调试信息

crash> mod -s oops  /root/rlk_lab/oops.ko
     MODULE       NAME                            SIZE  OBJECT FILE
ffffffffc0856000  oops                           12741  /root/rlk_lab/oops.ko

3. 查看出现问题的代码行号

ffffffffc0854019为堆栈信息中RIP所指地址。在输出信息中我们可以看到触发宕机的代码在oops_test.c文件的第16行。

crash> sym ffffffffc0854019
ffffffffc0854019 (T) create_oops+25 [oops] /root/rlk_lab/oops_test.c: 16

3. 查看宕机函数传入参数的值

堆栈中RDI，RSI分别指向了传入的第一个，第二个参数。

方法一：使用struct查看结构体的值

1. 使用dis -l命令找到宕机触发点的代码位置

crash> dis -l kfree+316 10
crash> dis -l ffffffffc0748019 10
/root/rlk_lab/oops_test.c: 16
0xffffffffc0748019 <create_oops+25>:    mov    0x50(%rax),%rax
0xffffffffc074801d <create_oops+29>:    mov    %rax,-0x8(%rbp)

2. 在源码中找到对应文件，查看传入的结构体类型

结合代码我们可以看到，传入的两个参数类型分别是struct vm_area_struct 和 struct mydev_priv。

int create_oops(struct vm_area_struct *vma, struct mydev_priv *priv)
{
	unsigned long flags;

	flags = vma->vm_flags;
	printk("flags=0x%lx, name=%s\n", flags, priv->name);
	
	return 0;
}

3. 使用struct命令结合结构体位置，查看传入的结构体数据

crash> struct  mydev_priv ffff95f18619bce4
struct mydev_priv {
  name = "benshushu\000\377\377\000\275\031\206\361\225\377\377j\342\303'\000\000\000\000\240\360\001\000\000\000\000\000\200a\350\224\361\225\377\377\000\320t\300\377\377\377\377 \200\241\263\377\377\377\377\300\036`\212",
  i = 10
}

参数的地址由堆栈中的RSI和RDI指出。

方法二：使用rd命令查看对应内存的值

crash> rd ffff95f18619bce4
ffff95f18619bce4:  68737568736e6562                    benshush