ubuntu kdump + crash使用

1. kdump简介

    coredump为用户态进程段错误等触发的crash现场转储机制, 可利用gdb进行corefile的调试.

    kdump为kernel crash/panic等而触发的转储机制, 一般使用crash工具进行调试.

 

2. kdump原理

     预留部分内存(如192MB) 供转储内核使用, 当panic/crash产生时 利用kexec系统调用直接拉起转储内核. 利用该内核进行crash现场的储存dump.

 

3. kdump使用

     ubuntu配置的内核已使能kdump的支持, 故仅需要安装linux-crashdump包即可,

     sudo apt-get install linux-crashdump

     该包包含crash, kdump-tools, grub等相关依赖.

     安装过程中会修改kernel cmdline已预留kdump转储内核内存空间. 以及使能kdump.

      reboot重启系统后,

      可用以下方式验证:

      1). kdump-config show

river@river-C-Z370M-DH:~$ kdump-config show DUMP_MODE:        kdump USE_KDUMP:        1 KDUMP_SYSCTL:     kernel.panic_on_oops=1 KDUMP_COREDIR:    /var/crash crashkernel addr: 0x    /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.0.21 kdump initrd:    /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.0.21 current state:    ready to kdump kexec command:   /sbin/kexec -p --command-line="BOOT_IMAGE=/vmlinuz-5.0.21 root=UUID=e4c42b5a-44ca-485a-9346-5628d0f46c97 ro quiet splash vt.handoff=1 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

     

 

 

 

 

 

 

 

 

 

      2). echo c > /proc/sysrq-trigger

     类似alt + sysrq + c键会触发一个空指针的kernel panic. 

可验证kdump机制是否运行.

  

=============================================================================================

4. crash使用

     crash system.map vmlinux kdumpfile

常用命令:

bt

crash> bt PID: 7944   TASK: ffffa0c8a1564680  CPU: 2   COMMAND: "insmod"  #0 [ffffbb6c021eb820] machine_kexec at ffffffffa186afa3  #1 [ffffbb6c021eb880] __crash_kexec at ffffffffa19425c2  #2 [ffffbb6c021eb950] crash_kexec at ffffffffa1943421  #3 [ffffbb6c021eb970] oops_end at ffffffffa1832ff7  #4 [ffffbb6c021eb998] no_context at ffffffffa187b68e  #5 [ffffbb6c021eba40] __bad_area_nosemaphore at ffffffffa187b8a0  #6 [ffffbb6c021eba88] bad_area_nosemaphore at ffffffffa187ba26  #7 [ffffbb6c021eba98] __do_page_fault at ffffffffa187c497  #8 [ffffbb6c021ebb00] do_page_fault at ffffffffa187c6ae  #9 [ffffbb6c021ebb30] page_fault at ffffffffa24010ae     [exception RIP: strcmp+16]     RIP: ffffffffa220dc90  RSP: ffffbb6c021ebbe8  RFLAGS: 00010246     RAX: 0000000000000000  RBX: ffffa0c8a53c6840  RCX: 0000000000000000     RDX: 0000000000000072  RSI: 0000000000000000  RDI: ffffffffa2b4dbb1     RBP: ffffbb6c021ebc10   R8: 00000000000271a0   R9: ffffffffa220709c     R10: fffff240c93aa600  R11: 0000000000000000  R12: ffffa0c8a525bf00     R13: 0000000000000000  R14: ffffa0c8a525bf10  R15: ffffffffc05d4400     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018 #10 [ffffbb6c021ebbe8] kset_find_obj at ffffffffa2205c1a #11 [ffffbb6c021ebc18] driver_find at ffffffffa1ea3853 #12 [ffffbb6c021ebc30] driver_register at ffffffffa1ea38cf #13 [ffffbb6c021ebc50] __platform_driver_register at ffffffffa1ea4cf6 #14 [ffffbb6c021ebc60] _MODULE_INIT_START_globalmem at ffffffffc05d703d [globalmem] #15 [ffffbb6c021ebc78] do_one_initcall at ffffffffa180279a #16 [ffffbb6c021ebcf0] do_init_module at ffffffffa193d000 #17 [ffffbb6c021ebd18] load_module at ffffffffa193f3f9 #18 [ffffbb6c021ebe58] __do_sys_finit_module at ffffffffa193f8dd #19 [ffffbb6c021ebf20] __x64_sys_finit_module at ffffffffa193f95a #20 [ffffbb6c021ebf30] do_syscall_64 at ffffffffa180429a #21 [ffffbb6c021ebf50] entry_SYSCALL_64_after_hwframe at ffffffffa240008c     RIP: 00007f15d269e2e9  RSP: 00007ffeaa9a66b8  RFLAGS: 00000246     RAX: ffffffffffffffda  RBX: 00005649ae149770  RCX: 00007f15d269e2e9     RDX: 0000000000000000  RSI: 00005649acedd338  RDI: 0000000000000003     RBP: 00005649acedd338   R8: 0000000000000000   R9: 00007f15d276ece0     R10: 0000000000000003  R11: 0000000000000246  R12: 0000000000000000     R13: 00005649ae14bf60  R14: 0000000000000000  R15: 0000000000000000     ORIG_RAX: 0000000000000139  CS: 0033  SS: 002b

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

dis -l

crash> dis -l ffffffffa1ea3853 /home/river/linux_workspace/kernel/linux/linux-5.0.21/drivers/base/driver.c: 215 0xffffffffa1ea3853 :    mov    %rax,%rbx

 

 

 

 

etc...