ARM64下 Kdump & Crash

1.工具准备

1.1 kexec

kexec 是一个 Linux 内核到内核的引导加载程序,可以帮助从第一个内核的上下文引导到第二个内核。kexec会关闭第一个内核,绕过BIOS或固件阶段,并跳转到第二个内核。在没有 BIOS 阶段的情况下,重新启动变得更快。

  1. 下载最新的kexec-tools源码包

kexec源码下载地址
2. 解压源码包

tar xvpzf kexec-tools.tar.gz
  1. 进入到kexec-tools目录中,并进行配置(静态编译)
LDFLAGS=-static ./configure ARCH=arm64 --build=x86_64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu --without-xen
  1. 编译
make
  1. 生成的工具即在build目录下,可拷贝至文件系统中

1.2 crash

crash是redhat的工程师开发的,主要用来离线分析linux内核转存文件,它整合了gdb工具,功能非常强大。可以查看堆栈,dmesg日志,内核数据结构,反汇编等等。crash支持多种工具生成的转存文件格式,如kdump,LKCD,netdump和diskdump,而且还可以分析虚拟机Xen和Kvm上生成的内核转存文件。同时crash还可以调试运行时系统,直接运行crash即可,ubuntu下内核映象存放在/proc/kcore。

  1. 下载

crash不同版本下载地址

  1. 解压编译
$ tar -vxzf crash-7.3.0.tar.gz
$ cd crash-7.3.0/
$ make target=arm64

首次编译时总是报一个错误,我的PC系统是ubuntu20.04,安装一些工具后解决

sudo apt-get install libaio-dev libncurses5-dev zlib1g-dev liblzma-dev flex bison byacc
  1. 验证
ycy@DESKTOP-IDHR9T5:~/tool/crash-7.3.0$ ./crash --buildinfo
   build_command: crash
      build_data: Wed Jun  9 11:41:35 CST 2021 by uid=1000(ycy) on DESKTOP-IDHR9T5
    build_target: ARM64
   build_version: 7.3.0
compiler version: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

2.整体流程

2.1 使能配置项

  • CONFIG_KEXEC=y
  • CONFIG_SYSFS=y
  • CONFIG_DEBUG_INFO=y
  • CONFIG_CRASH_DUMP=y
  • CONFIG_PROC_VMCORE=y

2.2 预留内存

在内核命令行中传递以下启动项,其中 @X 是可选的。

crashkernel=Y[@X]
// Y: 内核的大小
// X: 内核的起始地址(起始地址必须与2MiB (0x200000)对齐)

通过cat /proc/iomem 可以查看预留内存是否成

180000000-1efffffff : System RAM
  1a0000000-1a7ffffff : Crash kernel
  1eab38000-1eadfffff : reserved
  1eae11000-1efdfffff : reserved
  1efe30000-1efe4ffff : reserved
  1efe50000-1efe50fff : reserved
  1efe53000-1efe53fff : reserved
  1efe54000-1eff6ffff : reserved
  1eff70000-1efffffff : reserved

2.3 加载内核

kexec -p  \
   --initrd= \
   --append="root= "
   
//下面是当时使用的特定于arch的命令行选项

For arm64:
	"1 maxcpus=1 reset_devices"

2.4 检查捕获内核加载状况

//查看捕获内核加载状态 0:未加载  1:已加载
root@a1000:/# cat /sys/kernel/kexec_crash_loaded
//查看捕获内核加载大小
root@a1000:/# cat /sys/kernel/kexec_crash_size

2.5 启动捕获内核

在如前所述成功加载转储-捕获内核之后,如果触发系统崩溃,系统将重新引导到转储-捕获内核。触发点位于panic()、die()、die_nmi()和sysrq处理程序(ALT-SysRq-c)中。

可通过以下指令触发panic测试:

root@a1000:~# echo c > /proc/sysrq-trigger

2.6 拷贝vmcore

将proc目录下的vmcore文件拷贝至你的存储设备中即可,vmcore就是第一个内核崩溃时的内存镜像

  • 提供一个脚本:
#!/bin/sh
KDUMP_PATH="/home"
function save_dump_file()
{
#   datename=$(date +%Y-%m-%d-%H:%M:%S)
    coredir="${KDUMP_PATH}/kdump"
    mkdir -p $coredir
    if [ ! -f /sbin/vmcore-dmesg ];then
        echo “Skipping saving vmcore-dmesg.txt.File /sbin/vmcore-dmesg is not present”
        return;
    fi
    
    echo “kdump: saving vmcore-dmesg.txt to $coredir”
    
    if [ ! -f "$coredir/vmcore_1" ]; then
        /sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_1.txt
        cp /proc/vmcore $coredir/vmcore_1
    else
        if [ ! -f "$coredir/vmcore_2" ]; then
            /sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_2.txt
            cp /proc/vmcore $coredir/vmcore_2
        else
            rm $coredir/vmcore_1 $coredir/vmcore-dmesg_1.txt
            mv $coredir/vmcore_2 $coredir/vmcore_1
            mv $coredir/vmcore-dmesg_2.txt $coredir/vmcore-dmesg_1.txt
            /sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_2.txt
            cp /proc/vmcore $coredir/vmcore_2
        fi
    fi
}

if [ -s /proc/vmcore ];then
    echo “save core dump file…”
    save_dump_file
    reboot
else
# kexec setting
    /sbin/kexec -p --append="earlycon=uart8250,mmio32,0x20008000 console=ttyS0,115200n8 memreserve=64M@0xf8000000 rw boot_delay=32 1 maxcpus=1 reset_devices root=/dev/mmcblk0p9 boot_delay=32 rootwait init=/linuxrc" --dtb=/home/root/bsta1000-fadb.dtb /home/root/Image
fi

3.Crash使用详解

3.1 背景知识

crash是redhat的工程师开发的,主要用来离线分析linux内核转存文件,它整合了gdb工具,功能非常强大。可以查看堆栈,dmesg日志,内核数据结构,反汇编等等。crash支持多种工具生成的转存文件格式,如kdump,LKCD,netdump和diskdump,而且还可以分析虚拟机Xen和Kvm上生成的内核转存文件。同时crash还可以调试运行时系统,直接运行crash即可,ubuntu下内核映象存放在/proc/kcore

3.2 使用方法

crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]     (dumpfile form)
crash [OPTION]... [NAMELIST]                          (live system form)

ex:
ycy@DESKTOP-IDHR9T5:~/tool/crash-7.3.0$ ./crash vmlinux ~/te/vmcore

crash 7.3.0
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-elf-linux"...
... ...

crash> 

crash在加载内核转存文件是会输出系统基本信息,如出问题的进程(bash - 2613),系统内存大小(7.9GB),系统架构(x86_64)等等,可以看到这个dump是sysrq触发的一个panic系统崩溃。

This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-elf-linux"...
      KERNEL: vmlinux
    DUMPFILE: /home/ycy/te/vmcore
        CPUS: 8
        DATE: Sun May 16 16:50:05 CST 2021
      UPTIME: 00:15:09
LOAD AVERAGE: 5.06, 4.79, 3.15
       TASKS: 274
    NODENAME: a1000
     RELEASE: 4.19.66+1.0.0
     VERSION: #2 SMP Tue Jun 15 09:27:09 CST 2021
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 2.6 GB
       PANIC: "sysrq: SysRq : Trigger a crash"
         PID: 641
     COMMAND: "sh"
        TASK: ffff8001df128dc0  [THREAD_INFO: ffff8001df128dc0]
         CPU: 5
       STATE: TASK_RUNNING (SYSRQ)

crash>

3.3 常用命令

Crash命令列表

命令 功能
* 指针快捷健
alias 命令快捷键
ascii ASCII码转换和码表
bpf eBPF - extended Berkeley Filter
bt 堆栈查看
btop 地址页表转换
dev 设备数据查询
dis 反汇编
eval 计算器
exit 退出
extend 命令扩展
files 打开的文件查看
foreach 循环查看
fuser 文件使用者查看
gdb 调用gdb执行命令
help 帮助
ipcs 查看system V IPC工具
irq 查看irq数据
kmem 查看Kernel内存
list 查看链表
log 查看系统消息缓存
mach 查看平台信息
mod 加载符号表
mount Mount文件系统数据
net 网络命令
p 查看数据结构
ps 查看进程状态信息
pte 查看页表
ptob 页表地址转换
ptov 物理地址虚拟地址转换
rd 查看内存
repeat 重复执行
runq 查看run queue上的线程
search 搜索内存
set 设置线程环境和Crash内部变量
sig 查询线程消息
struct 查询结构体
swap 查看swap信息
sym 符号和虚拟地址转换
sys 查看系统信息
task 查看task_struct和thread_thread信息
timer 查看timer队列
tree 查看radix树和rb树
union 查看union结构体
vm 查看虚拟内存
vtop 虚拟地址物理地址转换
waitq 查看wait queue上的进程
whatis 符号表查询
wr 改写内存
q 退出

以下对常用命令进行详细介绍:

3.3.1 查看堆栈 bt
  • bt:默认会dump问题线程的场景
crash> bt
PID: 641    TASK: ffff8001df128dc0  CPU: 5   COMMAND: "sh"
 #0 [8caa7f99223ac900] machine_kexec at ffff0000082185ec
     PC: 0000ffff8552f2ec   LR: 0000ffff854dd6e0   SP: 0000ffffff551f40
    X29: 0000ffffff551f40  X28: 0000000000000000  X27: 0000000000000001
    X26: 00000000004cd320  X25: 000000000050b000  X24: 0000000000000002
    X23: 00000000020b54e0  X22: 0000000000000002  X21: 0000ffff855d2560
    X20: 00000000020b54e0  X19: 0000000000000001  X18: 0000000000000fff
    X17: 0000ffff854da1f0  X16: 00000000005030f0  X15: 0000000000000020
    X14: 0000000000000000  X13: 0000000000000018  X12: 0000000000000000
    X11: 0000000000000000  X10: 0000ffffff551f20   X9: 0000000000000001
     X8: 0000000000000040   X7: 0000000000000063   X6: 0000000000000001
     X5: 0000000000015551   X4: 0000000000000000   X3: 0000ffff855d6a78
     X2: 0000000000000002   X1: 00000000020b54e0   X0: 0000000000000001
    ORIG_X0: 0000000000000001  SYSCALLNO: 40  PSTATE: 20000000
  • -t/-T:把整个stack的信息dump出来
crash> bt -t
PID: 641    TASK: ffff8001df128dc0  CPU: 5   COMMAND: "sh"
              START: machine_kexec at ffff0000082185f0
  [ffff8001e450b6a0] machine_kexec at ffff0000082185f0
  [ffff8001e450b6e8] __crash_kexec at ffff0000082ba048
  [ffff8001e450b738] crash_kexec at ffff0000082ba15c
  [ffff8001e450b868] __handle_sysrq at ffff00000872a080
  [ffff8001e450b878] sysrq_handle_crash at ffff0000087295f4
  [ffff8001e450b8b0] sysrq_handle_crash at ffff0000087295f4
  [ffff8001e450b8c8] die at ffff00000820ce00
  [ffff8001e450b8f8] die_kernel_fault at ffff00000821ae10
  [ffff8001e450b938] __do_kernel_fault at ffff00000821aea8
  [ffff8001e450b968] do_page_fault at ffff00000821b0d0
  [ffff8001e450b998] do_translation_fault at ffff00000821b3d8
  [ffff8001e450ba18] cont_add at ffff00000827a9e8
  [ffff8001e450ba28] __up_console_sem at ffff00000827abf8
  [ffff8001e450ba48] console_unlock at ffff00000827b3e0
  [ffff8001e450ba68] vprintk_emit at ffff00000827cb6c
  [ffff8001e450ba98] do_mem_abort at ffff000008201364
  [ffff8001e450baa8] el1_ia at ffff0000082030d0
  [ffff8001e450bac8] sysrq_handle_crash at ffff0000087295f4
  [ffff8001e450bad8] vprintk_emit at ffff00000827ca64
  [ffff8001e450bae8] vprintk_default at ffff00000827cd60
  [ffff8001e450bb38] _cond_resched at ffff000008d32bb0
  [ffff8001e450bc60] __handle_sysrq at ffff00000872a080
  [ffff8001e450bc70] sysrq_handle_crash at ffff0000087295f4
  [ffff8001e450bca8] sysrq_handle_crash at ffff0000087295f4
  [ffff8001e450bcb8] write_sysrq_trigger at ffff00000872a65c
  [ffff8001e450bcf8] proc_reg_write at ffff00000840ffa0
  [ffff8001e450bd18] __vfs_write at ffff000008390ea8
  [ffff8001e450bd38] vfs_write at ffff0000083911a4
  [ffff8001e450bd58] f_dupfd at ffff0000083b48d8
  [ffff8001e450bd78] _cond_resched at ffff000008d32bb0
  [ffff8001e450bd98] __sb_start_write at ffff0000083934c0
  [ffff8001e450bda8] vfs_write at ffff000008391270
  [ffff8001e450bdd8] ksys_write at ffff00000839147c
  [ffff8001e450be18] __arm64_sys_write at ffff000008391510
  [ffff8001e450be48] __arm64_sys_close at ffff00000838e59c
  [ffff8001e450be68] el0_svc_common at ffff0000082139f4
  [ffff8001e450be78] el0_svc_handler at ffff000008213b04
  [ffff8001e450beb8] el0_svc at ffff000008203f88
  • -slf:加参数显示函数偏移,函数所在的文件和每一帧的具体内容,从而对照源码和汇编代码,查看函数入参和局部变量
crash> bt -slf
PID: 641    TASK: ffff8001df128dc0  CPU: 5   COMMAND: "sh"
 #0 [8caa7f99223ac900] machine_kexec+60 at ffff0000082185ec
    /home/ycy/kernel_tag/linux_kernel-Linux-v1.4/build/../arch/arm64/kernel/machine_kexec.c: 158
     PC: 0000ffff8552f2ec   LR: 0000ffff854dd6e0   SP: 0000ffffff551f40
    X29: 0000ffffff551f40  X28: 0000000000000000  X27: 0000000000000001
    X26: 00000000004cd320  X25: 000000000050b000  X24: 0000000000000002
    X23: 00000000020b54e0  X22: 0000000000000002  X21: 0000ffff855d2560
    X20: 00000000020b54e0  X19: 0000000000000001  X18: 0000000000000fff
    X17: 0000ffff854da1f0  X16: 00000000005030f0  X15: 0000000000000020
    X14: 0000000000000000  X13: 0000000000000018  X12: 0000000000000000
    X11: 0000000000000000  X10: 0000ffffff551f20   X9: 0000000000000001
     X8: 0000000000000040   X7: 0000000000000063   X6: 0000000000000001
     X5: 0000000000015551   X4: 0000000000000000   X3: 0000ffff855d6a78
     X2: 0000000000000002   X1: 00000000020b54e0   X0: 0000000000000001
    ORIG_X0: 0000000000000001  SYSCALLNO: 40  PSTATE: 20000000
  • -c/-a:查看指定cpu的堆栈或所有当前CPU
crash> bt -c 2
PID: 0      TASK: ffff8001e9f044c0  CPU: 2   COMMAND: "swapper/2"
 #0 [ffff8001eab77cc0] crash_save_cpu at ffff0000082ba430
 #1 [ffff8001eab77e80] handle_IPI at ffff000008212f08
 #2 [ffff8001eab77ec0] gic_handle_irq at ffff00000820189c
---  ---
 #3 [ffff8001e9f1bf50] el1_irq at ffff00000820332c
     PC: ffff000008206f18  [arch_cpu_idle+16]
     LR: ffff000008206f14  [arch_cpu_idle+12]
     SP: ffff8001e9f1bf60  PSTATE: 60c00009
    X29: ffff8001e9f1bf60  X28: 0000000000000000  X27: 0000000000000000
    X26: 0000000000000000  X25: 0000000000000000  X24: 0000000000000000
    X23: ffff8001e9f044c0  X22: ffff8001e9f044c0  X21: ffff0000092da748
    X20: 0000000000000002  X19: ffff0000092da6c8  X18: 0000000000000000
    X17: 0000000000000000  X16: 0000000000000000  X15: 0000000000000400
    X14: 0000000000000400  X13: 0000000000000001  X12: 0000000000000000
    X11: 000000000000bc06  X10: 00000000000009e0   X9: ffff8001e9f1beb0
     X8: ffff8001e9f04f00   X7: 0000000000000000   X6: 0000000000000002
     X5: 000000000000003f   X4: ffff8001e9f1bed0   X3: ffff0000092f04c0
     X2: 0000000000000000   X1: 00000000002e1ab8   X0: 0000000000000008
 #4 [ffff8001e9f1bf60] arch_cpu_idle at ffff000008206f14
 #5 [ffff8001e9f1bf70] do_idle at ffff000008254678
 #6 [ffff8001e9f1bfb0] cpu_startup_entry at ffff000008254888
 #7 [ffff8001e9f1bfd0] secondary_start_kernel at ffff0000082129cc

ps:也可以用set命令来改变线程环境,从而查看别的cpu上的堆栈情况

3.3.2 查看堆栈 log

log命令可以用来查看系统的日志,“log-a”可以读取还没有从内核日志缓存到用户空间日志缓存的日志。也可以重定向到文件(log > logfile)。

crash> log
[    7.141822] Mali:
[    7.141823] Mali device driver loaded
[    7.141929] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    7.146623] brd: module loaded
[    7.150647] loop: module loaded
[    7.150867] zram: Added device: zram0
[    7.151777] null: module loaded
[    7.151780] dummy-irq: no IRQ given.  Use irq=N
[    7.156258] i2c-core: driver [at24] registered
[    7.156278] i2c-core: driver [eeprom] registered
.......
3.3.3 反汇编 dis -r/-f
  • 显示从给定地址到例程结束的所有指令。(-f 正向,-r 反向)
crash> dis -f ffff00000820332c
d0xffff00000820332c :      blr     x1
0xffff000008203330 :       mov     sp, x19
0xffff000008203334 :       msr     daifset, #0xf
0xffff000008203338 :       ldr     x20, [sp,#288]
0xffff00000820333c :       str     x20, [x28,#8]
0xffff000008203340 :       ldp     x21, x22, [sp,#256]
0xffff000008203344 :       msr     elr_el1, x21
0xffff000008203348 :       msr     spsr_el1, x22
0xffff00000820334c :       ldp     x0, x1, [sp]
0xffff000008203350 :       ldp     x2, x3, [sp,#16]
0xffff000008203354 :       ldp     x4, x5, [sp,#32]
0xffff000008203358 :       ldp     x6, x7, [sp,#48]
0xffff00000820335c :       ldp     x8, x9, [sp,#64]
0xffff000008203360 :       ldp     x10, x11, [sp,#80]
0xffff000008203364 :       ldp     x12, x13, [sp,#96]
0xffff000008203368 :       ldp     x14, x15, [sp,#112]
0xffff00000820336c :       ldp     x16, x17, [sp,#128]
0xffff000008203370 :       ldp     x18, x19, [sp,#144]
0xffff000008203374 :       ldp     x20, x21, [sp,#160]
0xffff000008203378 :       ldp     x22, x23, [sp,#176]
0xffff00000820337c :       ldp     x24, x25, [sp,#192]
0xffff000008203380 :       ldp     x26, x27, [sp,#208]
0xffff000008203384 :       ldp     x28, x29, [sp,#224]
0xffff000008203388 :       ldr     x30, [sp,#240]
0xffff00000820338c :       add     sp, sp, #0x140
0xffff000008203390 :       eret
3.3.4 查看和搜索内存 rd
  • rd:查看内存内容
crash> rd ffff000008d40070 32
ffff000008d40070:  65762078756e694c 2e34206e6f697372   Linux version 4.
ffff000008d40080:  2e312b36362e3931 7963792820302e30   19.66+1.0.0 (ycy
ffff000008d40090:  504f544b53454440 355439524844492d   @DESKTOP-IDHR9T5
ffff000008d400a0:  7620636367282029 39206e6f69737265   ) (gcc version 9
ffff000008d400b0:  62552820302e332e 332e392075746e75   .3.0 (Ubuntu 9.3
ffff000008d400c0:  75627537312d302e 2e30327e3175746e   .0-17ubuntu1~20.
ffff000008d400d0:  2032232029293430 2065755420504d53   04)) #2 SMP Tue
ffff000008d400e0:  30203531206e754a 2039303a37323a39   Jun 15 09:27:09
ffff000008d400f0:  3132303220545343 000000000000000a   CST 2021........
ffff000008d40100:  0000000000000000 0000000000000000   ................
ffff000008d40110:  0000000000000000 0000000000000000   ................
ffff000008d40120:  0000000000000000 0000000000000000   ................
ffff000008d40130:  0000000000000000 0000000000000000   ................
ffff000008d40140:  0000000000000000 0000000000000000   ................
ffff000008d40150:  0000000000000000 0000000000000000   ................
ffff000008d40160:  0000000000000000 0000000000000000   ................
  • -a linux_banner:查看版本信息
crash> rd -a linux_banner
ffff000008d40070:  Linux version 4.19.66+1.0.0 (ycy@DESKTOP-IDHR9T5) (gcc versi
ffff000008d400ac:  on 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #2 SMP Tue Jun 15 0
ffff000008d400e8:  9:27:09 CST 2021
  • -s:打印符号表
crash> rd ffff8001e450b6a0 32 -s
ffff8001e450b6a0:  machine_kexec+64 0000000000000004
ffff8001e450b6b0:  0000000000000004 000000000000003f
ffff8001e450b6c0:  ffff8001a0000040 0000000000000001
ffff8001e450b6d0:  ffff8001a00000f8 d61f0220aa1f03e3
ffff8001e450b6e0:  ffff8001e450b730 __crash_kexec+120
ffff8001e450b6f0:  __stack_chk_guard crashk_res
ffff8001e450b700:  ffff8001e450bb70 posix_timers_hashtable+2040
ffff8001e450b710:  ffff8001e450b778 0000000000000000
ffff8001e450b720:  0000000056000000 posix_timers_hashtable+2040
ffff8001e450b730:  ffff8001e450b8c0 crash_kexec+108
ffff8001e450b740:  ffffffffffffffff __compound_literal.26+8
ffff8001e450b750:  ffff8001e450bb70 0000000000000000
ffff8001e450b760:  0000ffffffffffff kallsyms_token_index+6456
ffff8001e450b770:  ffff8001e450b860 0000000000000001
ffff8001e450b780:  0000000000000000 0000000000000000
ffff8001e450b790:  0000000000000006 0000000000000001
  • -e:查看指定内存区域内容
crash> rd ffff8001e450b6a0 -e ffff8001e450b6f0
ffff8001e450b6a0:  ffff0000082185f0 0000000000000004   ..!.............
ffff8001e450b6b0:  0000000000000004 000000000000003f   ........?.......
ffff8001e450b6c0:  ffff8001a0000040 0000000000000001   @...............
ffff8001e450b6d0:  ffff8001a00000f8 d61f0220aa1f03e3   ............ ...
ffff8001e450b6e0:  ffff8001e450b730 ffff0000082ba048   0.P.....H.+.....
  • search:搜索指定内存
crash> search -s ffff8001e450b6a0 -e ffff8001e450b6e0 d61f0220aa1f03e3
ffff8001e450b6d8: d61f0220aa1f03e3
3.3.5 查看线程状态 ps
  • ps:查看所有线程状态
crash> ps
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>     0      0   0  ffff0000092e5240  RU   0.0       0      0  [swapper/0]
>     0      0   1  ffff8001e9f06040  RU   0.0       0      0  [swapper/1]
>     0      0   2  ffff8001e9f044c0  RU   0.0       0      0  [swapper/2]
>     0      0   3  ffff8001e9f02940  RU   0.0       0      0  [swapper/3]
>     0      0   4  ffff8001e9f00000  RU   0.0       0      0  [swapper/4]
      0      0   5  ffff8001e9f03700  RU   0.0       0      0  [swapper/5]
>     0      0   6  ffff8001e9f01b80  RU   0.0       0      0  [swapper/6]
>     0      0   7  ffff8001e9f3b700  RU   0.0       0      0  [swapper/7]
      1      0   5  ffff8001e9ec6040  IN   0.1  157336   6688  systemd
      2      0   3  ffff8001e9ec44c0  IN   0.0       0      0  [kthreadd]
      3      2   0  ffff8001e9ec2940  ID   0.0       0      0  [rcu_gp]
      4      2   0  ffff8001e9ec0000  ID   0.0       0      0  [rcu_par_gp]
      7      2   3  ffff8001e9ec5280  ID   0.0       0      0  [kworker/u16:0]
... ...
  • -p:查看父线程树
crash> ps -p 167
PID: 0      TASK: ffff0000092e5240  CPU: 0   COMMAND: "swapper/0"
 PID: 2      TASK: ffff8001e9ec44c0  CPU: 3   COMMAND: "kthreadd"
  PID: 167    TASK: ffff8001e6a5ee00  CPU: 5   COMMAND: "bst_lw0"
  • -c:查看子线程
crash> ps -c 2
PID: 2      TASK: ffff8001e9ec44c0  CPU: 3   COMMAND: "kthreadd"
  PID: 3      TASK: ffff8001e9ec2940  CPU: 0   COMMAND: "rcu_gp"
  PID: 4      TASK: ffff8001e9ec0000  CPU: 0   COMMAND: "rcu_par_gp"
  PID: 7      TASK: ffff8001e9ec5280  CPU: 3   COMMAND: "kworker/u16:0"
  PID: 8      TASK: ffff8001e9ec0dc0  CPU: 0   COMMAND: "mm_percpu_wq"
  PID: 9      TASK: ffff8001e9ec6e00  CPU: 0   COMMAND: "ksoftirqd/0"
  PID: 10     TASK: ffff8001e9f05280  CPU: 1   COMMAND: "rcu_sched"
  PID: 11     TASK: ffff8001e9f00dc0  CPU: 0   COMMAND: "rcu_bh"
  PID: 12     TASK: ffff8001e9f06e00  CPU: 0   COMMAND: "migration/0"
... ...
  • -t:查看线程运行时间
crash> ps -t 167
PID: 167    TASK: ffff8001e6a5ee00  CPU: 5   COMMAND: "bst_lw0"
    RUN TIME: 00:15:01
  START TIME: 8177699104
       UTIME: 0
       STIME: 0
  • -A:查看活动线程
crash> ps -A
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>     0      0   0  ffff0000092e5240  RU   0.0       0      0  [swapper/0]
>     0      0   1  ffff8001e9f06040  RU   0.0       0      0  [swapper/1]
>     0      0   2  ffff8001e9f044c0  RU   0.0       0      0  [swapper/2]
>     0      0   3  ffff8001e9f02940  RU   0.0       0      0  [swapper/3]
>     0      0   4  ffff8001e9f00000  RU   0.0       0      0  [swapper/4]
>     0      0   6  ffff8001e9f01b80  RU   0.0       0      0  [swapper/6]
>     0      0   7  ffff8001e9f3b700  RU   0.0       0      0  [swapper/7]
>   641    478   5  ffff8001df128dc0  RU   0.0    7352   3764  sh
  • -k:查看内核线程
crash> ps -k
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>     0      0   0  ffff0000092e5240  RU   0.0       0      0  [swapper/0]
>     0      0   1  ffff8001e9f06040  RU   0.0       0      0  [swapper/1]
>     0      0   2  ffff8001e9f044c0  RU   0.0       0      0  [swapper/2]
>     0      0   3  ffff8001e9f02940  RU   0.0       0      0  [swapper/3]
>     0      0   4  ffff8001e9f00000  RU   0.0       0      0  [swapper/4]
      0      0   5  ffff8001e9f03700  RU   0.0       0      0  [swapper/5]
>     0      0   6  ffff8001e9f01b80  RU   0.0       0      0  [swapper/6]
>     0      0   7  ffff8001e9f3b700  RU   0.0       0      0  [swapper/7]
      2      0   3  ffff8001e9ec44c0  IN   0.0       0      0  [kthreadd]
      3      2   0  ffff8001e9ec2940  ID   0.0       0      0  [rcu_gp]
      4      2   0  ffff8001e9ec0000  ID   0.0       0      0  [rcu_par_gp]
... ...
  • -u:查看用户态线程
crash> ps -u
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
      1      0   5  ffff8001e9ec6040  IN   0.1  157336   6688  systemd
    259      1   1  ffff8001e86be040  IN   0.1   25836  10456  systemd-journal
    277      1   1  ffff8001e4468dc0  IN   0.0   13860   3152  systemd-udevd
    372      1   6  ffff8001e97a9b80  IN   0.1    7868   4276  systemd-network
    378      1   5  ffff8001e584c4c0  IN   0.1    7516   4132  systemd-resolve
    379      1   1  ffff8001e446e040  IN   0.0   81452   3936  systemd-timesyn
    381      1   4  ffff8001e8103700  IN   0.1    7492   4076  systemd-logind
    382      1   4  ffff8001e8105280  IN   0.0    3128   1620  klogd
    407      1   7  ffff8001e8106040  IN   0.0    3128   1596  syslogd
    408      1   7  ffff8001e81044c0  IN   0.1  602040   9028  SafetyService
    409      1   1  ffff8001e8102940  IN   0.0    4648   3184  dbus-daemon
... ...
  • -l:查看最后运行时间戳
crash> ps -l
[909022209204] [IN]  PID: 535    TASK: ffff8001df37c4c0  CPU: 2   COMMAND: "SecurityService"
[909021738959] [IN]  PID: 507    TASK: ffff8001e584a940  CPU: 3   COMMAND: "CameraService"
[909021687516] [IN]  PID: 510    TASK: ffff8001e5848dc0  CPU: 7   COMMAND: "SecurityService"
[909017588390] [IN]  PID: 514    TASK: ffff8001df18e040  CPU: 6   COMMAND: "PersistenceServ"
[909016957627] [IN]  PID: 561    TASK: ffff8001dd951b80  CPU: 0   COMMAND: "UpgradeService"
[909014829938] [IN]  PID: 470    TASK: ffff8001e2aa2940  CPU: 2   COMMAND: "DiagnoseService"
[909012664304] [IN]  PID: 490    TASK: ffff8001e0778dc0  CPU: 0   COMMAND: "SensorService"
... ...
  • -r:查看线程资源限制
crash> ps -r 167
PID: 167    TASK: ffff8001e6a5ee00  CPU: 5   COMMAND: "bst_lw0"
      RLIMIT     CURRENT       MAXIMUM
         CPU   (unlimited)   (unlimited)
       FSIZE   (unlimited)   (unlimited)
        DATA   (unlimited)   (unlimited)
       STACK     8388608     (unlimited)
        CORE        0        (unlimited)
         RSS   (unlimited)   (unlimited)
       NPROC      6630          6630
      NOFILE      1024          4096
     MEMLOCK      65536         65536
          AS   (unlimited)   (unlimited)
       LOCKS   (unlimited)   (unlimited)
  SIGPENDING      6630          6630
    MSGQUEUE     819200        819200
        NICE        0             0
      RTPRIO        0             0
      RTTIME   (unlimited)   (unlimited)
3.3.6 Context切换 set
  • 切换panic线程
crash> set -p
    PID: 641
COMMAND: "sh"
   TASK: ffff8001df128dc0  [THREAD_INFO: ffff8001df128dc0]
    CPU: 5
  STATE: TASK_RUNNING (SYSRQ)
crash> set ffff8001df128dc0
    PID: 641
COMMAND: "sh"
   TASK: ffff8001df128dc0  [THREAD_INFO: ffff8001df128dc0]
    CPU: 5
  STATE: TASK_RUNNING (SYSRQ)
3.3.7 加载module符号表 mod
  • mod:查看当前加载的module
crash> mod
     MODULE       NAME                BASE         SIZE  OBJECT FILE
ffff000000d42040  panic_module  ffff000000d40000  16384  (not loaded)  [CONFIG_KALLSYMS]
  • -S:加载所有module符号表

你可能感兴趣的:(linux,kernel,linux,内核)