ARM64下 Kdump & Crash
1.工具准备
1.1 kexec
kexec 是一个 Linux 内核到内核的引导加载程序，可以帮助从第一个内核的上下文引导到第二个内核。kexec会关闭第一个内核，绕过BIOS或固件阶段，并跳转到第二个内核。在没有 BIOS 阶段的情况下，重新启动变得更快。
- 下载最新的kexec-tools源码包
kexec源码下载地址
2. 解压源码包
tar xvpzf kexec-tools.tar.gz
- 进入到kexec-tools目录中，并进行配置（静态编译）
LDFLAGS=-static ./configure ARCH=arm64 --build=x86_64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu --without-xen
- 编译
make
- 生成的工具即在build目录下，可拷贝至文件系统中
1.2 crash
crash是redhat的工程师开发的，主要用来离线分析linux内核转存文件，它整合了gdb工具，功能非常强大。可以查看堆栈，dmesg日志，内核数据结构，反汇编等等。crash支持多种工具生成的转存文件格式，如kdump，LKCD，netdump和diskdump，而且还可以分析虚拟机Xen和Kvm上生成的内核转存文件。同时crash还可以调试运行时系统，直接运行crash即可，ubuntu下内核映象存放在/proc/kcore。
- 下载
crash不同版本下载地址
- 解压编译
$ tar -vxzf crash-7.3.0.tar.gz
$ cd crash-7.3.0/
$ make target=arm64
首次编译时总是报一个错误，我的PC系统是ubuntu20.04，安装一些工具后解决
sudo apt-get install libaio-dev libncurses5-dev zlib1g-dev liblzma-dev flex bison byacc
- 验证
ycy@DESKTOP-IDHR9T5:~/tool/crash-7.3.0$ ./crash --buildinfo
build_command: crash
build_data: Wed Jun 9 11:41:35 CST 2021 by uid=1000(ycy) on DESKTOP-IDHR9T5
build_target: ARM64
build_version: 7.3.0
compiler version: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
2.整体流程
2.1 使能配置项
- CONFIG_KEXEC=y
- CONFIG_SYSFS=y
- CONFIG_DEBUG_INFO=y
- CONFIG_CRASH_DUMP=y
- CONFIG_PROC_VMCORE=y
2.2 预留内存
在内核命令行中传递以下启动项，其中 @X 是可选的。
crashkernel=Y[@X]
// Y: 内核的大小
// X: 内核的起始地址（起始地址必须与2MiB (0x200000)对齐）
通过cat /proc/iomem 可以查看预留内存是否成
180000000-1efffffff : System RAM
1a0000000-1a7ffffff : Crash kernel
1eab38000-1eadfffff : reserved
1eae11000-1efdfffff : reserved
1efe30000-1efe4ffff : reserved
1efe50000-1efe50fff : reserved
1efe53000-1efe53fff : reserved
1efe54000-1eff6ffff : reserved
1eff70000-1efffffff : reserved
2.3 加载内核
kexec -p \
--initrd= \
--append="root= "
//下面是当时使用的特定于arch的命令行选项
For arm64:
"1 maxcpus=1 reset_devices"
2.4 检查捕获内核加载状况
//查看捕获内核加载状态 0：未加载 1：已加载
root@a1000:/# cat /sys/kernel/kexec_crash_loaded
//查看捕获内核加载大小
root@a1000:/# cat /sys/kernel/kexec_crash_size
2.5 启动捕获内核
在如前所述成功加载转储-捕获内核之后，如果触发系统崩溃，系统将重新引导到转储-捕获内核。触发点位于panic()、die()、die_nmi()和sysrq处理程序(ALT-SysRq-c)中。
可通过以下指令触发panic测试：
root@a1000:~# echo c > /proc/sysrq-trigger
2.6 拷贝vmcore
将proc目录下的vmcore文件拷贝至你的存储设备中即可，vmcore就是第一个内核崩溃时的内存镜像
- 提供一个脚本：
#!/bin/sh
KDUMP_PATH="/home"
function save_dump_file()
{
# datename=$(date +%Y-%m-%d-%H:%M:%S)
coredir="${KDUMP_PATH}/kdump"
mkdir -p $coredir
if [ ! -f /sbin/vmcore-dmesg ];then
echo “Skipping saving vmcore-dmesg.txt.File /sbin/vmcore-dmesg is not present”
return;
fi
echo “kdump: saving vmcore-dmesg.txt to $coredir”
if [ ! -f "$coredir/vmcore_1" ]; then
/sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_1.txt
cp /proc/vmcore $coredir/vmcore_1
else
if [ ! -f "$coredir/vmcore_2" ]; then
/sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_2.txt
cp /proc/vmcore $coredir/vmcore_2
else
rm $coredir/vmcore_1 $coredir/vmcore-dmesg_1.txt
mv $coredir/vmcore_2 $coredir/vmcore_1
mv $coredir/vmcore-dmesg_2.txt $coredir/vmcore-dmesg_1.txt
/sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_2.txt
cp /proc/vmcore $coredir/vmcore_2
fi
fi
}
if [ -s /proc/vmcore ];then
echo “save core dump file…”
save_dump_file
reboot
else
# kexec setting
/sbin/kexec -p --append="earlycon=uart8250,mmio32,0x20008000 console=ttyS0,115200n8 memreserve=64M@0xf8000000 rw boot_delay=32 1 maxcpus=1 reset_devices root=/dev/mmcblk0p9 boot_delay=32 rootwait init=/linuxrc" --dtb=/home/root/bsta1000-fadb.dtb /home/root/Image
fi
3.Crash使用详解
3.1 背景知识
crash是redhat的工程师开发的，主要用来离线分析linux内核转存文件，它整合了gdb工具，功能非常强大。可以查看堆栈，dmesg日志，内核数据结构，反汇编等等。crash支持多种工具生成的转存文件格式，如kdump，LKCD，netdump和diskdump，而且还可以分析虚拟机Xen和Kvm上生成的内核转存文件。同时crash还可以调试运行时系统，直接运行crash即可，ubuntu下内核映象存放在/proc/kcore
3.2 使用方法
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
ex:
ycy@DESKTOP-IDHR9T5:~/tool/crash-7.3.0$ ./crash vmlinux ~/te/vmcore
crash 7.3.0
Copyright (C) 2002-2021 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-elf-linux"...
... ...
crash>
crash在加载内核转存文件是会输出系统基本信息，如出问题的进程（bash - 2613），系统内存大小（7.9GB），系统架构（x86_64）等等，可以看到这个dump是sysrq触发的一个panic系统崩溃。
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-elf-linux"...
KERNEL: vmlinux
DUMPFILE: /home/ycy/te/vmcore
CPUS: 8
DATE: Sun May 16 16:50:05 CST 2021
UPTIME: 00:15:09
LOAD AVERAGE: 5.06, 4.79, 3.15
TASKS: 274
NODENAME: a1000
RELEASE: 4.19.66+1.0.0
VERSION: #2 SMP Tue Jun 15 09:27:09 CST 2021
MACHINE: aarch64 (unknown Mhz)
MEMORY: 2.6 GB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 641
COMMAND: "sh"
TASK: ffff8001df128dc0 [THREAD_INFO: ffff8001df128dc0]
CPU: 5
STATE: TASK_RUNNING (SYSRQ)
crash>
3.3 常用命令
Crash命令列表
| 命令 | 功能 |
|---|---|
| * | 指针快捷健 |
| alias | 命令快捷键 |
| ascii | ASCII码转换和码表 |
| bpf | eBPF - extended Berkeley Filter |
| bt | 堆栈查看 |
| btop | 地址页表转换 |
| dev | 设备数据查询 |
| dis | 反汇编 |
| eval | 计算器 |
| exit | 退出 |
| extend | 命令扩展 |
| files | 打开的文件查看 |
| foreach | 循环查看 |
| fuser | 文件使用者查看 |
| gdb | 调用gdb执行命令 |
| help | 帮助 |
| ipcs | 查看system V IPC工具 |
| irq | 查看irq数据 |
| kmem | 查看Kernel内存 |
| list | 查看链表 |
| log | 查看系统消息缓存 |
| mach | 查看平台信息 |
| mod | 加载符号表 |
| mount | Mount文件系统数据 |
| net | 网络命令 |
| p | 查看数据结构 |
| ps | 查看进程状态信息 |
| pte | 查看页表 |
| ptob | 页表地址转换 |
| ptov | 物理地址虚拟地址转换 |
| rd | 查看内存 |
| repeat | 重复执行 |
| runq | 查看run queue上的线程 |
| search | 搜索内存 |
| set | 设置线程环境和Crash内部变量 |
| sig | 查询线程消息 |
| struct | 查询结构体 |
| swap | 查看swap信息 |
| sym | 符号和虚拟地址转换 |
| sys | 查看系统信息 |
| task | 查看task_struct和thread_thread信息 |
| timer | 查看timer队列 |
| tree | 查看radix树和rb树 |
| union | 查看union结构体 |
| vm | 查看虚拟内存 |
| vtop | 虚拟地址物理地址转换 |
| waitq | 查看wait queue上的进程 |
| whatis | 符号表查询 |
| wr | 改写内存 |
| q | 退出 |
以下对常用命令进行详细介绍：
3.3.1 查看堆栈 bt
- bt:默认会dump问题线程的场景
crash> bt
PID: 641 TASK: ffff8001df128dc0 CPU: 5 COMMAND: "sh"
#0 [8caa7f99223ac900] machine_kexec at ffff0000082185ec
PC: 0000ffff8552f2ec LR: 0000ffff854dd6e0 SP: 0000ffffff551f40
X29: 0000ffffff551f40 X28: 0000000000000000 X27: 0000000000000001
X26: 00000000004cd320 X25: 000000000050b000 X24: 0000000000000002
X23: 00000000020b54e0 X22: 0000000000000002 X21: 0000ffff855d2560
X20: 00000000020b54e0 X19: 0000000000000001 X18: 0000000000000fff
X17: 0000ffff854da1f0 X16: 00000000005030f0 X15: 0000000000000020
X14: 0000000000000000 X13: 0000000000000018 X12: 0000000000000000
X11: 0000000000000000 X10: 0000ffffff551f20 X9: 0000000000000001
X8: 0000000000000040 X7: 0000000000000063 X6: 0000000000000001
X5: 0000000000015551 X4: 0000000000000000 X3: 0000ffff855d6a78
X2: 0000000000000002 X1: 00000000020b54e0 X0: 0000000000000001
ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 20000000
- -t/-T:把整个stack的信息dump出来
crash> bt -t
PID: 641 TASK: ffff8001df128dc0 CPU: 5 COMMAND: "sh"
START: machine_kexec at ffff0000082185f0
[ffff8001e450b6a0] machine_kexec at ffff0000082185f0
[ffff8001e450b6e8] __crash_kexec at ffff0000082ba048
[ffff8001e450b738] crash_kexec at ffff0000082ba15c
[ffff8001e450b868] __handle_sysrq at ffff00000872a080
[ffff8001e450b878] sysrq_handle_crash at ffff0000087295f4
[ffff8001e450b8b0] sysrq_handle_crash at ffff0000087295f4
[ffff8001e450b8c8] die at ffff00000820ce00
[ffff8001e450b8f8] die_kernel_fault at ffff00000821ae10
[ffff8001e450b938] __do_kernel_fault at ffff00000821aea8
[ffff8001e450b968] do_page_fault at ffff00000821b0d0
[ffff8001e450b998] do_translation_fault at ffff00000821b3d8
[ffff8001e450ba18] cont_add at ffff00000827a9e8
[ffff8001e450ba28] __up_console_sem at ffff00000827abf8
[ffff8001e450ba48] console_unlock at ffff00000827b3e0
[ffff8001e450ba68] vprintk_emit at ffff00000827cb6c
[ffff8001e450ba98] do_mem_abort at ffff000008201364
[ffff8001e450baa8] el1_ia at ffff0000082030d0
[ffff8001e450bac8] sysrq_handle_crash at ffff0000087295f4
[ffff8001e450bad8] vprintk_emit at ffff00000827ca64
[ffff8001e450bae8] vprintk_default at ffff00000827cd60
[ffff8001e450bb38] _cond_resched at ffff000008d32bb0
[ffff8001e450bc60] __handle_sysrq at ffff00000872a080
[ffff8001e450bc70] sysrq_handle_crash at ffff0000087295f4
[ffff8001e450bca8] sysrq_handle_crash at ffff0000087295f4
[ffff8001e450bcb8] write_sysrq_trigger at ffff00000872a65c
[ffff8001e450bcf8] proc_reg_write at ffff00000840ffa0
[ffff8001e450bd18] __vfs_write at ffff000008390ea8
[ffff8001e450bd38] vfs_write at ffff0000083911a4
[ffff8001e450bd58] f_dupfd at ffff0000083b48d8
[ffff8001e450bd78] _cond_resched at ffff000008d32bb0
[ffff8001e450bd98] __sb_start_write at ffff0000083934c0
[ffff8001e450bda8] vfs_write at ffff000008391270
[ffff8001e450bdd8] ksys_write at ffff00000839147c
[ffff8001e450be18] __arm64_sys_write at ffff000008391510
[ffff8001e450be48] __arm64_sys_close at ffff00000838e59c
[ffff8001e450be68] el0_svc_common at ffff0000082139f4
[ffff8001e450be78] el0_svc_handler at ffff000008213b04
[ffff8001e450beb8] el0_svc at ffff000008203f88
- -slf:加参数显示函数偏移，函数所在的文件和每一帧的具体内容，从而对照源码和汇编代码，查看函数入参和局部变量
crash> bt -slf
PID: 641 TASK: ffff8001df128dc0 CPU: 5 COMMAND: "sh"
#0 [8caa7f99223ac900] machine_kexec+60 at ffff0000082185ec
/home/ycy/kernel_tag/linux_kernel-Linux-v1.4/build/../arch/arm64/kernel/machine_kexec.c: 158
PC: 0000ffff8552f2ec LR: 0000ffff854dd6e0 SP: 0000ffffff551f40
X29: 0000ffffff551f40 X28: 0000000000000000 X27: 0000000000000001
X26: 00000000004cd320 X25: 000000000050b000 X24: 0000000000000002
X23: 00000000020b54e0 X22: 0000000000000002 X21: 0000ffff855d2560
X20: 00000000020b54e0 X19: 0000000000000001 X18: 0000000000000fff
X17: 0000ffff854da1f0 X16: 00000000005030f0 X15: 0000000000000020
X14: 0000000000000000 X13: 0000000000000018 X12: 0000000000000000
X11: 0000000000000000 X10: 0000ffffff551f20 X9: 0000000000000001
X8: 0000000000000040 X7: 0000000000000063 X6: 0000000000000001
X5: 0000000000015551 X4: 0000000000000000 X3: 0000ffff855d6a78
X2: 0000000000000002 X1: 00000000020b54e0 X0: 0000000000000001
ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 20000000
- -c/-a:查看指定cpu的堆栈或所有当前CPU
crash> bt -c 2
PID: 0 TASK: ffff8001e9f044c0 CPU: 2 COMMAND: "swapper/2"
#0 [ffff8001eab77cc0] crash_save_cpu at ffff0000082ba430
#1 [ffff8001eab77e80] handle_IPI at ffff000008212f08
#2 [ffff8001eab77ec0] gic_handle_irq at ffff00000820189c
--- ---
#3 [ffff8001e9f1bf50] el1_irq at ffff00000820332c
PC: ffff000008206f18 [arch_cpu_idle+16]
LR: ffff000008206f14 [arch_cpu_idle+12]
SP: ffff8001e9f1bf60 PSTATE: 60c00009
X29: ffff8001e9f1bf60 X28: 0000000000000000 X27: 0000000000000000
X26: 0000000000000000 X25: 0000000000000000 X24: 0000000000000000
X23: ffff8001e9f044c0 X22: ffff8001e9f044c0 X21: ffff0000092da748
X20: 0000000000000002 X19: ffff0000092da6c8 X18: 0000000000000000
X17: 0000000000000000 X16: 0000000000000000 X15: 0000000000000400
X14: 0000000000000400 X13: 0000000000000001 X12: 0000000000000000
X11: 000000000000bc06 X10: 00000000000009e0 X9: ffff8001e9f1beb0
X8: ffff8001e9f04f00 X7: 0000000000000000 X6: 0000000000000002
X5: 000000000000003f X4: ffff8001e9f1bed0 X3: ffff0000092f04c0
X2: 0000000000000000 X1: 00000000002e1ab8 X0: 0000000000000008
#4 [ffff8001e9f1bf60] arch_cpu_idle at ffff000008206f14
#5 [ffff8001e9f1bf70] do_idle at ffff000008254678
#6 [ffff8001e9f1bfb0] cpu_startup_entry at ffff000008254888
#7 [ffff8001e9f1bfd0] secondary_start_kernel at ffff0000082129cc
ps:也可以用set命令来改变线程环境，从而查看别的cpu上的堆栈情况
3.3.2 查看堆栈 log
log命令可以用来查看系统的日志，“log-a”可以读取还没有从内核日志缓存到用户空间日志缓存的日志。也可以重定向到文件（log > logfile）。
crash> log
[ 7.141822] Mali:
[ 7.141823] Mali device driver loaded
[ 7.141929] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 7.146623] brd: module loaded
[ 7.150647] loop: module loaded
[ 7.150867] zram: Added device: zram0
[ 7.151777] null: module loaded
[ 7.151780] dummy-irq: no IRQ given. Use irq=N
[ 7.156258] i2c-core: driver [at24] registered
[ 7.156278] i2c-core: driver [eeprom] registered
.......
3.3.3 反汇编 dis -r/-f
- 显示从给定地址到例程结束的所有指令。（-f 正向，-r 反向）
crash> dis -f ffff00000820332c
d0xffff00000820332c : blr x1
0xffff000008203330 : mov sp, x19
0xffff000008203334 : msr daifset, #0xf
0xffff000008203338 : ldr x20, [sp,#288]
0xffff00000820333c : str x20, [x28,#8]
0xffff000008203340 : ldp x21, x22, [sp,#256]
0xffff000008203344 : msr elr_el1, x21
0xffff000008203348 : msr spsr_el1, x22
0xffff00000820334c : ldp x0, x1, [sp]
0xffff000008203350 : ldp x2, x3, [sp,#16]
0xffff000008203354 : ldp x4, x5, [sp,#32]
0xffff000008203358 : ldp x6, x7, [sp,#48]
0xffff00000820335c : ldp x8, x9, [sp,#64]
0xffff000008203360 : ldp x10, x11, [sp,#80]
0xffff000008203364 : ldp x12, x13, [sp,#96]
0xffff000008203368 : ldp x14, x15, [sp,#112]
0xffff00000820336c : ldp x16, x17, [sp,#128]
0xffff000008203370 : ldp x18, x19, [sp,#144]
0xffff000008203374 : ldp x20, x21, [sp,#160]
0xffff000008203378 : ldp x22, x23, [sp,#176]
0xffff00000820337c : ldp x24, x25, [sp,#192]
0xffff000008203380 : ldp x26, x27, [sp,#208]
0xffff000008203384 : ldp x28, x29, [sp,#224]
0xffff000008203388 : ldr x30, [sp,#240]
0xffff00000820338c : add sp, sp, #0x140
0xffff000008203390 : eret
3.3.4 查看和搜索内存 rd
- rd:查看内存内容
crash> rd ffff000008d40070 32
ffff000008d40070: 65762078756e694c 2e34206e6f697372 Linux version 4.
ffff000008d40080: 2e312b36362e3931 7963792820302e30 19.66+1.0.0 (ycy
ffff000008d40090: 504f544b53454440 355439524844492d @DESKTOP-IDHR9T5
ffff000008d400a0: 7620636367282029 39206e6f69737265 ) (gcc version 9
ffff000008d400b0: 62552820302e332e 332e392075746e75 .3.0 (Ubuntu 9.3
ffff000008d400c0: 75627537312d302e 2e30327e3175746e .0-17ubuntu1~20.
ffff000008d400d0: 2032232029293430 2065755420504d53 04)) #2 SMP Tue
ffff000008d400e0: 30203531206e754a 2039303a37323a39 Jun 15 09:27:09
ffff000008d400f0: 3132303220545343 000000000000000a CST 2021........
ffff000008d40100: 0000000000000000 0000000000000000 ................
ffff000008d40110: 0000000000000000 0000000000000000 ................
ffff000008d40120: 0000000000000000 0000000000000000 ................
ffff000008d40130: 0000000000000000 0000000000000000 ................
ffff000008d40140: 0000000000000000 0000000000000000 ................
ffff000008d40150: 0000000000000000 0000000000000000 ................
ffff000008d40160: 0000000000000000 0000000000000000 ................
- -a linux_banner:查看版本信息
crash> rd -a linux_banner
ffff000008d40070: Linux version 4.19.66+1.0.0 (ycy@DESKTOP-IDHR9T5) (gcc versi
ffff000008d400ac: on 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #2 SMP Tue Jun 15 0
ffff000008d400e8: 9:27:09 CST 2021
- -s:打印符号表
crash> rd ffff8001e450b6a0 32 -s
ffff8001e450b6a0: machine_kexec+64 0000000000000004
ffff8001e450b6b0: 0000000000000004 000000000000003f
ffff8001e450b6c0: ffff8001a0000040 0000000000000001
ffff8001e450b6d0: ffff8001a00000f8 d61f0220aa1f03e3
ffff8001e450b6e0: ffff8001e450b730 __crash_kexec+120
ffff8001e450b6f0: __stack_chk_guard crashk_res
ffff8001e450b700: ffff8001e450bb70 posix_timers_hashtable+2040
ffff8001e450b710: ffff8001e450b778 0000000000000000
ffff8001e450b720: 0000000056000000 posix_timers_hashtable+2040
ffff8001e450b730: ffff8001e450b8c0 crash_kexec+108
ffff8001e450b740: ffffffffffffffff __compound_literal.26+8
ffff8001e450b750: ffff8001e450bb70 0000000000000000
ffff8001e450b760: 0000ffffffffffff kallsyms_token_index+6456
ffff8001e450b770: ffff8001e450b860 0000000000000001
ffff8001e450b780: 0000000000000000 0000000000000000
ffff8001e450b790: 0000000000000006 0000000000000001
- -e:查看指定内存区域内容
crash> rd ffff8001e450b6a0 -e ffff8001e450b6f0
ffff8001e450b6a0: ffff0000082185f0 0000000000000004 ..!.............
ffff8001e450b6b0: 0000000000000004 000000000000003f ........?.......
ffff8001e450b6c0: ffff8001a0000040 0000000000000001 @...............
ffff8001e450b6d0: ffff8001a00000f8 d61f0220aa1f03e3 ............ ...
ffff8001e450b6e0: ffff8001e450b730 ffff0000082ba048 0.P.....H.+.....
- search:搜索指定内存
crash> search -s ffff8001e450b6a0 -e ffff8001e450b6e0 d61f0220aa1f03e3
ffff8001e450b6d8: d61f0220aa1f03e3
3.3.5 查看线程状态 ps
- ps:查看所有线程状态
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffff0000092e5240 RU 0.0 0 0 [swapper/0]
> 0 0 1 ffff8001e9f06040 RU 0.0 0 0 [swapper/1]
> 0 0 2 ffff8001e9f044c0 RU 0.0 0 0 [swapper/2]
> 0 0 3 ffff8001e9f02940 RU 0.0 0 0 [swapper/3]
> 0 0 4 ffff8001e9f00000 RU 0.0 0 0 [swapper/4]
0 0 5 ffff8001e9f03700 RU 0.0 0 0 [swapper/5]
> 0 0 6 ffff8001e9f01b80 RU 0.0 0 0 [swapper/6]
> 0 0 7 ffff8001e9f3b700 RU 0.0 0 0 [swapper/7]
1 0 5 ffff8001e9ec6040 IN 0.1 157336 6688 systemd
2 0 3 ffff8001e9ec44c0 IN 0.0 0 0 [kthreadd]
3 2 0 ffff8001e9ec2940 ID 0.0 0 0 [rcu_gp]
4 2 0 ffff8001e9ec0000 ID 0.0 0 0 [rcu_par_gp]
7 2 3 ffff8001e9ec5280 ID 0.0 0 0 [kworker/u16:0]
... ...
- -p:查看父线程树
crash> ps -p 167
PID: 0 TASK: ffff0000092e5240 CPU: 0 COMMAND: "swapper/0"
PID: 2 TASK: ffff8001e9ec44c0 CPU: 3 COMMAND: "kthreadd"
PID: 167 TASK: ffff8001e6a5ee00 CPU: 5 COMMAND: "bst_lw0"
- -c:查看子线程
crash> ps -c 2
PID: 2 TASK: ffff8001e9ec44c0 CPU: 3 COMMAND: "kthreadd"
PID: 3 TASK: ffff8001e9ec2940 CPU: 0 COMMAND: "rcu_gp"
PID: 4 TASK: ffff8001e9ec0000 CPU: 0 COMMAND: "rcu_par_gp"
PID: 7 TASK: ffff8001e9ec5280 CPU: 3 COMMAND: "kworker/u16:0"
PID: 8 TASK: ffff8001e9ec0dc0 CPU: 0 COMMAND: "mm_percpu_wq"
PID: 9 TASK: ffff8001e9ec6e00 CPU: 0 COMMAND: "ksoftirqd/0"
PID: 10 TASK: ffff8001e9f05280 CPU: 1 COMMAND: "rcu_sched"
PID: 11 TASK: ffff8001e9f00dc0 CPU: 0 COMMAND: "rcu_bh"
PID: 12 TASK: ffff8001e9f06e00 CPU: 0 COMMAND: "migration/0"
... ...
- -t:查看线程运行时间
crash> ps -t 167
PID: 167 TASK: ffff8001e6a5ee00 CPU: 5 COMMAND: "bst_lw0"
RUN TIME: 00:15:01
START TIME: 8177699104
UTIME: 0
STIME: 0
- -A:查看活动线程
crash> ps -A
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffff0000092e5240 RU 0.0 0 0 [swapper/0]
> 0 0 1 ffff8001e9f06040 RU 0.0 0 0 [swapper/1]
> 0 0 2 ffff8001e9f044c0 RU 0.0 0 0 [swapper/2]
> 0 0 3 ffff8001e9f02940 RU 0.0 0 0 [swapper/3]
> 0 0 4 ffff8001e9f00000 RU 0.0 0 0 [swapper/4]
> 0 0 6 ffff8001e9f01b80 RU 0.0 0 0 [swapper/6]
> 0 0 7 ffff8001e9f3b700 RU 0.0 0 0 [swapper/7]
> 641 478 5 ffff8001df128dc0 RU 0.0 7352 3764 sh
- -k:查看内核线程
crash> ps -k
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffff0000092e5240 RU 0.0 0 0 [swapper/0]
> 0 0 1 ffff8001e9f06040 RU 0.0 0 0 [swapper/1]
> 0 0 2 ffff8001e9f044c0 RU 0.0 0 0 [swapper/2]
> 0 0 3 ffff8001e9f02940 RU 0.0 0 0 [swapper/3]
> 0 0 4 ffff8001e9f00000 RU 0.0 0 0 [swapper/4]
0 0 5 ffff8001e9f03700 RU 0.0 0 0 [swapper/5]
> 0 0 6 ffff8001e9f01b80 RU 0.0 0 0 [swapper/6]
> 0 0 7 ffff8001e9f3b700 RU 0.0 0 0 [swapper/7]
2 0 3 ffff8001e9ec44c0 IN 0.0 0 0 [kthreadd]
3 2 0 ffff8001e9ec2940 ID 0.0 0 0 [rcu_gp]
4 2 0 ffff8001e9ec0000 ID 0.0 0 0 [rcu_par_gp]
... ...
- -u:查看用户态线程
crash> ps -u
PID PPID CPU TASK ST %MEM VSZ RSS COMM
1 0 5 ffff8001e9ec6040 IN 0.1 157336 6688 systemd
259 1 1 ffff8001e86be040 IN 0.1 25836 10456 systemd-journal
277 1 1 ffff8001e4468dc0 IN 0.0 13860 3152 systemd-udevd
372 1 6 ffff8001e97a9b80 IN 0.1 7868 4276 systemd-network
378 1 5 ffff8001e584c4c0 IN 0.1 7516 4132 systemd-resolve
379 1 1 ffff8001e446e040 IN 0.0 81452 3936 systemd-timesyn
381 1 4 ffff8001e8103700 IN 0.1 7492 4076 systemd-logind
382 1 4 ffff8001e8105280 IN 0.0 3128 1620 klogd
407 1 7 ffff8001e8106040 IN 0.0 3128 1596 syslogd
408 1 7 ffff8001e81044c0 IN 0.1 602040 9028 SafetyService
409 1 1 ffff8001e8102940 IN 0.0 4648 3184 dbus-daemon
... ...
- -l:查看最后运行时间戳
crash> ps -l
[909022209204] [IN] PID: 535 TASK: ffff8001df37c4c0 CPU: 2 COMMAND: "SecurityService"
[909021738959] [IN] PID: 507 TASK: ffff8001e584a940 CPU: 3 COMMAND: "CameraService"
[909021687516] [IN] PID: 510 TASK: ffff8001e5848dc0 CPU: 7 COMMAND: "SecurityService"
[909017588390] [IN] PID: 514 TASK: ffff8001df18e040 CPU: 6 COMMAND: "PersistenceServ"
[909016957627] [IN] PID: 561 TASK: ffff8001dd951b80 CPU: 0 COMMAND: "UpgradeService"
[909014829938] [IN] PID: 470 TASK: ffff8001e2aa2940 CPU: 2 COMMAND: "DiagnoseService"
[909012664304] [IN] PID: 490 TASK: ffff8001e0778dc0 CPU: 0 COMMAND: "SensorService"
... ...
- -r:查看线程资源限制
crash> ps -r 167
PID: 167 TASK: ffff8001e6a5ee00 CPU: 5 COMMAND: "bst_lw0"
RLIMIT CURRENT MAXIMUM
CPU (unlimited) (unlimited)
FSIZE (unlimited) (unlimited)
DATA (unlimited) (unlimited)
STACK 8388608 (unlimited)
CORE 0 (unlimited)
RSS (unlimited) (unlimited)
NPROC 6630 6630
NOFILE 1024 4096
MEMLOCK 65536 65536
AS (unlimited) (unlimited)
LOCKS (unlimited) (unlimited)
SIGPENDING 6630 6630
MSGQUEUE 819200 819200
NICE 0 0
RTPRIO 0 0
RTTIME (unlimited) (unlimited)
3.3.6 Context切换 set
- 切换panic线程
crash> set -p
PID: 641
COMMAND: "sh"
TASK: ffff8001df128dc0 [THREAD_INFO: ffff8001df128dc0]
CPU: 5
STATE: TASK_RUNNING (SYSRQ)
crash> set ffff8001df128dc0
PID: 641
COMMAND: "sh"
TASK: ffff8001df128dc0 [THREAD_INFO: ffff8001df128dc0]
CPU: 5
STATE: TASK_RUNNING (SYSRQ)
3.3.7 加载module符号表 mod
- mod:查看当前加载的module
crash> mod
MODULE NAME BASE SIZE OBJECT FILE
ffff000000d42040 panic_module ffff000000d40000 16384 (not loaded) [CONFIG_KALLSYMS]
- -S:加载所有module符号表