Crash工具介绍和常见命令使用
1. 介绍
本文主要介绍crash工具的使用以及常用的命令。crash工具，常用来分析内核的coredump以及应用的coredump，功能非常强大。
crash工具官方介绍
使用crash分析内核crash情况，需要准备以下内容：
- 内核crash时生成的coredump文件或者raw data
- 运行内核对应的符号表vmlinux
- crash 工具
2. crash基本使用
2.1 解释命令
使用crash工具解析内核crash产生的raw data，命令如下：
crash cluster-vmlinux ramdump@0x57400000
解析成功后，会进入crash工具cmdline模式，如下图所示：
使用crash工具解析内核crash产生的coredump文件，命令如下：
crash vmlinux vmcore
解析成功后，也会进入crash工具的cmdline模式
2.2 crash工具基本输出
KERNEL: cluster-vmlinux [TAINTED]
DUMPFILES: /var/tmp/ramdump_elf_Tp0C6d [temporary ELF header]
ramdump
CPUS: 6
DATE: Mon Apr 18 11:33:58 CST 2022 //panic时间
UPTIME: 01:28:40 //panic时，已运行时间
LOAD AVERAGE: 1.31, 1.32, 1.29
TASKS: 134 //panic时，进程总数
NODENAME: x9h_ms //硬件信息
RELEASE: 4.14.61 //内核版本信息
VERSION: #1 SMP PREEMPT Fri Apr 15 14:25:47 UTC 2022
MACHINE: aarch64 (unknown Mhz)
MEMORY: 544 MB //Linux系统内存数
PANIC: "sysrq: SysRq : Trigger a crash" //panic原因
PID: 2496 //panic进程号
COMMAND: "sh" //panic进程名词
TASK: ffff800023c5b800 [THREAD_INFO: ffff800023c5b800] //panic进程task和thread_info信息
CPU: 5 //系统cpu核数
STATE: TASK_RUNNING (SYSRQ) //panic时任务状态
2.2.1 crash常用命令
- help
该命令是查看crash工具的帮助信息，譬如支持的命令、具体某个命令的使用方法等。
crash> help
* extend log rd task
alias files mach repeat timer
ascii foreach mod runq tree
bpf fuser mount search union
bt gdb net set vm
btop help p sig vtop
dev ipcs ps struct waitq
dis irq pte swap whatis
eval kmem ptob sym wr
exit list ptov sys q
crash version: 8.0.0++ gdb version: 10.2
For help on any command above, enter "help ".
For help on input options, enter "help input".
For help on output options, enter "help output".
//查看某个命令使用方法
crash> help ps
NAME
ps - display process status information
SYNOPSIS
ps [-k|-u|-G|-y policy] [-s] [-p|-c|-t|-[l|m][-C cpu]|-a|-g|-r|-S|-A]
[pid | task | command] ...
DESCRIPTION
This command displays process status for selected, or all, processes
in the system. If no arguments are entered, the process data is
is displayed for all processes. Specific processes may be selected
by using the following identifier formats:
pid a process PID.
task a hexadecimal task_struct pointer.
command a command name. If a command name is made up of letters that
are all numerical values, precede the name string with a "\".
If the command string is enclosed within "'" characters, then
the encompassed string must be a POSIX extended regular expression
that will be used to match task names.
The process list may be further restricted by the following options:
-k restrict the output to only kernel threads.
-u restrict the output to only user tasks.
-G display only the thread group leader in a thread group.
-y policy restrict the output to tasks having a specified scheduling policy
expressed by its integer value or by its (case-insensitive) name;
multiple policies may be entered in a comma-separated list:
0 or NORMAL
1 or FIFO
2 or RR
3 or BATCH
4 or ISO
5 or IDLE
6 or DEADLINE
The process identifier types may be mixed. For each task, the following
items are displayed:
1. the process PID.
2. the parent process PID.
3. the CPU number that the task ran on last.
4. the task_struct address or the kernel stack pointer of the process.
(see -s option below)
5. the task state (RU, IN, UN, ZO, ST, TR, DE, SW, WA, PA, ID, NE).
6. the percentage of physical memory being used by this task.
7. the virtual address size of this task in kilobytes.
8. the resident set size of this task in kilobytes.
9. the command name.
The default output shows the task_struct address of each process under a
column titled "TASK". This can be changed to show the kernel stack
pointer under a column titled "KSTACKP".
-s replace the TASK column with the KSTACKP column.
On SMP machines, the active task on each CPU will be highlighted by an
angle bracket (">") preceding its information. If the crash variable
"offline" is set to "hide", the active task on an offline CPU will
be highlighted by a "-" preceding its information.
Alternatively, information regarding parent-child relationships,
per-task time usage data, argument/environment data, thread groups,
or resource limits may be displayed:
-p display the parental hierarchy of selected, or all, tasks.
-c display the children of selected, or all, tasks.
-t display the task run time, start time, and cumulative user
and system times.
-l display the task's last-run timestamp value, using either the
task_struct's last_run value, the task_struct's timestamp value
or the task_struct's sched_entity last_arrival value, whichever
applies, of selected, or all, tasks; the list is sorted with the
most recently-run task (with the largest timestamp) shown first,
followed by the task's current state.
-m similar to -l, but the timestamp value is translated into days,
hours, minutes, seconds, and milliseconds since the task was
last run on a cpu.
-C cpus only usable with the -l or -m options, dump the timestamp data
in per-cpu blocks, where the cpu[s] can be specified as "1,3,5",
"1-3", "1,3,5-7,10", "all", or "a" (shortcut for "all").
-a display the command line arguments and environment strings of
selected, or all, user-mode tasks.
-g display tasks by thread group, of selected, or all, tasks.
-r display resource limits (rlimits) of selected, or all, tasks.
-S display a summary consisting of the number of tasks in a task state.
-A display only the active task on each cpu.
EXAMPLES
Show the process status of all current tasks:
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 3 c024c000 RU 0.0 0 0 [swapper]
> 0 0 0 c0dce000 RU 0.0 0 0 [swapper]
0 0 1 c0fa8000 RU 0.0 0 0 [swapper]
> 0 0 2 c009a000 RU 0.0 0 0 [swapper]
1 0 1 c0098000 IN 0.0 1096 476 init
2 1 1 c0090000 IN 0.0 0 0 [kflushd]
3 1 1 c000e000 IN 0.0 0 0 [kpiod]
4 1 3 c000c000 IN 0.0 0 0 [kswapd]
5 1 1 c0008000 IN 0.0 0 0 [mdrecoveryd]
253 1 2 fbc4c000 IN 0.0 1088 376 portmap
- bt命令
该命令是查看进程栈信息的，可以通过pid/cpu等选项指定相关信息
crash> bt
PID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"
#0 [ffff000013573800] __crash_kexec at ffff00000817cee8
#1 [ffff000013573890] (null) at 108142390
#2 [ffff000013573920] psci_sys_reset at ffff000008965ae0
#3 [ffff000013573940] machine_restart at ffff000008085b94
#4 [ffff000013573960] emergency_restart at ffff0000080fddc4
#5 [ffff000013573970] panic at ffff0000080d8c04
#6 [ffff000013573a50] die at ffff00000808ae10
#7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8
#8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc
#9 [ffff000013573b30] do_translation_fault at ffff0000080a0390
#10 [ffff000013573b40] do_mem_abort at ffff00000808130c
#11 [ffff000013573d20] el1_ia at ffff000008083050
PC: ffff0000086095b0 [sysrq_handle_crash+32]
LR: ffff00000860959c [sysrq_handle_crash+12]
SP: ffff000013573d30 PSTATE: 60400145
X29: ffff000013573d30 X28: ffff800023c5b800 X27: ffff000008cc1000
X26: 0000000000000040 X25: 0000000000000124 X24: 0000000000000000
X23: 0000000000000004 X22: ffff000009217000 X21: ffff000009217f10
X20: 0000000000000063 X19: ffff00000919e000 X18: 0000000000000010
X17: 000000000049b4d8 X16: ffff000008277b80 X15: ffffffffffffffff
X14: ffff0000892fc7a7 X13: ffff0000092fc7b5 X12: ffff00000919e000
X11: ffff00000917ae78 X10: ffff000008434f78 X9: 00000000ffffffd0
X8: 0000000000000015 X7: 6767697254203a20 X6: 00000000000004a3
X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000
X2: ffff80002731efb8 X1: 0000000000000000 X0: 0000000000000001
#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac
#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c
#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100
#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c
#16 [ffff000013573dc0] __vfs_write at ffff00000827760c
#17 [ffff000013573e40] vfs_write at ffff000008277900
#18 [ffff000013573e80] sys_write at ffff000008277bc4
#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abc
PC: 0000ffff9e5aaeac LR: 0000000000408cc4 SP: 0000ffffeea390d0
X29: 0000ffffeea390d0 X28: 0000000000000000 X27: 0000000000000000
X26: 00000000071a3cc5 X25: 000000000049b000 X24: 00000000071a6372
X23: 0000000000000000 X22: 0000000000000001 X21: 00000000071a6370
X20: 0000000000000002 X19: 0000000000000001 X18: 00000000000002c8
X17: 000000000049b4d8 X16: 0000ffff9e5aae80 X15: 0000ffff9e4ebde0
X14: 0000ffff9e4f92c8 X13: 000000000000270f X12: 0101010101010101
X11: 0000000000000000 X10: 0101010101010101 X9: fffffffffffffff0
X8: 0000000000000040 X7: 7f7f7f7f7f7f7f7f X6: 0080000080808080
X5: 0000000000000000 X4: 00000000071a0063 X3: 0000ffff9e63f190
X2: 0000000000000002 X1: 00000000071a6370 X0: 0000000000000001
ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 80000000
- set
该命令获取crash的进程信息
crash> set
PID: 2496
COMMAND: "sh"
TASK: ffff800023c5b800 [THREAD_INFO: ffff800023c5b800]
CPU: 5
STATE: TASK_RUNNING (SYSRQ)
- sym
该命令查看符号和符号对应的地址信息，可以根据符号给出对应的地址，也可以根据给出的地址给出对应的符号
crash> sym proc_reg_write
ffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224
crash> sym ffff0000082ea1d0
ffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224
crash> sym __log_buf
ffff0000092ff020 (b) __log_buf
crash> sym ffff0000092ff020
ffff0000092ff020 (b) __log_buf
- ps
查看系统的进程信息，类似linux的ps
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffff000009186300 RU 0.0 0 0 [swapper/0]
> 0 0 1 ffff800025ba9c00 RU 0.0 0 0 [swapper/1]
> 0 0 2 ffff800025baaa00 RU 0.0 0 0 [swapper/2]
> 0 0 3 ffff800025bab800 RU 0.0 0 0 [swapper/3]
> 0 0 4 ffff800025bac600 RU 0.0 0 0 [swapper/4]
0 0 5 ffff800025bad400 RU 0.0 0 0 [swapper/5]
1 0 2 ffff800025b70000 IN 0.1 1824 1276 init
2 0 2 ffff800025b70e00 IN 0.0 0 0 [kthreadd]
4 2 0 ffff800025b72a00 ID 0.0 0 0 [kworker/0:0H]
6 2 0 ffff800025b74600 ID 0.0 0 0 [mm_percpu_wq]
7 2 0 ffff800025b75400 IN 0.0 0 0 [ksoftirqd/0]
8 2 5 ffff800025b76200 ID 0.0 0 0 [rcu_preempt]
9 2 1 ffff800025b77000 ID 0.0 0 0 [rcu_sched]
10 2 0 ffff800025ba8000 ID 0.0 0 0 [rcu_bh]
11 2 0 ffff800025ba8e00 IN 0.0 0 0 [migration/0]
12 2 0 ffff800025bae200 IN 0.0 0 0 [cpuhp/0]
13 2 1 ffff800025baf000 IN 0.0 0 0 [cpuhp/1]
14 2 1 ffff800025be8000 IN 0.0 0 0 [migration/1]
...
- vtop
根据一个虚拟地址转换成对应的物理地址
crash> vtop ffff800025065400
VIRTUAL PHYSICAL
ffff800025065400 65065400
PAGE DIRECTORY: ffff00000938b000
PGD: ffff00000938b800 => 673f8803
PUD: ffff8000273f8000 => 673f7803
PMD: ffff8000273f7940 => f8000065000f11
PAGE: 65000000 (2MB)
PTE PHYSICAL FLAGS
f8000065000f11 65000000 (VALID|SHARED|AF|NG|PXN|UXN|DIRTY)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffff7e0000941940 65065000 dead000000000400 0 0 fffc00000000000
- ptov
根据一个物理地址，转存成对应的内核态虚拟地址
crash> ptov 65065400
VIRTUAL PHYSICAL
ffff800025065400 65065400
- struct
查看结构体信息，如成员偏移，某个结构体变量内存中的值等
crash> struct thread_info -o
struct thread_info {
[0] unsigned long flags;
[8] mm_segment_t addr_limit;
[16] int preempt_count;
}
SIZE: 24
crash> struct thread_info -o -x
struct thread_info {
[0x0] unsigned long flags;
[0x8] mm_segment_t addr_limit;
[0x10] int preempt_count;
}
SIZE: 0x18
crash> struct thread_info 0xffff000009186300
struct thread_info {
flags = 32,
addr_limit = 281474976710655,
preempt_count = 65537
}
crash> task_struct.thread_info 0xffff000009186300 -p
thread_info = {
flags = 32,
addr_limit = 281474976710655,
preempt_count = 65537
},
- dis
反汇编，根据给出的PC地址，反编译出汇编源码，一般结合bt提供的栈信息和寄存器信息和反编译的汇编源码信息分析。
crash> bt
PID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"
#0 [ffff000013573800] __crash_kexec at ffff00000817cee8
#1 [ffff000013573890] (null) at 108142390
#2 [ffff000013573920] psci_sys_reset at ffff000008965ae0
#3 [ffff000013573940] machine_restart at ffff000008085b94
#4 [ffff000013573960] emergency_restart at ffff0000080fddc4
#5 [ffff000013573970] panic at ffff0000080d8c04
#6 [ffff000013573a50] die at ffff00000808ae10
#7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8
#8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc
#9 [ffff000013573b30] do_translation_fault at ffff0000080a0390
#10 [ffff000013573b40] do_mem_abort at ffff00000808130c
#11 [ffff000013573d20] el1_ia at ffff000008083050
PC: ffff0000086095b0 [sysrq_handle_crash+32]
LR: ffff00000860959c [sysrq_handle_crash+12]
SP: ffff000013573d30 PSTATE: 60400145
X29: ffff000013573d30 X28: ffff800023c5b800 X27: ffff000008cc1000
X26: 0000000000000040 X25: 0000000000000124 X24: 0000000000000000
X23: 0000000000000004 X22: ffff000009217000 X21: ffff000009217f10
X20: 0000000000000063 X19: ffff00000919e000 X18: 0000000000000010
X17: 000000000049b4d8 X16: ffff000008277b80 X15: ffffffffffffffff
X14: ffff0000892fc7a7 X13: ffff0000092fc7b5 X12: ffff00000919e000
X11: ffff00000917ae78 X10: ffff000008434f78 X9: 00000000ffffffd0
X8: 0000000000000015 X7: 6767697254203a20 X6: 00000000000004a3
X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000
X2: ffff80002731efb8 X1: 0000000000000000 X0: 0000000000000001
#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac
#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c
#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100
#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c
#16 [ffff000013573dc0] __vfs_write at ffff00000827760c
#17 [ffff000013573e40] vfs_write at ffff000008277900
#18 [ffff000013573e80] sys_write at ffff000008277bc4
#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abc
PC: 0000ffff9e5aaeac LR: 0000000000408cc4 SP: 0000ffffeea390d0
X29: 0000ffffeea390d0 X28: 0000000000000000 X27: 0000000000000000
X26: 00000000071a3cc5 X25: 000000000049b000 X24: 00000000071a6372
X23: 0000000000000000 X22: 0000000000000001 X21: 00000000071a6370
X20: 0000000000000002 X19: 0000000000000001 X18: 00000000000002c8
X17: 000000000049b4d8 X16: 0000ffff9e5aae80 X15: 0000ffff9e4ebde0
X14: 0000ffff9e4f92c8 X13: 000000000000270f X12: 0101010101010101
X11: 0000000000000000 X10: 0101010101010101 X9: fffffffffffffff0
X8: 0000000000000040 X7: 7f7f7f7f7f7f7f7f X6: 0080000080808080
X5: 0000000000000000 X4: 00000000071a0063 X3: 0000ffff9e63f190
X2: 0000000000000002 X1: 00000000071a6370 X0: 0000000000000001
ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 80000000
//查看栈数据
crash> bt -f 2
PID: 2 TASK: ffff800179d30000 CPU: 1 COMMAND: "kthreadd"
#0 [ffff00000a56bd90] __switch_to at ffff000008086690
ffff00000a56bd90: ffff00000a56bdb0 ffff000008ea5fe8
ffff00000a56bda0: ffff80017feae280 ffff800179d6e580
#1 [ffff00000a56bdb0] __schedule at ffff000008ea5fe4
ffff00000a56bdb0: ffff00000a56be40 ffff000008ea6708
ffff00000a56bdc0: ffff800179d30000 ffff000009a2bda0
ffff00000a56bdd0: ffff000009b8df60 ffff8000f4276600
ffff00000a56bde0: 0000000000000000 ffff000008108b68
ffff00000a56bdf0: 0000000000000001 ffff000008108000
ffff00000a56be00: 0000000000000000 0000000000000000
ffff00000a56be10: ffff000008109f54 ffff00000a56be60
ffff00000a56be20: ffff000008109e80 80838e611e66f600
ffff00000a56be30: ffff000000000004 80838e611e66f600
#2 [ffff00000a56be40] schedule at ffff000008ea6704
ffff00000a56be40: ffff00000a56be60 ffff000008109f54
ffff00000a56be50: ffff8000f4276628 ffff000008109e80
#3 [ffff00000a56be60] kthreadd at ffff000008109f50
//查看异常栈
crash> bt -e
PID: 1803 TASK: ffff8001794fab80 CPU: 5 COMMAND: "pvr_defer_free"
KERNEL-MODE EXCEPTION FRAME AT: ffff00000e7dbbc0
PC: ffff00000876d444 [DeviceMemSet+140]
LR: ffff0000087609c0 [_ZeroPageArray+120]
SP: ffff00000e7dbd00 PSTATE: 20c00145
X29: ffff00000e7dbd00 X28: ffff80015bc9fb00 X27: 00000000000186a0
X26: 0000000000000000 X25: 00e8000000000f0f X24: 0000000000000080
X23: ffff80012d0db000 X22: ffff00001fb7c000 X21: 0000000000000000
X20: 0000000000080000 X19: ffff00001fbaf000 X18: 0000ffffac000bcc
X17: 0000ffff97c22ec8 X16: ffff00000818ded0 X15: 0000ffffac000bc8
X14: 0140000000000000 X13: ffff00001fbfc000 X12: 0000000000000000
X11: 0000000000000000 X10: 0000000000000040 X9: 0040000000000041
X8: 0040000000000001 X7: 0000000000000001 X6: 000000017fffd7e8
X5: ffff8001398feb98 X4: ffff8001398feb98 X3: ffff00001fbfbfff
X2: 0000000000000000 X1: 0000000000000000 X0: ffff00001fbfc000
- irq
查看系统的中断信息
crash> irq
IRQ IRQ_DESC/_DATA IRQACTION NAME
0 (unused) (unused)
1 ffff800025807400 (unused)
2 ffff800025807800 (unused)
3 ffff800025807a00 (unused)
4 ffff800025807c00 ffff80002588df00 "arch_timer"
5 ffff800025807e00 (unused)
6 ffff800024c7e400 (unused)
7 ffff800024c7e600 ffff800024fcc100 "ttyS0"
8 ffff800024c7e800 (unused)
9 ffff800024c7ea00 ffff800024b10680 "30b00000.i2c"
10 ffff800024c7ec00 ffff800024efc200 "30b10000.i2c"
11 ffff800024c7ee00 ffff800024c72f00 "30b30000.i2c"
...
- files
查看进程打开的文件
crash> files
PID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"
ROOT: / CWD: /home/root
FD FILE DENTRY INODE TYPE PATH
0 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS0
1 ffff800024de9500 ffff8000239b29c0 ffff80002399e508 REG /proc/sysrq-trigger
2 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS0
10 ffff8000236a3b00 ffff8000245ff300 ffff800025095668 CHR /dev/tty
11 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS0
crash> foreach files -R /dev/console //结合foreach命令搜索打开/dev/console文件的进程
PID: 2331 TASK: ffff800024a1b800 CPU: 2 COMMAND: "messagecenter"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2460 TASK: ffff800024f8aa00 CPU: 5 COMMAND: "adbd"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2462 TASK: ffff80002421f000 CPU: 2 COMMAND: "usb ffs open"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2463 TASK: ffff80002421aa00 CPU: 5 COMMAND: "server socket"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2464 TASK: ffff800024a1aa00 CPU: 5 COMMAND: "->transport"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2465 TASK: ffff800024a1c600 CPU: 5 COMMAND: "<-transport"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2478 TASK: ffff800024a1d400 CPU: 1 COMMAND: "start_getty"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
0 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
1 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
- dev
查看系统注册的设备信息
crash> dev
CHRDEV NAME CDEV OPERATIONS
1 mem ffff8000253f5b00 memory_fops
2 pty ffff800024a6ed80 tty_fops
3 ttyp ffff8000249e3280 tty_fops
4 /dev/vc/0 ffff00000934b7a8 console_fops
4 tty ffff80002487e180 tty_fops
4 ttyS ffff800024f02e80 tty_fops
5 /dev/tty ffff00000934a2d0 tty_fops
5 /dev/console ffff00000934a338 console_fops
5 /dev/ptmx ffff00000934a4b0 ptmx_fops
...
BLKDEV NAME GENDISK OPERATIONS
259 blkext (none)
7 loop ffff800024e85000 lo_fops
8 sd (none)
9 md ffff800023db8800 md_fops
31 mtdblock (none)
...
- mount
查看系统挂载的文件系统
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff800025b4c000 ffff800025812000 rootfs rootfs /
ffff800023d0c000 ffff800023db9000 ext4 /dev/root /
ffff800023d0c1c0 ffff8000250b8000 devtmpfs devtmpfs /dev
ffff800023e0c000 ffff800025815000 proc proc /proc
ffff800023e0c1c0 ffff800023e60000 sysfs sysfs /sys
ffff800023e0c380 ffff800025817000 debugfs debugfs /sys/kernel/debug
ffff800023e0c540 ffff80002403f800 configfs configfs /sys/kernel/config
ffff800023e0c700 ffff800023e60800 tmpfs tmpfs /run
ffff800023e0c8c0 ffff800023e61000 tmpfs tmpfs /var/volatile
ffff8000250a8540 ffff8000234e4000 devpts devpts /dev/pts
ffff8000250a8700 ffff800023e61000 tmpfs tmpfs /var/lib
ffff800023e0ca80 ffff80002403f800 configfs none /config
ffff800023e0cc40 ffff800023e62000 functionfs adb /dev/usb-ffs/adb
ffff800023d0c380 ffff800024e19800 tracefs tracefs /sys/kernel/debug/tracing
ffff8000250a88c0 ffff8000234e6800 ext4 /dev/block/by-name/userdata /data
- kmem
查看内核态下内存的使用情况，如slab信息、vmalloc信息等
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 120690 471.4 MB ----
FREE 76395 298.4 MB 63% of TOTAL MEM
USED 44295 173 MB 36% of TOTAL MEM
SHARED 7202 28.1 MB 5% of TOTAL MEM
BUFFERS 649 2.5 MB 0% of TOTAL MEM
CACHED 10890 42.5 MB 9% of TOTAL MEM
SLAB 7869 30.7 MB 6% of TOTAL MEM
TOTAL HUGE 0 0 ----
HUGE FREE 0 0 0% of TOTAL HUGE
TOTAL SWAP 0 0 ----
SWAP USED 0 0 0% of TOTAL SWAP
SWAP FREE 0 0 0% of TOTAL SWAP
COMMIT LIMIT 60345 235.7 MB ----
COMMITTED 43807 171.1 MB 72% of TOTAL LIMIT
crash> kmem -s //查看kmalloc信息
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ffff800024ada180 136 3052 3060 102 4k ext4_groupinfo_1k
ffff800023c96000 144 3 28 1 4k ext4_groupinfo_4k
ffff800024164480 1408 2 46 2 32k UDPv6
ffff800024164300 232 0 0 0 4k tw_sock_TCPv6
ffff800024164180 304 0 0 0 8k request_sock_TCPv6
ffff800024164000 2112 5 75 5 32k TCPv6
ffff8000241c7e00 224 233 882 49 4k nf_conntrack_expect
ffff8000241c7c80 256 0 0 0 8k nf_conntrack
ffff8000241c7b00 312 0 0 0 8k ashmem_area_cache
ffff8000241c7980 3312 0 0 0 32k kcopyd_job
ffff8000241c7800 120 0 0 0 4k dm_rq_target_io
ffff8000241c7680 72 0 0 0 4k isp1760_qtd
ffff8000241c7500 112 13 216 6 4k cfq_io_cq
ffff8000241c7380 240 14 102 6 4k cfq_queue
ffff8000241c7200 216 0 0 0 4k bsg_cmd
ffff8000241c7080 896 1 17 1 16k mqueue_inode_cache
ffff8000241c6f00 656 0 0 0 16k v9fs_inode_cache
...
crash> kmem -S //查看所有slab对象信息
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ffff800024ada180 136 3052 3060 102 4k ext4_groupinfo_1k
CPU 0 KMEM_CACHE_CPU:
ffff7dffbfecbfb0
CPU 0 SLAB:
(empty)
CPU 0 PARTIAL:
(empty)
CPU 1 KMEM_CACHE_CPU:
ffff7dffbfee4fb0
CPU 1 SLAB:
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ffff7e0000721f80 ffff80001c87e000 0 30 22 8
FREE / [ALLOCATED]
[ffff80001c87e000]
[ffff80001c87e088]
...
crash> kmem -p //查看page信息
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffff7e0000000000 40000000 0 0 0 0
ffff7e0000000040 40001000 0 0 0 0
ffff7e0000000080 40002000 0 0 0 0
ffff7e00000000c0 40003000 0 0 0 0
ffff7e0000000100 40004000 0 0 0 0
ffff7e0000000140 40005000 0 0 0 0
...
- foreach
遍历系统上task的信息，如查看系统上处于UNINTERRUPT状态的task的栈信息
crash> foreach UN bt
PID: 561 TASK: ffff8000e3d33a00 CPU: 0 COMMAND: "kworker/u12:1"
#0 [ffff00002038b5f0] __switch_to at ffff000008086690
#1 [ffff00002038b610] __schedule at ffff000008ea5fe4
#2 [ffff00002038b6a0] schedule at ffff000008ea6704
#3 [ffff00002038b6c0] wait_transaction_locked at ffff0000083b5d8c
#4 [ffff00002038b730] add_transaction_credits at ffff0000083b602c
#5 [ffff00002038b7a0] start_this_handle at ffff0000083b624c
#6 [ffff00002038b840] jbd2__journal_start at ffff0000083b6a50
#7 [ffff00002038b8a0] __ext4_journal_start_sb at ffff0000083564d8
#8 [ffff00002038b8f0] ext4_writepages at ffff0000083760cc
#9 [ffff00002038ba40] do_writepages at ffff000008225400
#10 [ffff00002038bac0] __writeback_single_inode at ffff0000082ee774
#11 [ffff00002038bb10] writeback_sb_inodes at ffff0000082ef180
#12 [ffff00002038bc10] __writeback_inodes_wb at ffff0000082ef4b0
#13 [ffff00002038bc60] wb_writeback at ffff0000082ef828
#14 [ffff00002038bd10] wb_workfn at ffff0000082f0400
#15 [ffff00002038bdd0] process_one_work at ffff000008101950
#16 [ffff00002038be20] worker_thread at ffff000008101c6c
#17 [ffff00002038be70] kthread at ffff000008108c9c
PID: 814 TASK: ffff8001793f8000 CPU: 0 COMMAND: "saftey_heart_be"
#0 [ffff00000c053c10] __switch_to at ffff000008086690
#1 [ffff00000c053c30] __schedule at ffff000008ea5fe4
#2 [ffff00000c053cc0] schedule at ffff000008ea6704
#3 [ffff00000c053ce0] schedule_timeout at ffff000008eaa428
#4 [ffff00000c053d80] msleep at ffff000008178d64
#5 [ffff00000c053da0] saftey_hb_thread at ffff000008ad925c
#6 [ffff00000c053e70] kthread at ffff000008108c9c
PID: 1767 TASK: ffff800020776580 CPU: 0 COMMAND: "AVDemuxer::Read"
#0 [ffff000015dc3780] __switch_to at ffff000008086690
#1 [ffff000015dc37a0] __schedule at ffff000008ea5fe4
#2 [ffff000015dc3830] schedule at ffff000008ea6704
#3 [ffff000015dc3850] wait_transaction_locked at ffff0000083b5d8c
#4 [ffff000015dc38c0] add_transaction_credits at ffff0000083b602c
#5 [ffff000015dc3930] start_this_handle at ffff0000083b624c
#6 [ffff000015dc39d0] jbd2__journal_start at ffff0000083b6a50
#7 [ffff000015dc3a30] __ext4_journal_start_sb at ffff0000083564d8
#8 [ffff000015dc3a80] ext4_da_write_begin at ffff000008378448
#9 [ffff000015dc3c10] generic_perform_write at ffff00000821263c
#10 [ffff000015dc3ca0] __generic_file_write_iter at ffff000008215884
#11 [ffff000015dc3cf0] ext4_file_write_iter at ffff000008362760
#12 [ffff000015dc3d80] __vfs_write at ffff0000082b5180
#13 [ffff000015dc3e30] vfs_write at ffff0000082b53e0
#14 [ffff000015dc3e70] sys_write at ffff0000082b56f8
#15 [ffff000015dc3ff0] el0_svc_naked at ffff000008083af4
PC: 0000ffffa8eaee60 LR: 0000ffffa8eaee48 SP: 0000ffff93ffcdc0
X29: 0000ffff93ffcdc0 X28: 0000000000008000 X27: 0000000002faf080
X26: 0000000000000000 X25: 0000000000000000 X24: 0000000000001000
X23: 0000ffff84004de0 X22: 0000000000001000 X21: 0000ffff84004de0
X20: 0000000000001000 X19: 000000000000001f X18: 0000ffffa8f5ba70
X17: 0000ffffa8e50398 X16: 0000ffffa9b00420 X15: 0000074895555910
X14: 0000000000000000 X13: 52e6647d9bae90b9 X12: 936c08cc82e6992c
X11: 2de7fcffe5393767 X10: 41a928c963510c66 X9: 44377654e95c11b2
X8: 0000000000000040 X7: 042d1f3b698dddd1 X6: 0000000000000000
X5: 0000ffff93ffede0 X4: 00000000ffffffbb X3: 0000000000000000
X2: 0000000000001000 X1: 0000ffff84004de0 X0: 000000000000001f
ORIG_X0: 000000000000001f SYSCALLNO: 40 PSTATE: 80000000
...
- p
打印变量或者表达式的值
crash> p jiffies
jiffies = $1 = 4298283231
crash> p vm_event_states
PER-CPU DATA TYPE:
struct vm_event_state vm_event_states;
PER-CPU ADDRESSES:
[0]: ffff80017fe91ab8
[1]: ffff80017feaaab8
[2]: ffff80017fec3ab8
[3]: ffff80017fedcab8
[4]: ffff80017fef5ab8
[5]: ffff80017ff0eab8
//查看cpu1对应的变量的值
crash> p vm_event_states:1
per_cpu(vm_event_states, 1) = $3 = {
event = {318552, 739584, 0, 0, 558108, 2889268, 0, 0, 0, 0, 0, 0, 0, 4189877, 38713, 21935, 255, 2454551, 259, 0, 27971, 34455, 0, 63883, 0, 0, 0, 0, 62080, 233, 2, 1, 14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4576, 0, 0, 0, 46844, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 622, 34, 48, 0, 0, 0, 0, 0, 2119, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0}
}
//查看数组元素
crash> p &page_wait_table
$4 = (wait_queue_head_t (*)[256]) 0xffff000009a07d80
crash> p page_wait_table[4]
$5 = {
lock = {
{
rlock = {
raw_lock = {
owner = 6378,
next = 6378
}
}
}
},
head = {
next = 0xffff000009a07de8 ,
prev = 0xffff000009a07de8
}
}
- list
遍历由指针或者链表连接起来的内容
//查看挂在panic_notifier_list的notify
crash> p &panic_notifier_list
$3 = (struct atomic_notifier_head *) 0xffff000009b8a670
crash> struct atomic_notifier_head
struct atomic_notifier_head {
spinlock_t lock;
struct notifier_block *head;
}
SIZE: 16
crash> struct notifier_block
struct notifier_block {
notifier_fn_t notifier_call;
struct notifier_block *next;
int priority;
}
SIZE: 24
crash> list atomic_notifier_head.head 0xffff000009b8a670 -s notifier_block.next
ffff000009b8a670
next = 0xffff000009afaad0 ,
ffff000009afaad0
next = 0xffff000009a3b540 ,
ffff000009a3b540
next = 0xffff000009a322a0 ,
ffff000009a322a0
next = 0xffff000009a3ad50 ,
ffff000009a3ad50
next = 0xffff000009aa5b80 ,
ffff000009aa5b80
next = 0xffff000009a1b190 ,
ffff000009a1b190
next = 0xffff000009a1b588 ,
ffff000009a1b588
next = 0xffff000009a1d3b0 ,
ffff000009a1d3b0
next = 0xffff000009afd460 ,
ffff000009afd460
next = 0x0,