Crash-utility command help 汇总（1）
NAME
extend - extend the crash command set
SYNOPSIS
extend [shared-object …] | [-u [shared-object …]]
DESCRIPTION
This command dynamically loads or unloads crash extension shared object
libraries:
shared-object load the specified shared object file; more than one
one object file may be entered.
-u shared-object unload the specified shared object file; if no file
arguments are specified, unload all objects.
If the shared-object filename is not expressed with a fully-qualified
pathname, the following directories will be searched in the order shown,
and the first instance of the file that is found will be selected:
1. the current working directory
2. the directory specified in the CRASH_EXTENSIONS environment variable
3. /usr/lib64/crash/extensions (64-bit architectures)
4. /usr/lib/crash/extensions
If no arguments are entered, the current set of shared object files and
a list of their commands will be displayed. The registered commands
contained in each shared object file will appear automatically in the
“help” command screen.
An example of a shared object prototype file, and how to compile it
into a shared object, is appended below.
EXAMPLES
Load two shared object files:
crash> extend extlib1.so extlib2.so
./extlib1.so: shared object loaded
./extlib2.so: shared object loaded
Display the current set of shared object files and their commands:
crash> extend
SHARED OBJECT COMMANDS
./extlib1.so echo util bin
./extlib2.so smp show
Unload one of the shared object files:
crash> extend -u extlib1.so
./extlib1.so: shared object unloaded
Unload all currently-loaded object files:
crash> extend -u
./extlib2.so: shared object unloaded
CREATING A SHARED OBJECT
The extend command loads shared object files using dlopen(3), which in
turn calls the shared object’s constructor function. The shared object’s
constructor function should register its command set by calling
register_extension(), passing it a pointer to an array of one or more
structures of the following type:
struct command_table_entry {
char *name;
cmd_func_t func;
char **help_data,
ulong flags;
};
Each command_table_entry structure contains the ASCII name of a command,
the command’s function address, a pointer to an array of help data strings,
and a flags field. The help_data field is optional; if it is non-NULL, it
should point to an array of character strings used by the “help”
command, and during command failures. The flags field currently has two
available bit settings, REFRESH_TASK_TABLE, which should be set if it is
preferable to reload the current set of running processes just prior to
executing the command (on a live system) and MINIMAL, which should be
set if the command should be available in minimal mode. Terminate the array
of command_table_entry structures with an entry with a NULL command name.
Below is an example shared object file consisting of just one command,
called “echo”, which simply echoes back all arguments passed to it.
Note the comments contained within it for further details. Cut and paste
the following output into a file, and call it, for example, “echo.c”.
Then compiled in either of two manners. Either manually like so:
gcc -shared -rdynamic -o echo.so echo.c -fPIC -D $(TARGET_CFLAGS)
where must be one of the MACHINE_TYPE #define’s in defs.h,
and where $(TARGET_CFLAGS) is the same as it is declared in the top-level
Makefile after a build is completed. Or alternatively, the “echo.c” file
can be copied into the “extensions” subdirectory, and compiled automatically
like so:
make extensions
The echo.so file may be dynamically linked into crash during runtime, or
during initialization by putting “extend echo.so” into a .crashrc file
located in the current directory, or in the user’s $HOME directory.
---------------------------------- cut here ----------------------------------
#include "defs.h" /* From the crash source top-level directory */
void echo_init(void); /* constructor function */
void echo_fini(void); /* destructor function (optional) */
void cmd_echo(void); /* Declare the commands and their help data. */
char *help_echo[];
static struct command_table_entry command_table[] = {
{ "echo", cmd_echo, help_echo, 0}, /* One or more commands, */
{ NULL }, /* terminated by NULL, */
};
void __attribute__((constructor))
echo_init(void) /* Register the command set. */
{
register_extension(command_table);
}
/*
* This function is called if the shared object is unloaded.
* If desired, perform any cleanups here.
*/
void __attribute__((destructor))
echo_fini(void) { }
/*
* Arguments are passed to the command functions in the global args[argcnt]
* array. See getopt(3) for info on dash arguments. Check out defs.h and
* other crash commands for usage of the myriad of utility routines available
* to accomplish what your task.
*/
void
cmd_echo(void)
{
int c;
while ((c = getopt(argcnt, args, "")) != EOF) {
switch(c)
{
default:
argerrs++;
break;
}
}
if (argerrs)
cmd_usage(pc->curcmd, SYNOPSIS);
while (args[optind])
fprintf(fp, "%s ", args[optind++]);
fprintf(fp, "\n");
}
/*
* The optional help data is simply an array of strings in a defined format.
* For example, the "help echo" command will use the help_echo[] string
* array below to create a help page that looks like this:
*
*
* NAME
* echo - echoes back its arguments
*
* SYNOPSIS
* echo arg ...
*
* DESCRIPTION
* This command simply echoes back its arguments.
*
* EXAMPLE
* Echo back all command arguments:
*
* crash> echo hello, world
* hello, world
*
*/
char *help_echo[] = {
"echo", /* command name */
"echoes back its arguments", /* short description */
"arg ...", /* argument synopsis, or " " if none */
" This command simply echoes back its arguments.",
"\nEXAMPLE",
" Echo back all command arguments:\n",
" crash> echo hello, world",
" hello, world",
NULL
};
///
NAME
log - dump system message buffer
SYNOPSIS
log [-tdma]
DESCRIPTION
This command dumps the kernel log_buf contents in chronological order. The
command supports the older log_buf formats, which may or may not contain a
timestamp inserted prior to each message, as well as the newer variable-length
record format, where the timestamp is contained in each log entry’s header.
-t Display the message text without the timestamp; only applicable to the
variable-length record format.
-d Display the dictionary of key/value pair properties that are optionally
appended to a message by the kernel's dev_printk() function; only
applicable to the variable-length record format.
-m Display the message log level in brackets preceding each message. For
the variable-length record format, the level will be displayed in
hexadecimal, and depending upon the kernel version, also contains the
facility or flags bits.
-a Dump the audit logs remaining in kernel audit buffers that have not
been copied out to the user-space audit daemon.
EXAMPLES
Dump the kernel message buffer:
crash> log
Linux version 2.2.5-15smp (root@mclinux1) (gcc version egcs-2.91.66 19990
314/Linux (egcs-1.1.2 release)) #1 SMP Thu Aug 26 11:04:37 EDT 1999
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: DELL Product ID: WS 410 APIC at: 0xFEE00000
Processor #0 Pentium(tm) Pro APIC version 17
Processor #1 Pentium(tm) Pro APIC version 17
I/O APIC #2 Version 17 at 0xFEC00000.
Processors: 2
mapped APIC to ffffe000 (fee00000)
mapped IOAPIC to ffffd000 (fec00000)
Detected 447696347 Hz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 445.64 BogoMIPS
...
8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
MII transceiver found at address 24, status 782d.
Enabling bus-master transmits and whole-frame receives.
Installing knfsd (copyright (C) 1996 [email protected]).
nfsd_init: initialized fhcache, entries=256
...
Do the same thing, but also show the log level preceding each message:
crash> log -m
<4>Linux version 2.2.5-15smp (root@mclinux1) (gcc version egcs-2.91.66 19990
314/Linux (egcs-1.1.2 release)) #1 SMP Thu Aug 26 11:04:37 EDT 1999
<4>Intel MultiProcessor Specification v1.4
<4> Virtual Wire compatibility mode.
<4>OEM ID: DELL Product ID: WS 410 APIC at: 0xFEE00000
<4>Processor #0 Pentium(tm) Pro APIC version 17
<4>Processor #1 Pentium(tm) Pro APIC version 17
<4>I/O APIC #2 Version 17 at 0xFEC00000.
<4>Processors: 2
<4>mapped APIC to ffffe000 (fee00000)
<4>mapped IOAPIC to ffffd000 (fec00000)
<4>Detected 447696347 Hz processor.
<4>Console: colour VGA+ 80x25
<4>Calibrating delay loop... 445.64 BogoMIPS
...
<6> 8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
<6> MII transceiver found at address 24, status 782d.
<6> Enabling bus-master transmits and whole-frame receives.
<6>Installing knfsd (copyright (C) 1996 [email protected]).
<7>nfsd_init: initialized fhcache, entries=256
...
On a system with the variable-length record format, and whose log_buf has been
filled and wrapped around, display the log with timestamp data:
crash> log
[ 0.467730] pci 0000:ff:02.0: [8086:2c10] type 00 class 0x060000
[ 0.467749] pci 0000:ff:02.1: [8086:2c11] type 00 class 0x060000
[ 0.467769] pci 0000:ff:02.4: [8086:2c14] type 00 class 0x060000
[ 0.467788] pci 0000:ff:02.5: [8086:2c15] type 00 class 0x060000
[ 0.467809] pci 0000:ff:03.0: [8086:2c18] type 00 class 0x060000
[ 0.467828] pci 0000:ff:03.1: [8086:2c19] type 00 class 0x060000
...
Display the same message text as above, without the timestamp data:
crash> log -t
pci 0000:ff:02.0: [8086:2c10] type 00 class 0x060000
pci 0000:ff:02.1: [8086:2c11] type 00 class 0x060000
pci 0000:ff:02.4: [8086:2c14] type 00 class 0x060000
pci 0000:ff:02.5: [8086:2c15] type 00 class 0x060000
pci 0000:ff:03.0: [8086:2c18] type 00 class 0x060000
pci 0000:ff:03.1: [8086:2c19] type 00 class 0x060000
...
Display the same message text as above, with appended dictionary data:
crash> log -td
pci 0000:ff:02.0: [8086:2c10] type 00 class 0x060000
SUBSYSTEM=pci
DEVICE=+pci:0000:ff:02.0
pci 0000:ff:02.1: [8086:2c11] type 00 class 0x060000
SUBSYSTEM=pci
DEVICE=+pci:0000:ff:02.1
pci 0000:ff:02.4: [8086:2c14] type 00 class 0x060000
SUBSYSTEM=pci
DEVICE=+pci:0000:ff:02.4
pci 0000:ff:02.5: [8086:2c15] type 00 class 0x060000
SUBSYSTEM=pci
DEVICE=+pci:0000:ff:02.5
pci 0000:ff:03.0: [8086:2c18] type 00 class 0x060000
SUBSYSTEM=pci
DEVICE=+pci:0000:ff:03.0
pci 0000:ff:03.1: [8086:2c19] type 00 class 0x060000
SUBSYSTEM=pci
DEVICE=+pci:0000:ff:03.1
...
Dump the kernel audit logs:
crash> log -a
type=1320 audit(1489384479.809:4342):
type=1300 audit(1489384479.809:4343): arch=c000003e syscall=0 success=yes
exit=0 a0=4 a1=7f84154a2000 a2=400 a3=22 items=0 ppid=2560 pid=2591 auid=0
uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=ttyS0 ses=1
comm="pidof" exe="/usr/sbin/killall5"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=1320 audit(1489384479.809:4343):
type=1300 audit(1489384479.809:4344): arch=c000003e syscall=3 success=yes
exit=0 a0=4 a1=1 a2=8 a3=0 items=0 ppid=2560 pid=2591 auid=0 uid=0 gid=0
euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=ttyS0 ses=1 comm="pidof"
exe="/usr/sbin/killall5"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=1320 audit(1489384479.809:4344):
type=1300 audit(1489384479.809:4345): arch=c000003e syscall=11
success=yes exit=0 a0=7f84154a2000 a1=1000 a2=0 a3=0 items=0 ppid=2560
pid=2591 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
tty=ttyS0 ses=1 comm="pidof" exe="/usr/sbin/killall5"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=1320 audit(1489384479.809:4345):
type=1300 audit(1489384479.809:4346): arch=c000003e syscall=2 success=yes
exit=4 a0=7ffcfd20f5a0 a1=0 a2=1b6 a3=24 items=1 ppid=2560 pid=2591 auid=0
uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=ttyS0 ses=1
comm="pidof" exe="/usr/sbin/killall5"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=1307 audit(1489384479.809:4346): cwd="/proc"
...
///
NAME
rd - read memory
SYNOPSIS
rd [-adDsSupxmfNR][-8|-16|-32|-64][-o offs][-e addr][-r file][address|symbol]
[count]
DESCRIPTION
This command displays the contents of memory, with the output formatted
in several different manners. The starting address may be entered either
symbolically or by address. The default output size is the size of a long
data type, and the default output format is hexadecimal. When hexadecimal
output is used, the output will be accompanied by an ASCII translation.
-p address argument is a physical address.
-u address argument is a user virtual address; only required on
processors with common user and kernel virtual address spaces.
-m address argument is a xen host machine address.
-f address argument is a dumpfile offset.
-d display output in signed decimal format (default is hexadecimal).
-D display output in unsigned decimal format (default is hexadecimal).
-s displays output symbolically when appropriate.
-S[S] displays output symbolically when appropriate; if the memory
contents reference a slab cache object, the name of the slab cache
will be displayed in brackets. If -S is entered twice, and the
memory contents reference a slab cache object, both the memory
contents and the name of the slab cache will be displayed in
brackets.
-x do not display ASCII translation at end of each line.
-8 display output in 8-bit values.
-16 display output in 16-bit values.
-32 display output in 32-bit values (default on 32-bit machines).
-64 display output in 64-bit values (default on 64-bit machines).
-a display output in ASCII characters if the memory contains printable
ASCII characters; if no count argument is entered, stop at the first
non-printable character.
-N display output in network byte order (only valid for 16- and 32-bit
values)
-R display memory in reverse order; memory will be displayed up to and
including the address argument, requiring the count argument to be
greater than 1 in order to display memory before the specified
address.
-o offs offset the starting address by offs.
-e addr display memory until reaching specified ending hexadecimal address.
-r file dumps raw data to the specified output file; the number of bytes that
are copied to the file must be specified either by a count argument
or by the -e option.
address starting hexadecimal address:
1. the default presumes a kernel virtual address.
2. -p specifies a physical address.
3. -u specifies a user virtual address, but is only necessary on
processors with common user and kernel virtual address spaces.
symbol symbol of starting address to read.
count number of memory locations to display; if entered, it must be the
last argument on the command line; if not entered, the count defaults
to 1, or unlimited for -a; when used with the -r option, it is the
number of bytes to be written to the file.
EXAMPLES
Display the kernel’s version string:
crash> rd -a linux_banner
c082a020: Linux version 2.6.32-119.el6.i686 ([email protected]
c082a05c: d.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13)
c082a098: (GCC) ) #1 SMP Tue Mar 1 18:16:57 EST 2011
Display the same block of memory, first without symbols, again
with symbols, and then with symbols and slab cache references:
crash> rd f6e31f70 28
f6e31f70: f6e31f6c f779c180 c04a4032 00a9dd40 l.....y.2@J.@...
f6e31f80: 00000fff c0472da0 f6e31fa4 f779c180 .....-G.......y.
f6e31f90: fffffff7 00a9b70f f6e31000 c04731ee .............1G.
f6e31fa0: f6e31fa4 00000000 00000000 00000000 ................
f6e31fb0: 00000000 00a9dd40 c0404f17 00000000 [email protected]@.....
f6e31fc0: 00a9dd40 00000fff 00a9dd40 00a9b70f @.......@.......
f6e31fd0: bf9e2718 ffffffda c040007b 0000007b .'......{.@.{...
crash> rd -s f6e31f70 28
f6e31f70: f6e31f6c f779c180 kmsg_read 00a9dd40
f6e31f80: 00000fff vfs_read+159 f6e31fa4 f779c180
f6e31f90: fffffff7 00a9b70f f6e31000 sys_read+60
f6e31fa0: f6e31fa4 00000000 00000000 00000000
f6e31fb0: 00000000 00a9dd40 syscall_call+7 00000000
f6e31fc0: 00a9dd40 00000fff 00a9dd40 00a9b70f
f6e31fd0: bf9e2718 ffffffda startup_32+123 0000007b
crash> rd -S f6e31f70 28
f6e31f70: [size-4096] [filp] kmsg_read 00a9dd40
f6e31f80: 00000fff vfs_read+159 [size-4096] [filp]
f6e31f90: fffffff7 00a9b70f [size-4096] sys_read+60
f6e31fa0: [size-4096] 00000000 00000000 00000000
f6e31fb0: 00000000 00a9dd40 syscall_call+7 00000000
f6e31fc0: 00a9dd40 00000fff 00a9dd40 00a9b70f
f6e31fd0: bf9e2718 ffffffda startup_32+123 0000007b
crash> rd -SS f6e31f70 28
f6e31f70: [f6e31f6c:size-4096] [f779c180:filp] kmsg_read 00a9dd40
f6e31f80: 00000fff vfs_read+159 [f6e31fa4:size-4096] [f779c180:filp]
f6e31f90: fffffff7 00a9b70f [f6e31000:size-4096] sys_read+60
f6e31fa0: [f6e31fa4:size-4096] 00000000 00000000 00000000
f6e31fb0: 00000000 00a9dd40 syscall_call+7 00000000
f6e31fc0: 00a9dd40 00000fff 00a9dd40 00a9b70f
f6e31fd0: bf9e2718 ffffffda startup_32+123 0000007b
Read jiffies in hexadecimal and decimal format:
crash> rd jiffies
c0213ae0: 0008cc3a :...
crash> rd -d jiffies
c0213ae0: 577376
Access the same memory in different sizes:
crash> rd -64 kernel_version
c0226a6c: 35312d352e322e32 2.2.5-15
crash> rd -32 kernel_version 2
c0226a6c: 2e322e32 35312d35 2.2.5-15
crash> rd -16 kernel_version 4
c0226a6c: 2e32 2e32 2d35 3531 2.2.5-15
crash> rd -8 kernel_version 8
c0226a6c: 32 2e 32 2e 35 2d 31 35 2.2.5-15
Read the range of memory from c009bf2c to c009bf60:
crash> rd c009bf2c -e c009bf60
c009bf2c: c009bf64 c01328c3 c009bf64 c0132838 d....(..d...8(..
c009bf3c: 0000002a 00000004 c57d77e8 00000104 *........w}.....
c009bf4c: 0000000b c009a000 7fffffff 00000000 ................
c009bf5c: 00000000 ....
///
NAME
task - task_struct and thread_info contents
SYNOPSIS
task [-R member[,member]] [-dx] [pid | taskp] …
DESCRIPTION
This command dumps a formatted display of the contents of a task’s
task_struct and thread_info structures. Multiple task or PID numbers
may be entered; if no arguments are entered, the task_struct and
thread_info structures of the current context are displayed. The -R option,
which may also be invoked indirectly via “foreach task”, pares the output
down to one or more structure members.
pid a process PID.
taskp a hexadecimal task_struct pointer.
-R member
a comma-separated list of one or more task_struct and/or
thread_info structure members. If any member contains an embedded
structure, or is an array, the output may be restricted to the
embedded structure or an array element by expressing the member
argument as "member.member" or "member[index]"; embedded member
specifications may extend beyond one level deep, by expressing the
member argument as "member.member.member...".
-x override default output format with hexadecimal format.
-d override default output format with decimal format.
EXAMPLES
Dump the task_struct and thread_info structures of the current context
in hexadecimal format:
crash> task -x
PID: 3176 TASK: f2451550 CPU: 1 COMMAND: "memtest"
struct task_struct {
state = 0x0,
stack = 0xf05b6000,
usage = {
counter = 0x2
},
flags = 0x402040,
ptrace = 0x0,
lock_depth = 0xffffffff,
prio = 0x78,
static_prio = 0x78,
normal_prio = 0x78,
rt_priority = 0x0,
...
perf_event_ctxp = {0x0, 0x0},
memcg_batch = {
do_batch = 0x0,
memcg = 0x0,
bytes = 0x0,
memsw_bytes = 0x0
}
}
struct thread_info {
task = 0xf2451550,
exec_domain = 0xc0a60860,
flags = 0x88,
status = 0x0,
cpu = 0x1,
preempt_count = 0x4010000,
addr_limit = {
seg = 0xc0000000
},
restart_block = {
...
Display the ngroups and groups task_struct members for PID 2958:
crash> task -R ngroups,groups 2958
PID: 2958 TASK: c6718000 CPU: 0 COMMAND: "bash"
ngroups = 6,
groups = {504, 8, 9, 1000, 1007, 1006, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
Display the embedded sched_entity structure’s on_rq member:
crash> task -R se.on_rq
PID: 6529 TASK: ffff880116538790 CPU: 1 COMMAND: "bash"
se.on_rq = 1,
Display the 3rd pid_link structure in the embedded pids[] array:
crash> task -R pids[2]
PID: 6529 TASK: ffff880116538790 CPU: 0 COMMAND: "bash"
pids[2] = {
node = {
next = 0xffff8801165391b0,
pprev = 0xffff880209d011b0
},
pid = 0xffff8801f0876e00
}
NOTE: When this command is invoked directly (i.e., not from “foreach”), it
is not necessary to include the “-R” before the task_struct/thread_info
member name(s).
///
NAME
ascii - translate a hexadecimal string to ASCII
SYNOPSIS
ascii value …
DESCRIPTION
Translates 32-bit or 64-bit hexadecimal values to ASCII. If no argument
is entered, an ASCII chart is displayed.
EXAMPLES
Translate the hexadecimal value of 0x62696c2f7273752f to ASCII:
crash> ascii 62696c2f7273752f
62696c2f7273752f: /usr/lib
Display an ASCII chart:
crash> ascii
0 1 2 3 4 5 6 7
+-------------------------------
0 | NUL DLE SP 0 @ P ' p
1 | SOH DC1 ! 1 A Q a q
2 | STX DC2 " 2 B R b r
3 | ETX DC3 # 3 C S c s
4 | EOT DC4 $ 4 D T d t
5 | ENQ NAK % 5 E U e u
6 | ACK SYN & 6 F V f v
7 | BEL ETB ` 7 G W g w
8 | BS CAN ( 8 H X h x
9 | HT EM ) 9 I Y i y
A | LF SUB * : J Z j z
B | VT ESC + ; K [ k {
C | FF FS , < L \ l |
D | CR GS _ = M ] m }
E | SO RS . > N ^ n ~
F | SI US / ? O - o DEL
///
NAME
foreach - display command data for multiple tasks in the system
SYNOPSIS
foreach [[pid | taskp | name | state | [kernel | user | gleader]] …]
command [flag] [argument]
DESCRIPTION
This command allows for an examination of various kernel data associated
with any, or all, tasks in the system, without having to set the context
to each targeted task.
pid perform the command(s) on this PID.
taskp perform the command(s) on task referenced by this hexadecimal
task_struct pointer.
name perform the command(s) on all tasks with this name. If the
task name can be confused with a foreach command name, then
precede the name string with a "\". If the name string is
enclosed within "'" characters, then the encompassed string
must be a POSIX extended regular expression that will be used
to match task names.
user perform the command(s) on all user (non-kernel) threads.
gleader perform the command(s) on all user (non-kernel) thread group leaders.
kernel perform the command(s) on all kernel threads.
active perform the command(s) on the active thread on each CPU.
state perform the command(s) on all tasks in the specified state, which
may be one of: RU, IN, UN, ST, ZO, TR, SW, DE, WA, PA, ID or NE.
If none of the task-identifying arguments above are entered, the command
will be performed on all tasks.
command select one or more of the following commands to be run on the tasks
selected, or on all tasks:
bt run the "bt" command (optional flags: -r -t -l -e -R -f -F
-o -s -x -d)
vm run the "vm" command (optional flags: -p -v -m -R -d -x)
task run the "task" command (optional flags: -R -d -x)
files run the "files" command (optional flag: -c -R)
net run the "net" command (optional flags: -s -S -R -d -x)
set run the "set" command
ps run the "ps" command (optional flags: -G -s -p -c -t -l -a
-g -r -y)
sig run the "sig" command (optional flag: -g)
vtop run the "vtop" command (optional flags: -c -u -k)
flag Pass this optional flag to the command selected.
argument Pass this argument to the command selected.
A header containing the PID, task address, cpu and command name will be
pre-pended before the command output for each selected task. Consult the
help page of each of the command types above for details.
EXAMPLES
Display the stack traces for all tasks:
crash> foreach bt
PID: 4752 TASK: c7680000 CPU: 1 COMMAND: "xterm"
#0 [c7681edc] schedule at c01135f6
(void)
#1 [c7681f34] schedule_timeout at c01131ff
(24)
#2 [c7681f64] do_select at c0132838
(5, c7681fa4, c7681fa0)
#3 [c7681fbc] sys_select at c0132dad
(5, 8070300, 8070380, 0, 0)
#4 [bffffb0c] system_call at c0109944
EAX: 0000008e EBX: 00000005 ECX: 08070300 EDX: 08070380
DS: 002b ESI: 00000000 ES: 002b EDI: 00000000
SS: 002b ESP: bffffadc EBP: bffffb0c
CS: 0023 EIP: 402259ee ERR: 0000008e EFLAGS: 00000246
PID: 557 TASK: c5600000 CPU: 0 COMMAND: "nfsd"
#0 [c5601f38] schedule at c01135f6
(void)
#1 [c5601f90] schedule_timeout at c01131ff
(c5600000)
#2 [c5601fb8] svc_recv at c805363a
(c0096f40, c5602800, 7fffffff, 100, c65c9f1c)
#3 [c5601fec] (nfsd module) at c806e303
(c5602800, c5602800, c0096f40, 6c6e0002, 50)
#4 [c65c9f24] kernel_thread at c010834f
(0, 0, ext2_file_inode_operations)
PID: 824 TASK: c7c84000 CPU: 0 COMMAND: "mingetty"
...
Display the task_struct structure for each “bash” command:
crash> foreach bash task
...
Display the open files for all tasks:
crash> foreach files
...
Display the state of tasks whose name contains a match to “event.*”:
crash> foreach 'event.*' task -R state
PID: 99 TASK: ffff8804750d5500 CPU: 0 COMMAND: "events/0"
state = 1,
PID: 100 TASK: ffff8804750d4ac0 CPU: 1 COMMAND: "events/1"
state = 1,
PID: 101 TASK: ffff8804750d4080 CPU: 2 COMMAND: "events/2"
state = 1,
...
Display the stack traces for all blocked (TASK_UNINTERRUPTIBLE) tasks:
crash> foreach UN bt
PID: 428 TASK: ffff880036b6c560 CPU: 1 COMMAND: "jbd2/dm-1-8"
#0 [ffff880035779a70] __schedule at ffffffff815df272
#1 [ffff880035779b08] schedule at ffffffff815dfacf
#2 [ffff880035779b18] io_schedule at ffffffff815dfb7f
#3 [ffff880035779b38] sleep_on_page at ffffffff81119a4e
#4 [ffff880035779b48] __wait_on_bit at ffffffff815e039f
#5 [ffff880035779b98] wait_on_page_bit at ffffffff81119bb8
#6 [ffff880035779be8] filemap_fdatawait_range at ffffffff81119ccc
#7 [ffff880035779cd8] filemap_fdatawait at ffffffff81119d8b
#8 [ffff880035779ce8] jbd2_journal_commit_transaction at ffffffff8123a99c
#9 [ffff880035779e58] kjournald2 at ffffffff8123ee7b
#10 [ffff880035779ee8] kthread at ffffffff8108fb9c
#11 [ffff880035779f48] kernel_thread_helper at ffffffff815ebaf4
...
///
NAME
mod - module information and loading of symbols and debugging data
SYNOPSIS
mod -s module [objfile] | -d module | -S [directory] [-D|-t|-r|-R|-o|-g]
DESCRIPTION
With no arguments, this command displays basic information of the currently
installed modules, consisting of the module address, name, size, the
object file name (if known), and whether the module was compiled with
CONFIG_KALLSYMS.
The arguments are concerned with with the loading or deleting of symbolic
and debugging data from a module’s object file. A modules’s object file
always contains symbolic data (symbol names and addresses), but contains
debugging data only if the module was compiled with the -g CFLAG. In
addition, the module may have compiled with CONFIG_KALLSYMS, which means
that the module’s symbolic data will have been loaded into the kernel’s
address space when it was installed. If the module was not compiled with
CONFIG_KALLSYMS, then only the module’s exported symbols will be loaded
into the kernel’s address space. Therefore, for the purpose of this
command, it should noted that a kernel module may have been compiled in
one of following manners:
-
If the module was built without CONFIG_KALLSYMS and without the -g CFLAG,
then the loading of the module’s additional non-exported symbols can
be accomplished with this command. -
If the module was built with CONFIG_KALLSYMS, but without the -g CFLAG,
then there is no benefit in loading the symbols from the module object
file, because all of the module’s symbols will have been loaded into the
kernel’s address space when it was installed. -
If the module was built with CONFIG_KALLSYMS and with the the -g CFLAG,
then the loading of the module’s debugging data can be accomplished
with this command. -
If the module was built without CONFIG_KALLSYMS but with the -g CFLAG,
then the loading of the both module’s symbolic and debugging data can
be accomplished with this command.-s module [objfile] Loads symbolic and debugging data from the object file for the module specified. If no objfile argument is appended, a search will be made for an object file consisting of the module name with a .o or .ko suffix, starting at the /lib/modules/directory on the host system, or if not found there, starting at the directory containing the kernel namelist file. If an objfile argument is appended, then that file will be used. -d module Deletes the symbolic and debugging data of the module specified. -S [directory] Load symbolic and debugging data from the object file for all loaded modules. For each module, a search will be made for an object file consisting of the module name with a .o or .ko suffix, starting at the /lib/modules/ directory of the host system, or if not found there, starting at the directory containing the kernel namelist file. If a directory argument is appended, then the search will be restricted to that directory. -D Deletes the symbolic and debugging data of all modules. -t Display the contents of the module's "taints" bitmask if it is non-zero. When possible, the "taints" bits are translated to symbolic letters of the taint type; otherwise the hexadecimal value is shown. In older kernels, the contents of the "license_gplok" field is displayed in hexadecimal; the field may be either a bitmask or a boolean, depending upon the kernel version. The relevant kernel sources should be consulted for the meaning of the letter(s) or hexadecimal bit value(s). For modules that have a "gpgsig_ok" field that is zero (unsigned), the notation "(U)" is shown. -r Passes the -readnow flag to the embedded gdb module, which will override the two-stage strategy that it uses for reading symbol tables from module object files. -R Reinitialize module data. All currently-loaded symbolic and debugging data will be deleted, and the installed module list will be updated (live system only). -g When used with -s or -S, add a module object's section start and end addresses to its symbol list. -o Load module symbols with old mechanism.
If the crash session was invoked with the "–mod " option, or
a CRASH_MODULE_PATH environment variable exists, then /lib/modules/
will be overridden as the default directory tree that is searched for module
object files.
After symbolic and debugging data have been loaded, backtraces and text
disassembly will be displayed appropriately. Depending upon the processor
architecture, data may also printed symbolically with the “p” command;
at a minimum, the “rd” command may be used with module data symbols.
If crash can recognize that the set of modules has changed while running a
session on a live kernel, the module data will be reinitialized the next
time this command is run; the -r option forces the reinitialization.
EXAMPLES
Display the currently-installed modules:
crash> mod
MODULE NAME SIZE OBJECT FILE
c8019000 soundcore 2788 (not loaded)
c801b000 soundlow 336 (not loaded)
c801d000 sound 59864 (not loaded)
c802d000 ad1848 15728 (not loaded)
c8032000 uart401 6000 (not loaded)
c8035000 cs4232 2472 (not loaded)
c8043000 opl3 11048 (not loaded)
c8047000 3c59x 18152 (not loaded)
c804d000 sunrpc 53796 (not loaded)
c805c000 lockd 31528 (not loaded)
c8065000 nfsd 151896 (not loaded)
c8092000 nfs 29752 (not loaded)
Display the currently-installed modules on a system where all modules were
compiled with CONFIG_KALLSYMS:
crash> mod
MODULE NAME SIZE OBJECT FILE
e080d000 jbd 57016 (not loaded) [CONFIG_KALLSYMS]
e081e000 ext3 92360 (not loaded) [CONFIG_KALLSYMS]
e0838000 usbcore 83168 (not loaded) [CONFIG_KALLSYMS]
e0850000 usb-uhci 27532 (not loaded) [CONFIG_KALLSYMS]
e085a000 ehci-hcd 20904 (not loaded) [CONFIG_KALLSYMS]
e0865000 input 6208 (not loaded) [CONFIG_KALLSYMS]
e086a000 hid 22404 (not loaded) [CONFIG_KALLSYMS]
e0873000 mousedev 5688 (not loaded) [CONFIG_KALLSYMS]
e0878000 keybdev 2976 (not loaded) [CONFIG_KALLSYMS]
e08fd000 cdrom 34144 (not loaded) [CONFIG_KALLSYMS]
e0909000 ide-cd 35776 (not loaded) [CONFIG_KALLSYMS]
e0915000 scsi_mod 117928 (not loaded) [CONFIG_KALLSYMS]
e0935000 ide-scsi 12752 (not loaded) [CONFIG_KALLSYMS]
e093c000 microcode 5248 (not loaded) [CONFIG_KALLSYMS]
e0943000 sr_mod 18136 (not loaded) [CONFIG_KALLSYMS]
e0956000 floppy 59056 (not loaded) [CONFIG_KALLSYMS]
e0966000 sg 38060 (not loaded) [CONFIG_KALLSYMS]
e0971000 ip_tables 16544 (not loaded) [CONFIG_KALLSYMS]
e097d000 iptable_filter 2412 (not loaded) [CONFIG_KALLSYMS]
e097f000 e1000 76096 (not loaded) [CONFIG_KALLSYMS]
e09ba000 autofs 13780 (not loaded) [CONFIG_KALLSYMS]
e09c1000 parport 39072 (not loaded) [CONFIG_KALLSYMS]
e09ce000 lp 9220 (not loaded) [CONFIG_KALLSYMS]
e09d4000 parport_pc 19204 (not loaded) [CONFIG_KALLSYMS]
e09e2000 agpgart 59128 (not loaded) [CONFIG_KALLSYMS]
e0a1a000 radeon 117156 (not loaded) [CONFIG_KALLSYMS]
e2dc7000 sunrpc 91996 (not loaded) [CONFIG_KALLSYMS]
e2de1000 lockd 60624 (not loaded) [CONFIG_KALLSYMS]
e2df3000 nfs 96880 (not loaded) [CONFIG_KALLSYMS]
Load the symbolic and debugging data of all modules:
crash> mod -S
MODULE NAME SIZE OBJECT FILE
c8019000 soundcore 2788 /lib/modules/2.2.5-15/misc/soundcore.o
c801b000 soundlow 336 /lib/modules/2.2.5-15/misc/soundlow.o
c801d000 sound 59864 /lib/modules/2.2.5-15/misc/sound.o
c802d000 ad1848 15728 /lib/modules/2.2.5-15/misc/ad1848.o
c8032000 uart401 6000 /lib/modules/2.2.5-15/misc/uart401.o
c8035000 cs4232 2472 /lib/modules/2.2.5-15/misc/cs4232.o
c8043000 opl3 11048 /lib/modules/2.2.5-15/misc/opl3.o
c8047000 3c59x 18152 /lib/modules/2.2.5-15/net/3c59x.o
c804d000 sunrpc 53796 /lib/modules/2.2.5-15/misc/sunrpc.o
c805c000 lockd 31528 /lib/modules/2.2.5-15/fs/lockd.o
c8065000 nfsd 151896 /lib/modules/2.2.5-15/fs/nfsd.o
c8092000 nfs 29752 /lib/modules/2.2.5-15/fs/nfs.o
Load the symbolic and debugging data of the soundcore module from its
known location:
crash> mod -s soundcore
MODULE NAME SIZE OBJECT FILE
c8019000 soundcore 2788 /lib/modules/2.2.5-15/misc/soundcore.o
Delete the current symbolic and debugging data of the soundcore module,
and then re-load it from a specified object file:
crash> mod -d soundcore
crash> mod -s soundcore /tmp/soundcore.o
MODULE NAME SIZE OBJECT FILE
c8019000 soundcore 2788 /tmp/soundcore.o
After installing a new kernel module on a live system, reinitialize the
installed module list:
crash> !insmod mdacon
crash> mod
mod: NOTE: modules have changed on this system -- reinitializing
MODULE NAME SIZE OBJECT FILE
c8019000 soundcore 2788 (not loaded)
c801b000 soundlow 336 (not loaded)
c801d000 sound 59864 (not loaded)
c802d000 ad1848 15728 (not loaded)
c8032000 uart401 6000 (not loaded)
c8035000 cs4232 2472 (not loaded)
c8043000 opl3 11048 (not loaded)
c8047000 3c59x 18152 (not loaded)
c804d000 sunrpc 53796 (not loaded)
c805c000 lockd 31528 (not loaded)
c8065000 nfs 29752 (not loaded)
c806e000 autofs 9316 (not loaded)
c8072000 nfsd 151896 (not loaded)
c80a1000 mdacon 3556 (not loaded)
Display modules that are “tainted”, where in this case
where they are proprietary and unsigned:
crash> mod -t
NAME TAINT
vxspec P(U)
vxportal P(U)
fdd P(U)
vxfs P(U)
vxdmp P(U)
vxio P(U)
vxglm P(U)
vxgms P(U)
vxodm P(U)
///
NAME
runq - run queue
SYNOPSIS
runq [-t] [-T] [-m] [-g] [-c cpu(s)]
DESCRIPTION
With no argument, this command displays the tasks on the run queues
of each cpu.
-t Display the timestamp information of each cpu's runqueue, which is the
rq.clock, rq.most_recent_timestamp or rq.timestamp_last_tick value,
whichever applies; following each cpu timestamp is the last_run or
timestamp value of the active task on that cpu, whichever applies,
along with the task identification.
-T Display the time lag of each CPU relative to the most recent runqueue
timestamp.
-m Display the amount of time that the active task on each cpu has been
running, expressed in a format consisting of days, hours, minutes,
seconds and milliseconds.
-g Display tasks hierarchically by task_group. The task_group line shows
the task_group address, the cfs_rq or rt_rq address, the task_group
name (if any), and whether the task_group is throttled.
-c cpu restrict the output to the run queue data of one or more CPUs,
which can be specified using the format "3", "1,8,9", "1-23",
or "1,8,9-14".
EXAMPLES
Display the tasks on an O(1) scheduler run queue:
crash> runq
CPU 0 RUNQUEUE: ffff880001cdb460
CURRENT: PID: 2739 TASK: ffff8800320fa7e0 COMMAND: "bash"
ACTIVE PRIO_ARRAY: ffff880001cdb4d8
[115] PID: 2739 TASK: ffff8800320fa7e0 COMMAND: "bash"
PID: 1776 TASK: ffff88003217d820 COMMAND: "syslogd"
EXPIRED PRIO_ARRAY: ffff880001cdbdb8
[no tasks queued]
CPU 1 RUNQUEUE: ffff880001ce3460
CURRENT: PID: 1779 TASK: ffff88003207a860 COMMAND: "klogd"
ACTIVE PRIO_ARRAY: ffff880001ce34d8
[115] PID: 1779 TASK: ffff88003207a860 COMMAND: "klogd"
EXPIRED PRIO_ARRAY: ffff880001ce3db8
[no tasks queued]
Display the tasks on a CFS run queue:
crash> runq
CPU 0 RUNQUEUE: ffff8800090436c0
CURRENT: PID: 588 TASK: ffff88007e4877a0 COMMAND: "udevd"
RT PRIO_ARRAY: ffff8800090437c8
[no tasks queued]
CFS RB_ROOT: ffff880009043740
[118] PID: 2110 TASK: ffff88007d470860 COMMAND: "check-cdrom.sh"
[118] PID: 2109 TASK: ffff88007f1247a0 COMMAND: "check-cdrom.sh"
[118] PID: 2114 TASK: ffff88007f20e080 COMMAND: "udevd"
CPU 1 RUNQUEUE: ffff88000905b6c0
CURRENT: PID: 2113 TASK: ffff88007e8ac140 COMMAND: "udevd"
RT PRIO_ARRAY: ffff88000905b7c8
[no tasks queued]
CFS RB_ROOT: ffff88000905b740
[118] PID: 2092 TASK: ffff88007d7a4760 COMMAND: "MAKEDEV"
[118] PID: 1983 TASK: ffff88007e59f140 COMMAND: "udevd"
[118] PID: 2064 TASK: ffff88007e40f7a0 COMMAND: "udevd"
[115] PID: 2111 TASK: ffff88007e4278a0 COMMAND: "kthreadd"
Display run queue timestamp data:
crash> runq -t
CPU 0: 2680990637359
2680986653330 PID: 28228 TASK: ffff880037ca2ac0 COMMAND: "loop"
CPU 1: 2680940618478
2680940618478 PID: 28167 TASK: ffff880078130040 COMMAND: "bash"
CPU 2: 2680990763425
2680986785772 PID: 28227 TASK: ffff8800787780c0 COMMAND: "loop"
CPU 3: 2680990954469
2680986059540 PID: 28226 TASK: ffff880078778b00 COMMAND: "loop"
Display the amount of time the active task on each cpu has been running:
crash> runq -m
CPU 0: [0 00:00:00.014] PID: 5275 TASK: f5dbcaa0 COMMAND: "sh"
CPU 1: [0 00:00:00.002] PID: 5203 TASK: f5c7baa0 COMMAND: "cat"
CPU 2: [0 00:00:00.014] PID: 7971 TASK: f5c6c550 COMMAND: "khelper"
CPU 3: [0 00:00:00.002] PID: 0 TASK: f4ccd000 COMMAND: "swapper"
Display tasks hierarchically by task_group:
crash> runq -g
CPU 0
CURRENT: PID: 14734 TASK: ffff88010626f500 COMMAND: "sh"
ROOT_TASK_GROUP: ffffffff81ed93e0 RT_RQ: ffff880028216808
[ 0] TASK_GROUP: ffff88022c6bbc00 RT_RQ: ffff880139fc9800 (THROTTLED)
[ 0] PID: 14750 TASK: ffff88013a4dd540 COMMAND: "rtloop99"
[ 1] PID: 14748 TASK: ffff88013bbca040 COMMAND: "rtloop98"
[ 1] TASK_GROUP: ffff88012b0fb400 RT_RQ: ffff880089029000
[ 1] PID: 14752 TASK: ffff880088abf500 COMMAND: "rtloop98"
[ 54] PID: 14749 TASK: ffff880037a4e080 COMMAND: "rtloop45"
[ 98] PID: 14746 TASK: ffff88012678c080 COMMAND: "rtloop1"
ROOT_TASK_GROUP: ffffffff81ed93e0 CFS_RQ: ffff8800282166e8
[120] PID: 14740 TASK: ffff88013b1e6080 COMMAND: "sh"
[120] PID: 14738 TASK: ffff88012678d540 COMMAND: "sh"
[120] PID: 14734 TASK: ffff88010626f500 COMMAND: "sh" [CURRENT]
TASK_GROUP: ffff884052bc9800 CFS_RQ: ffff8831e4a1b000 (THROTTLED)
[120] PID: 14732 TASK: ffff88013bbcb500 COMMAND: "sh"
[120] PID: 14728 TASK: ffff8800b3496080 COMMAND: "sh"
[120] PID: 14730 TASK: ffff880037833540 COMMAND: "sh"
TASK_GROUP: ffff884058f1d000 CFS_RQ: ffff88120a101600 (THROTTLED)
[120] PID: 14726 TASK: ffff880138d42aa0 COMMAND: "sh"
...
Display tasks hierarchically by task_group for cpu 3 only:
crash> runq -g -c3
CPU 3
CURRENT: PID: 2948 TASK: ffff88022af2a100 COMMAND: "bash"
INIT_TASK_GROUP: ffffffff81e1a780 RT_RQ: ffff880028216148
[no tasks queued]
INIT_TASK_GROUP: ffffffff81e1a780 CFS_RQ: ffff880028216028
[120] PID: 2948 TASK: ffff88022af2a100 COMMAND: "bash" [CURRENT]
TASK_GROUP: ffff88012b880800 CFS_RQ: ffff88012c5d1000
TASK_GROUP: ffff88012c078000 CFS_RQ: ffff88012c663e00
TASK_GROUP: ffff88022c7f4c00 CFS_RQ: ffff88012bb56000
TASK_GROUP: ffff88022b621400 CFS_RQ: ffff88012b012000
[120] PID: 3248 TASK: ffff88012a9d4100 COMMAND: "qemu-kvm"
///
NAME
tree - display radix tree, XArray or red-black tree
SYNOPSIS
tree [-t [radix|xarray|rbtree]] [-r offset] [-[s|S] struct[.member[,member]]]
-[x|d] [-o offset] [-l] [-p] [-N] start
DESCRIPTION
This command dumps the contents of a radix tree, an XAarray, or a red-black
tree. The arguments are as follows:
-t type The type of tree to dump; the type string can be one of
"radix", "rbtree", or "xarray", or alternatively, "ra",
"rb" or "x" are acceptable. If not specified, rbtree is the
default type.
-r offset If the "start" argument is the address of a data structure that
contains an radix_tree_root, xarray or rb_root structure, then this
is the offset to that structure member. If the offset is non-zero,
then this option is required. The offset may be entered in either
of two manners:
1. In "structure.member" format.
2. A number of bytes.
-o offset For red-black trees only, the offset of the rb_node within its
containing data structure; if the offset is non-zero, then this
option is required. The offset may be entered in either of two
manners:
1. In "structure.member" format.
2. A number of bytes.
This option is not applicable to radix trees.
-s struct For each entry in a tree, format and print it as this type of data
structure; use the "struct.member" format in order to display a
particular member of the structure. To display multiple members
of a structure, use a comma-separated list of members. If any
structure member contains an embedded structure or is an array, the
the output may be restricted to the embedded structure or an array
element by expressing the member argument as "struct.member.member"
or "struct.member[index]"; embedded member specifications may
extend beyond one level deep by expressing the struct argument as
"struct.member.member.member...".
-S struct Similar to -s, but instead of parsing gdb output, member values
are read directly from memory, so the command works much faster
for 1-, 2-, 4-, and 8-byte members.
-l For red-black trees, dump the tree sorted in linear order starting
with the leftmost node and progressing to the right. This option
does not apply to radix trees.
-p Display the node's position information, showing the relationship
between it and the root. For red-black trees, a position that
indicates "root/l/r" means that the node is the right child
of the left child of the root node. For radix trees and xarrays,
the index, the height, and the slot index values are shown with
respect to the root.
-x Override default output format with hexadecimal format.
-d Override default output format with decimal format.
The meaning of the “start” argument, which can be expressed either in
hexadecimal format or symbolically, depends upon whether the -N option
is prepended:
start The address of a radix_tree_root, xarray or rb_root structure, or
the address of a structure containing the radix_tree_root, xarray
or rb_root structure; if the latter, then the "-r offset" option
must be used if the member offset of the root structure is
non-zero.
-N start The address of a radix_tree_node, xa_node or rb_node structure,
bypassing the radix_tree_root, xarray, or rb_root that points
to it.
EXAMPLES
The vmap_area_root is a standalone rb_root structure. Display the
virtual addresses of each vmap_area in its red-black tree:
crash> whatis vmap_area_root
struct rb_root vmap_area_root;
crash> tree -t rbtree -o vmap_area.rb_node vmap_area_root
ffff880128c508c0
ffff88012cb68140
ffff88012c9afec0
ffff88012d65c440
...
Display the vmap_area’s va_start and va_end members of each of
the entries above expressing the vmap_area.rb_node offset as a
number of bytes:
crash> tree -t rbtree -o 24 vmap_area_root -s vmap_area.va_start,va_end
ffff880128c508c0
va_start = 0xffffc90014900000
va_end = 0xffffc90014921000
ffff88012cb68140
va_start = 0xffffc900110c0000
va_end = 0xffffc900110d1000
ffff88012c9afec0
va_start = 0xffffc90000640000
va_end = 0xffffc90000642000
ffff88012d65c440
va_start = 0xffffc90000620000
va_end = 0xffffc90000622000
...
Alternatively, use the -N option with the rb_node address contained
in the vmap_area_root structure:
crash> p vmap_area_root
vmap_area_root = $8 = {
rb_node = 0xffff880128c508d8
}
crash> tree -t rbtree -o vmap_area.rb_node -N 0xffff880128c508d8
ffff880128c508c0
ffff88012cb68140
ffff88012c9afec0
ffff88012d65c440
Display the virtual address of each vm_area_struct in the red-black
tree that has its root inside an mm_struct located at ffff880128b5a300.
The vm_area_struct.vm_rb rb_node member has an offset of 0x38 bytes:
crash> tree -t rbtree -r mm_struct.mm_rb ffff880128b5a300 -o 0x38
ffff88012a0de080
ffff880123e3ac78
ffff880123e3a700
ffff88012b2837c8
...
ffff880128c02ed0
ffff8801292e7958
ffff880123e3a318
ffff880123e3ad40
Add the -p option to the command above to show position information:
crash> tree -t rbtree -r mm_struct.mm_rb ffff880128b5a300 -o 0x38 -p
ffff88012a0de080
position: root
ffff880123e3ac78
position: root/l
ffff880123e3a700
position: root/l/l
ffff88012b2837c8
position: root/l/l/l
...
ffff880128c02ed0
position: root/r/r/l/r
ffff8801292e7958
position: root/r/r/l/r/r
ffff880123e3a318
position: root/r/r/r
ffff880123e3ad40
position: root/r/r/r/r
Given an mm_struct address of 0xffff880074b5be80, list the VMA tree in linear
order from the leftmost node progressing to the right using the -l option:
crash> tree -ls vm_area_struct.vm_start -o vm_area_struct.vm_rb \
-r mm_struct.mm_rb 0xffff880074b5be80 | paste - -
ffff88001f2c50e0 vm_start = 0x400000
ffff88001f2c5290 vm_start = 0xceb000
ffff880074bfc6c0 vm_start = 0xcec000
ffff88001f2c4bd0 vm_start = 0xd10000
ffff880074bfc948 vm_start = 0x1fe9000
ffff880036e54510 vm_start = 0x7ff6aa296000
ffff88001f2c5bd8 vm_start = 0x7ff6aa298000
ffff880036e54af8 vm_start = 0x7ff6aa497000
ffff880036e54f30 vm_start = 0x7ff6aa498000
ffff88000e06aa20 vm_start = 0x7ff6aa499000
ffff88000e06b368 vm_start = 0x7ff6ab95f000
...
ffff88001f2c5e60 vm_start = 0x7ff6bc1af000
ffff88001f2c4ca8 vm_start = 0x7ff6bc1b6000
ffff88001f2c5008 vm_start = 0x7ff6bc200000
ffff88001f2c5d88 vm_start = 0x7ff6bc205000
ffff880074bfd6c8 vm_start = 0x7ff6bc206000
ffff88001f2c4288 vm_start = 0x7ff6bc207000
ffff88001f2c4510 vm_start = 0x7ffc7a5fc000
ffff88001f2c5b00 vm_start = 0x7ffc7a6d1000
Compared to the top/down root/leaves order:
crash> tree -s vm_area_struct.vm_start -o vm_area_struct.vm_rb \
-r mm_struct.mm_rb 0xffff880074b5be80 | paste - -
ffff88001f2c5a28 vm_start = 0x7ff6bbbb9000
ffff88001f2c55f0 vm_start = 0x7ff6bb252000
ffff88000e06a360 vm_start = 0x7ff6ac6c3000
ffff88001f2c4bd0 vm_start = 0xd10000
ffff88001f2c5290 vm_start = 0xceb000
ffff88001f2c50e0 vm_start = 0x400000
ffff880074bfc6c0 vm_start = 0xcec000
ffff88000e06b368 vm_start = 0x7ff6ab95f000
ffff88001f2c5bd8 vm_start = 0x7ff6aa298000
ffff880074bfc948 vm_start = 0x1fe9000
ffff880036e54510 vm_start = 0x7ff6aa296000
ffff880036e54f30 vm_start = 0x7ff6aa498000
ffff880036e54af8 vm_start = 0x7ff6aa497000
ffff88000e06aa20 vm_start = 0x7ff6aa499000
ffff88000e06ae58 vm_start = 0x7ff6ac1df000
ffff88000e06ba28 vm_start = 0x7ff6abefc000
ffff88000e06a6c0 vm_start = 0x7ff6ac41b000
ffff88001f2c4000 vm_start = 0x7ff6bac75000
ffff88000e06bd88 vm_start = 0x7ff6b2d00000
ffff88000e06b440 vm_start = 0x7ff6b28de000
...
ffff880074bfd6c8 vm_start = 0x7ff6bc206000
ffff88001f2c4510 vm_start = 0x7ffc7a5fc000
ffff88001f2c5b00 vm_start = 0x7ffc7a6d1000
Display a list of the page structs in the radix tree of an address_space
structure located at ffff88012d364de0:
crash> tree -t radix -r address_space.page_tree ffff88012d364de0
ffffea00040d12c0
ffffea00040d9a60
ffffea00040d9b08
ffffea000407eda8
ffffea0004084288
...
ffffea000407bc70
ffffea00040baf48
ffffea0004043f48
ffffea000407de58
Add the -p option to the command above to show position information:
crash> tree -t radix -r address_space.page_tree ffff88012d364de0 -p
ffffea00040d12c0
index: 0 position: root/0/0
ffffea00040d9a60
index: 1 position: root/0/1
ffffea00040d9b08
index: 2 position: root/0/2
ffffea000407eda8
index: 3 position: root/0/3
ffffea0004084288
index: 4 position: root/0/4
...
ffffea000407bc70
index: 217 position: root/3/25
ffffea00040baf48
index: 218 position: root/3/26
ffffea0004043f48
index: 219 position: root/3/27
ffffea000407de58
index: 220 position: root/3/28
Alternatively, take the address of the radix_tree_node from the
radix_tree_root structure in the address_space structure above,
and display the tree with the -N option:
crash> struct address_space.page_tree ffff88012d364de0
page_tree = {
height = 0x2,
gfp_mask = 0x20,
rnode = 0xffff8801238add71
}
crash> tree -t radix -N 0xffff8801238add71
ffffea00040d12c0
ffffea00040d9a60
ffffea00040d9b08
ffffea000407eda8
ffffea0004084288
ffffea00040843a0
...
Using the same radix tree as above, display the flags and _count
members of each page struct in the list, and force the output format
to be hexadecimal:
crash> tree -t radix -N 0xffff8801238add71 -s page.flags,_count -x
ffffea00040d12c0
flags = 0x4000000002006c
_count = {
counter = 0x7
}
ffffea00040d9a60
flags = 0x4000000002006c
_count = {
counter = 0x7
}
ffffea00040d9b08
flags = 0x4000000002006c
_count = {
counter = 0x7
}
ffffea000407eda8
flags = 0x4000000002006c
_count = {
counter = 0x7
}
...
In more recent kernels, the XArray facility has replaced radix trees.
Display a list of the page structs in the XArray of an address_space
structure located at 0xffff94c235e76828, where the i_pages field is
an embedded xarray structure:
crash> tree -t xarray -r address_space.i_pages 0xffff94c235e76828
fffffcc005aa8380
fffffcc005cafa80
fffffcc005a79c80
fffffcc005ccad80
fffffcc005a72ec0
fffffcc005e27c00
fffffcc005ce3100
fffffcc005ff8dc0
fffffcc005c9a100
fffffcc005a49e40
fffffcc005c95a80
Add the -p option to the command above to show position information:
crash> tree -t xarray -r address_space.i_pages 0xffff94c235e76828 -p
fffffcc005aa8380
index: 90 position: root/1/26
fffffcc005cafa80
index: 91 position: root/1/27
fffffcc005a79c80
index: 92 position: root/1/28
fffffcc005ccad80
index: 93 position: root/1/29
fffffcc005a72ec0
index: 94 position: root/1/30
fffffcc005e27c00
index: 95 position: root/1/31
fffffcc005ce3100
index: 96 position: root/1/32
fffffcc005ff8dc0
index: 97 position: root/1/33
fffffcc005c9a100
index: 98 position: root/1/34
fffffcc005a49e40
index: 99 position: root/1/35
fffffcc005c95a80
index: 100 position: root/1/36
Alternatively, take the value found in the xa_head field from
the xarray structure, and display the tree with the -N option:
crash> address_space.i_pages 0xffff94c235e76828
i_pages = {
... [ xa_lock field not shown ] ...
xa_flags = 1,
xa_head = 0xffff94c23c1566ca
}
crash> tree -t x -N 0xffff94c23c1566ca
fffffcc005aa8380
fffffcc005cafa80
fffffcc005a79c80
fffffcc005ccad80
fffffcc005a72ec0
fffffcc005e27c00
fffffcc005ce3100
fffffcc005ff8dc0
fffffcc005c9a100
fffffcc005a49e40
fffffcc005c95a80
Using the same xarray command as above, display the flags and _refcount
members of each page struct in the list, and force the output format
to be hexadecimal:
crash> tree -t x -N 0xffff94c23c1566ca -s page.flags,_refcount -x
fffffcc005aa8380
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005cafa80
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005a79c80
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005ccad80
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005a72ec0
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005e27c00
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005ce3100
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005ff8dc0
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005c9a100
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005a49e40
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
fffffcc005c95a80
flags = 0x57ffffc0000014
_refcount = {
counter = 0x1
}
///
NAME
alias - command aliases
SYNOPSIS
alias [alias] [command string]
DESCRIPTION
This command creates an alias for a given command string. If no arguments
are entered, the current list of aliases are displayed. If one argument is
entered, the command string for that alias, if any, is displayed.
alias the single word to be used as an alias
command string the word(s) that will be substituted for the alias
Aliases may be created in four manners:
1. entering the alias in $HOME/.crashrc.
2. entering the alias in .crashrc in the current directory.
3. executing an input file containing the alias command.
4. during runtime with this command.
During initialization, $HOME/.crashrc is read first, followed by the
.crashrc file in the current directory. Aliases in the .crashrc file
in the current directory override those in $HOME/.crashrc. Aliases
entered with this command or by runtime input file override those
defined in either .crashrc file. Aliases may be deleted by entering an
empty string for the second argument. If redirection characters are to
be part of the command string, the command string must be enclosed by
quotation marks.
Note that there are a number of helpful built-in aliases – see the
first example below.
EXAMPLES
Display the currently-defined aliases, which in this example, only
consist of the built-in aliases:
crash> alias
ORIGIN ALIAS COMMAND
builtin man help
builtin ? help
builtin quit q
builtin sf set scroll off
builtin sn set scroll on
builtin hex set radix 16
builtin dec set radix 10
builtin g gdb
builtin px p -x
builtin pd p -d
builtin for foreach
builtin size *
builtin dmesg log
builtin lsmod mod
builtin last ps -l
Create a new alias to be added to the list:
crash> alias kp kmem -p
ORIGIN ALIAS COMMAND
runtime kp kmem -p
Create an alias with redirection characters:
crash> alias ksd "kmem -p | grep slab | grep DMA"
ORIGIN ALIAS COMMAND
runtime ksd kmem -p | grep slab | grep DMA
Remove an alias:
crash> alias kp ""
alias deleted: kp
///
NAME
files - open files
SYNOPSIS
files [-d dentry] | [-p inode] | [-c] [-R reference] [pid | taskp] …
DESCRIPTION
This command displays information about open files of a context.
It prints the context’s current root directory and current working
directory, and then for each open file descriptor it prints a pointer
to its file struct, a pointer to its dentry struct, a pointer to the
inode, the file type, and the pathname. If no arguments are entered,
the current context is used. The -R option, typically invoked from
“foreach files”, searches for references to a supplied number, address,
or filename argument, and prints only the essential information leading
up to and including the reference. The -d option is not context
specific, and only shows the data requested.
-d dentry given a hexadecimal dentry address, display its inode,
super block, file type, and full pathname.
-p inode given a hexadecimal inode address, dump all of its pages
that are in the page cache.
-c for each open file descriptor, prints a pointer to its
inode, a pointer to the inode's i_mapping address_space
structure, the number of pages of the inode that are in
the page cache, the file type, and the pathname.
-R reference search for references to this file descriptor number,
filename, dentry, inode, address_space, or file structure
address.
pid a process PID.
taskp a hexadecimal task_struct pointer.
EXAMPLES
Display the open files of the current context:
crash> files
PID: 720 TASK: c67f2000 CPU: 1 COMMAND: "innd"
ROOT: / CWD: /var/spool/news/articles
FD FILE DENTRY INODE TYPE PATH
0 c6b9c740 c7cc45a0 c7c939e0 CHR /dev/null
1 c6b9c800 c537bb20 c54d0000 REG /var/log/news/news
2 c6df9600 c537b420 c5c36360 REG /var/log/news/errlog
3 c74182c0 c6ede260 c6da3d40 PIPE
4 c6df9720 c696c620 c69398c0 SOCK
5 c6b9cc20 c68e7000 c6938d80 SOCK
6 c6b9c920 c7cc45a0 c7c939e0 CHR /dev/null
7 c6b9c680 c58fa5c0 c58a1200 REG /var/lib/news/history
8 c6df9f00 c6ede760 c6da3200 PIPE
9 c6b9c6e0 c58fa140 c5929560 REG /var/lib/news/history.dir
10 c7fa9320 c7fab160 c7fafd40 CHR /dev/console
11 c6b9c7a0 c58fa5c0 c58a1200 REG /var/lib/news/history
12 c377ec60 c58fa5c0 c58a1200 REG /var/lib/news/history
13 c4528aa0 c58fa6c0 c52fbb00 REG /var/lib/news/history.pag
14 c6df9420 c68e7700 c6938360 SOCK
15 c6df9360 c68e7780 c6938120 SOCK
16 c6b9c0e0 c68e7800 c6772000 SOCK
17 c6b9c200 c6b5f9c0 c6b5cea0 REG /var/lib/news/active
21 c6b9c080 c6ede760 c6da3200 PIPE
Display the files opened by the “crond” daemon, which is PID 462:
crash> files 462
PID: 462 TASK: f7220000 CPU: 2 COMMAND: "crond"
ROOT: / CWD: /var/spool
FD FILE DENTRY INODE TYPE PATH
0 f7534ae0 f7538de0 f7518dc0 CHR /dev/console
1 f7368f80 f72c7a40 f72f27e0 FIFO pipe:/[1456]
2 f74f3c80 f72c79c0 f72f2600 FIFO pipe:/[1457]
3 f7368b60 f72a5be0 f74300c0 REG /var/run/crond.pid
4 f7534360 f73408c0 f72c2840 REG /var/log/cron
7 f7368ce0 f72c7940 f72f2420 FIFO pipe:/[1458]
8 f7295de0 f72c7940 f72f2420 FIFO pipe:/[1458]
21 f74f36e0 f747cdc0 f747e840 CHR /dev/null
The -R option is typically invoked from “foreach files”. This example
shows all tasks that have “/dev/pts/4” open:
crash> foreach files -R pts/4
PID: 18633 TASK: c310a000 CPU: 0 COMMAND: "crash"
ROOT: / CWD: /home/CVS_pool/crash
FD FILE DENTRY INODE TYPE PATH
0 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
1 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
2 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
PID: 18664 TASK: c2392000 CPU: 1 COMMAND: "less"
ROOT: / CWD: /home/CVS_pool/crash
FD FILE DENTRY INODE TYPE PATH
1 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
2 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
PID: 23162 TASK: c5088000 CPU: 1 COMMAND: "bash"
ROOT: / CWD: /home/CVS_pool/crash
FD FILE DENTRY INODE TYPE PATH
0 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
1 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
2 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
255 c1412850 c2cb96d0 c2cad430 CHR /dev/pts/4
PID: 23159 TASK: c10fc000 CPU: 1 COMMAND: "xterm"
ROOT: / CWD: /homes/anderson/
FD FILE DENTRY INODE TYPE PATH
5 c1560da0 c2cb96d0 c2cad430 CHR /dev/pts/4
Display information about the dentry at address f745fd60:
crash> files -d f745fd60
DENTRY INODE SUPERBLK TYPE PATH
f745fd60 f7284640 f73a3e00 REG /var/spool/lpd/lpd.lock
For each open file, display the number of pages that are in the page cache:
crash> files -c 1954
PID: 1954 TASK: f7a28000 CPU: 1 COMMAND: "syslogd"
ROOT: / CWD: /
FD INODE I_MAPPING NRPAGES TYPE PATH
0 cb3ae868 cb3ae910 0 SOCK socket:/[4690]
2 f2721c5c f2721d04 461 REG /var/log/messages
3 cbda4884 cbda492c 47 REG /var/log/secure
4 e48092c0 e4809368 58 REG /var/log/maillog
5 f65192c0 f6519368 48 REG /var/log/cron
6 e4809e48 e4809ef0 0 REG /var/log/spooler
7 d9c43884 d9c4392c 0 REG /var/log/boot.log
For the inode at address f59b90fc, display all of its pages that are in
the page cache:
crash> files -p f59b90fc
INODE NRPAGES
f59b90fc 6
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ca3353e0 39a9f000 f59b91ac 0 2 82c referenced,uptodate,lru,private
ca22cb20 31659000 f59b91ac 1 2 82c referenced,uptodate,lru,private
ca220160 3100b000 f59b91ac 2 2 82c referenced,uptodate,lru,private
ca1ddde0 2eeef000 f59b91ac 3 2 82c referenced,uptodate,lru,private
ca36b300 3b598000 f59b91ac 4 2 82c referenced,uptodate,lru,private
ca202680 30134000 f59b91ac 5 2 82c referenced,uptodate,lru,private
///
NAME
mach - machine specific data
SYNOPSIS
mach [-m | -c -[xd] | -o]
DESCRIPTION
This command displays data specific to a machine type.
-m Display the physical memory map (x86, x86_64 and ia64 only).
-c Display each cpu's cpuinfo structure (x86, x86_64 and ia64 only).
Display each cpu's x8664_pda structure (x86_64 only),
Display the hwrpb_struct, and each cpu's percpu_struct (alpha only).
-x override default output format with hexadecimal format.
-d override default output format with decimal format.
-o Display the OPAL console log (ppc64 only).
EXAMPLES
crash> mach
MACHINE TYPE: i686
MEMORY SIZE: 512 MB
CPUS: 2
HYPERVISOR: KVM
PROCESSOR SPEED: 1993 Mhz
HZ: 100
PAGE SIZE: 4096
KERNEL VIRTUAL BASE: c0000000
KERNEL VMALLOC BASE: e0800000
KERNEL STACK SIZE: 8192
Display the system physical memory map:
crash> mach -m
PHYSICAL ADDRESS RANGE TYPE
0000000000000000 - 00000000000a0000 E820_RAM
00000000000f0000 - 0000000000100000 E820_RESERVED
0000000000100000 - 000000001ff75000 E820_RAM
000000001ff75000 - 000000001ff77000 E820_NVS
000000001ff77000 - 000000001ff98000 E820_ACPI
000000001ff98000 - 0000000020000000 E820_RESERVED
00000000fec00000 - 00000000fec90000 E820_RESERVED
00000000fee00000 - 00000000fee10000 E820_RESERVED
00000000ffb00000 - 0000000100000000 E820_RESERVED
Display the OPAL console log:
crash> mach -o
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...
[ 65.219065872,5] initial console log level: memory 7, driver 5
[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)
[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200
[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff
[ 65.219607955,5] FDT: Parsing fdt @0xff00000
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
///
NAME
repeat - repeat a command
SYNOPSIS
repeat [-seconds] command
DESCRIPTION
This command repeats a command indefinitely, optionally delaying a given
number of seconds between each command execution.
-seconds The number of seconds to delay between command executions.
This option must precede the command name to be executed.
Command execution may be stopped with CTRL-C, or if scrolling is in effect,
by entering “q”. This command is meant for use on a live system; it is
hard to conceive of a reason to use it when debugging a crash dump.
EXAMPLES
Display the value of jiffies once per second:
crash> repeat -1 p jiffies
jiffies = $1 = 155551079
jiffies = $2 = 155551180
jiffies = $3 = 155551281
jiffies = $4 = 155551382
jiffies = $5 = 155551483
jiffies = $6 = 155551584
jiffies = $7 = 155551685
jiffies = $8 = 155551786
jiffies = $9 = 155551887
jiffies = $10 = 155551988
jiffies = $11 = 155552089
jiffies = $12 = 155552190
jiffies = $13 = 155552291
jiffies = $14 = 155552392
jiffies = $15 = 155552493
jiffies = $16 = 155552594
jiffies = $17 = 155552695
jiffies = $18 = 155552796
...
///
NAME
timer - timer queue data
SYNOPSIS
timer [-r][-C cpu]
DESCRIPTION
This command displays the timer queue entries, both old- and new-style,
in chronological order. In the case of the old-style timers, the
timer_table array index is shown; in the case of the new-style timers,
the timer_list address is shown. On later kernels, the timer data is
per-cpu.
-r Display hrtimer timer queue entries, both old- and new-style, in
chronological order. In the case of the old-style hrtimers, the
expiration time is a single value; in the new-style hrtimers, the
expiration time is a range.
-C cpu Restrict the output to one or more CPUs, where multiple cpu[s] can
be specified, for example, as "1,3,5", "1-3", or "1,3,5-7,10".
EXAMPLES
Display the timer queue on an SMP system:
crash> timer
JIFFIES
4296291038
...
TIMER_BASES[1][BASE_STD]: ffff9801aba5aa00
EXPIRES TTE TIMER_LIST FUNCTION
4296282997 -8041 ffff9801aba55ce0 ffffffff83a3bda0
4296283104 -7934 ffff97fd84bd35e0 ffffffff83ac6b70
4296291061 23 ffffa6b283967de0 ffffffff83b29880
4296291112 74 ffff9800c9b62ad8 ffffffff83e6b550
4296291345 307 ffff980186d5ef88 ffffffff84146b80
4296291484 446 ffff9801a7c54740 ffffffff84147f50
4296291997 959 ffffffffc073f880 ffffffff83ac6b70
4296296213 5175 ffffa6b28339be18 ffffffff83b29880
4296304383 13345 ffff980194ca72a8 ffffffff8412e4e0
4296305724 14686 ffff980194ca6918 ffffffff8412e4e0
4296306036 14998 ffff980194ca6d58 ffffffff8412e4e0
4296306883 15845 ffff980194ca7e58 ffffffff8412e4e0
4296307588 16550 ffff9801aaa27e58 ffffffff8412e4e0
4296307625 16587 ffff980194ca6a28 ffffffff8412e4e0
4296313542 22504 ffff980194ca7c38 ffffffff8412e4e0
4296317680 26642 ffff9800c9149c58 ffffffff840da870
4296317744 26706 ffff9801a5354468 ffffffff83ac6b70
4296343322 52284 ffff980194ca63c8 ffffffff8412e4e0
4296343581 52543 ffff980194ca7088 ffffffff8412e4e0
4296343597 52559 ffff9801aaa274c8 ffffffff8412e4e0
4296714205 423167 ffffffff84caf3c0 ffffffff83ac6b70
TIMER_BASES[1][BASE_DEF]: ffff9801aba5bc80
EXPIRES TTE TIMER_LIST FUNCTION
4296291264 226 ffffffff855eb238 ffffffff83c08fb0
4296319997 28959 ffffffffc06ede40 ffffffff83ac6b70
4296506084 215046 ffff9801aba629c8 ffffffff83ac5ea0
...
Display a new-style hrtimer queue:
crash> timer -r
...
CPU: 2 HRTIMER_CPU_BASE: ffff9801aba9cf00
CLOCK: 0 HRTIMER_CLOCK_BASE: ffff9801aba9cf40 [ktime_get]
CURRENT
1623742000000
SOFTEXPIRES EXPIRES TTE HRTIMER FUNCTION
1623741000000 1623741000000 -1000000 ffff9801aba9d540 ffffffff83b3c8e0
1624024000000 1624024000000 282000000 ffff9801aba9d720 ffffffff83b7e7a0
1626000939806 1626010929804 2268929804 ffffa6b28399fa40 ffffffff83b2c1e0
1627576915615 1627576915615 3834915615 ffff9801a5727978 ffffffff83b365c0
1627637194488 1627647194487 3905194487 ffffa6b283977db0 ffffffff83b2c1e0
1629937423000 1629937423000 6195423000 ffff9801a9af2900 ffffffff83cf3d30
CLOCK: 1 HRTIMER_CLOCK_BASE: ffff9801aba9cf80 [ktime_get_real]
CURRENT
1558362388334558243
SOFTEXPIRES EXPIRES TTE HRTIMER FUNCTION
1558362389331238000 1558362389331288000 996729757 ffffa6b28574bcf0 ffffffff83b2c1e0
1558364372000000000 1558364372000000000 1983665441757 ffff9801a3513278 ffffffff83b365c0
CLOCK: 2 HRTIMER_CLOCK_BASE: ffff9801aba9cfc0 [ktime_get_boottime]
(empty)
...
///
NAME
bpf - extended Berkeley Packet Filter (eBPF)
SYNOPSIS
bpf [[-p ID | -P] [-tTj]] [[-m ID] | -M] [-s] [-xd]
DESCRIPTION
This command provides information on currently-loaded eBPF programs and maps.
With no arguments, basic information about each loaded eBPF program and map
is displayed. For each eBPF program, its ID number, the addresses of its
bpf_prog and bpf_prog_aux data structures, its type, tag, and the IDs of the
eBPF maps that it uses are displayed. For each eBPF map, its ID number, the
address of its bpf_map data structure, its type, and the hexadecimal value of
its map_flags are displayed.
-p ID displays the basic information specific to the program ID, plus the
size in bytes of its translated bytecode, the size in bytes of its
jited code, the number of bytes locked into memory, the time that
the program was loaded, whether it is GPL compatible, and its UID.
-P same as -p, but displays the basic and extra data for all programs.
-m ID displays the basic information specific to the map ID, plus the
size in bytes of its key and value, the maximum number of key-value
pairs that can be stored within the map, the number of bytes locked
into memory, its name string, and its UID.
-M same as -m, but displays the basic and extra data for all maps.
-t translate the bytecode of the specified program ID.
-T same as -t, but also dump the bytecode of each instruction.
-j disassemble the jited code of the specified program ID.
-s with -p or -P, dump the bpf_prog and bpf_prog_aux data structures.
with -m or -M, dump the bpf_map structure.
-x with -s, override default output format with hexadecimal format.
-d with -s, override default output format with decimal format.
EXAMPLES
Display all loaded eBPF programs and maps:
crash> bpf
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
13 ffffbc00c06d1000 ffff9ff260f0c400 CGROUP_SKB 7be49e3934a125ba 13,14
14 ffffbc00c0761000 ffff9ff260f0f600 CGROUP_SKB 2a142ef67aaad174 13,14
15 ffffbc00c001d000 ffff9ff2618f9e00 CGROUP_SKB 7be49e3934a125ba 15,16
16 ffffbc00c06c9000 ffff9ff2618f9400 CGROUP_SKB 2a142ef67aaad174 15,16
19 ffffbc00c0d39000 ffff9ff2610fa000 CGROUP_SKB 7be49e3934a125ba 19,20
20 ffffbc00c0d41000 ffff9ff2610f8e00 CGROUP_SKB 2a142ef67aaad174 19,20
30 ffffbc00c065f000 ffff9ff1b64de200 KPROBE 69fed6de18629d7a 32
31 ffffbc00c065b000 ffff9ff1b64df200 KPROBE 69fed6de18629d7a 37
32 ffffbc00c0733000 ffff9ff1b64dc600 KPROBE 69fed6de18629d7a 38
33 ffffbc00c0735000 ffff9ff1b64dca00 KPROBE 69fed6de18629d7a 39
34 ffffbc00c0737000 ffff9ff1b64dfc00 KPROBE 4abbddae72a6ee17 33,36,34
36 ffffbc00c0839000 ffff9ff1b64dd000 KPROBE da4fc6a3f41761a2 32
41 ffffbc00c07ec000 ffff9ff207b70400 TRACEPOINT e2094f9f46284bf6 55,54
44 ffffbc00c07ee000 ffff9ff1b64dc800 PERF_EVENT 19578a12836c4115 62
46 ffffbc00c07f0000 ffff9ff207b70400 SOCKET_FILTER 1fcfc04afd689133 64
ID BPF_MAP BPF_MAP_TYPE MAP_FLAGS
13 ffff9ff260f0ec00 LPM_TRIE 00000001
14 ffff9ff260f0de00 LPM_TRIE 00000001
15 ffff9ff2618fbe00 LPM_TRIE 00000001
16 ffff9ff2618fb800 LPM_TRIE 00000001
19 ffff9ff2610faa00 LPM_TRIE 00000001
20 ffff9ff2610fb800 LPM_TRIE 00000001
32 ffff9ff260d74000 HASH 00000000
33 ffff9ff260d76400 LRU_HASH 00000000
34 ffff9ff260d70000 LRU_HASH 00000002
35 ffff9ff260d73800 LRU_HASH 00000004
36 ffff9ff1b4f44000 ARRAY_OF_MAPS 00000000
37 ffff9ff260d77c00 PERCPU_HASH 00000000
38 ffff9ff260d70800 HASH 00000001
39 ffff9ff260d76c00 PERCPU_HASH 00000001
54 ffff9ff260dd2c00 HASH 00000000
55 ffff9ff260dd1400 HASH 00000000
62 ffff9ff1ae784000 HASH 00000000
64 ffff9ff1aea15000 ARRAY 00000000
Display additional data about program ID 20:
crash> bpf -p 20
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
20 ffffbc00c0d41000 ffff9ff2610f8e00 CGROUP_SKB 2a142ef67aaad174 19,20
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:21 2018
GPL_COMPATIBLE: yes UID: 0
Display additional data about map ID 34:
crash> bpf -m 34
ID BPF_MAP BPF_MAP_TYPE MAP_FLAGS
34 ffff9ff260d70000 LRU_HASH 00000000
KEY_SIZE: 4 VALUE_SIZE: 8 MAX_ENTRIES: 10000 MEMLOCK: 1953792
NAME: "lru_hash_map" UID: 0
Disassemble the jited program of program ID 20:
crash> bpf -p 20 -j
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
20 ffffbc00c0d41000 ffff9ff2610f8e00 CGROUP_SKB 2a142ef67aaad174 19,20
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:21 2018
GPL_COMPATIBLE: yes UID: 0
0xffffffffc06887a2: push %rbp
0xffffffffc06887a3: mov %rsp,%rbp
0xffffffffc06887a6: sub $0x40,%rsp
0xffffffffc06887ad: sub $0x28,%rbp
0xffffffffc06887b1: mov %rbx,0x0(%rbp)
0xffffffffc06887b5: mov %r13,0x8(%rbp)
0xffffffffc06887b9: mov %r14,0x10(%rbp)
0xffffffffc06887bd: mov %r15,0x18(%rbp)
0xffffffffc06887c1: xor %eax,%eax
0xffffffffc06887c3: mov %rax,0x20(%rbp)
0xffffffffc06887c7: mov %rdi,%rbx
0xffffffffc06887ca: movzwq 0xc0(%rbx),%r13
0xffffffffc06887d2: xor %r14d,%r14d
0xffffffffc06887d5: cmp $0x8,%r13
0xffffffffc06887d9: jne 0xffffffffc068881b
0xffffffffc06887db: mov %rbx,%rdi
0xffffffffc06887de: mov $0xc,%esi
0xffffffffc06887e3: mov %rbp,%rdx
0xffffffffc06887e6: add $0xfffffffffffffffc,%rdx
0xffffffffc06887ea: mov $0x4,%ecx
0xffffffffc06887ef: callq 0xffffffffb0865340
0xffffffffc06887f4: movabs $0xffff9ff2610faa00,%rdi
0xffffffffc06887fe: mov %rbp,%rsi
0xffffffffc0688801: add $0xfffffffffffffff8,%rsi
0xffffffffc0688805: movl $0x20,0x0(%rsi)
0xffffffffc068880c: callq 0xffffffffb01fcba0
0xffffffffc0688811: cmp $0x0,%rax
0xffffffffc0688815: je 0xffffffffc068881b
0xffffffffc0688817: or $0x2,%r14d
0xffffffffc068881b: cmp $0xdd86,%r13
0xffffffffc0688822: jne 0xffffffffc0688864
0xffffffffc0688824: mov %rbx,%rdi
0xffffffffc0688827: mov $0x8,%esi
0xffffffffc068882c: mov %rbp,%rdx
0xffffffffc068882f: add $0xfffffffffffffff0,%rdx
0xffffffffc0688833: mov $0x10,%ecx
0xffffffffc0688838: callq 0xffffffffb0865340
0xffffffffc068883d: movabs $0xffff9ff2610fb800,%rdi
0xffffffffc0688847: mov %rbp,%rsi
0xffffffffc068884a: add $0xffffffffffffffec,%rsi
0xffffffffc068884e: movl $0x80,0x0(%rsi)
0xffffffffc0688855: callq 0xffffffffb01fcba0
0xffffffffc068885a: cmp $0x0,%rax
0xffffffffc068885e: je 0xffffffffc0688864
0xffffffffc0688860: or $0x2,%r14d
0xffffffffc0688864: mov $0x1,%eax
0xffffffffc0688869: cmp $0x2,%r14
0xffffffffc068886d: jne 0xffffffffc0688871
0xffffffffc068886f: xor %eax,%eax
0xffffffffc0688871: mov 0x0(%rbp),%rbx
0xffffffffc0688875: mov 0x8(%rbp),%r13
0xffffffffc0688879: mov 0x10(%rbp),%r14
0xffffffffc068887d: mov 0x18(%rbp),%r15
0xffffffffc0688881: add $0x28,%rbp
0xffffffffc0688885: leaveq
0xffffffffc0688886: retq
Translate each bytecode instruction of program ID 13:
crash> bpf -p 13 -t
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
13 ffffbc00c06d1000 ffff9ff260f0c400 CGROUP_SKB 7be49e3934a125ba 13,14
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:11 2018
GPL_COMPATIBLE: yes UID: 0
0: (bf) r6 = r1
1: (69) r7 = *(u16 *)(r6 +192)
2: (b4) (u32) r8 = (u32) 0
3: (55) if r7 != 0x8 goto pc+14
4: (bf) r1 = r6
5: (b4) (u32) r2 = (u32) 16
6: (bf) r3 = r10
7: (07) r3 += -4
8: (b4) (u32) r4 = (u32) 4
9: (85) call bpf_skb_load_bytes#6793152
10: (18) r1 = map[id:13]
12: (bf) r2 = r10
13: (07) r2 += -8
14: (62) *(u32 *)(r2 +0) = 32
15: (85) call bpf_map_lookup_elem#73760
16: (15) if r0 == 0x0 goto pc+1
17: (44) (u32) r8 |= (u32) 2
18: (55) if r7 != 0xdd86 goto pc+14
19: (bf) r1 = r6
20: (b4) (u32) r2 = (u32) 24
21: (bf) r3 = r10
22: (07) r3 += -16
23: (b4) (u32) r4 = (u32) 16
24: (85) call bpf_skb_load_bytes#6793152
25: (18) r1 = map[id:14]
27: (bf) r2 = r10
28: (07) r2 += -20
29: (62) *(u32 *)(r2 +0) = 128
30: (85) call bpf_map_lookup_elem#73760
31: (15) if r0 == 0x0 goto pc+1
32: (44) (u32) r8 |= (u32) 2
33: (b7) r0 = 1
34: (55) if r8 != 0x2 goto pc+1
35: (b7) r0 = 0
36: (95) exit
Translate, and then dump each bytecode instruction of program ID 13:
crash> bpf -p 13 -T
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
13 ffffbc00c06d1000 ffff9ff260f0c400 CGROUP_SKB 7be49e3934a125ba 13,14
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:11 2018
GPL_COMPATIBLE: yes UID: 0
0: (bf) r6 = r1
bf 16 00 00 00 00 00 00
1: (69) r7 = *(u16 *)(r6 +192)
69 67 c0 00 00 00 00 00
2: (b4) (u32) r8 = (u32) 0
b4 08 00 00 00 00 00 00
3: (55) if r7 != 0x8 goto pc+14
55 07 0e 00 08 00 00 00
4: (bf) r1 = r6
bf 61 00 00 00 00 00 00
5: (b4) (u32) r2 = (u32) 16
b4 02 00 00 10 00 00 00
6: (bf) r3 = r10
bf a3 00 00 00 00 00 00
7: (07) r3 += -4
07 03 00 00 fc ff ff ff
8: (b4) (u32) r4 = (u32) 4
b4 04 00 00 04 00 00 00
9: (85) call bpf_skb_load_bytes#6793152
85 00 00 00 c0 a7 67 00
10: (18) r1 = map[id:13]
18 01 00 00 00 7a 96 61 00 00 00 00 b2 9d ff ff
12: (bf) r2 = r10
bf a2 00 00 00 00 00 00
13: (07) r2 += -8
07 02 00 00 f8 ff ff ff
14: (62) *(u32 *)(r2 +0) = 32
62 02 00 00 20 00 00 00
15: (85) call bpf_map_lookup_elem#73760
85 00 00 00 20 20 01 00
16: (15) if r0 == 0x0 goto pc+1
15 00 01 00 00 00 00 00
17: (44) (u32) r8 |= (u32) 2
44 08 00 00 02 00 00 00
18: (55) if r7 != 0xdd86 goto pc+14
55 07 0e 00 86 dd 00 00
19: (bf) r1 = r6
bf 61 00 00 00 00 00 00
20: (b4) (u32) r2 = (u32) 24
b4 02 00 00 18 00 00 00
21: (bf) r3 = r10
bf a3 00 00 00 00 00 00
22: (07) r3 += -16
07 03 00 00 f0 ff ff ff
23: (b4) (u32) r4 = (u32) 16
b4 04 00 00 10 00 00 00
24: (85) call bpf_skb_load_bytes#6793152
85 00 00 00 c0 a7 67 00
25: (18) r1 = map[id:14]
18 01 00 00 00 68 96 61 00 00 00 00 b2 9d ff ff
27: (bf) r2 = r10
bf a2 00 00 00 00 00 00
28: (07) r2 += -20
07 02 00 00 ec ff ff ff
29: (62) *(u32 *)(r2 +0) = 128
62 02 00 00 80 00 00 00
30: (85) call bpf_map_lookup_elem#73760
85 00 00 00 20 20 01 00
31: (15) if r0 == 0x0 goto pc+1
15 00 01 00 00 00 00 00
32: (44) (u32) r8 |= (u32) 2
44 08 00 00 02 00 00 00
33: (b7) r0 = 1
b7 00 00 00 01 00 00 00
34: (55) if r8 != 0x2 goto pc+1
55 08 01 00 02 00 00 00
35: (b7) r0 = 0
b7 00 00 00 00 00 00 00
36: (95) exit
95 00 00 00 00 00 00 00
Display the bpf_map data structure for map ID 13:
crash> bpf -m 13 -s
ID BPF_MAP BPF_MAP_TYPE MAP_FLAGS
13 ffff9ff260f0ec00 LPM_TRIE 00000001
KEY_SIZE: 8 VALUE_SIZE: 8 MAX_ENTRIES: 1 MEMLOCK: 4096
NAME: (unused) UID: 0
struct bpf_map {
ops = 0xffffffffb0e36720,
inner_map_meta = 0x0,
security = 0xffff9ff26873a158,
map_type = BPF_MAP_TYPE_LPM_TRIE,
key_size = 8,
value_size = 8,
max_entries = 1,
map_flags = 1,
pages = 1,
id = 13,
numa_node = -1,
unpriv_array = false,
user = 0xffffffffb14578a0,
refcnt = {
counter = 3
},
usercnt = {
counter = 1
},
work = {
data = {
counter = 0
},
entry = {
next = 0x0,
prev = 0x0
},
func = 0x0,
lockdep_map = {
key = 0x0,
class_cache = {0x0, 0x0},
name = 0x0,
cpu = 0,
ip = 0
}
},
name = "
}
Display the bpf_prog and bpf_prog_aux structures for program ID 13:
crash> bpf -p 13 -s
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
13 ffffbc00c06d1000 ffff9ff260f0c400 CGROUP_SKB 7be49e3934a125ba 13,14
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:10 2018
GPL_COMPATIBLE: yes UID: 0
struct bpf_prog {
pages = 1,
jited = 1,
jit_requested = 1,
locked = 1,
gpl_compatible = 1,
cb_access = 0,
dst_needed = 0,
blinded = 0,
is_func = 0,
kprobe_override = 0,
type = BPF_PROG_TYPE_CGROUP_SKB,
len = 37,
jited_len = 229,
tag = "{\344\236\071\064\241%\272",
aux = ffff9ff260f0c400,
orig_prog = 0x0,
bpf_func = 0xffffffffc0218a59,
{
insns = 0xffffb0cf406d1030,
insnsi = 0xffffb0cf406d1030
}
}
struct bpf_prog_aux {
refcnt = {
counter = 2
},
used_map_cnt = 2,
max_ctx_offset = 20,
stack_depth = 20,
id = 13,
func_cnt = 0,
offload_requested = false,
func = 0x0,
jit_data = 0x0,
ksym_tnode = {
node = {{
__rb_parent_color = 18446635988194065457,
rb_right = 0x0,
rb_left = 0x0
}, {
__rb_parent_color = 18446635988194065481,
rb_right = 0x0,
rb_left = 0x0
}}
},
ksym_lnode = {
next = 0xffff9db261966460,
prev = 0xffffffffb85d1150
},
ops = 0xffffffffb7f09060,
used_maps = 0xffff9db261e03600,
prog = 0xffffb0cf406d1000,
user = 0xffffffffb84578a0,
load_time = 23962237943,
name = "
security = 0xffff9db266f9cf50,
offload = 0x0,
{
work = {
data = {
counter = 0
},
entry = {
next = 0x0,
prev = 0x0
},
func = 0x0,
lockdep_map = {
key = 0x0,
class_cache = {0x0, 0x0},
name = 0x0,
cpu = 0,
ip = 0
}
},
rcu = {
next = 0x0,
func = 0x0
}
}
}
Display the extra data about all programs:
crash> bpf -P
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
13 ffffbc00c06d1000 ffff9ff260f0c400 CGROUP_SKB 7be49e3934a125ba 13,14
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:10 2018
GPL_COMPATIBLE: yes UID: 0
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
14 ffffbc00c0761000 ffff9ff260f0f600 CGROUP_SKB 2a142ef67aaad174 13,14
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:10 2018
GPL_COMPATIBLE: yes UID: 0
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
15 ffffbc00c001d000 ffff9ff2618f9e00 CGROUP_SKB 7be49e3934a125ba 15,16
XLATED: 296 JITED: 229 MEMLOCK: 4096
LOAD_TIME: Fri Apr 20 19:39:11 2018
GPL_COMPATIBLE: yes UID: 0
...
ID BPF_PROG BPF_PROG_AUX BPF_PROG_TYPE TAG USED_MAPS
75 ffffbc00c0ed1000 ffff9ff2429c6400 KPROBE da4fc6a3f41761a2 107
XLATED: 5168 JITED: 2828 MEMLOCK: 8192
LOAD_TIME: Fri Apr 27 14:54:40 2018
GPL_COMPATIBLE: yes UID: 0
Display the extra data for all maps:
crash> bpf -M
ID BPF_MAP BPF_MAP_TYPE MAP_FLAGS
13 ffff9ff260f0ec00 LPM_TRIE 00000001
KEY_SIZE: 8 VALUE_SIZE: 8 MAX_ENTRIES: 1 MEMLOCK: 4096
NAME: (unused) UID: 0
ID BPF_MAP BPF_MAP_TYPE MAP_FLAGS
14 ffff9ff260f0de00 LPM_TRIE 00000001
KEY_SIZE: 20 VALUE_SIZE: 8 MAX_ENTRIES: 1 MEMLOCK: 4096
NAME: (unused) UID: 0
...
ID BPF_MAP BPF_MAP_TYPE MAP_FLAGS
108 ffff9ff1aeab9400 LRU_HASH 00000000
KEY_SIZE: 4 VALUE_SIZE: 8 MAX_ENTRIES: 1000 MEMLOCK: 147456
NAME: "lru_hash_lookup" UID: 0
To display all possible information that this command offers about
all programs and maps, enter:
crash> bpf -PM -jTs
///
NAME
fuser - file users
SYNOPSIS
fuser [pathname | inode]
DESCRIPTION
This command displays the tasks using specified files or sockets.
Tasks will be listed that reference the file as the current working
directory, root directory, an open file descriptor, or that mmap the
file. If the file is held open in the kernel by the lockd server on
behalf of a client discretionary file lock, the client hostname is
listed.
pathname the full pathname of the file.
inode the hexadecimal inode address for the file.
EXAMPLES
Display the tasks using file /usr/lib/libkfm.so.2.0.0
crash> fuser /usr/lib/libkfm.so.2.0.0
PID TASK COMM USAGE
779 c5e82000 "kwm" mmap
808 c5a8e000 "krootwm" mmap
806 c5b42000 "kfm" mmap
809 c5dde000 "kpanel" mmap
///
NAME
mount - mounted filesystem data
SYNOPSIS
mount [-f][-i] [-n pid|task] [mount|vfsmount|superblock|dev|dir|dentry|inode]
DESCRIPTION
This command displays basic information about the currently-mounted
filesystems. The per-filesystem dirty inode list or list of open
files for the filesystem may also be displayed.
-f dump dentries and inodes for open files in each filesystem; only
supported on kernels prior to Linux 3.13.
-i dump all dirty inodes associated with each filesystem; only
supported on kernels prior to Linux 2.6.32.
For kernels supporting namespaces, the -n option may be used to
display the mounted filesystems with respect to the namespace of a
specified task:
-n pid a process PID.
-n task a hexadecimal task_struct pointer.
Specific filesystems may be selected using the following forms:
vfsmount hexadecimal address of a filesystem vfsmount structure.
mount hexadecimal address of a filesystem mount structure (Linux 3.3
and later).
superblock hexadecimal address of a filesystem super_block structure.
dev device name of a filesystem.
dir directory where a filesystem is mounted.
dentry hexadecimal address of an open dentry of a filesystem.
inode hexadecimal address of an open inode of a filesystem.
The first column of the command output displays the filesystem’s vfsmount
structure address for kernels prior to Linux 3.3. For Linux 3.3 and later
kernels, the first column displays the filesystem’s mount structure address,
which contains an embedded vfsmount structure.
EXAMPLES
Display mounted filesystem data:
crash> mount
VFSMOUNT SUPERBLK TYPE DEVNAME DIRNAME
c0089ea0 c0088a00 ext2 /dev/root /
c0089cf0 c0088c00 proc /proc /proc
c0089e10 c0088800 ext2 /dev/sda5 /boot
c0089d80 c0088600 ext2 /dev/sda6 /usr
c0089f30 c0088400 devpts none /dev/pts
c3f4b010 c0088200 ext2 /dev/sda1 /home
On Linux 3.3 and later kernels, the filesystem’s mount structure address
is shown:
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff880212fb8200 ffff880212fc0800 rootfs rootfs /
ffff88020ffbea00 ffff880212fc2000 proc proc proc
ffff880211db7f00 ffff88020e01a800 sysfs sysfs /sys
ffff88020ffe1300 ffff880212a40000 devtmpfs devtmpfs /dev
ffff88020ff15000 ffff880212bbc800 devpts devpts /dev/pts
ffff88020e542800 ffff88020e62b800 tmpfs tmpfs /dev/shm
...
Display the open files associated with each mounted filesystem:
crash> mount -f
VFSMOUNT SUPERBLK TYPE DEVNAME DIRNAME
c7fb2b80 c7fb3200 ext2 /dev/root /
OPEN FILES:
DENTRY INODE TYPE PATH
c6d02200 c6d0f7a0 REG usr/X11R6/lib/libX11.so.6.1
c6d02100 c6d0f9e0 REG usr/X11R6/lib/libXext.so.6.3
c6d02000 c6d0fc20 REG usr/X11R6/lib/libICE.so.6.3
c6d02680 c6d0f320 REG usr/X11R6/bin/xfs
c7106580 c70c5440 CHR dev/psaux
...
Display the dirty inodes associated with each mounted filesystem:
crash> mount -i
VFSMOUNT SUPERBLK TYPE DEVNAME DIRNAME
c0089ea0 c0088a00 ext2 /dev/root /
DIRTY INODES
c7ad4008
c2233438
c72c4008
c7d6b548
c3af1a98
c7d6b768
c3c4e228
...
Display the mounted filesystem containing inode c5000aa8:
crash> mount c5000aa8
VFSMOUNT SUPERBLK TYPE DEVNAME DIRNAME
c0089f30 c0088600 ext2 /dev/sda6 /usr
Display the mounted filesystem containing inode ffff8801f4245e40:
crash> mount ffff8801f4245e40
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff88020ffbea00 ffff880212fc2000 proc proc /proc
///
NAME
search - search memory
SYNOPSIS
search [-s start] [ -[kKV] | -u | -p | -t | -T ] [-e end | -l length] [-m mask]
[-x count] -[cwh] [value | (expression) | symbol | string] …
DESCRIPTION
This command searches for a given value within a range of user virtual, kernel
virtual, or physical memory space. If no end nor length value is entered,
then the search stops at the end of user virtual, kernel virtual, or physical
address space, whichever is appropriate.
An optional mask value may be entered to mask off “don’t care” bits.
-s start Start the search at this hexadecimal user or kernel virtual
address, physical address, or kernel symbol. The start address
must be appropriate for the memory type specified; if no memory
type is specified, the default is kernel virtual address space.
-k If no start address is specified, start the search at the base
of kernel virtual address space. This option is the default.
-K Same as -k, except that mapped kernel virtual memory that was
allocated by vmalloc(), module memory, or virtual mem_map regions
will not be searched.
-V Same as -k, except that unity-mapped kernel virtual memory and
mapped kernel-text/static-data (x86_64 and ia64) will not be
searched.
-u If no start address is specified, start the search at the base
of the current context's user virtual address space. If a start
address is specified, then this option specifies that the start
address is a user virtual address.
-p If no start address is specified, start the search at the base
of physical address space. If a start address is specified,
then this option specifies that the start address is a physical
address.
-t Search only the kernel stack pages of every task. If one or more
matches are found in a task's kernel stack, precede the output
with a task-identifying header.
-T Same as -t, except only the active task(s) are considered.
-e end Stop the search at this hexadecimal user or kernel virtual
address, kernel symbol, or physical address. The end address
must be appropriate for the memory type specified.
-l length Length in bytes of address range to search.
-m mask Ignore the bits that are set in the hexadecimal mask value.
-c Search for character string values instead of unsigned longs. If
the string contains any space(s), it must be encompassed by double
quotes.
-w Search for unsigned hexadecimal ints instead of unsigned longs.
This is only meaningful on 64-bit systems in order to search both
the upper and lower 32-bits of each 64-bit long for the value.
-h Search for unsigned hexadecimal shorts instead of unsigned longs.
-x count Display the memory contents before and after any found value. The
before and after memory context will consist of "count" memory
items of the same size as the "value" argument. This option is
not applicable with the -c option.
value Search for this hexadecimal long, unless modified by the -c, -w,
or -h options.
(expression) Search for the value of this expression; the expression value must
not overflow the designated size when -h or -w are used; not
applicable when used with the -c option.
symbol Search for this symbol value; the symbol value must not overflow
the designated size when -h or -w are used; not applicable when
used with the -c option.
string Search for character string values; if the string contains any
space(s), it must be encompassed by double quotes; only applicable
with the -c option.
If -k, -K, -V, -u, -p or -t are not used, then the search defaults to kernel
virtual address space. The starting address must be long-word aligned.
Address ranges that start in user space and end in kernel space are not
accepted.
EXAMPLES
Search the current context’s address space for all instances of 0xdeadbeef:
crash> search -u deadbeef
81aba5c: deadbeef
81abaa8: deadbeef
bfffc698: deadbeef
bffff390: deadbeef
Search all kernel memory above the kernel text space for all instances
of 0xabcd occurring in the lower 16-bits of each 32-bit word:
crash> search -s _etext -m ffff0000 abcd
c071481c: abcd
c0c2b0fc: 804abcd
c0cf5e74: 7489abcd
c17c0b44: c012abcd
c1dac730: 3dbeabcd
c226d0e8: ffffabcd
c23ed5dc: abcd
c3022544: 3dbeabcd
c3069b58: 3dbeabcd
c3e86e84: aabcd
c3e88ed0: aabcd
c3e8ee5c: aabcd
c3e9df50: aabcd
c3e9e930: aabcd
c440a778: 804abcd
c486eb44: 3dbeabcd
c578f0fc: 804abcd
c6394f90: 8ababcd
c65219f0: 3abcd
c661399c: abcd
c68514ac: 8abcd
c7e036bc: 3dbeabcd
c7e12568: 5abcd
c7e1256c: 5abcd
Search the 4K page at c532c000 for all instances of 0xffffffff:
crash> search -s c532c000 -l 4096 ffffffff
c532c33c: ffffffff
c532c3fc: ffffffff
Search the static kernel data area for all instances of c2d400eb:
crash> search -s _etext -e _edata c2d400eb
c022b550: c2d400eb
c022b590: c2d400eb
c022b670: c2d400eb
c022b6e0: c2d400eb
c022b7b0: c2d400eb
c022b7e0: c2d400eb
c022b8b0: c2d400eb
Search physical memory for all instances of 0xbabe occurring in the
upper 16 bits of each 32-bit word:
crash> search -p babe0000 -m ffff
2a1dc4: babe671e
2b6928: babe3de1
2f99ac: babe0d54
31843c: babe70b9
3ba920: babeb5d7
413ce4: babe7540
482747c: babe2600
48579a4: babe2600
4864a68: babe2600
...
Search physical memory for all instances of 0xbabe occurring in the
upper 16 bits of each 32-bit word on a 64-bit system:
crash> search -p babe0000 -m ffff -w
102e248: babe1174
11d2f90: babe813d
122d3ad70: babe6b27
124d8cd30: babe3dc8
124d8eefc: babef981
124d8f060: babe3dc8
124d8f17c: babefc81
...
Search kernel memory for all instances of 32-bit value 0xbabe1174
on a 64-bit system:
crash> search -k -w babe1174
ffff88000102e248: babe1174
ffffffff8102e248: babe1174
Search kernel memory for two strings:
crash> search -k -c "can't allocate memory" "Failure to"
ffff8800013ddec1: can't allocate memory for key lists..<3>%s %s: error con
ffff8801258be748: Failure to install fence: %d..<3>[drm:%s] *ERROR* Failed
ffff880125f07ec9: can't allocate memory..<3>ACPI: Invalid data..Too many d
ffffffff813ddec1: can't allocate memory for key lists..<3>%s %s: error con
Search the kernel stacks of all tasks for those that contain the inode
address ffff81002c0a3050:
crash> search -t ffff81002c0a3050
PID: 4876 TASK: ffff81003e9f5860 CPU: 7 COMMAND: "automount"
ffff8100288fbe98: ffff81002c0a3050
PID: 4880 TASK: ffff81003ce967a0 CPU: 0 COMMAND: "automount"
ffff81002c0fbdd8: ffff81002c0a3050
ffff81002c0fbe78: ffff81002c0a3050
When a kernel symbol or an (expression) is used an argument, both the
resultant value and the input string are displayed:
crash> search anon_inode_inode (__down_interruptible+191)
ffff81000222a728: ffffffff80493d60 (anon_inode_inode)
ffff810005a1e918: ffffffff800649d6 (__down_interruptible+191)
ffff810005a1e9d0: ffffffff800649d6 (__down_interruptible+191)
ffff810005a1eb48: ffffffff800649d6 (__down_interruptible+191)
ffff81000b409c60: ffffffff80493d60 (anon_inode_inode)
ffff81000c155b98: ffffffff80493d60 (anon_inode_inode)
ffff8100194fac70: ffffffff80493d60 (anon_inode_inode)
ffff81001daa1008: ffffffff80493d60 (anon_inode_inode)
ffff810028b95830: ffffffff800649d6 (__down_interruptible+191)
ffff81002cea0c70: ffffffff80493d60 (anon_inode_inode)
ffff810031327268: ffffffff80493d60 (anon_inode_inode)
ffff810031327270: ffffffff800649d6 (__down_interruptible+191)
ffff810034b1ccd0: ffffffff800649d6 (__down_interruptible+191)
ffff8100399565a8: ffffffff80493d60 (anon_inode_inode)
ffff81003a278cd0: ffffffff800649d6 (__down_interruptible+191)
ffff81003cc23e08: ffffffff800649d6 (__down_interruptible+191)
///
NAME
union - union contents
SYNOPSIS
union union_name[.member[,member]] [-o][-l offset][-rfuxdp]
[address | symbol][:cpuspec] [count | -c count]
DESCRIPTION
This command displays either a union definition, or a formatted display
of the contents of a union at a specified address. When no address is
specified, the union definition is shown along with the union size.
A union member may be appended to the structure name in order to limit
the scope of the data displayed to that particular member; when no address
is specified, the member’s offset (always 0) and definition are shown.
union_name name of a C-code union used by the kernel.
.member name of a union member; to display multiple members of a
union, use a comma-separated list of members. If any member
contains an embedded structure, or the member is an array, the
output may be restricted to just the embedded structure or an
array element by expressing the argument as "member.member"
or "member[index]"; embedded member specifications may extend
beyond one level deep, by expressing the member argument as
"member.member.member...".
-o show member offsets when displaying union definitions; the
offset is always 0 unless used with an address or symbol
argument, in which case each member will be preceded by its
virtual address.
-l offset if the address argument is a pointer to a list_head structure
that is embedded in the target union structure, the offset
to the list_head member may be entered in either of the
following manners:
1. in "structure.member" format.
2. a number of bytes.
-r raw dump of union data.
-f address argument is a dumpfile offset.
-x override default output format with hexadecimal format.
-d override default output format with decimal format.
-p if a union member is a pointer value, show the member's
data type on the output line; and on the subsequent line(s),
dereference the pointer, display the pointer target's symbol
value in brackets if appropriate, and if possible, display the
target data; requires an address argument.
-u address argument is a user virtual address in the current
context.
address hexadecimal address of a union; if the address points
to an embedded list_head structure contained within the
target union structure, then the "-l" option must be used.
symbol symbolic reference to the address of a union.
:cpuspec CPU specification for a per-cpu address or symbol:
: CPU of the currently selected task.
:a[ll] all CPUs.
:#[-#][,...] CPU list(s), e.g. "1,3,5", "1-3",
or "1,3,5-7,10".
count count of unions to dump from an array of unions; if used,
this must be the last argument entered.
-c count "-c" is only required if "count" is not the last argument
entered or if a negative number is entered; if a negative
value is entered, the (positive) "count" structures that
lead up to and include the target structure will be displayed.
Union data, sizes, and member offsets are shown in the current output radix
unless the -x or -d option is specified.
Please note that in the vast majority of cases, the “union” command
name may be dropped; if the union name does not conflict with any crash
or gdb command name, then the “union_name[.member]” argument will be
recognized as a union name, and this command automatically executed.
See the NOTE below.
EXAMPLES
Display the bdflush_param union definition, and then an instance of it:
crash> union bdflush_param
union bdflush_param {
struct {
int nfract;
int ndirty;
int nrefill;
int nref_dirt;
int dummy1;
int age_buffer;
int age_super;
int dummy2;
int dummy3;
} b_un;
unsigned int data[9];
}
SIZE: 36 (0x24)
crash> union bdflush_param bdf_prm
union bdflush_param {
b_un = {
nfract = 40,
ndirty = 500,
nrefill = 64,
nref_dirt = 256,
dummy1 = 15,
age_buffer = 3000,
age_super = 500,
dummy2 = 1884,
dummy3 = 2
},
data = {40, 500, 64, 256, 15, 3000, 500, 1884, 2}
}
NOTE
If the union name does not conflict with any crash command name, the
“union” command may be dropped. Accordingly, the examples above could
also have been accomplished like so:
crash> bdflush_param
crash> bdflush_param bdf_prm
Lastly, the short-cut “*” (pointer-to) command may also be used to negate
the need to enter the “union” command name (enter “help *” for details).