02_静态链接和简单宕机分享

ARM64寄存器

Arm64提供31个64bit通用寄存器 汇编用x表示64位宽 w32位宽

X0~X7: 用于传递子程序参数和结果，使用时不需要保存，多余参数采用堆栈传递，64位返回结果采用x0表示，128位 返回结果采用X1:X0
表示。
X24到x28 看得出来子函数调用的时候会进行保存 ,也就是说使用寄存器一定要保存寄存器在栈中
X29 为FP寄存器 通常叫帧寄存器 每一个栈是一个帧 帧栈寄存器
X30 LR寄存器为链接寄存器 返回地址有关
X31 堆栈指针寄存器
PC：程序计数器，俗称PC指针，总是指向即将要执行的下一条指令

程序运作时候的跳转和链接过程

用来理解背景知识 LR寄存器 和SP寄存器 和标准的栈帧长什么样的

先用一段小程序作为程序运行时候的栈帧变化图

int m=0;

int funa(int a, int b)
{
    int ret = 0 ;
    ret = a+b;
    return ret;

}

int funb(int c, int d)
{
    int ret = c+d ;
    ret = funa(c, ret);
    return ret;
}

int main(void)
{
    int i=1,j=2, r;
    m=6;
    r = funb(i,j);
    return r;
}

查看栈帧变化过程

1 程序加载如入内存后最开始的样子

2全局变量m赋值

程序加载到内存后，全局变量放在data段，已经有初始化的值。

单独看调用函数main

1.main函数自己开辟栈空间，同时保存caller的FP和LR
PC跳转到main函数运行时。为当前函数开辟栈空间。
由于main函数不是叶子函数，会修改x29(FP)和（LR）寄存器的值，需要将这两个寄存器的值保存到当前栈，以便返回时恢复。

2.更新栈帧寄存器FP

SP寄存器是当前函数栈指针，指向栈顶。
FP寄存器是当前函数栈帧指针，指向栈底。
对当前函数来说，FP=SP。FP指向当前函数的栈帧基地址。每个函数都要执行该动作。

3.mian函数的局部变量依次入栈保存

将当前函数局部变量，依次从栈底往栈顶顺序(高地址—>低地址)压栈保存。

4.函数内部更新全局变量的值

将子函数返回赋值给局部变量

子函数执行完成后返回，将返回结果保存在寄存器w0。
由于子函数返回结果赋值给局部变量r了，因此将寄存器w0的值保存在栈，即给局部变量r赋值。

main函数释放栈，并从栈上恢复FP和LR寄存器，并返回

当前函数自己分配栈，返回前自己做栈平衡。在释放栈前，将保存在栈上的caller的FP和LR恢复。

3有calller和callee的子函数funb栈变化

因为main函数里面执行了funb,接下来分析一下在
从main函数跳转到funb函数时，栈的变化情况。从上面4.1可知，当时main给funb传递两个参数w0和w1，并需要funcb返回一个值。
这时候跳转前

1.函数funb为自己分配栈空间，并保存LR和FP到栈顶

funb函数为自己分配栈空间，在其caller的底部，栈向下生长。
由于函数不是叶子函数，因此还要调用子函数，会修改FP和LR寄存器的值，因此需要将其caller（main函数）的执行现场FP和LR寄存器保存到栈顶。

2.函数funb更新栈帧FP寄存器

FP指向自己的栈帧，FP=SP。

3.funb函数保存寄存器传参的值

从caller传过来的参数保存在寄存器w0和w1, 接下来要用这两个寄存器。因此，先将传参保存到栈上。
先保存w0，后保存w1
先存在高地址，后存在低地址。压栈按照从高到低顺序(栈底——>栈顶)

funb函数中将运算结果赋值给局部变量

为局部变量ret分配空间，存放计算结果。
ret作为funb函数的第一个局部变量，放在栈底的位置。
注意这时PC刚好指向funb的入口

funb跳转到funa

从funb函数跳转到funa时，栈的情况如下：

1.funa函数为自己分配栈

由于函数funa()是叶子函数，没有callee。就不会修改FP和LR寄存器的值了，因此无需保存这两个寄存器了。

2.funa函数不需要更新FP栈帧寄存器，FP指向栈底

由于它是叶子函数，用SP已经可以表示栈帧了。FP在这里表示栈底，指向caller的栈帧。
这里一个关系
可通过判断FP ≠ SP来判断函数是叶子函数。

3.funa函数保存形参到栈

先保存w0，后保存w1
先存在高地址，后存在低地址。压栈按照从高到低顺序(栈底——>栈顶)

4.funa函数为局部变量分配空间

为局部变量ret 分配空间，并将运算结果赋值给ret。

5.funa执行完成,函数准备返回值到w0,并处理栈平衡

先将返回值放在wo寄存器，再释放自己的栈。

总结1:最后典型的栈结构

每个函数（假设同时具有caller和callee）的栈结构格式为：

栈顶保存的是自己的FP(栈底)
栈顶+8 处保存的是LR寄存器的值，也就是自己return后要从哪里开始执行。也叫该函数返回的下一条指令的地址
然后保存的是局部变量的值。
然后，保存的是从n,…,3,2,1 ，从低往高，的传参的值。
到上一个函数的FP为止，函数的栈结束。
这就是一个完整的栈帧。

每个函数（假设同时具有caller和callee）进入后典型的栈操作：

首先会将caller的FP(栈帧地址）保存到栈的顶部(SP+0)
然后，然后将LR寄存器(返回地址)保存在自己的栈(SP+8).
函数总会执行FP=SP操作。因此，对arm64来说，当前函数的FP=SP。

每个函数（假设同时具有caller和callee）返回前典型的栈操作：

将当前栈中保存的LR赋值给LR寄存器，然后ret。

根据当前函数的FP，推断函数调用栈

核心点在于：FP寄存器，又叫栈帧寄存器。
所有函数调用栈都会组成一个链表。
每个栈有两个地址来构成这个链表。两个64位宽的地址。
低地址(栈顶)存放了指向上一个栈帧的基地址FP，类似链表的prev指针。
高地址存放了LR寄存器，当前函数的返回地址。

实战操作

源码下载,同时虚拟机需要网盘里的,使用的qemu内核需要这个git里编译出来的 不然无法出触发crash

环境配置

环境配置开始:
使用网盘的虚拟机  在git中下载源码进行解压
源码在/home/rlk/rlk/linux-5.0-kdump-master/kmodules
编译内核  ./run_debian_arm64.sh build_kernel

能死机的代码放在 kmodule里面
Makefile要记得更改 不知道为啥指定的内核源码路径不对,需要指定编译qemu目录

编译下面代码触发系统宕机

#include 
#include 
#include 
#include 
#include 

struct mydev_priv {
	char name[64];
	int i;
};

int create_oops(struct vm_area_struct *vma, struct mydev_priv *priv)
{
	unsigned long flags;

	flags = vma->vm_flags;
	printk("flags=0x%lx, name=%s\n", flags, priv->name);
	
	return 0;
}

int __init my_oops_init(void)
{
	int ret;
	struct vm_area_struct *vma = NULL;
	struct mydev_priv priv;

	vma = kmalloc(sizeof (*vma), GFP_KERNEL);
	if (!vma)
		return -ENOMEM;

	kfree(vma);
	vma = NULL;

	smp_mb();

	memcpy(priv.name, "figo", sizeof("figo"));
	priv.i = 10;

	ret = create_oops(vma, &priv);

	return 0;
}

void __exit my_oops_exit(void)
{
	printk("goodbye\n");
}

module_init(my_oops_init);
module_exit(my_oops_exit);
MODULE_LICENSE("GPL");

宕机后内核信息
可以看见执行了内核崩溃后 马上使用备份内核来记录当前信息 记录完后开始重启

root@benshushu:/mnt/01_oops# insmod ./oops.
oops.ko     oops.mod.o  oops.o      
root@benshushu:/mnt/01_oops# insmod ./oops.ko 
[ 6816.922583] oops: loading out-of-tree module taints kernel.
[ 6816.938146] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000050
[ 6816.939936] Mem abort info:
[ 6816.940122]   ESR = 0x96000004
[ 6816.940336]   Exception class = DABT (current EL), IL = 32 bits
[ 6816.940664]   SET = 0, FnV = 0
[ 6816.940865]   EA = 0, S1PTW = 0
[ 6816.941026] Data abort info:
[ 6816.941425]   ISV = 0, ISS = 0x00000004
[ 6816.941621]   CM = 0, WnR = 0
[ 6816.942191] user pgtable: 4k pages, 48-bit VAs, pgdp = 000000001d3b724b
[ 6816.942626] [0000000000000050] pgd=0000000000000000
[ 6816.943324] Internal error: Oops: 96000004 [#1] SMP
[ 6816.944043] Modules linked in: oops(OE+) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) gf128mul(E) aes_arm64(E) sha2_ce(E) evdev(E) sha256_arm64(E) sha1_ce(E) cfg80211(E) rfkill(E) 8021q(E) garp(E) mrp(E) virtio_net(E) stp(E) llc(E) gpio_keys(E) net_failover(E) failover(E) 9p(E) fscache(E) ip_tables(E) x_tables(E) autofs4(E)
[ 6816.946956] CPU: 1 PID: 2346 Comm: insmod Kdump: loaded Tainted: G           OE     5.0.0-rlk #2
[ 6816.947761] Hardware name: linux,dummy-virt (DT)
[ 6816.948208] pstate: 80000005 (Nzcv daif -PAN -UAO)
[ 6816.949505] pc : create_oops+0x20/0x4c [oops]
[ 6816.950007] lr : my_oops_init+0xa0/0x1000 [oops]
[ 6816.950524] sp : ffff00001352bb20
[ 6816.951172] x29: ffff00001352bb20 x28: ffff000008cf81d0 
[ 6816.951700] x27: ffff000008cf8180 x26: ffff00001352bdc0 
[ 6816.952185] x25: ffff000008cf8198 x24: ffff000008cf8008 
[ 6816.952645] x23: 0000000000000000 x22: ffff800023600e80 
[ 6816.952886] x21: ffff000008cf8018 x20: 0000000000000000 
[ 6816.953113] x19: ffff000008bbe000 x18: 0000000000000000 
[ 6816.953483] x17: 0000000000000000 x16: 0000000000000000 
[ 6816.954128] x15: ffff800023601370 x14: ffffffffffffffff 
[ 6816.954868] x13: 0000000000000040 x12: 0000000000000228 
[ 6816.955479] x11: 0000000000000000 x10: 0000000000000000 
[ 6816.956565] x9 : 0000000000000000 x8 : 00000000000007a7 
[ 6816.957786] x7 : ffff80002a803b00 x6 : ffff00001352bb89 
[ 6816.958644] x5 : ffff800029319e00 x4 : ffff80002fdaeb00 
[ 6816.959826] x3 : 0000000000000000 x2 : ffff000008bbe0a0 
[ 6816.960361] x1 : ffff00001352bb84 x0 : 0000000000000000 
[ 6816.960862] Process insmod (pid: 2346, stack limit = 0x000000006111c3da)
[ 6816.961319] Call trace:
[ 6816.961614]  create_oops+0x20/0x4c [oops]
[ 6816.961978]  my_oops_init+0xa0/0x1000 [oops]
[ 6816.962775]  do_one_initcall+0x50/0x1d8
[ 6816.963198]  do_init_module+0x60/0x1e8
[ 6816.963729]  load_module+0x1bd4/0x1ea8
[ 6816.964134]  __se_sys_finit_module+0x9c/0xf8
[ 6816.964611]  __arm64_sys_finit_module+0x24/0x30
[ 6816.964899]  el0_svc_common+0x78/0x120
[ 6816.965216]  el0_svc_handler+0x38/0x78
[ 6816.965656]  el0_svc+0x8/0xc
[ 6816.966127] Code: f9000be1 aa0203e0 d503201f f9400fe0 (f9402800) 
[ 6816.968053] SMP: stopping secondary CPUs
[ 6816.970667] Starting crashdump kernel...
[ 6816.971029] Bye!
[    0.000000] Booting Linux on physical CPU 0x0000000001 [0x411fd070]
[    0.000000] Linux version 5.0.0-rlk (rlk@rlk) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu1)) #2 SMP Sat Sep 30 12:37:16 CST 2023
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] Reserving 1KB of memory at 0x7fdff000 for elfcorehdr
[    0.000000] cma: Reserved 64 MiB at 0x000000007bc00000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x000000006fe00000-0x000000007fdfffff]
[    0.000000] NUMA: NODE_DATA [mem 0x7fd70840-0x7fd71fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x000000006fe00000-0x000000007fdfffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x000000006fe00000-0x000000007fdfffff]
[    0.000000] Initmem setup node 0 [mem 0x000000006fe00000-0x000000007fdfffff]
[    0.000000] On node 0 totalpages: 65536
[    0.000000]   DMA32 zone: 1024 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 65536 pages, LIFO batch:15
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] Number of cores (4) exceeds configured maximum of 1 - clipping
[    0.000000] random: get_random_bytes called from start_kernel+0xa8/0x4bc with crng_init=0
[    0.000000] percpu: Embedded 25 pages/cpu @(____ptrval____) s64152 r8192 d30056 u102400
[    0.000000] pcpu-alloc: s64152 r8192 d30056 u102400 alloc=25*4096
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: ARM erratum 832075
[    0.000000] CPU features: detected: ARM erratum 834220
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64512
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: noinintrd root=/dev/vda rootfstype=ext4 rw loglevel=8 nr_cpus=1 systemd.unit=kdump-tools.service
[    0.000000] Memory: 149572K/262144K available (10876K kernel code, 1804K rwdata, 4316K rodata, 5312K init, 572K bss, 47036K reserved, 65536K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] ftrace: allocating 37102 entries in 145 pages
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv2m: range[mem 0x08020000-0x08020fff], SPI[80:143]
[    0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
[    0.000086] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
[    0.006431] Console: colour dummy device 80x25
[    0.007578] printk: console [tty0] enabled
[    0.009233] Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=250000)
[    0.009368] pid_max: default: 32768 minimum: 301
[    0.010701] LSM: Security Framework initializing
[    0.010849] Yama: becoming mindful.
[    0.013670] AppArmor: AppArmor initialized
[    0.014233] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.014436] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.014661] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
[    0.014707] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
[    0.047388] ASID allocator initialised with 32768 entries
[    0.048221] rcu: Hierarchical SRCU implementation.
[    0.054431] EFI services will not be available.
[    0.056548] smp: Bringing up secondary CPUs ...
[    0.056615] smp: Brought up 1 node, 1 CPU
[    0.056645] SMP: Total of 1 processors activated.
[    0.056716] CPU features: detected: 32-bit EL0 Support
[    0.056796] CPU features: detected: CRC32 instructions
[    0.062390] CPU: All CPU(s) started at EL1
[    0.062594] alternatives: patching kernel code
[    0.079605] devtmpfs: initialized
[    0.089402] Registered cp15_barrier emulation handler
[    0.089494] Registered setend emulation handler
[    0.090986] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.091195] futex hash table entries: 256 (order: 2, 16384 bytes)
[    0.097162] xor: measuring software checksum speed
[    0.144408]    8regs     :  2661.000 MB/sec
[    0.194198]    32regs    :  2655.000 MB/sec
[    0.244098]    arm64_neon:  2390.000 MB/sec
[    0.244191] xor: using function: 8regs (2661.000 MB/sec)
[    0.244442] pinctrl core: initialized pinctrl subsystem
[    0.261449] DMI not present or invalid.
[    0.265698] NET: Registered protocol family 16
[    0.268161] audit: initializing netlink subsys (disabled)
[    0.272393] cpuidle: using governor ladder
[    0.272511] cpuidle: using governor menu
[    0.273325] vdso: 2 pages (1 code @ (____ptrval____), 1 data @ (____ptrval____))
[    0.273433] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.274230] audit: type=2000 audit(0.216:1): state=initialized audit_enabled=0 res=1
[    0.279738] DMA: preallocated 256 KiB pool for atomic allocations
[    0.281309] Serial: AMBA PL011 UART driver
[    0.300876] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 39, base_baud = 0) is a PL011 rev1
[    0.328947] printk: console [ttyAMA0] enabled
[    0.355404] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.355736] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.356027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.356338] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.442378] raid6: neonx8   gen()  2807 MB/s
[    0.525096] raid6: neonx8   xor()  1184 MB/s
[    0.606834] raid6: neonx4   gen()  2954 MB/s
[    0.689711] raid6: neonx4   xor()  1186 MB/s
[    0.774389] raid6: neonx2   gen()  2425 MB/s
[    0.858148] raid6: neonx2   xor()  1027 MB/s
[    0.941982] raid6: neonx1   gen()  1502 MB/s
[    1.027063] raid6: neonx1   xor()   812 MB/s
[    1.111003] raid6: int64x8  gen()  1950 MB/s
[    1.195022] raid6: int64x8  xor()  1317 MB/s
[    1.278308] raid6: int64x4  gen()  2420 MB/s
[    1.362703] raid6: int64x4  xor()  1049 MB/s
[    1.448529] raid6: int64x2  gen()  1881 MB/s
[    1.532336] raid6: int64x2  xor()  1286 MB/s
[    1.617522] raid6: int64x1  gen()  1358 MB/s
[    1.702134] raid6: int64x1  xor()   920 MB/s
[    1.702274] raid6: using algorithm neonx4 gen() 2954 MB/s
[    1.702519] raid6: .... xor() 1186 MB/s, rmw enabled
[    1.702689] raid6: using neon recovery algorithm
[    1.704940] ACPI: Interpreter disabled.
[    1.706598] vgaarb: loaded
[    1.708377] SCSI subsystem initialized
[    1.709298] libata version 3.00 loaded.
[    1.709704] pps_core: LinuxPPS API ver. 1 registered
[    1.709800] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    1.710180] PTP clock support registered
[    1.710484] EDAC MC: Ver: 3.0.0
[    1.723926] clocksource: Switched to clocksource arch_sys_counter
[    1.865135] VFS: Disk quotas dquot_6.6.0
[    1.865439] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.871416] AppArmor: AppArmor Filesystem Enabled
[    1.872132] pnp: PnP ACPI: disabled
[    1.894791] NET: Registered protocol family 2
[    1.901370] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes)
[    1.901680] TCP established hash table entries: 2048 (order: 2, 16384 bytes)
[    1.902021] TCP bind hash table entries: 2048 (order: 3, 32768 bytes)
[    1.902328] TCP: Hash tables configured (established 2048 bind 2048)
[    1.903691] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    1.904382] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    1.906200] NET: Registered protocol family 1
[    1.906684] PCI: CLS 0 bytes, default 64
[    1.911244] Unpacking initramfs...
[    2.307736] Freeing initrd memory: 7012K
[    2.310437] hw perfevents: enabled with armv8_pmuv3 PMU driver, 5 counters available
[    2.311177] kvm [1]: HYP mode not available
[    2.315296] Initialise system trusted keyrings
[    2.316953] workingset: timestamp_bits=44 max_order=16 bucket_order=0
[    2.329396] zbud: loaded
[    3.957379] Key type asymmetric registered
[    3.957577] Asymmetric key parser 'x509' registered
[    3.957930] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    3.958713] io scheduler mq-deadline registered
[    3.968748] pl061_gpio 9030000.pl061: PL061 GPIO chip @0x0000000009030000 registered
[    3.970605] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    3.971539] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:
[    3.972477] pci-host-generic 4010000000.pcie:    IO 0x3eff0000..0x3effffff -> 0x00000000
[    3.973163] pci-host-generic 4010000000.pcie:   MEM 0x10000000..0x3efeffff -> 0x10000000
[    3.973335] pci-host-generic 4010000000.pcie:   MEM 0x8000000000..0xffffffffff -> 0x8000000000
[    3.974265] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]
[    3.975515] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00
[    3.975925] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.976170] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.976779] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]
[    3.976932] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[    3.978257] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[    3.982482] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000
[    3.985671] pci 0000:00:01.0: reg 0x10: [io  0x1040-0x105f]
[    3.986627] pci 0000:00:01.0: reg 0x14: [mem 0x10040000-0x10040fff]
[    3.989618] pci 0000:00:01.0: reg 0x20: [mem 0x8000000000-0x8000003fff 64bit pref]
[    3.990594] pci 0000:00:01.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    3.991448] pci 0000:00:02.0: [1af4:1009] type 00 class 0x000200
[    3.992576] pci 0000:00:02.0: reg 0x10: [io  0x1000-0x103f]
[    3.993777] pci 0000:00:02.0: reg 0x14: [mem 0x10041000-0x10041fff]
[    3.998034] pci 0000:00:02.0: reg 0x20: [mem 0x8000004000-0x8000007fff 64bit pref]
[    4.002673] pci 0000:00:01.0: BAR 6: assigned [mem 0x10000000-0x1003ffff pref]
[    4.003128] pci 0000:00:01.0: BAR 4: assigned [mem 0x8000000000-0x8000003fff 64bit pref]
[    4.004084] pci 0000:00:02.0: BAR 4: assigned [mem 0x8000004000-0x8000007fff 64bit pref]
[    4.005281] pci 0000:00:01.0: BAR 1: assigned [mem 0x10040000-0x10040fff]
[    4.005532] pci 0000:00:02.0: BAR 1: assigned [mem 0x10041000-0x10041fff]
[    4.005714] pci 0000:00:02.0: BAR 0: assigned [io  0x1000-0x103f]
[    4.005872] pci 0000:00:01.0: BAR 0: assigned [io  0x1040-0x105f]
[    4.027220] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    4.030415] Serial: AMBA driver
[    4.031114] msm_serial: driver initialized
[    4.032959] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    4.045019] virtio_blk virtio0: [vda] 16777216 512-byte logical blocks (8.59 GB/8.00 GiB)
[    4.067351] mousedev: PS/2 mouse device common for all mice
[    4.071453] rtc-pl031 9010000.pl031: registered as rtc0
[    4.074618] ledtrig-cpu: registered to indicate activity on CPUs
[    4.079818] NET: Registered protocol family 10
[    4.197561] Segment Routing with IPv6
[    4.198098] mip6: Mobile IPv6
[    4.198411] NET: Registered protocol family 17
[    4.199226] 9pnet: Installing 9P2000 support
[    4.204038] mpls_gso: MPLS GSO support
[    4.205932] registered taskstats version 1
[    4.206083] Loading compiled-in X.509 certificates
[    4.206997] zswap: loaded using pool lzo/zbud
[    4.212171] Btrfs loaded, crc32c=crc32c-generic
[    4.214696] AppArmor: AppArmor sha1 policy hashing enabled
[    4.217749] rtc-pl031 9010000.pl031: setting system clock to 2023-09-30T08:03:52 UTC (1696061032)
[    4.221638] uart-pl011 9000000.pl011: no DMA platform data
[    4.523117] Freeing unused kernel memory: 5312K
[    4.524475] Run /init as init process
Loading, please wait...
Starting version 241
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
done.
Warning: fsck not present, so skipping root file system
[    7.541410] random: fast init done
[    7.735652] EXT4-fs (vda): recovery complete
[    7.748160] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.
[    8.388934] autofs4: module verification failed: signature and/or required key missing - tainting kernel
[    8.406350] systemd[1]: Inserted module 'autofs4'
[    8.497339] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[    8.499822] systemd[1]: Detected virtualization qemu.
[    8.500212] systemd[1]: Detected architecture arm64.

Welcome to Debian GNU/Linux buster/sid!

[    8.529838] systemd[1]: Set hostname to <benshushu>.
[    9.433756] systemd-system-update-generator[165]: Offline system update overridden by kernel command line systemd.unit= setting
[   10.040658] random: systemd: uninitialized urandom read (16 bytes read)
[   10.055024] random: systemd: uninitialized urandom read (16 bytes read)
[   10.059842] systemd[1]: Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket (/dev/log).
[   10.066862] random: systemd: uninitialized urandom read (16 bytes read)
[   10.067933] systemd[1]: Listening on Journal Audit Socket.
[  OK  ] Listening on Journal Audit Socket.
[   10.069720] systemd[1]: Reached target Swap.
[  OK  ] Reached target Swap.
[   10.071890] systemd[1]: Listening on udev Kernel Socket.
[  OK  ] Listening on udev Kernel Socket.
[   10.074894] systemd[1]: Listening on Journal Socket.
[  OK  ] Listening on Journal Socket.
[   10.105730] systemd[1]: Mounting Huge Pages File System...
         Mounting Huge Pages File System...
         Starting Create list of re…odes for the current kernel...
         Starting Remount Root and Kernel File Systems...
         Starting Journal Service...
         Starting Load Kernel Modules...
         Mounting Kernel Debug File System...
[  OK  ] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
         Mounting POSIX Message Queue File System...
[  OK  ] Set up automount Arbitrary…s File System Automount Point.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Started Create list of req… nodes for the current kernel.
[FAILED] Failed to start Remount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
[  OK  ] Started Load Kernel Modules.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Started Journal Service.
         Starting Apply Kernel Variables...
         Starting Load/Save Random Seed...
         Starting Create System Users...
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Create System Users.
[   12.186241] systemd-journald[171]: Received request to flush runtime journal from PID 1
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
         Mounting /mnt...
         Starting udev Kernel Device Manager...
[  OK  ] Started udev Coldplug all Devices.
         Starting Helper to synchronize boot up for ifupdown...
[  OK  ] Started Helper to synchronize boot up for ifupdown.
[  OK  ] Started udev Kernel Device Manager.
[   13.345191] FS-Cache: Loaded
[   13.402334] 9p: Installing v9fs 9p2000 file system support
[   13.403232] FS-Cache: Netfs '9p' registered for caching
[  OK  ] Mounted /mnt.
[  OK  ] Reached target Local File Systems.
         Starting Raise network interfaces...
         Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
         Starting Network Time Synchronization...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Network Time Synchronization.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target System Time Synchronized.
[  OK  ] Started Raise network interfaces.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
         Starting Kernel crash dump capture service...
[   17.786977] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[   18.515478] kdump-tools[236]: Starting kdump-tools:      #第二个内核启动kdump-tools
[  OK  ] Stopped Kernel crash dump capture service.
         Starting Kernel crash dump capture service...
[   19.308167] virtio_net virtio1 enp0s1: renamed from eth0
[  OK  ] Found device Virtio network device.
[  OK  ] Started ifup for enp0s1.
[   20.787668] kdump-tools[255]: Starting kdump-tools: running makedumpfile -c -d 31 /proc/vmcore /var/crash/202309300804/dump-incomplete.    #核心转存文件放在的位置
Copying data                                      : [100.0 %] |           eta: 0s
[   50.664493] kdump-tools[255]: The kernel version is not supported.
[   50.671122] kdump-tools[255]: The makedumpfile operation may be incomplete.
[   50.673516] kdump-tools[255]: The dumpfile is saved to /var/crash/202309300804/dump-incomplete.  
[   50.677696] kdump-tools[255]: makedumpfile Completed.
[   50.753120] kdump-tools[255]: kdump-tools: saved vmcore in /var/crash/202309300804.
[   50.893158] kdump-tools[255]: running makedumpfile --dump-dmesg /proc/vmcore /var/crash/202309300804/dmesg.202309300804.
[   50.974048] kdump-tools[255]: The kernel version is not supported.
[   50.975674] kdump-tools[255]: The makedumpfile operation may be incomplete.
[   50.977360] kdump-tools[255]: The dmesg log is saved to /var/crash/202309300804/dmesg.202309300804.
[   50.981244] kdump-tools[255]: makedumpfile Completed.
[   50.990062] kdump-tools[255]: kdump-tools: saved dmesg content in /var/crash/202309300804.
[   51.125962] kdump-tools[255]: Sat, 30 Sep 2023 08:04:39 +0000
[   51.185245] kdump-tools[255]: Rebooting.
[   51.521006] reboot: Restarting system
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[    0.000000] Linux version 5.0.0-rlk (rlk@rlk) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu1)) #2 SMP Sat Sep 30 12:37:16 CST 2023
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] crashkernel reserved: 0x000000006fe00000 - 0x000000007fe00000 (256 MB)
[    0.000000] cma: Reserved 64 MiB at 0x000000006bc00000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x6fdf1840-0x6fdf2fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] On node 0 totalpages: 262144
[    0.000000]   DMA32 zone: 4096 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 262144 pages, LIFO batch:63
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] random: get_random_bytes called from start_kernel+0xa8/0x4bc with crng_init=0
[    0.000000] percpu: Embedded 25 pages/cpu @(____ptrval____) s64152 r8192 d30056 u102400
[    0.000000] pcpu-alloc: s64152 r8192 d30056 u102400 alloc=25*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: ARM erratum 832075
[    0.000000] CPU features: detected: ARM erratum 834220
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: noinintrd root=/dev/vda rootfstype=ext4 rw crashkernel=256M loglevel=8
[    0.000000] Memory: 678024K/1048576K available (10876K kernel code, 1804K rwdata, 4316K rodata, 5312K init, 572K bss, 305016K reserved, 65536K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 37102 entries in 145 pages
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv2m: range[mem 0x08020000-0x08020fff], SPI[80:143]
[    0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
[    0.000093] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
[    0.006713] Console: colour dummy device 80x25
[    0.007932] printk: console [tty0] enabled
[    0.009758] Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=250000)
[    0.009901] pid_max: default: 32768 minimum: 301
[    0.011042] LSM: Security Framework initializing
[    0.011202] Yama: becoming mindful.
[    0.013708] AppArmor: AppArmor initialized
[    0.015098] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.015817] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.016024] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.016074] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.046297] ASID allocator initialised with 32768 entries
[    0.047425] rcu: Hierarchical SRCU implementation.
[    0.052564] EFI services will not be available.
[    0.055204] smp: Bringing up secondary CPUs ...
[    0.058002] Detected PIPT I-cache on CPU1
[    0.058637] CPU1: Booted secondary processor 0x0000000001 [0x411fd070]
[    0.063208] Detected PIPT I-cache on CPU2
[    0.063356] CPU2: Booted secondary processor 0x0000000002 [0x411fd070]
[    0.065266] Detected PIPT I-cache on CPU3
[    0.065407] CPU3: Booted secondary processor 0x0000000003 [0x411fd070]
[    0.067068] smp: Brought up 1 node, 4 CPUs
[    0.067176] SMP: Total of 4 processors activated.
[    0.067235] CPU features: detected: 32-bit EL0 Support
[    0.067328] CPU features: detected: CRC32 instructions
[    0.103715] CPU: All CPU(s) started at EL1
[    0.113122] alternatives: patching kernel code
[    0.131758] devtmpfs: initialized
[    0.141863] Registered cp15_barrier emulation handler
[    0.142070] Registered setend emulation handler
[    0.143597] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.143788] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.149133] xor: measuring software checksum speed
[    0.195376]    8regs     :  2402.000 MB/sec
[    0.244348]    32regs    :  2545.000 MB/sec
[    0.293207]    arm64_neon:  2333.000 MB/sec
[    0.293305] xor: using function: 32regs (2545.000 MB/sec)
[    0.293574] pinctrl core: initialized pinctrl subsystem
[    0.303644] DMI not present or invalid.
[    0.308098] NET: Registered protocol family 16
[    0.311271] audit: initializing netlink subsys (disabled)
[    0.313205] audit: type=2000 audit(0.240:1): state=initialized audit_enabled=0 res=1
[    0.317131] cpuidle: using governor ladder
[    0.317314] cpuidle: using governor menu
[    0.318346] vdso: 2 pages (1 code @ (____ptrval____), 1 data @ (____ptrval____))
[    0.318465] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.327891] DMA: preallocated 256 KiB pool for atomic allocations
[    0.329514] Serial: AMBA PL011 UART driver
[    0.348452] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 39, base_baud = 0) is a PL011 rev1
[    0.377054] printk: console [ttyAMA0] enabled
[    0.401037] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.401531] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.401925] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.402217] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.487726] raid6: neonx8   gen()  2679 MB/s
[    0.570470] raid6: neonx8   xor()  1058 MB/s
[    0.651458] raid6: neonx4   gen()  2516 MB/s
[    0.734013] raid6: neonx4   xor()  1067 MB/s
[    0.817795] raid6: neonx2   gen()  2113 MB/s
[    0.899670] raid6: neonx2   xor()   915 MB/s
[    0.981716] raid6: neonx1   gen()  1351 MB/s
[    1.063985] raid6: neonx1   xor()   750 MB/s
[    1.146009] raid6: int64x8  gen()  1569 MB/s
[    1.228138] raid6: int64x8  xor()  1170 MB/s
[    1.334874] raid6: int64x4  gen()  2171 MB/s
[    1.420115] raid6: int64x4  xor()  1061 MB/s
[    1.503350] raid6: int64x2  gen()  1797 MB/s
[    1.588089] raid6: int64x2  xor()  1243 MB/s
[    1.670899] raid6: int64x1  gen()  1257 MB/s
[    1.753645] raid6: int64x1  xor()   884 MB/s
[    1.753866] raid6: using algorithm neonx8 gen() 2679 MB/s
[    1.754138] raid6: .... xor() 1058 MB/s, rmw enabled
[    1.754440] raid6: using neon recovery algorithm
[    1.756527] ACPI: Interpreter disabled.
[    1.758425] vgaarb: loaded
[    1.760435] SCSI subsystem initialized
[    1.764519] libata version 3.00 loaded.
[    1.765036] pps_core: LinuxPPS API ver. 1 registered
[    1.765149] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    1.765724] PTP clock support registered
[    1.766150] EDAC MC: Ver: 3.0.0
[    1.785998] clocksource: Switched to clocksource arch_sys_counter
[    1.938926] VFS: Disk quotas dquot_6.6.0
[    1.940072] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.947132] AppArmor: AppArmor Filesystem Enabled
[    1.949304] pnp: PnP ACPI: disabled
[    1.992237] NET: Registered protocol family 2
[    1.999826] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes)
[    2.000372] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[    2.001010] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    2.001546] TCP: Hash tables configured (established 8192 bind 8192)
[    2.003553] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    2.004063] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    2.006233] NET: Registered protocol family 1
[    2.007376] PCI: CLS 0 bytes, default 64
[    2.019738] hw perfevents: enabled with armv8_pmuv3 PMU driver, 5 counters available
[    2.021698] kvm [1]: HYP mode not available
[    2.028091] Initialise system trusted keyrings
[    2.030368] workingset: timestamp_bits=44 max_order=18 bucket_order=0
[    2.043612] zbud: loaded
[    2.615595] Key type asymmetric registered
[    2.615948] Asymmetric key parser 'x509' registered
[    2.616385] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    2.617483] io scheduler mq-deadline registered
[    2.629849] pl061_gpio 9030000.pl061: PL061 GPIO chip @0x0000000009030000 registered
[    2.633124] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    2.634248] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:
[    2.635661] pci-host-generic 4010000000.pcie:    IO 0x3eff0000..0x3effffff -> 0x00000000
[    2.636457] pci-host-generic 4010000000.pcie:   MEM 0x10000000..0x3efeffff -> 0x10000000
[    2.637327] pci-host-generic 4010000000.pcie:   MEM 0x8000000000..0xffffffffff -> 0x8000000000
[    2.641164] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]
[    2.643122] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00
[    2.643539] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.643753] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    2.643942] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]
[    2.644274] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[    2.645804] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[    2.653036] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000
[    2.653771] pci 0000:00:01.0: reg 0x10: [io  0x0000-0x001f]
[    2.661308] pci 0000:00:01.0: reg 0x14: [mem 0x00000000-0x00000fff]
[    2.662367] pci 0000:00:01.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[    2.662586] pci 0000:00:01.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[    2.663455] pci 0000:00:02.0: [1af4:1009] type 00 class 0x000200
[    2.663927] pci 0000:00:02.0: reg 0x10: [io  0x0000-0x003f]
[    2.664222] pci 0000:00:02.0: reg 0x14: [mem 0x00000000-0x00000fff]
[    2.664396] pci 0000:00:02.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[    2.667807] pci 0000:00:01.0: BAR 6: assigned [mem 0x10000000-0x1003ffff pref]
[    2.668243] pci 0000:00:01.0: BAR 4: assigned [mem 0x8000000000-0x8000003fff 64bit pref]
[    2.668635] pci 0000:00:02.0: BAR 4: assigned [mem 0x8000004000-0x8000007fff 64bit pref]
[    2.668900] pci 0000:00:01.0: BAR 1: assigned [mem 0x10040000-0x10040fff]
[    2.669285] pci 0000:00:02.0: BAR 1: assigned [mem 0x10041000-0x10041fff]
[    2.669496] pci 0000:00:02.0: BAR 0: assigned [io  0x1000-0x103f]
[    2.669682] pci 0000:00:01.0: BAR 0: assigned [io  0x1040-0x105f]
[    2.688110] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)
[    2.690960] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)
[    2.694873] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.699567] Serial: AMBA driver
[    2.700857] msm_serial: driver initialized
[    2.703714] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    2.726154] virtio_blk virtio0: [vda] 16777216 512-byte logical blocks (8.59 GB/8.00 GiB)
[    2.755391] mousedev: PS/2 mouse device common for all mice
[    2.759747] rtc-pl031 9010000.pl031: registered as rtc0
[    2.764905] ledtrig-cpu: registered to indicate activity on CPUs
[    2.771465] NET: Registered protocol family 10
[    2.785626] Segment Routing with IPv6
[    2.787928] mip6: Mobile IPv6
[    2.788633] NET: Registered protocol family 17
[    2.790244] 9pnet: Installing 9P2000 support
[    2.797362] mpls_gso: MPLS GSO support
[    2.803076] registered taskstats version 1
[    2.803881] Loading compiled-in X.509 certificates
[    2.805752] zswap: loaded using pool lzo/zbud
[    2.813309] Btrfs loaded, crc32c=crc32c-generic
[    2.816493] AppArmor: AppArmor sha1 policy hashing enabled
[    2.821800] rtc-pl031 9010000.pl031: setting system clock to 2023-09-30T08:04:43 UTC (1696061083)
[    2.834910] uart-pl011 9000000.pl011: no DMA platform data
[    2.915463] EXT4-fs (vda): recovery complete
[    2.918204] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[    2.918875] VFS: Mounted root (ext4 filesystem) on device 254:0.
[    3.238316] Freeing unused kernel memory: 5312K
[    3.239331] Run /sbin/init as init process
[    3.315940] random: fast init done
[    3.616973] autofs4: module verification failed: signature and/or required key missing - tainting kernel
[    3.636358] systemd[1]: Inserted module 'autofs4'
[    3.720093] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[    3.728643] systemd[1]: Detected virtualization qemu.
[    3.729982] systemd[1]: Detected architecture arm64.

Welcome to Debian GNU/Linux buster/sid!

[    3.762873] systemd[1]: Set hostname to <benshushu>.
[    4.879495] random: systemd: uninitialized urandom read (16 bytes read)
[    4.903484] random: systemd: uninitialized urandom read (16 bytes read)
[    4.908399] systemd[1]: Listening on udev Control Socket.
[  OK  ] Listening on udev Control Socket.
[    4.912469] random: systemd: uninitialized urandom read (16 bytes read)
[    4.929852] systemd[1]: Created slice system-serial\x2dgetty.slice.
[  OK  ] Created slice system-serial\x2dgetty.slice.
[    4.932443] systemd[1]: Reached target Swap.
[  OK  ] Reached target Swap.
[    4.936708] systemd[1]: Listening on Syslog Socket.
[  OK  ] Listening on Syslog Socket.
[    4.938432] systemd[1]: Reached target Remote File Systems.
[  OK  ] Reached target Remote File Systems.
[    4.945594] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[  OK  ] Set up automount Arbitrary…s File System Automount Point.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Listening on Journal Audit Socket.
[  OK  ] Started Forward Password R…uests to Wall Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
         Starting Journal Service...
         Starting Load Kernel Modules...
         Mounting Kernel Debug File System...
         Starting Create list of re…odes for the current kernel...
         Starting udev Coldplug all Devices...
         Mounting Huge Pages File System...
         Starting Remount Root and Kernel File Systems...
         Mounting POSIX Message Queue File System...
[  OK  ] Created slice User and Session Slice.
[  OK  ] Reached target Slices.
[  OK  ] Started Load Kernel Modules.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Started Create list of req… nodes for the current kernel.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted POSIX Message Queue File System.
         Starting Apply Kernel Variables...
[FAILED] Failed to start Remount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
         Starting Load/Save Random Seed...
         Starting Create System Users...
[  OK  ] Started Apply Kernel Variables.
[    6.161478] systemd-journald[138]: Received request to flush runtime journal from PID 1
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
         Mounting /mnt...
         Starting udev Kernel Device Manager...
[  OK  ] Started udev Coldplug all Devices.
         Starting Helper to synchronize boot up for ifupdown...
[    6.788550] FS-Cache: Loaded
[    6.831728] 9p: Installing v9fs 9p2000 file system support
[    6.833048] FS-Cache: Netfs '9p' registered for caching
[  OK  ] Started Helper to synchronize boot up for ifupdown.
[  OK  ] Mounted /mnt.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Raise network interfaces...
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Network Time Synchronization...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Network Time Synchronization.
[  OK  ] Reached target System Time Synchronized.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Started Periodic ext4 Onli…ata Check for All Filesystems.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting System Logging Service...
[  OK  ] Started D-Bus System Message Bus.
         Starting WPA supplicant...
         Starting Avahi mDNS/DNS-SD Stack...
[  OK  ] Started Regular background program processing daemon.
         Starting Remove Stale Onli…t4 Metadata Check Snapshots...
         Starting Login Service...
         Starting LSB: Execute the …-e command to reboot system...
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Reached target Timers.
         Starting DHCP Client Daemon...
[  OK  ] Started System Logging Service.
[  OK  ] Found device /dev/ttyAMA0.
[  OK  ] Started Avahi mDNS/DNS-SD Stack.
[    9.728376] 8021q: 802.1Q VLAN Support v1.8
[  OK  ] Started WPA supplicant.
[  OK  ] Started Login Service.
[  OK  ] Started LSB: Execute the k…c -e command to reboot system.
         Starting LSB: Load kernel image with kexec...
[   10.289702] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[  OK  ] Started Raise network interfaces.
[   10.711046] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[   10.720927] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[   10.725929] cfg80211: failed to load regulatory.db
[   10.799337] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[  OK  ] Started Remove Stale Onlin…ext4 Metadata Check Snapshots.
[  OK  ] Started DHCP Client Daemon.
[  OK  ] Reached target Network.
         Starting Permit User Sessions...
         Starting OpenBSD Secure Shell server...
[  OK  ] Reached target Network is Online.
         Starting Kernel crash dump capture service...
[  OK  ] Started LSB: Load kernel image with kexec.
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Serial Getty on ttyAMA0.
[   11.748122] 8021q: adding VLAN 0 to HW filter on device eth0
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
[  OK  ] Stopped Kernel crash dump capture service.
[  OK  ] Listening on Load/Save RF …itch Status /dev/rfkill Watch.
         Starting Kernel crash dump capture service...
[   13.229288] cryptd: max_cpu_qlen set to 1000
[   13.558988] kdump-tools[271]: Starting kdump-tools: Creating symlink /var/lib/kdump/vmlinuz.
[   13.626842] kdump-tools[271]: Creating symlink /var/lib/kdump/initrd.img.
[   15.285353] kdump-tools[271]: loaded kdump kernel.
[  OK  ] Started Kernel crash dump capture service.

Debian GNU/Linux buster/sid benshushu ttyAMA0

benshushu login: 

crash工具入手

先直接进来 使用crash加载等待三分钟
crash ./var/crash/202309300804/dump.202309300804 /mnt/vmlinux

      KERNEL: /mnt/vmlinux                      
    DUMPFILE: /var/crash/202309300804/dump.202309300804  [PARTIAL DUMP]
        CPUS: 4
        DATE: Sat Sep 30 08:03:41 2023
      UPTIME: 04:36:31
LOAD AVERAGE: 0.00, 0.00, 0.00
       TASKS: 86
    NODENAME: benshushu
     RELEASE: 5.0.0-rlk
     VERSION: #2 SMP Sat Sep 30 12:37:16 CST 2023
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 1 GB
## 这里说指针,给力一个overview
       PANIC: "Unable to handle kernel NULL pointer dereference at virtual address 0000000000000050"
         PID: 2346
     COMMAND: "insmod"
        TASK: ffff800023600e80  [THREAD_INFO: ffff800023600e80]
         CPU: 1
       STATE: TASK_RUNNING (PANIC)

bt
使用bt查看调用栈 查看发生crash的时候 每个寄存器的值

crash> bt
PID: 2346   TASK: ffff800023600e80  CPU: 1   COMMAND: "insmod"
 #0 [ffff00001352b660] machine_kexec at ffff0000100a0448
 #1 [ffff00001352b6c0] __crash_kexec at ffff000010198380
 #2 [ffff00001352b850] crash_kexec at ffff000010198494
 #3 [ffff00001352b880] die at ffff00001008ec10
 #4 [ffff00001352b8c0] die_kernel_fault at ffff0000100a39bc
 #5 [ffff00001352b8f0] __do_kernel_fault at ffff0000100a3a5c
 #6 [ffff00001352b920] do_page_fault at ffff000010b05f5c
 #7 [ffff00001352b980] do_translation_fault at ffff000010b062a4
 #8 [ffff00001352b9b0] do_mem_abort at ffff0000100815bc
 #9 [ffff00001352bb10] el1_ia at ffff00001008318c
     PC: ffff000008cf6020  [create_oops+32]
     LR: ffff000008bbe0a0  [_MODULE_INIT_START_oops+160]
     SP: ffff00001352bb20  PSTATE: 80000005
    X29: ffff00001352bb20  X28: ffff000008cf81d0  X27: ffff000008cf8180
    X26: ffff00001352bdc0  X25: ffff000008cf8198  X24: ffff000008cf8008
    X23: 0000000000000000  X22: ffff800023600e80  X21: ffff000008cf8018
    X20: 0000000000000000  X19: ffff000008bbe000  X18: 0000000000000000
    X17: 0000000000000000  X16: 0000000000000000  X15: ffff800023601370
    X14: ffffffffffffffff  X13: 0000000000000040  X12: 0000000000000228
    X11: 0000000000000000  X10: 0000000000000000   X9: 0000000000000000
     X8: 00000000000007a7   X7: ffff80002a803b00   X6: ffff00001352bb89
     X5: ffff800029319e00   X4: ffff80002fdaeb00   X3: 0000000000000000
     X2: ffff000008bbe0a0   X1: ffff00001352bb84   X0: 0000000000000000
#10 [ffff00001352bb20] create_oops at ffff000008cf601c [oops]
#11 [ffff00001352bb50] _MODULE_INIT_START_oops at ffff000008bbe09c [oops]
#12 [ffff00001352bbd0] do_one_initcall at ffff00001008486c
#13 [ffff00001352bc60] do_init_module at ffff0000101925b4
#14 [ffff00001352bc90] load_module at ffff000010194428
#15 [ffff00001352bd80] __se_sys_finit_module at ffff000010194a30
#16 [ffff00001352be40] __arm64_sys_finit_module at ffff000010194ae0
Arm中看PC指针  出错函数在create_opps()+32字节的地方
SP寄存器为当时的栈指针
X29为FP寄存器
X0传递参数1 x1传递参数2

mod
使用mod装载外部符号表
因为是vmlinux的符号表没有额外装载的内核模块符号 通过mod -s 装载符号

crash> mod -s oops ./oops.ko
     MODULE       NAME             SIZE  OBJECT FILE
ffff000008cf8000  oops            16384  /mnt/01_oops/oops.ko 

dis
通过dis查看某个地址对应的反汇编指令
此时上面出错的PC寄存器值为 ffff000008cf6020 直接通过dis查看

crash> dis ffff000008cf6020
0xffff000008cf6020 <create_oops+32>:    ldr     x0, [x0,#80]

0xffff000008cf6020地址为函数create_oops()偏移32字节的地方
此时汇编代码为ldr x0, [x0,#80]

---

struct
前面说,x1,x0,保存了传参 宕机代码中create_oops()函数传入了 struct vm_area_struct *
和自己的写的结构体struct mydev_priv priv
查看符号vm_area_struct 和它的成员相对 符号 vm_area_struct的偏移

crash> hex  (切换为16进制)
output radix: 16 (hex)
crash> struct -o vm_area_struct  (查看结构体和对应成员的偏移)
struct vm_area_struct {
   [0x0] unsigned long vm_start;
   [0x8] unsigned long vm_end;
  [0x10] struct vm_area_struct *vm_next;
  [0x18] struct vm_area_struct *vm_prev;
  [0x20] struct rb_node vm_rb;
  [0x38] unsigned long rb_subtree_gap;
  [0x40] struct mm_struct *vm_mm;
  [0x48] pgprot_t vm_page_prot;
  [0x50] unsigned long vm_flags;
         struct {
             struct rb_node rb;
             unsigned long rb_subtree_last;
  [0x58] } shared;
  [0x78] struct list_head anon_vma_chain;

vm_area_struct成员vm_flags 的偏移量为0x50 也是#80
和反汇编出来的数字一样 通过x0偏移80字节
这里看见x0寄存器存放的是函数第一个参数 此时x0为0000000000000000
通过把结构体vm_area_struct映射到内存的 0x0000000000000000
这里表示为一个无效的结构体 (宕机原因)

crash> struct -o vm_area_struct 0x0000000000000000
struct: invalid kernel virtual address: 0x0000000000000000

如果是个正常地址这个 指令怎么使用
X1存放的是第二个参数地址 0xffff00001352bb84
rd
使用rd读取这个地址能解析出字符串

crash> rd  0xffff00001352bb84
ffff00001352bb84:  1352bd006f676966                    figo..R.

同时使用struct 去用对应的结构体mydev_priv解析这个地址 下面是能打印出每个变量的 也证明了这个地址的有效

crash> struct mydev_priv 0xffff00001352bb84
struct mydev_priv {
  name = "figo\000\275R\023\000\000\377\377\200\201\317\b\000\000\377\377Ё\317\b\000\000\377\377#>\006\000\000\000\000\000\200\016`#\000\200\377\377\200\016`#\000\200\377\377\000\200\317\b\000\000\377\377P\261L\021", 
  i = 0xa
}

实操2手动恢复调用栈

上面一到我们在core_dump的时候有崩溃时候的寄存器

[ 6816.949505] pc : create_oops+0x20/0x4c [oops]
[ 6816.950007] lr : my_oops_init+0xa0/0x1000 [oops]
[ 6816.950524] sp : ffff00001352bb20
[ 6816.951172] x29: ffff00001352bb20 x28: ffff000008cf81d0 
[ 6816.951700] x27: ffff000008cf8180 x26: ffff00001352bdc0 
[ 6816.952185] x25: ffff000008cf8198 x24: ffff000008cf8008 
[ 6816.952645] x23: 0000000000000000 x22: ffff800023600e80 
[ 6816.952886] x21: ffff000008cf8018 x20: 0000000000000000 
[ 6816.953113] x19: ffff000008bbe000 x18: 0000000000000000 
[ 6816.953483] x17: 0000000000000000 x16: 0000000000000000 
[ 6816.954128] x15: ffff800023601370 x14: ffffffffffffffff 
[ 6816.954868] x13: 0000000000000040 x12: 0000000000000228 
[ 6816.955479] x11: 0000000000000000 x10: 0000000000000000 
[ 6816.956565] x9 : 0000000000000000 x8 : 00000000000007a7 
[ 6816.957786] x7 : ffff80002a803b00 x6 : ffff00001352bb89 
[ 6816.958644] x5 : ffff800029319e00 x4 : ffff80002fdaeb00 
[ 6816.959826] x3 : 0000000000000000 x2 : ffff000008bbe0a0 
[ 6816.960361] x1 : ffff00001352bb84 x0 : 0000000000000000 

崩溃的时候sp和fp都为一个值 接下来反推父函数当时保存的栈帧寄存器的值
读取fp寄存器的值,使用rd读取这个地址,得到了父栈帧的rd地址 0xffff00001352bb50

crash> rd ffff00001352bb20
ffff00001352bb20:  ffff00001352bb50                    P.R.....

这个和bt打出来的值是一样的 所以bt前面的数字也就是每个函数保存的fp寄存器的值

crash> bt
PID: 2346   TASK: ffff800023600e80  CPU: 1   COMMAND: "insmod"
	#每个栈帧的fp       符号名称         调用函数pc的值
 #0 [ffff00001352b660] machine_kexec at ffff0000100a0448
#11 [ffff00001352bb50] _MODULE_INIT_START_oops at ffff000008bbe09c [oops]

推断子函数的名称
子函数崩溃的时候 fp寄存器值为 ffff00001352bb20 fp也指向 0xffff00001352bb20地址处
2.得到栈帧上保存的LR，ffff00001352bb20 + 8 =ffff00001352bb28 。
读取这个地址得出 子函数的后返回地址

crash> rd ffff00001352bb28
ffff00001352bb28:  ffff000008bbe0a0                    ........

ffff000008bbe0a0就是此时create_oops()结束后的地址(返回地址)
*lr-4 得到的物理地址就是 pc的值= ffff000008bbe09c (为什么是-4查看树自我修养中104页)
大概是 create_oops()的地址 会存放在create_oops()运行完下一条地址,再减4
这时候反汇编这个地址就能得到执行的函数为 create_oops()
反汇编这个地址存放的值得出来就是这时候开始执行create_oops()

crash> dis ffff000008bbe09c
0xffff000008bbe09c <_MODULE_INIT_START_oops+0x9c>:      bl      0xffff000008cf6000 <create_oops>

上面的结果表明0xffff000008bbe09c地址为函数_MODULE_INIT_START_oops()
偏移0x9c 对这个地址的值反汇编结果为 bl跳转到地址0xffff000008cf6000

实操3 我刚刚传输什么值进函数啊

从上面的问题继续引发思考 假如程序中有锁导致机器hung死咋办

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,11,0)
#include 
#endif

struct mydev_priv {
	char name[64];
	int i;
	struct mm_struct *mm;
	struct rw_semaphore *sem;
};

int create_oops(struct vm_area_struct *vma, struct mydev_priv *priv, struct rw_semaphore *sem)
{
	unsigned long flags;

	down_read(sem);

	flags = vma->vm_flags;
	printk("flags=0x%lx, name=%s\n", flags, priv->name);
	
	return 0;
}

int __init my_oops_init(void)
{
	int ret;
	struct vm_area_struct *vma = NULL;
	struct mydev_priv priv;
	struct mm_struct *mm;

	mm = get_task_mm(current);

	priv.mm = mm;
	priv.sem = &mm->mmap_sem;

	down_write(&mm->mmap_sem);

	vma = kmalloc(sizeof (*vma), GFP_KERNEL);
	if (!vma)
		return -ENOMEM;

	kfree(vma);
	vma = NULL;

	smp_mb();

	memcpy(priv.name, "figo", sizeof("figo"));
	priv.i = 10;

	ret = create_oops(vma, &priv, &mm->mmap_sem);

	return 0;
}

void __exit my_oops_exit(void)
{
	printk("goodbye\n");
}

module_init(my_oops_init);
module_exit(my_oops_exit);
MODULE_LICENSE("GPL");

抛出一个能hung代码

#打开hung死触发panic  设置超过30s判断软件为hung死
echo 1 > /proc/sys/kernel/hung_task_panic 1
echo 1 > /proc/sys/kernel/softlockup_panic 1
echo 30 > /proc/sys/kernel/hung_task_timeout_secs 
sysctl -a | grep panic

下面是painc时候内核的打印信息

root@benshushu:/mnt/03_hung_duck# insmod ./hung_duck.ko 
[  600.012622] hung_duck: loading out-of-tree module taints kernel.
[  636.176050] INFO: task insmod:578 blocked for more than 30 seconds.
[  636.176536]       Tainted: G           OE     5.0.0-rlk #2
[  636.176798] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  636.177893] insmod          D    0   578    464 0x00000020
[  636.178321] Call trace:
[  636.178742]  __switch_to+0xb4/0x1b8
[  636.179112]  __schedule+0x294/0x8c0
[  636.179644]  schedule+0x34/0x80
[  636.179897]  rwsem_down_read_failed+0x10c/0x1a0
[  636.180471]  down_read+0x54/0x70
[  636.182293]  create_oops+0x28/0x58 [hung_duck]
[  636.182738]  my_oops_init+0xe0/0x1000 [hung_duck]
[  636.183581]  do_one_initcall+0x50/0x1d8
[  636.183920]  do_init_module+0x60/0x1e8
[  636.184210]  load_module+0x1bd4/0x1ea8
[  636.185180]  __se_sys_finit_module+0x9c/0xf8
[  636.186018]  __arm64_sys_finit_module+0x24/0x30
[  636.186611]  el0_svc_common+0x78/0x120
[  636.187441]  el0_svc_handler+0x38/0x78
[  636.187917]  el0_svc+0x8/0xc
[  636.189146] Kernel panic - not syncing: hung_task: blocked tasks
[  636.189983] CPU: 0 PID: 35 Comm: khungtaskd Kdump: loaded Tainted: G           OE     5.0.0-rlk #2
[  636.191290] Hardware name: linux,dummy-virt (DT)
[  636.191533] Call trace:
[  636.192049]  dump_backtrace+0x0/0x178
[  636.192657]  show_stack+0x24/0x30
[  636.192822]  dump_stack+0x88/0xac
[  636.192987]  panic+0x134/0x2d8
[  636.193083]  watchdog+0x27c/0x3e0
[  636.193177]  kthread+0x108/0x138
[  636.193274]  ret_from_fork+0x10/0x1c
[  636.194319] SMP: stopping secondary CPUs
[  636.209329] Starting crashdump kernel...
[  636.209945] Bye!
机器就死在里面了 这时候机器开始重启

拿到这次的coredump 上crash工具

crash ./dump.202310011624 ../../../mnt/vmlinux
      KERNEL: ../../../mnt/vmlinux              
    DUMPFILE: ./dump.202310011624  [PARTIAL DUMP]
        CPUS: 4
        DATE: Sun Oct  1 16:24:02 2023
      UPTIME: 00:18:59
LOAD AVERAGE: 0.50, 0.27, 0.18
       TASKS: 85
    NODENAME: benshushu
     RELEASE: 5.0.0-rlk
     VERSION: #2 SMP Sat Sep 30 12:37:16 CST 2023
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 1 GB
       PANIC: "Kernel panic - not syncing: hung_task: blocked tasks"
         PID: 35
     COMMAND: "khungtaskd"
        TASK: ffff80002a0dab80  [THREAD_INFO: ffff80002a0dab80]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)
#此时加载出来的panic的是因为有进程block导致panic
#上面也能看出因为insmod驱动的时候  导致insmod进程hung死同时insmod pid578

我们最想知道的是 这个驱动里面传递什么参数 让insmod如此难过
此时需要查看insmod进程的调用栈

crash> bt 578
PID: 578    TASK: ffff800023444880  CPU: 3   COMMAND: "insmod"
 #0 [ffff000011deb970] __switch_to at ffff000010087e90
 #1 [ffff000011deb990] __schedule at ffff000010afede8
 #2 [ffff000011deba20] schedule at ffff000010aff448
 #3 [ffff000011deba40] rwsem_down_read_failed at ffff000010b028d0
 #4 [ffff000011debad0] down_read at ffff000010b01e60
 #5 [ffff000011debaf0] create_oops at ffff000008d30024 [hung_duck] 
 #6 [ffff000011debb30] _MODULE_INIT_START_hung_duck at ffff000008be20dc [hung_duck]
 #7 [ffff000011debbd0] do_one_initcall at ffff00001008486c
 #8 [ffff000011debc60] do_init_module at ffff0000101925b4
 #9 [ffff000011debc90] load_module at ffff000010194428
#10 [ffff000011debd80] __se_sys_finit_module at ffff000010194a30
#11 [ffff000011debe40] __arm64_sys_finit_module at ffff000010194ae0
#12 [ffff000011debe60] el0_svc_common at ffff0000100966f4
#13 [ffff000011debea0] el0_svc_handler at ffff0000100967d4

通过调用栈看得出来最后在 函数create_oops()的时候传递了什么参数 让程序死在里面了
源码: ret = create_oops(vma, &priv, &mm->mmap_sem);
#mod装载模块对应coredump的符号表
crash> mod -s hung_duck hung_duck.ko
     MODULE       NAME             SIZE  OBJECT FILE
ffff000008d32000  hung_duck       16384  hung_duck.ko 
#有符号表后 通过dis进行反汇编这个create_oops()函数
# 下面初始汇编不同看参考文章
crash> dis create_oops
0xffff000008d30000 <create_oops>:       stp     x29, x30, [sp,#-64]!  #第一步创建栈帧把sp压栈底,创建这个的栈帧
0xffff000008d30004 <create_oops+4>:     mov     x29, sp  #fp也指过去,此时sp=fp
0xffff000008d30008 <create_oops+8>:     mov     x3, x30
0xffff000008d3000c <create_oops+12>:    str     x0, [sp,#40] #参数1在sp偏移#40,参数1放入x0
0xffff000008d30010 <create_oops+16>:    str     x1, [sp,#32] #参数2在sp偏移#32,参数2放入x1
0xffff000008d30014 <create_oops+20>:    str     x2, [sp,#24] #参数3在sp偏移#24,参数3放入x2
0xffff000008d30018 <create_oops+24>:    mov     x0, x3
0xffff000008d3001c <create_oops+28>:    nop
0xffff000008d30020 <create_oops+32>:    ldr     x0, [sp,#24] #这里使用了参数3 也就是&mm->mmap_sem 给下面函数down_read()
0xffff000008d30024 <create_oops+36>:    bl      0xffff000010b01e10 <down_read>
0xffff000008d30028 <create_oops+40>:    ldr     x0, [sp,#40]
0xffff000008d3002c <create_oops+44>:    ldr     x0, [x0,#80]
0xffff000008d30030 <create_oops+48>:    str     x0, [sp,#56]
0xffff000008d30034 <create_oops+52>:    ldr     x0, [sp,#32]
0xffff000008d30038 <create_oops+56>:    mov     x2, x0
0xffff000008d3003c <create_oops+60>:    ldr     x1, [sp,#56]

从上面downread()使用参数3 也说明这个函数调用栈中sp+24 就是存放了参数mm->mmap_sem的地址
因为sp=0xffff000011debaf0 参数mm->mmap_sem地址=0xffff000011debaf0+0x18=0xFFFF 0000 11DE BB08

#直接就能读出当前传入参数的值
crash> struct vm_area_struct ffff000011debb08
struct vm_area_struct {
  vm_start = 18446603336834076512, 
  vm_end = 18446462599032650608, 
  vm_next = 0x0, 
  vm_prev = 0xffff000008d32018 <__this_module+24>, 
  vm_rb = {
    __rb_parent_color = 18446462598880894984, 
    rb_right = 0xffff000011debbd0, 
    rb_left = 0xffff000010084870 <do_one_initcall+80>
   .......

同理第一个第二个第三个三处都能这样找到

总结

下面解释create_oops()函数的栈

参考文献

https://blog.csdn.net/heshuangzong/article/details/126911474 crash分析,函数每一步调用栈的过程