移植Kdump至嵌入式ARM64环境

工欲善其事，必先利其器。操作系统崩溃是高级Linux运维工程师很头疼的事情，但也不要害怕，前辈们已经给我们磨了好多把刀，其中有一把叫做Kdump。目前在主流的Linux发行版如Ubuntu、Centos等已经默认安装或者仓库中含有工具包，但是经过裁剪的嵌入式Linux里面没有包含Kdump。本文章使用busybox建立最小文件系统，讲述移植Kdump至嵌入式ARM64的环境中。
一、使用busybox建立最小文件系统
这个我就不讲了，度娘上一大堆文章。建立rootfs文件夹，将busybox下的 _install内容全部拷贝至rootfs中。
二、编译Kdump工具
（1）从下面的网站下载最新的kexec-tools源码包。
http://kernel.org/pub/linux/utils/kernel/kexec/kexec-tools.tar.gz
（2）解压源码包。
tar xvpzf kexec-tools.tar.gz
（3）进入到kexec-tools中，并进行配置。
LDFLAGS=-static ./configure ARCH=arm64 --build=x86_64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu --without-xen
这里使用静态编译。
（4）然后使用make进行编译。
Make
（5）将build目录下生成文件拷贝至rootfs文件夹中。
三、内核配置
（1）在"Processor type and features.“下使能"kexec system call”。
CONFIG_KEXEC=y
（2）在"Filesystem" -> “Pseudo filesystems.“下使能"sysfs file system support”。
CONFIG_SYSFS=y
（3）在"Kernel hacking.“下使能"Compile the kernel with debug info” 。
CONFIG_DEBUG_INFO=Y
（4）在"Processor type and features"下使能"kernel crash dumps”。
CONFIG_CRASH_DUMP=y
（5）在"Filesystems" -> “Pseudo filesystems"下使能”/proc/vmcore support"。
CONFIG_PROC_VMCORE=y
然后重新编译内核，生成Image。
四、建立文件系统镜像
（1）建立IMG文件。
dd if=/dev/zero of=rootfs.img bs=1M count=4k oflag=direct
（2）将IMG格式化为ext4格式。
mkfs.ext4 rootfs.img
（3）建立ubuntu-mount文件夹。
mkdir ubuntu-mount
（4）将IMG与文件夹进行挂载。
mount rootfs.img ubuntu-mount
（5）将rootfs里面的内容复制到IMG中。
cp -rfp rootfs/* ubuntu-mount/
（6）取消IMG和文件夹的挂载。
umount ubuntu-mount
（7）检查文件系统的正确性。
e2fsck -p -f rootfs.img
五、实验操作
（1）当我们做完以上的工作后，就可以通过QEMU启动ARM64的最小系统了。命令如下：
qemu-system-aarch64 -machine virt -M virt,gic_version=3 -cpu cortex-a57 -machine type=virt -nographic -smp 2 -m 1024 -kernel ./Image -hda ./rootfs.img -append “console=ttyAMA0 root=/dev/vda rw crashkernel=128M nr_cpus=2”
以下是我实验的启动流程：
root@ubuntu:/home/alvin/share/busybox_arm64# qemu-system-aarch64 -machine virt -M virt,gic_version=3 -cpu cortex-a57 -machine type=virt -nographic -smp 2 -m 1024 -kernel ./Image -hda ./rootfs.img -append “console=ttyAMA0 root=/dev/vda rw crashkernel=128M nr_cpus=2”
WARNING: Image format was not specified for ‘./rootfs.img’ and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the ‘raw’ format explicitly to remove the restrictions.
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 5.0.7 (root@ubuntu) (gcc version 7.3.0 (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04)) #12 SMP PREEMPT Sat May 11 15:14:29 PDT 2019
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] crashkernel reserved: 0x0000000077e00000 - 0x000000007fe00000 (128 MB)
[ 0.000000] cma: Reserved 32 MiB at 0x0000000075c00000
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x77df4840-0x77df5fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv0.2 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: Trusted OS migration not required
[ 0.000000] random: get_random_bytes called from start_kernel+0xa8/0x40c with crng_init=0
[ 0.000000] percpu: Embedded 23 pages/cpu @(ptrval) s56024 r8192 d29992 u94208
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: detected: ARM erratum 832075
[ 0.000000] CPU features: detected: ARM erratum 834220
[ 0.000000] CPU features: detected: EL2 vector hardening
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 258048
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: console=ttyAMA0 root=/dev/vda rw crashkernel=128M nr_cpus=2
[ 0.000000] Memory: 845240K/1048576K available (11004K kernel code, 1604K rwdata, 5332K rodata, 1408K init, 384K bss, 170568K reserved, 32768K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GICv3: Distributor has no Range Selector support
[ 0.000000] GICv3: no VLPI support, no direct LPI support
[ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000
[ 0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
[ 0.000174] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
[ 0.008201] Console: colour dummy device 80x25
[ 0.010926] Calibrating delay loop (skipped), value calculated using timer frequency… 125.00 BogoMIPS (lpj=250000)
[ 0.011187] pid_max: default: 32768 minimum: 301
[ 0.012558] LSM: Security Framework initializing
[ 0.014434] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.015704] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.015999] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.016112] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.082658] ASID allocator initialised with 32768 entries
[ 0.091943] rcu: Hierarchical SRCU implementation.
[ 0.107577] EFI services will not be available.
[ 0.116690] smp: Bringing up secondary CPUs …
[ 0.158098] Detected PIPT I-cache on CPU1
[ 0.158751] GICv3: CPU1: found redistributor 1 region 0:0x00000000080c0000
[ 0.159082] CPU1: Booted secondary processor 0x0000000001 [0x411fd070]
[ 0.163232] smp: Brought up 1 node, 2 CPUs
[ 0.163311] SMP: Total of 2 processors activated.
[ 0.163389] CPU features: detected: GIC system register CPU interface
[ 0.163424] CPU features: detected: 32-bit EL0 Support
[ 0.163520] CPU features: detected: CRC32 instructions
[ 0.171461] CPU: All CPU(s) started at EL1
[ 0.171777] alternatives: patching kernel code
[ 0.200862] devtmpfs: initialized
[ 0.220424] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.220680] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 0.228215] pinctrl core: initialized pinctrl subsystem
[ 0.251460] DMI not present or invalid.
[ 0.262311] NET: Registered protocol family 16
[ 0.266512] audit: initializing netlink subsys (disabled)
[ 0.269487] audit: type=2000 audit(0.208:1): state=initialized audit_enabled=0 res=1
[ 0.285140] cpuidle: using governor menu
[ 0.287626] vdso: 2 pages (1 code @ (ptrval), 1 data @ (ptrval))
[ 0.287785] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.297348] DMA: preallocated 256 KiB pool for atomic allocations
[ 0.313825] Serial: AMBA PL011 UART driver
[ 0.381830] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 39, base_baud = 0) is a PL011 rev1
[ 0.490940] printk: console [ttyAMA0] enabled
[ 0.651007] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.651932] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.652686] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.653521] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[ 0.660698] cryptd: max_cpu_qlen set to 1000
[ 0.676782] ACPI: Interpreter disabled.
[ 0.699521] vgaarb: loaded
[ 0.703209] SCSI subsystem initialized
[ 0.718287] usbcore: registered new interface driver usbfs
[ 0.720782] usbcore: registered new interface driver hub
[ 0.723444] usbcore: registered new device driver usb
[ 0.733534] pps_core: LinuxPPS API ver. 1 registered
[ 0.734813] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [email protected]
[ 0.736854] PTP clock support registered
[ 0.746659] EDAC MC: Ver: 3.0.0
[ 0.757551] Advanced Linux Sound Architecture Driver Initialized.
[ 0.776621] clocksource: Switched to clocksource arch_sys_counter
[ 0.782977] VFS: Disk quotas dquot_6.6.0
[ 0.784515] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.794319] pnp: PnP ACPI: disabled
[ 0.867760] NET: Registered protocol family 2
[ 0.878383] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes)
[ 0.879454] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.880970] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[ 0.881576] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.884265] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.885425] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.887818] NET: Registered protocol family 1
[ 0.896387] RPC: Registered named UNIX socket transport module.
[ 0.910707] RPC: Registered udp transport module.
[ 0.911119] RPC: Registered tcp transport module.
[ 0.912991] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.923200] hw perfevents: enabled with armv8_pmuv3 PMU driver, 5 counters available
[ 0.925509] kvm [1]: HYP mode not available
[ 0.992384] Initialise system trusted keyrings
[ 0.997978] workingset: timestamp_bits=44 max_order=18 bucket_order=0
[ 1.060250] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 1.069144] NFS: Registering the id_resolver key type
[ 1.069994] Key type id_resolver registered
[ 1.070221] Key type id_legacy registered
[ 1.070831] nfs4filelayout_init: NFSv4 File Layout Driver Registering…
[ 1.073270] 9p: Installing v9fs 9p2000 file system support
[ 1.129431] Key type asymmetric registered
[ 1.129886] Asymmetric key parser ‘x509’ registered
[ 1.130961] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[ 1.131756] io scheduler mq-deadline registered
[ 1.134068] io scheduler kyber registered
[ 1.193670] pl061_gpio 9030000.pl061: PL061 GPIO chip @0x0000000009030000 registered
[ 1.206992] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:
[ 1.208269] pci-host-generic 4010000000.pcie: IO 0x3eff0000…0x3effffff -> 0x00000000
[ 1.210415] pci-host-generic 4010000000.pcie: MEM 0x10000000…0x3efeffff -> 0x10000000
[ 1.211020] pci-host-generic 4010000000.pcie: MEM 0x8000000000…0xffffffffff -> 0x8000000000
[ 1.213354] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]
[ 1.216257] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00
[ 1.218452] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.219166] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 1.219642] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]
[ 1.220051] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[ 1.236339] pci 0000:00:01.0: BAR 6: assigned [mem 0x10000000-0x1003ffff pref]
[ 1.238306] pci 0000:00:01.0: BAR 4: assigned [mem 0x8000000000-0x8000003fff 64bit pref]
[ 1.239271] pci 0000:00:02.0: BAR 4: assigned [mem 0x8000004000-0x8000007fff 64bit pref]
[ 1.240016] pci 0000:00:02.0: BAR 0: assigned [io 0x1000-0x107f]
[ 1.241310] pci 0000:00:01.0: BAR 0: assigned [io 0x1080-0x109f]
[ 1.263130] EINJ: ACPI disabled.
[ 1.320069] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)
[ 1.325531] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)
[ 1.372240] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 1.391891] SuperH (H)SCI(F) driver initialized
[ 1.396244] msm_serial: driver initialized
[ 1.407937] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 1.466535] loop: module loaded
[ 1.478452] virtio_blk virtio1: [vda] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB)
[ 1.543860] libphy: Fixed MDIO Bus: probed
[ 1.552503] tun: Universal TUN/TAP device driver, 1.6
[ 1.570756] thunder_xcv, ver 1.0
[ 1.571914] thunder_bgx, ver 1.0
[ 1.573689] nicpf, ver 1.0
[ 1.579220] hclge is initializing
[ 1.581401] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[ 1.582293] hns3: Copyright © 2017 Huawei Corporation.
[ 1.583795] e1000e: Intel® PRO/1000 Network Driver - 3.2.6-k
[ 1.585393] e1000e: Copyright© 1999 - 2015 Intel Corporation.
[ 1.587092] igb: Intel® Gigabit Ethernet Network Driver - version 5.4.0-k
[ 1.589120] igb: Copyright © 2007-2014 Intel Corporation.
[ 1.590132] igbvf: Intel® Gigabit Virtual Function Network Driver - version 2.4.0-k
[ 1.591243] igbvf: Copyright © 2009 - 2012 Intel Corporation.
[ 1.596258] sky2: driver version 1.30
[ 1.601774] VFIO - User Level meta-driver version: 0.3
[ 1.617845] ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
[ 1.619470] ehci-pci: EHCI PCI platform driver
[ 1.621804] ehci-platform: EHCI generic platform driver
[ 1.623722] ehci-orion: EHCI orion driver
[ 1.625425] ehci-exynos: EHCI EXYNOS driver
[ 1.627358] ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver
[ 1.629208] ohci-pci: OHCI PCI platform driver
[ 1.630863] ohci-platform: OHCI generic platform driver
[ 1.634266] ohci-exynos: OHCI EXYNOS driver
[ 1.640986] usbcore: registered new interface driver usb-storage
[ 1.670598] rtc-pl031 9010000.pl031: registered as rtc0
[ 1.677241] i2c /dev entries driver
[ 1.715681] sdhci: Secure Digital Host Controller Interface driver
[ 1.718305] sdhci: Copyright© Pierre Ossman
[ 1.723334] Synopsys Designware Multimedia Card Interface Driver
[ 1.731635] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.742333] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.767205] usbcore: registered new interface driver usbhid
[ 1.768038] usbhid: USB HID core driver
[ 1.801497] NET: Registered protocol family 17
[ 1.805294] 9pnet: Installing 9P2000 support
[ 1.806524] Key type dns_resolver registered
[ 1.812974] registered taskstats version 1
[ 1.813598] Loading compiled-in X.509 certificates
[ 1.827961] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[ 1.834872] rtc-pl031 9010000.pl031: setting system clock to 2019-05-14T12:07:42 UTC (1557835662)
[ 1.846911] ALSA device list:
[ 1.847890] No soundcards found.
[ 1.857713] uart-pl011 9000000.pl011: no DMA platform data
[ 1.916602] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[ 1.918270] VFS: Mounted root (ext4 filesystem) on device 254:0.
[ 1.923838] devtmpfs: mounted
[ 2.225180] Freeing unused kernel memory: 1408K
[ 2.227492] Run /sbin/init as init process

Please press Enter to activate this console.

Processing /etc/profile… Done

/ #
在QEMU的启动参数里面配置crashkernel=128M，通过cat /proc/iomem 可以查看预留内存是否成功。
/ # cat /proc/iomem
09000000-09000fff : pl011@9000000
09000000-09000fff : pl011@9000000
09010000-09010fff : pl031@9010000
09010000-09010fff : rtc-pl031
09030000-09030fff : pl061@9030000
09030000-09030fff : pl061@9030000
10000000-3efeffff : pcie@10000000
10000000-1003ffff : 0000:00:01.0
40000000-7fffffff : System RAM
40080000-4107ffff : Kernel code
41080000-411dffff : reserved
411e0000-413d4fff : Kernel data
48000000-480fffff : reserved
74c00000-77bfffff : reserved
77dbe000-77dbefff : reserved
77dbf000-77decfff : reserved
77def000-77df0fff : reserved
77df1000-77df1fff : reserved
77df2000-77df6fff : reserved
77df7000-77dfefff : reserved
77dff000-77dfffff : reserved
77e00000-7fdfffff : Crash kernel
7fe00000-7fffffff : reserved
4010000000-401fffffff : PCI ECAM
8000000000-ffffffffff : pcie@10000000
8000000000-8000003fff : 0000:00:01.0
8000000000-8000003fff : virtio-pci-modern
8000004000-8000007fff : 0000:00:02.0
8000004000-8000007fff : virtio-pci-modern

Crashkernel的内存选择可以看以下内核Documentation\kdump\ kdump.txt文件。
Extended crashkernel syntax
While the “crashkernel=size[@offset]” syntax is sufficient for most
configurations, sometimes it’s handy to have the reserved memory dependent
on the value of System RAM – that’s mostly for distributors that pre-setup
the kernel command line to avoid a unbootable system after some memory has
been removed from the machine.
The syntax is:
crashkernel=:[,:,…][@offset]
range=start-[end]

For example:

crashkernel=512M-2G:64M,2G-:128M

This would mean:
1) if the RAM is smaller than 512M, then don’t reserve anything
(this is the “rescue” case)
2) if the RAM size is between 512M and 2G (exclusive), then reserve 64M
3) if the RAM size is larger than 2G, then reserve 128M

（2）然后输入以下命令
/ # kexec -p --command-line="console=ttyAMA0 root=/dev/vda rw nr_cpus=2 nr_cpus=1 " /Image

（3）进行触发崩溃的操作
echo c > /proc/sysrq-trigger
当触发完成后，会打印崩溃信息，并快速启动第二个内核。打印流程如下：
/ # echo c > /proc/sysrq-trigger
[ 127.702257] sysrq: SysRq : Trigger a crash
[ 127.703189] Kernel panic - not syncing: sysrq triggered crash
[ 127.704394] CPU: 0 PID: 1477 Comm: sh Kdump: loaded Not tainted 5.0.7 #12
[ 127.705172] Hardware name: linux,dummy-virt (DT)
[ 127.706310] Call trace:
[ 127.706555] dump_backtrace+0x0/0x178
[ 127.707455] show_stack+0x14/0x20
[ 127.708767] dump_stack+0x90/0xb4
[ 127.710017] panic+0x13c/0x2ec
[ 127.711068] sysrq_handle_reboot+0x0/0x18
[ 127.711595] __handle_sysrq+0xa8/0x1a0
[ 127.712287] write_sysrq_trigger+0x64/0x80
[ 127.712856] proc_reg_write+0x60/0xa0
[ 127.713542] __vfs_write+0x30/0x180
[ 127.713927] vfs_write+0xa4/0x1b0
[ 127.714907] ksys_write+0x60/0xd8
[ 127.715649] __arm64_sys_write+0x18/0x20
[ 127.716322] el0_svc_common+0x60/0x100
[ 127.716675] el0_svc_handler+0x2c/0x80
[ 127.717526] el0_svc+0x8/0xc
[ 127.719019] SMP: stopping secondary CPUs
[ 127.721121] Starting crashdump kernel…
[ 127.721999] Bye!
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 5.0.7 (root@ubuntu) (gcc version 7.3.0 (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04)) #12 SMP PREEMPT Sat May 11 15:14:29 PDT 2019
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] Reserving 1KB of memory at 0x7fdff000 for elfcorehdr
[ 0.000000] cma: Reserved 32 MiB at 0x000000007dc00000
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000077e00000-0x000000007fdfffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x7fdb3840-0x7fdb4fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000077e00000-0x000000007fdfffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000077e00000-0x000000007fdfffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000077e00000-0x000000007fdfffff]
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv0.2 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: Trusted OS migration not required
[ 0.000000] Number of cores (2) exceeds configured maximum of 1 - clipping
[ 0.000000] random: get_random_bytes called from start_kernel+0xa8/0x40c with crng_init=0
[ 0.000000] percpu: Embedded 23 pages/cpu @(ptrval) s56024 r8192 d29992 u94208
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: detected: ARM erratum 832075
[ 0.000000] CPU features: detected: ARM erratum 834220
[ 0.000000] CPU features: detected: EL2 vector hardening
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 32256
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: console=ttyAMA0 root=/dev/vda rw nr_cpus=2 nr_cpus=1
[ 0.000000] Memory: 61708K/131072K available (11004K kernel code, 1604K rwdata, 5332K rodata, 1408K init, 384K bss, 36596K reserved, 32768K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GICv3: Distributor has no Range Selector support
[ 0.000000] GICv3: no VLPI support, no direct LPI support
[ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000
[ 0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
[ 0.000105] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
[ 0.008090] Console: colour dummy device 80x25
[ 0.010680] Calibrating delay loop (skipped), value calculated using timer frequency… 125.00 BogoMIPS (lpj=250000)
[ 0.010848] pid_max: default: 32768 minimum: 301
[ 0.012233] LSM: Security Framework initializing
[ 0.012974] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
[ 0.013178] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.013406] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
[ 0.013453] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
[ 0.077223] ASID allocator initialised with 32768 entries
[ 0.086200] rcu: Hierarchical SRCU implementation.
[ 0.102112] EFI services will not be available.
[ 0.111226] smp: Bringing up secondary CPUs …
[ 0.111363] smp: Brought up 1 node, 1 CPU
[ 0.111411] SMP: Total of 1 processors activated.
[ 0.111561] CPU features: detected: GIC system register CPU interface
[ 0.111608] CPU features: detected: 32-bit EL0 Support
[ 0.111703] CPU features: detected: CRC32 instructions
[ 0.117106] CPU: All CPU(s) started at EL1
[ 0.117762] alternatives: patching kernel code
[ 0.140863] devtmpfs: initialized
[ 0.162106] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.162323] futex hash table entries: 256 (order: 2, 16384 bytes)
[ 0.175144] pinctrl core: initialized pinctrl subsystem
[ 0.219055] DMI not present or invalid.
[ 0.229449] NET: Registered protocol family 16
[ 0.233400] audit: initializing netlink subsys (disabled)
[ 0.237532] audit: type=2000 audit(0.180:1): state=initialized audit_enabled=0 res=1
[ 0.259222] cpuidle: using governor menu
[ 0.266291] vdso: 2 pages (1 code @ (ptrval), 1 data @ (ptrval))
[ 0.267049] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.277356] DMA: preallocated 256 KiB pool for atomic allocations
[ 0.298623] Serial: AMBA PL011 UART driver
[ 0.395531] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 39, base_baud = 0) is a PL011 rev1
[ 0.491048] printk: console [ttyAMA0] enabled
[ 0.740496] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.741311] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.742767] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.743413] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[ 0.751955] cryptd: max_cpu_qlen set to 1000
[ 0.764345] ACPI: Interpreter disabled.
[ 0.788136] vgaarb: loaded
[ 0.795412] SCSI subsystem initialized
[ 0.811495] usbcore: registered new interface driver usbfs
[ 0.812958] usbcore: registered new interface driver hub
[ 0.814128] usbcore: registered new device driver usb
[ 0.832093] pps_core: LinuxPPS API ver. 1 registered
[ 0.832387] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [email protected]
[ 0.835593] PTP clock support registered
[ 0.837876] EDAC MC: Ver: 3.0.0
[ 0.863939] Advanced Linux Sound Architecture Driver Initialized.
[ 0.886023] clocksource: Switched to clocksource arch_sys_counter
[ 0.888000] VFS: Disk quotas dquot_6.6.0
[ 0.888922] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.897723] pnp: PnP ACPI: disabled
[ 1.009366] NET: Registered protocol family 2
[ 1.019164] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes)
[ 1.020359] TCP established hash table entries: 1024 (order: 1, 8192 bytes)
[ 1.021504] TCP bind hash table entries: 1024 (order: 2, 16384 bytes)
[ 1.022935] TCP: Hash tables configured (established 1024 bind 1024)
[ 1.025222] UDP hash table entries: 256 (order: 1, 8192 bytes)
[ 1.026914] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[ 1.029588] NET: Registered protocol family 1
[ 1.050538] RPC: Registered named UNIX socket transport module.
[ 1.052753] RPC: Registered udp transport module.
[ 1.053121] RPC: Registered tcp transport module.
[ 1.053526] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1.069153] hw perfevents: enabled with armv8_pmuv3 PMU driver, 5 counters available
[ 1.071736] kvm [1]: HYP mode not available
[ 1.131574] Initialise system trusted keyrings
[ 1.134596] workingset: timestamp_bits=44 max_order=15 bucket_order=0
[ 1.235184] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 1.249084] NFS: Registering the id_resolver key type
[ 1.249858] Key type id_resolver registered
[ 1.252719] Key type id_legacy registered
[ 1.253450] nfs4filelayout_init: NFSv4 File Layout Driver Registering…
[ 1.256423] 9p: Installing v9fs 9p2000 file system support
[ 1.303254] Key type asymmetric registered
[ 1.303654] Asymmetric key parser ‘x509’ registered
[ 1.305720] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[ 1.308168] io scheduler mq-deadline registered
[ 1.309265] io scheduler kyber registered
[ 1.407589] pl061_gpio 9030000.pl061: PL061 GPIO chip @0x0000000009030000 registered
[ 1.427945] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:
[ 1.429773] pci-host-generic 4010000000.pcie: IO 0x3eff0000…0x3effffff -> 0x00000000
[ 1.433178] pci-host-generic 4010000000.pcie: MEM 0x10000000…0x3efeffff -> 0x10000000
[ 1.435046] pci-host-generic 4010000000.pcie: MEM 0x8000000000…0xffffffffff -> 0x8000000000
[ 1.437441] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]
[ 1.441159] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00
[ 1.443696] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.444684] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 1.445111] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]
[ 1.445505] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[ 1.477322] pci 0000:00:01.0: BAR 6: assigned [mem 0x10000000-0x1003ffff pref]
[ 1.479077] pci 0000:00:01.0: BAR 4: assigned [mem 0x8000000000-0x8000003fff 64bit pref]
[ 1.480625] pci 0000:00:02.0: BAR 4: assigned [mem 0x8000004000-0x8000007fff 64bit pref]
[ 1.482593] pci 0000:00:02.0: BAR 0: assigned [io 0x1000-0x107f]
[ 1.483426] pci 0000:00:01.0: BAR 0: assigned [io 0x1080-0x109f]
[ 1.513825] EINJ: ACPI disabled.
[ 1.668632] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 1.703495] SuperH (H)SCI(F) driver initialized
[ 1.709393] msm_serial: driver initialized
[ 1.721495] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 1.780444] loop: module loaded
[ 1.791479] virtio_blk virtio1: [vda] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB)
[ 1.886915] libphy: Fixed MDIO Bus: probed
[ 1.893437] tun: Universal TUN/TAP device driver, 1.6
[ 1.923676] thunder_xcv, ver 1.0
[ 1.928028] thunder_bgx, ver 1.0
[ 1.929089] nicpf, ver 1.0
[ 1.935420] hclge is initializing
[ 1.936083] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[ 1.937151] hns3: Copyright © 2017 Huawei Corporation.
[ 1.941014] e1000e: Intel® PRO/1000 Network Driver - 3.2.6-k
[ 1.943745] e1000e: Copyright© 1999 - 2015 Intel Corporation.
[ 1.944904] igb: Intel® Gigabit Ethernet Network Driver - version 5.4.0-k
[ 1.945775] igb: Copyright © 2007-2014 Intel Corporation.
[ 1.948065] igbvf: Intel® Gigabit Virtual Function Network Driver - version 2.4.0-k
[ 1.948854] igbvf: Copyright © 2009 - 2012 Intel Corporation.
[ 1.954323] sky2: driver version 1.30
[ 1.965789] VFIO - User Level meta-driver version: 0.3
[ 1.995451] ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
[ 1.996575] ehci-pci: EHCI PCI platform driver
[ 2.000118] ehci-platform: EHCI generic platform driver
[ 2.002008] ehci-orion: EHCI orion driver
[ 2.005424] ehci-exynos: EHCI EXYNOS driver
[ 2.022126] ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver
[ 2.023849] ohci-pci: OHCI PCI platform driver
[ 2.026236] ohci-platform: OHCI generic platform driver
[ 2.028953] ohci-exynos: OHCI EXYNOS driver
[ 2.035819] usbcore: registered new interface driver usb-storage
[ 2.077659] rtc-pl031 9010000.pl031: registered as rtc0
[ 2.086105] i2c /dev entries driver
[ 2.147938] sdhci: Secure Digital Host Controller Interface driver
[ 2.148616] sdhci: Copyright© Pierre Ossman
[ 2.155715] Synopsys Designware Multimedia Card Interface Driver
[ 2.168046] sdhci-pltfm: SDHCI platform and OF driver helper
[ 2.186261] ledtrig-cpu: registered to indicate activity on CPUs
[ 2.212538] usbcore: registered new interface driver usbhid
[ 2.226491] usbhid: USB HID core driver
[ 2.264339] NET: Registered protocol family 17
[ 2.269099] 9pnet: Installing 9P2000 support
[ 2.271692] Key type dns_resolver registered
[ 2.278302] registered taskstats version 1
[ 2.280976] Loading compiled-in X.509 certificates
[ 2.299717] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[ 2.306642] rtc-pl031 9010000.pl031: setting system clock to 2019-05-14T12:09:54 UTC (1557835794)
[ 2.312154] ALSA device list:
[ 2.312482] No soundcards found.
[ 2.317908] uart-pl011 9000000.pl011: no DMA platform data
[ 2.397763] EXT4-fs (vda): recovery complete
[ 2.402662] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[ 2.404383] VFS: Mounted root (ext4 filesystem) on device 254:0.
[ 2.409448] devtmpfs: mounted
[ 2.712434] Freeing unused kernel memory: 1408K
[ 2.715067] Run /sbin/init as init process

Please press Enter to activate this console.

Processing /etc/profile… Done

（4）然后将proc目录下的vmcore文件拷贝至你的存储设备中即可，vmcore就是第一个内核崩溃时的内存镜像。
这里提供一个脚本
#!/bin/sh
KDUMP_PATH="/home"
function save_dump_file()
{
coredir="${KDUMP_PATH}/127.0.0.1-date +"%Y-%m-%d-%H:%M""
mkdir -p $coredir
if [ ! -f /sbin/vmcore-dmesg ];then
echo “Skipping saving vmcore-dmesg.txt. File /sbin/vmcore-dmesg is not present”
return;
fi
echo “kdump: saving vmcore-dmesg.txt to $coredir”
/sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg.txt
cp /proc/vmcore $coredir
}

if [ -s /proc/vmcore ];then
echo “save core dump file…”
save_dump_file
reboot
else
# kexec setting
kexec -p --command-line="console=ttyAMA0 root=/dev/vda rw nr_cpus=2 nr_cpus=1 " /Image
fi
该脚本的首先判断如果存在/proc/vmcore，则转储core dump文件到dump目录，否则就使用kexec命令来加载捕捉内核。可以将以上脚本设置为系统启动自动运行即可，dump出来的文件会保存在KDUMP_PATH目录下。
接下来就是通过crash或者gdb工具进行分析。
ubuntu和centos等操作系统比较还提供一个工具叫makedumpfile的工具对vmcore进行压缩，以减少体积。这个工具我只试了X86_64下的，在arm64下会有点难搞，具体原由可以看其README。