副本技能-记录一次OOM的异常处理
问题定位
1.服务挂的现象出现！
2.检查发现并未出现OOM的溢出DUMP文件产生
3.检查是否是被Linux系统给杀了
服务器环境查询步骤
- 服务启动脚本查看
// 查看Shell脚本
[vevor@v169 bmp-crm-manager]$ cat start.sh
# 脚本内容如下
#!/bin/bash
DATE=$(date +%Y%m%d)
source /etc/profile
DIR=/home/vevor/vevor-bmp-packages/bmp-crm-manager/
cd $DIR
# 获取最新的jar包
JAR=`ls -lt *-spring-boot.jar | head -1 | awk '{print $9}'`
# 获取当前项目的进程的进程号
PID=`ps -ef | grep bmp-crm-service | grep -v grep | awk '{print $2}'`
# 项目dump文件根据启动时候的日期确定
DUMPNAME=`date "+%Y_%m_%d_%H_%M_%S_DUMP"`
# 有进程号就杀进程后再启动新的，没有就直接启动
if [ "$PID" != "" ];then
kill -9 $PID
nohup java -jar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DIR$DUMPNAME \
-Dspring.cloud.consul.host=172.19.225.168 \
-Dspring.cloud.consul.port=8500 \
-Dspring.cloud.consul.config.default-context=test \
-Dspring.cloud.consul.config.data-key=lichenghuai $DIR$JAR > /dev/null 2>&1 &
else
nohup java -jar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DIR$DUMPNAME \
-Dspring.cloud.consul.host=172.19.225.168 \
-Dspring.cloud.consul.port=8500 \
-Dspring.cloud.consul.config.default-context=test \
-Dspring.cloud.consul.config.data-key=lichenghuai $DIR$JAR > /dev/null 2>&1 &
fi
2.启动后的服务进程号查看并记录
3. 等待服务挂后查看日志【1.么有DUMP产生】
4. 扩大日志搜索范围【看看是不是Linux干的好事】
命令：dmesg | egrep -i -B100 ‘Kill process’
我们的进程号：9445
发现咱的进程确实被杀，可是到底啥时候被杀的呢？
命令：date -d “1970-01-01 UTC echo "$(date +%s)-$(cat /proc/uptime|cut -f 1 -d' ')+25094516.126539"|bcseconds”
时间转化成可以看懂的！！！
total-vm是指所有可用内存，约8G
anon-rss + file-rss 是该进程实际使用的内存，约2.5G，
这个进程实际使用了3.5GB左右内存，为什么会导致被oom killer杀掉呢？
应该是其他进程导致的，我注意到其中有个vim进程（7788），占用的rss内存页个数是814178，内存页大小为4KB，那么这个vim进程占用了814178*4/1024 = 3180MB，那么原因就很明显了，有人用vim打开了一个很大的文件，导致内存耗尽，最终oom killer登场杀死了java进程。
6. 死亡原因报告说明
oom killer是如何来选择一个被终止掉的进程？
原来oom killer巡视一遍所有进程后就给每个进程分配一个分数，也就是"badness score"，坏蛋分，分数最高的那个将被kill掉，打分的原则简单讲就是占用内存越高的那个进程得分就越高，很显然我们的进程就是得分最高的那个，然后就被kill掉了，没有留下任何线索。
【处理方案】
限制内存使用参数，避免被进程杀死：nohup java -Xms1g -Xmx1g -jar
【完整日志】
[25094516.126455] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[25094516.126462] [ 628] 0 628 13962 100 32 0 0 systemd-journal
[25094516.126465] [ 654] 0 654 11165 118 22 0 -1000 systemd-udevd
[25094516.126467] [ 987] 0 987 13882 113 28 0 -1000 auditd
[25094516.126470] [ 1052] 999 1052 153086 1658 62 0 0 polkitd
[25094516.126472] [ 1058] 0 1058 6701 219 19 0 0 systemd-logind
[25094516.126475] [ 1059] 81 1059 14530 158 32 0 -900 dbus-daemon
[25094516.126477] [ 1063] 998 1063 29482 133 28 0 0 chronyd
[25094516.126479] [ 1296] 0 1296 25724 515 48 0 0 dhclient
[25094516.126481] [ 1354] 0 1354 143549 3329 99 0 0 tuned
[25094516.126483] [ 1500] 0 1500 177430 446 156 0 0 rsyslogd
[25094516.126485] [ 1509] 0 1509 6477 53 17 0 0 atd
[25094516.126488] [ 1511] 0 1511 31573 162 20 0 0 crond
[25094516.126490] [ 1517] 0 1517 27527 34 12 0 0 agetty
[25094516.126492] [ 1518] 0 1518 27527 33 10 0 0 agetty
[25094516.126494] [ 2038] 0 2038 5717 88 15 0 0 argusagent
[25094516.126497] [ 2040] 0 2040 253445 2881 54 0 0 /usr/local/clou
[25094516.126499] [ 3104] 0 3104 28230 256 57 0 -1000 sshd
[25094516.126501] [27621] 1000 27621 1189200 180693 458 0 0 java
[25094516.126503] [ 1690] 1000 1690 1248350 398534 925 0 0 java
[25094516.126505] [ 5052] 0 5052 109229 148 27 0 0 AliSecGuard
[25094516.126507] [ 947] 0 947 201651 1497 13 0 0 aliyun-service
[25094516.126509] [ 1069] 0 1069 4470 122 12 0 0 assist_daemon
[25094516.126511] [30180] 1000 30180 1187045 168005 436 0 0 java
[25094516.126513] [21557] 1000 21557 1229125 219482 551 0 0 java
[25094516.126516] [ 1364] 1000 1364 1233480 222741 560 0 0 java
[25094516.126518] [12203] 0 12203 20685 8176 38 0 0 AliYunDunUpdate
[25094516.126521] [22307] 1000 22307 1244476 270965 668 0 0 java
[25094516.126523] [ 2117] 1000 2117 1238545 255050 666 0 0 java
[25094516.126525] [17180] 1000 17180 1292373 245993 771 0 0 java
[25094516.126527] [ 915] 1000 915 1391175 269320 962 0 0 java
[25094516.126529] [11040] 1000 11040 1192170 180695 468 0 0 java
[25094516.126531] [14645] 0 14645 32641 1370 64 0 0 AliYunDun
[25094516.126533] [ 3603] 1000 3603 1233254 219157 587 0 0 java
[25094516.126535] [28583] 1000 28583 1201707 171353 483 0 0 java
[25094516.126537] [ 9415] 1000 9415 2073112 923377 2090 0 0 java
[25094516.126539] Out of memory: Kill process 9415 (java) score 237 or sacrifice child
[25094516.127920] Killed process 9415 (java), UID 1000, total-vm:8292448kB, anon-rss:3693380kB, file-rss:128kB, shmem-rss:0kB
[25140556.796701] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[25140556.796705] java cpuset=/ mems_allowed=0
[25140556.796708] CPU: 1 PID: 1549 Comm: java Tainted: G OE ------------ 3.10.0-1062.18.1.el7.x86_64 #1
[25140556.796710] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS e623647 04/01/2014
[25140556.796711] Call Trace:
[25140556.796720] [<ffffffff8597b416>] dump_stack+0x19/0x1b
[25140556.796728] [<ffffffff85975d8d>] dump_header+0x90/0x229
[25140556.796733] [<ffffffff85306392>] ? ktime_get_ts64+0x52/0xf0
[25140556.796737] [<ffffffff853c196e>] oom_kill_process+0x25e/0x3f0
[25140556.796740] [<ffffffff853330c1>] ? cpuset_mems_allowed_intersects+0x21/0x30
[25140556.796743] [<ffffffff853c13cd>] ? oom_unkillable_task+0xcd/0x120
[25140556.796745] [<ffffffff853c1476>] ? find_lock_task_mm+0x56/0xc0
[25140556.796747] [<ffffffff853c21c6>] out_of_memory+0x4b6/0x4f0
[25140556.796749] [<ffffffff859768aa>] __alloc_pages_slowpath+0x5db/0x729
[25140556.796751] [<ffffffff853c8636>] __alloc_pages_nodemask+0x436/0x450
[25140556.796755] [<ffffffff85416c58>] alloc_pages_current+0x98/0x110
[25140556.796757] [<ffffffff853bd877>] __page_cache_alloc+0x97/0xb0
[25140556.796760] [<ffffffff853c0430>] filemap_fault+0x270/0x420
[25140556.796774] [<ffffffffc0577356>] ext4_filemap_fault+0x36/0x50 [ext4]
[25140556.796778] [<ffffffff853ebdba>] __do_fault.isra.61+0x8a/0x100
[25140556.796780] [<ffffffff853ec36c>] do_read_fault.isra.63+0x4c/0x1b0
[25140556.796782] [<ffffffff853f3bd0>] handle_mm_fault+0xa20/0xfb0
[25140556.796787] [<ffffffff85312110>] ? futex_wake+0x90/0x180
[25140556.796792] [<ffffffff85988653>] __do_page_fault+0x213/0x500
[25140556.796794] [<ffffffff85988a26>] trace_do_page_fault+0x56/0x150
[25140556.796797] [<ffffffff85987fa2>] do_async_page_fault+0x22/0xf0
[25140556.796799] [<ffffffff859847a8>] async_page_fault+0x28/0x30
[25140556.796800] Mem-Info:
[25140556.796805] active_anon:3757967 inactive_anon:78 isolated_anon:0
active_file:999 inactive_file:2553 isolated_file:99
unevictable:0 dirty:19 writeback:0 unstable:0
slab_reclaimable:15833 slab_unreclaimable:11246
mapped:277 shmem:230 pagetables:10853 bounce:0
free:33154 free_pcp:949 free_cma:0
[25140556.796808] Node 0 DMA free:15908kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[25140556.796814] lowmem_reserve[]: 0 2813 15178 15178
[25140556.796816] Node 0 DMA32 free:61844kB min:12516kB low:15644kB high:18772kB active_anon:2739356kB inactive_anon:64kB active_file:448kB inactive_file:2688kB unevictable:0kB isolated(anon):0kB isolated(file):328kB present:3111608kB managed:2881176kB mlocked:0kB dirty:12kB writeback:0kB mapped:40kB shmem:208kB slab_reclaimable:22436kB slab_unreclaimable:11368kB kernel_stack:11536kB pagetables:5840kB unstable:0kB bounce:0kB free_pcp:2120kB local_pcp:632kB free_cma:0kB writeback_tmp:0kB pages_scanned:5702 all_unreclaimable? yes
[25140556.796821] lowmem_reserve[]: 0 0 12364 12364
[25140556.796824] Node 0 Normal free:54864kB min:54996kB low:68744kB high:82492kB active_anon:12292512kB inactive_anon:248kB active_file:3548kB inactive_file:7524kB unevictable:0kB isolated(anon):0kB isolated(file):68kB present:12931072kB managed:12661380kB mlocked:0kB dirty:64kB writeback:76kB mapped:1068kB shmem:712kB slab_reclaimable:40896kB slab_unreclaimable:33616kB kernel_stack:30512kB pagetables:37572kB unstable:0kB bounce:0kB free_pcp:1676kB local_pcp:68kB free_cma:0kB writeback_tmp:0kB pages_scanned:17171 all_unreclaimable? yes
[25140556.796828] lowmem_reserve[]: 0 0 0 0
[25140556.796831] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[25140556.796840] Node 0 DMA32: 190*4kB (UE) 246*8kB (UEM) 422*16kB (UEM) 194*32kB (UEM) 240*64kB (UEM) 144*128kB (UEM) 39*256kB (UEM) 1*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 62024kB
[25140556.796850] Node 0 Normal: 13699*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54796kB
[25140556.796857] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[25140556.796858] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[25140556.796859] 4105 total pagecache pages
[25140556.796861] 0 pages in swap cache
[25140556.796862] Swap cache stats: add 0, delete 0, find 0/0
[25140556.796863] Free swap = 0kB
[25140556.796864] Total swap = 0kB
[25140556.796865] 4014668 pages RAM
[25140556.796866] 0 pages HighMem/MovableOnly
[25140556.796867] 125052 pages reserved
[25140556.796868] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[25140556.796875] [ 628] 0 628 13962 102 33 0 0 systemd-journal
[25140556.796878] [ 654] 0 654 11165 118 22 0 -1000 systemd-udevd
[25140556.796880] [ 987] 0 987 13882 113 28 0 -1000 auditd
[25140556.796882] [ 1052] 999 1052 153086 1658 62 0 0 polkitd
[25140556.796885] [ 1058] 0 1058 6701 213 19 0 0 systemd-logind
[25140556.796887] [ 1059] 81 1059 14530 156 32 0 -900 dbus-daemon
[25140556.796889] [ 1063] 998 1063 29482 145 28 0 0 chronyd
[25140556.796891] [ 1296] 0 1296 25724 515 48 0 0 dhclient
[25140556.796893] [ 1354] 0 1354 143549 3329 99 0 0 tuned
[25140556.796896] [ 1500] 0 1500 177430 443 157 0 0 rsyslogd
[25140556.796898] [ 1509] 0 1509 6477 53 17 0 0 atd
[25140556.796900] [ 1511] 0 1511 31573 163 20 0 0 crond
[25140556.796902] [ 1517] 0 1517 27527 34 12 0 0 agetty
[25140556.796904] [ 1518] 0 1518 27527 33 10 0 0 agetty
[25140556.796906] [ 2038] 0 2038 5717 88 15 0 0 argusagent
[25140556.796908] [ 2040] 0 2040 253446 2893 54 0 0 /usr/local/clou
[25140556.796910] [ 3104] 0 3104 28230 256 57 0 -1000 sshd
[25140556.796913] [27621] 1000 27621 1189200 180693 458 0 0 java
[25140556.796915] [ 1690] 1000 1690 1248350 398534 925 0 0 java
[25140556.796917] [ 5052] 0 5052 109229 148 27 0 0 AliSecGuard
[25140556.796919] [ 947] 0 947 201651 1512 13 0 0 aliyun-service
[25140556.796921] [ 1069] 0 1069 4470 122 12 0 0 assist_daemon
[25140556.796923] [30180] 1000 30180 1187045 168008 436 0 0 java
[25140556.796925] [21557] 1000 21557 1229125 219506 551 0 0 java
[25140556.796927] [ 1364] 1000 1364 1233480 222750 560 0 0 java
[25140556.796929] [12203] 0 12203 20685 8176 38 0 0 AliYunDunUpdate
[25140556.796931] [22307] 1000 22307 1244476 271006 668 0 0 java
[25140556.796934] [ 2117] 1000 2117 1238545 262222 678 0 0 java
[25140556.796936] [17180] 1000 17180 1292373 247051 773 0 0 java
[25140556.796938] [ 915] 1000 915 1391175 269373 962 0 0 java
[25140556.796940] [11040] 1000 11040 1192170 181123 469 0 0 java
[25140556.796942] [14645] 0 14645 32641 1349 64 0 0 AliYunDun
[25140556.796944] [ 3603] 1000 3603 1233254 219423 587 0 0 java
[25140556.796946] [28583] 1000 28583 1201707 172433 483 0 0 java
[25140556.796948] [ 9445] 1000 9445 2070967 618667 1498 0 0 java
[25140556.796950] [14314] 0 14314 38677 346 78 0 0 sshd
[25140556.796951] [14317] 1000 14317 38677 326 75 0 0 sshd
[25140556.796953] [14318] 1000 14318 28895 140 15 0 0 bash
[25140556.796955] [15150] 0 15150 38677 346 77 0 0 sshd
[25140556.796957] [15153] 1000 15153 38907 550 75 0 0 sshd
[25140556.796959] [15154] 1000 15154 28895 137 13 0 0 bash
[25140556.796960] [15273] 1000 15273 317269 280581 578 0 0 vim
[25140556.796962] Out of memory: Kill process 9445 (java) score 159 or sacrifice child
[25140556.798288] Killed process 9445 (java), UID 1000, total-vm:8283868kB, anon-rss:2474472kB, file-rss:196kB, shmem-rss:0kB
[25140578.512344] AliYunDunUpdate invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[25140578.512348] AliYunDunUpdate cpuset=/ mems_allowed=0
[25140578.512351] CPU: 3 PID: 12203 Comm: AliYunDunUpdate Tainted: G OE ------------ 3.10.0-1062.18.1.el7.x86_64 #1
[25140578.512353] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS e623647 04/01/2014
[25140578.512354] Call Trace:
[25140578.512364] [<ffffffff8597b416>] dump_stack+0x19/0x1b
[25140578.512368] [<ffffffff85975d8d>] dump_header+0x90/0x229
[25140578.512373] [<ffffffff85306392>] ? ktime_get_ts64+0x52/0xf0
[25140578.512378] [<ffffffff853c196e>] oom_kill_process+0x25e/0x3f0
[25140578.512381] [<ffffffff853330c1>] ? cpuset_mems_allowed_intersects+0x21/0x30
[25140578.512384] [<ffffffff853c13cd>] ? oom_unkillable_task+0xcd/0x120
[25140578.512386] [<ffffffff853c1476>] ? find_lock_task_mm+0x56/0xc0
[25140578.512389] [<ffffffff853c21c6>] out_of_memory+0x4b6/0x4f0
[25140578.512391] [<ffffffff859768aa>] __alloc_pages_slowpath+0x5db/0x729
[25140578.512394] [<ffffffff853c8636>] __alloc_pages_nodemask+0x436/0x450
[25140578.512399] [<ffffffff85416c58>] alloc_pages_current+0x98/0x110
[25140578.512401] [<ffffffff853bd877>] __page_cache_alloc+0x97/0xb0
[25140578.512404] [<ffffffff853c0430>] filemap_fault+0x270/0x420
[25140578.512420] [<ffffffffc0577356>] ext4_filemap_fault+0x36/0x50 [ext4]
[25140578.512424] [<ffffffff853ebdba>] __do_fault.isra.61+0x8a/0x100
[25140578.512428] [<ffffffff852e6601>] ? put_prev_entity+0x31/0x400
[25140578.512430] [<ffffffff853ec36c>] do_read_fault.isra.63+0x4c/0x1b0
[25140578.512433] [<ffffffff853f3bd0>] handle_mm_fault+0xa20/0xfb0
[25140578.512438] [<ffffffff852ca42d>] ? hrtimer_start_range_ns+0x1fd/0x3c0
[25140578.512444] [<ffffffff85988653>] __do_page_fault+0x213/0x500
[25140578.512447] [<ffffffff85988a26>] trace_do_page_fault+0x56/0x150
[25140578.512450] [<ffffffff85987fa2>] do_async_page_fault+0x22/0xf0
[25140578.512452] [<ffffffff859847a8>] async_page_fault+0x28/0x30
[25140578.512454] Mem-Info:
[25140578.512460] active_anon:3764487 inactive_anon:76 isolated_anon:0
active_file:929 inactive_file:2890 isolated_file:83
unevictable:0 dirty:21 writeback:14 unstable:0
slab_reclaimable:15787 slab_unreclaimable:11254
mapped:293 shmem:222 pagetables:10595 bounce:0
free:33118 free_pcp:430 free_cma:0
[25140578.512464] Node 0 DMA free:15908kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[25140578.512470] lowmem_reserve[]: 0 2813 15178 15178
[25140578.512474] Node 0 DMA32 free:61748kB min:12516kB low:15644kB high:18772kB active_anon:2745676kB inactive_anon:64kB active_file:916kB inactive_file:4044kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:3111608kB managed:2881176kB mlocked:0kB dirty:52kB writeback:36kB mapped:172kB shmem:208kB slab_reclaimable:22340kB slab_unreclaimable:11368kB kernel_stack:10688kB pagetables:5836kB unstable:0kB bounce:0kB free_pcp:608kB local_pcp:236kB free_cma:0kB writeback_tmp:0kB pages_scanned:9201 all_unreclaimable? yes
[25140578.512480] lowmem_reserve[]: 0 0 12364 12364
[25140578.512482] Node 0 Normal free:54816kB min:54996kB low:68744kB high:82492kB active_anon:12312272kB inactive_anon:240kB active_file:2800kB inactive_file:7516kB unevictable:0kB isolated(anon):0kB isolated(file):204kB present:12931072kB managed:12661380kB mlocked:0kB dirty:32kB writeback:20kB mapped:1000kB shmem:680kB slab_reclaimable:40808kB slab_unreclaimable:33648kB kernel_stack:25488kB pagetables:36544kB unstable:0kB bounce:0kB free_pcp:1112kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:16466 all_unreclaimable? yes
[25140578.512488] lowmem_reserve[]: 0 0 0 0
[25140578.512491] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[25140578.512502] Node 0 DMA32: 380*4kB (UEM) 243*8kB (UEM) 468*16kB (UEM) 297*32kB (UE) 220*64kB (UEM) 109*128kB (UE) 24*256kB (UEM) 10*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 61800kB
[25140578.512514] Node 0 Normal: 13710*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54840kB
[25140578.512522] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[25140578.512524] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[25140578.512525] 4250 total pagecache pages
[25140578.512527] 0 pages in swap cache
[25140578.512528] Swap cache stats: add 0, delete 0, find 0/0
[25140578.512529] Free swap = 0kB
[25140578.512531] Total swap = 0kB
[25140578.512532] 4014668 pages RAM
[25140578.512533] 0 pages HighMem/MovableOnly
[25140578.512534] 125052 pages reserved
[25140578.512535] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[25140578.512543] [ 628] 0 628 13962 98 33 0 0 systemd-journal
[25140578.512546] [ 654] 0 654 11165 118 22 0 -1000 systemd-udevd
[25140578.512549] [ 987] 0 987 13882 113 28 0 -1000 auditd
[25140578.512552] [ 1052] 999 1052 153086 1658 62 0 0 polkitd
[25140578.512555] [ 1058] 0 1058 6701 197 19 0 0 systemd-logind
[25140578.512557] [ 1059] 81 1059 14530 139 32 0 -900 dbus-daemon
[25140578.512560] [ 1063] 998 1063 29482 141 28 0 0 chronyd
[25140578.512562] [ 1296] 0 1296 25724 515 48 0 0 dhclient
[25140578.512565] [ 1354] 0 1354 143549 3329 99 0 0 tuned
[25140578.512567] [ 1500] 0 1500 177430 443 157 0 0 rsyslogd
[25140578.512570] [ 1509] 0 1509 6477 53 17 0 0 atd
[25140578.512573] [ 1511] 0 1511 31573 160 20 0 0 crond
[25140578.512575] [ 1517] 0 1517 27527 34 12 0 0 agetty
[25140578.512578] [ 1518] 0 1518 27527 33 10 0 0 agetty
[25140578.512580] [ 2038] 0 2038 5717 88 15 0 0 argusagent
[25140578.512583] [ 2040] 0 2040 253445 2893 54 0 0 /usr/local/clou
[25140578.512585] [ 3104] 0 3104 28230 256 57 0 -1000 sshd
[25140578.512588] [27621] 1000 27621 1189200 180693 458 0 0 java
[25140578.512592] [ 1690] 1000 1690 1248350 398534 925 0 0 java
[25140578.512611] [ 5052] 0 5052 109229 148 27 0 0 AliSecGuard
[25140578.512613] [ 947] 0 947 201651 1512 13 0 0 aliyun-service
[25140578.512616] [ 1069] 0 1069 4470 122 12 0 0 assist_daemon
[25140578.512618] [30180] 1000 30180 1187045 168008 436 0 0 java
[25140578.512621] [21557] 1000 21557 1229125 219506 551 0 0 java
[25140578.512623] [ 1364] 1000 1364 1233480 222750 560 0 0 java
[25140578.512625] [12203] 0 12203 20685 8176 38 0 0 AliYunDunUpdate
[25140578.512628] [22307] 1000 22307 1244476 271006 668 0 0 java
[25140578.512630] [ 2117] 1000 2117 1238545 262222 678 0 0 java
[25140578.512632] [17180] 1000 17180 1292373 247051 773 0 0 java
[25140578.512634] [ 915] 1000 915 1391175 269373 962 0 0 java
[25140578.512637] [11040] 1000 11040 1192170 181123 469 0 0 java
[25140578.512639] [14645] 0 14645 32641 1349 64 0 0 AliYunDun
[25140578.512642] [ 3603] 1000 3603 1233254 219423 587 0 0 java
[25140578.512644] [28583] 1000 28583 1201707 172483 483 0 0 java
[25140578.512646] [14314] 0 14314 38677 339 78 0 0 sshd
[25140578.512648] [14317] 1000 14317 38677 336 75 0 0 sshd
[25140578.512650] [14318] 1000 14318 28895 135 15 0 0 bash
[25140578.512653] [15150] 0 15150 38677 339 77 0 0 sshd
[25140578.512655] [15153] 1000 15153 38907 543 75 0 0 sshd
[25140578.512657] [15154] 1000 15154 28895 132 13 0 0 bash
[25140578.512659] [15273] 1000 15273 950884 914042 1818 0 0 vim
[25140578.512661] Out of memory: Kill process 15273 (vim) score 235 or sacrifice child
[25140578.514172] Killed process 15273 (vim), UID 1000, total-vm:3803536kB, anon-rss:3656016kB, file-rss:152kB, shmem-rss:0kB
【神坑】
当系统内存不足时oom killer会基于一套算法选出一个进程并杀掉！不要问我啥算法！也不要问我为啥挂的都是俺！
【参考资料】
内存溢出（OOM）问题分析
一个嘴贱引发的内存异常bug