解决：kernel: Out of memory: Killed process 15967, UID 27, (mysqld).

参考：http://www.weiruoyu.cn/?p=477

最近服务器突然有一段时间：ssh连不上，查看日志发现有大有问题，去网上查了，发现是内存不够了，可以使用以为网友提供的添加一个1Gswap分区，原服务器确实已经删除了swap分区。

Mar 20 16:19:23 localhost kernel: Mem-info:
Mar 20 16:19:23 localhost kernel: DMA per-cpu:
Mar 20 16:19:23 localhost kernel: cpu 0 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 0 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 1 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 1 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 2 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 2 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 3 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 3 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 4 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 4 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 5 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 5 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 6 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 6 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 7 hot: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: cpu 7 cold: high 0, batch 1 used:0
Mar 20 16:19:23 localhost kernel: DMA32 per-cpu: empty
Mar 20 16:19:23 localhost kernel: Normal per-cpu:

Mar 20 16:19:23 localhost kernel: cpu 7 cold: high 62, batch 15 used:14
Mar 20 16:19:23 localhost kernel: Free pages: 315072kB (472kB HighMem)
Mar 20 16:19:23 localhost kernel: Active:1969387 inactive:0 dirty:0 writeback:0 unstable:0 free:78768 slab:11435 mapped-file:3 mapped-anon:1970742 pagetables:6141
Mar 20 16:19:23 localhost kernel: DMA free:12036kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
Mar 20 16:19:23 localhost kernel: lowmem_reserve[]: 0 0 880 10224
Mar 20 16:19:23 localhost kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Mar 20 16:19:23 localhost kernel: lowmem_reserve[]: 0 0 880 10224
Mar 20 16:19:23 localhost kernel: Normal free:302564kB min:3756kB low:4692kB high:5632kB active:457996kB inactive:0kB present:901120kB pages_scanned:76768585 all_unreclaimable? yes
Mar 20 16:19:23 localhost kernel: lowmem_reserve[]: 0 0 0 74752
Mar 20 16:19:23 localhost kernel: HighMem free:472kB min:512kB low:10492kB high:20472kB active:7419752kB inactive:0kB present:9568256kB pages_scanned:33425047 all_unreclaimable? yes
Mar 20 16:19:23 localhost kernel: lowmem_reserve[]: 0 0 0 0
Mar 20 16:19:23 localhost kernel: DMA: 11*4kB 13*8kB 11*16kB 12*32kB 9*64kB 6*128kB 5*256kB 5*512kB 2*1024kB 0*2048kB 1*4096kB = 12036kB
Mar 20 16:19:23 localhost kernel: DMA32: empty
Mar 20 16:19:23 localhost kernel: Normal: 41*4kB 3026*8kB 2549*16kB 2899*32kB 1702*64kB 241*128kB 5*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 302564kB
Mar 20 16:19:23 localhost kernel: HighMem: 46*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
Mar 20 16:19:23 localhost kernel: 1095 pagecache pages
Mar 20 16:19:23 localhost kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Mar 20 16:19:23 localhost kernel: Free swap = 0kB
Mar 20 16:19:23 localhost kernel: Total swap = 0kB
Mar 20 16:19:23 localhost kernel: Free swap: 0kB
Mar 20 16:19:23 localhost kernel: 2621440 pages of RAM
Mar 20 16:19:23 localhost kernel: 2392064 pages of HIGHMEM
Mar 20 16:19:23 localhost kernel: 548922 reserved pages
Mar 20 16:19:23 localhost kernel: 5664 pages shared
Mar 20 16:19:23 localhost kernel: 0 pages swap cached
Mar 20 16:19:23 localhost kernel: 0 pages dirty
Mar 20 16:19:23 localhost kernel: 0 pages writeback
Mar 20 16:19:23 localhost kernel: 3 pages mapped
Mar 20 16:19:23 localhost kernel: 11435 pages slab
Mar 20 16:19:23 localhost kernel: 6141 pages pagetables
Mar 20 16:19:23 localhost kernel: Out of memory: Killed process 15967, UID 27, (mysqld).

红色标记为：目前判断为故障发生时，内核还处于可运行状态，但由于内存不足，触发了OOM killer机制，但一直在杀进程又一直有新的进程创建，导致了死循环。

根源是内存不够，可以考虑临时添加一个swap分区，因为我本身就没有，所以添加，这是临时的，想根本解决就得添加内存了。

添加swap分区：

1.加1G ，swap分区

dd if=/dev/zero of=/opt/swapfile bs=1M count=1024
#mkswap /opt/swapfile
#swapon /opt/swapfile
#swapon -s

1. Filename Type Size Used Priority

2. /dev/mapper/VolGroup00-LogVol00 partition 2097144 87352 -1

3. /opt/swapfile file 2097144 0 -2

2.在fstab增加相关记录，否则重启就没了

更详细的请参考：http://www.weiruoyu.cn/?p=477