内存MCE错误导致系统崩溃的问题分析

今天服务器因为内存问题而崩溃，通过mcelog工具分析是在读内存的时候Error overflow（虽然是ECC内存，但也无奈错误太多），估计是内存硬件故障，如果再次出现的话就得考虑更换内存。

最终原因：硬件故障，应该是主板问题，因为是线上服务器为减少计划内停机时间，同时更换主板和内存解决。

# more /var/log/messages

Oct 31 14:19:36 pingu_fd kernel: sbridge: HANDLING MCE MEMORY ERROR

Oct 31 14:19:36 pingu_fd kernel: CPU 0: Machine Check Exception: 0 Bank 5: 8c00004000010092
Oct 31 14:19:36 pingu_fd kernel: TSC 0 ADDR 428fc8840 MISC 204808e886 PROCESSOR 0:206d6 TIME 1383200376 SOCKET 0 APIC 0
Oct 31 14:19:36 pingu_fd kernel: sbridge: HANDLING MCE MEMORY ERROR
Oct 31 14:19:36 pingu_fd kernel: CPU 0: Machine Check Exception: 0 Bank 10: 8800004800800092

Oct 31 14:19:36 pingu_fd kernel: TSC 0 ADDR 0 MISC 4900030243025000 PROCESSOR 0:206d6 TIME 1383200376 SOCKET 0 APIC 0

通过mcelog翻译message的内容如下：

# mcelog sandybridge-ep --ascii < mcelog-manu.txt   
sbridge: HANDLING MCE MEMORY ERROR
Hardware event. This is not a software error.
CPU 0 BANK 5 
MISC 244076f686 ADDR 1a6bca040 
TIME 1383200376 Thu Oct 31 14:19:36 2013
MCG status:
MCi status:
Error overflow
Corrected error
MCi_MISC register valid
MCi_ADDR register valid
MCA: MEMORY CONTROLLER RD_CHANNEL2_ERR
Transaction: Memory read error
STATUS cc0000c000010092 MCGSTATUS 0
CPUID Vendor Intel Family 6 Model 45
SOCKET 0 APIC 0