我的博客已迁移到xdoujiang.com请去那边和我交流
早上发现有台机器宕机了 查看日志发现
May 12 05:01:09 10.1.1.1 [4269014.583472] BUG: scheduling while atomic: swapper/0/0x10000100
May 12 05:01:09 10.1.1.1 0000000000015780
May 12 05:01:09 10.1.1.1 0000000000015780
May 12 05:01:09 10.1.1.1 0000000000015780
May 12 05:01:09 10.1.1.1
May 12 05:01:09 10.1.1.1 [4269014.961827] <0>
May 12 05:01:09 10.1.1.1 ffff8802615e9900
May 12 05:01:09 10.1.1.1 ffffffff8126315f
May 12 05:01:09 10.1.1.1 ffff880829b7cc00
May 12 05:01:09 10.1.1.1 ffff880829558a00
May 12 05:01:09 10.1.1.1
May 12 05:01:09 10.1.1.1 [4269014.969988] Call Trace:
May 12 05:01:09 10.1.1.1 [4269014.972630]
May 12 05:01:09 10.1.1.1
May 12 05:01:09 10.1.1.1 [4269014.974944] [
May 12 05:01:09 10.1.1.1 [4269014.980981] [
May 12 05:01:09 10.1.1.1 [4269014.987276] [
May 12 05:01:09 10.1.1.1 [4269014.993746] [
May 12 05:01:09 10.1.1.1 [4269014.999521] [
May 12 05:01:09 10.1.1.1 [4269015.005558] [
May 12 05:01:09 10.1.1.1 [4269015.011245] [
May 12 05:01:09 10.1.1.1 [4269015.016759] [
May 12 05:01:09 10.1.1.1 [4269015.022099] [
May 12 05:01:09 10.1.1.1 [4269015.028830] [
May 12 05:01:09 10.1.1.1 [4269015.035209]
May 12 05:01:09 10.1.1.1
May 12 05:01:09 10.1.1.1 [4269015.037526] [
May 12 05:01:09 10.1.1.1 [4269015.044968] [
May 12 05:01:09 10.1.1.1 [4269015.052411] [
May 12 05:01:09 10.1.1.1 [4269015.058534] [
May 12 05:01:09 10.1.1.1 [
May 12 05:01:09 10.1.1.1 [4269015.090513] RSP
May 12 05:01:09 10.1.1.1 [4269015.094550] ---[ end trace 2ff8aecaebc3aefc ]---
May 12 05:01:09 10.1.1.1 [4269015.094557] BUG: scheduling while atomic: swapper/0/0x10000100
May 12 05:01:09 10.1.1.1 [4269015.094562] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler netconsole configfs usbhid hid ohci_hcd loop snd_pcm snd_timer snd soundcore snd_page_alloc evdev dcdbas psmouse pcspkr iTCO_wdt iTCO_vendor_support serio_raw wmi power_meter button processor ext3 jbd mbcache uhci_hcd ses sd_mod crc_t10dif enclosure megaraid_sas ehci_hcd scsi_mod thermal thermal_sys usbcore nls_base bnx2 [last unloaded: scsi_wait_scan]
May 12 05:01:09 10.1.1.1 [4269015.094592] CPU 1:
May 12 05:01:09 10.1.1.1 [4269015.094594] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler netconsole configfs usbhid hid ohci_hcd loop snd_pcm snd_timer snd soundcore snd_page_alloc evdev dcdbas psmouse pcspkr iTCO_wdt iTCO_vendor_support serio_raw wmi power_meter button processor ext3 jbd mbcache uhci_hcd ses sd_mod crc_t10dif enclosure megaraid_sas ehci_hcd scsi_mod thermal thermal_sys usbcore nls_base bnx2 [last unloaded: scsi_wait_scan]
May 12 05:01:09 10.1.1.1 [4269015.094623] Pid: 0, comm: swapper Tainted: G D 2.6.32-5-amd64 #1 PowerEdge R510
May 12 05:01:09 10.1.1.1 [4269015.094626] RIP: 0010:[
May 12 05:01:09 10.1.1.1 [4269015.094636] RSP: 0018:ffff88082ccadea8 EFLAGS: 00000202
May 12 05:01:09 10.1.1.1 [4269015.094639] RAX: 0000000000010360 RBX: ffff88082abe9530 RCX: 00000000000f4240
May 12 05:01:09 10.1.1.1 [4269015.094642] RDX: ffff88001dc20000 RSI: 0000000000000000 RDI: 00000000003c8aab
May 12 05:01:09 10.1.1.1 [4269015.094646] RBP: ffffffff8101166e R08: 0000000000000f65 R09: 13dd46c52c8402dd
May 12 05:01:09 10.1.1.1 [4269015.094649] R10: 000000024c0a03b2 R11: 0000000000000001 R12: ffff88001dc35780
May 12 05:01:09 10.1.1.1 [4269015.094652] R13: ffffffff814634c0 R14: ffffffff81041ade R15: ffff880829608748
May 12 05:01:09 10.1.1.1 [4269015.094656] FS: 0000000000000000(0000) GS:ffff88001dc20000(0000) knlGS:0000000000000000
May 12 05:01:09 10.1.1.1 [4269015.094660] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
May 12 05:01:09 10.1.1.1 [4269015.094663] CR2: 00007f9edc6e34a8 CR3: 0000000001001000 CR4: 00000000000006e0
May 12 05:01:09 10.1.1.1 [4269015.094665] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 12 05:01:09 10.1.1.1 [4269015.094669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 12 05:01:09 10.1.1.1 [4269015.094671] Call Trace:
May 12 05:01:09 10.1.1.1 [4269015.094678] [
May 12 05:01:09 10.1.1.1 [4269015.094685] [
May 12 05:01:09 10.1.1.1 [4269015.094691] [
May 12 05:01:09 10.1.1.1 [4269015.095372] divide error: 0000 [#2] SMP
May 12 05:01:09 10.1.1.1 [4269015.095375] last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/0000:02:00.0/host0/scsi_host/host0/proc_name
May 12 05:01:09 10.1.1.1 [4269015.095379] CPU 1
May 12 05:01:09 10.1.1.1 [4269015.095380] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler netconsole configfs usbhid hid ohci_hcd loop snd_pcm snd_timer snd soundcore snd_page_alloc evdev dcdbas psmouse pcspkr iTCO_wdt iTCO_vendor_support serio_raw wmi power_meter button processor ext3 jbd mbcache uhci_hcd ses sd_mod crc_t10dif enclosure megaraid_sas ehci_hcd scsi_mod thermal thermal_sys usbcore nls_base bnx2 [last unloaded: scsi_wait_scan]
May 12 05:01:09 10.1.1.1 [4269015.095411] Pid: 0, comm: swapper Tainted: G D 2.6.32-5-amd64 #1 PowerEdge R510
May 12 05:01:09 10.1.1.1 [4269015.095414] RIP: 0010:[
May 12 05:01:09 10.1.1.1 [4269015.095422] RSP: 0018:ffff88001dc23c30 EFLAGS: 00010206
May 12 05:01:09 10.1.1.1 [4269015.095425] RAX: 0000000000003c00 RBX: 000000000000000f RCX: 0000000000000000
May 12 05:01:10 10.1.1.1 [4269015.095428] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000200
May 12 05:01:10 10.1.1.1 [4269015.095431] RBP: 0000000000015780 R08: 0000000000000000 R09: ffff88001dc4fc40
May 12 05:01:10 10.1.1.1 [4269015.095434] R10: 7fffffffffffffff R11: ffffffff813ce9f4 R12: ffff88001dc2fb30
May 12 05:01:10 10.1.1.1 [4269015.095437] R13: 000000000000000f R14: 0000000000015780 R15: ffff88001dc4fbf0
May 12 05:01:10 10.1.1.1 [4269015.095441] FS: 0000000000000000(0000) GS:ffff88001dc20000(0000) knlGS:0000000000000000
May 12 05:01:10 10.1.1.1 [4269015.095444] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
May 12 05:01:10 10.1.1.1 [4269015.095447] CR2: 0000000001ca30d0 CR3: 0000000819b21000 CR4: 00000000000006e0
May 12 05:01:10 10.1.1.1 [4269015.095450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 12 05:01:10 10.1.1.1 [4269015.095453] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 12 05:01:10 10.1.1.1 [4269015.095457] Process swapper (pid: 0, threadinfo ffff88082ccac000, task ffff88082cc83880)
May 12 05:01:10 10.1.1.1 [4269015.095459] Stack:
May 12 05:01:10 10.1.1.1 [4269015.095461] ffff88081c199498 0000000000015788 0000000000015780 0000000000000008
May 12 05:01:10 10.1.1.1 [4269015.095465] <0> 0000000000015780 0000000000015780 0000000000015780 0000000000015780
May 12 05:01:10 10.1.1.1 [4269015.095469] <0> ffff88081c199600 0000000000000082 00000000000000c1 ffff88001dc357e8
May 12 05:01:10 10.1.1.1 [4269015.095473] Call Trace:
May 12 05:01:10 10.1.1.1 [4269015.095475]
May 12 05:01:10 10.1.1.1 [4269015.095482] [
May 12 05:01:10 10.1.1.1 [4269015.095488] [
May 12 05:01:10 10.1.1.1 [4269015.095494] [
May 12 05:01:10 10.1.1.1 [4269015.095499] [
May 12 05:01:10 10.1.1.1 [4269015.095510] [
May 12 05:01:10 10.1.1.1 [4269015.095515] [
May 12 05:01:10 10.1.1.1 [4269015.095519] [
May 12 05:01:10 10.1.1.1 [4269015.095524] [
May 12 05:01:10 10.1.1.1 [4269015.095528] [
May 12 05:01:10 10.1.1.1 [4269015.095533] [
May 12 05:01:10 10.1.1.1 [4269015.095535]
May 12 05:01:10 10.1.1.1 [4269015.095542] [
May 12 05:01:10 10.1.1.1 [4269015.095548] [
May 12 05:01:10 10.1.1.1 [4269015.095554] [
May 12 05:01:10 10.1.1.1 [4269015.095559] [
May 12 05:01:10 10.1.1.1 [4269015.095562] Code: 01 00 00 00 74 13 48 8b 94 24 f0 01 00 00 c7 02 00 00 00 00 e9 91 00 00 00 41 8b 77 08 48 8b 84 24 78 01 00 00 31 d2 48 c1 e0 0a <48> f7 f6 48 8b b4 24 80 01 00 00 48 89 84 24 70 01 00 00 31 c0
May 12 05:01:10 10.1.1.1 [4269015.095589] RIP [
May 12 05:01:10 10.1.1.1 [4269015.095594] RSP
May 12 05:01:10 10.1.1.1 [4269015.095598] ---[ end trace 2ff8aecaebc3aefd ]---
May 12 05:01:10 10.1.1.1 [4269015.095604] divide error: 0000 [#3]
May 12 05:01:10 10.1.1.1 [4269015.095608] Kernel panic - not syncing: Fatal exception in interrupt
May 12 05:01:10 10.1.1.1 [4269015.095612] SMP
May 12 05:01:10 10.1.1.1 [4269015.095615] last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/0000:02:00.0/host0/scsi_host/host0/proc_name
May 12 05:01:10 10.1.1.1 [4269015.095621] Pid: 0, comm: swapper Tainted: G D 2.6.32-5-amd64 #1
May 12 05:01:10 10.1.1.1 [4269015.095626] CPU 3 Call Trace:
May 12 05:01:10 10.1.1.1 [4269015.095629]
May 12 05:01:10 10.1.1.1 [4269015.095631]
May 12 05:01:10 10.1.1.1 [4269015.095634] Modules linked in: ipmi_devintf [
May 12 05:01:10 10.1.1.1 [4269015.095644] ipmi_si ipmi_msghandler [
May 12 05:01:10 10.1.1.1 [4269015.095654] netconsole configfs [
May 12 05:01:10 10.1.1.1 [4269015.095664] usbhid hid [
May 12 05:01:10 10.1.1.1 [4269015.095673] ohci_hcd loop [
May 12 05:01:10 10.1.1.1 [4269015.095682] snd_pcm snd_timer [
May 12 05:01:10 10.1.1.1 [4269015.095690] snd soundcore [
May 12 05:01:10 10.1.1.1 [4269015.095699] snd_page_alloc evdev [
May 12 05:01:10 10.1.1.1 [4269015.095709] dcdbas psmouse [
May 12 05:01:10 10.1.1.1 [4269015.095718] pcspkr iTCO_wdt [
May 12 05:01:10 10.1.1.1 [4269015.095726] iTCO_vendor_support serio_raw [
May 12 05:01:10 10.1.1.1 [4269015.095736] wmi power_meter button [
May 12 05:01:10 10.1.1.1 [4269015.095745] processor ext3 [
May 12 05:01:10 10.1.1.1 [4269015.095754] jbd mbcache [
May 12 05:01:10 10.1.1.1 [4269015.095764] uhci_hcd ses [
May 12 05:01:10 10.1.1.1 [4269015.095772] sd_mod crc_t10dif [
May 12 05:01:10 10.1.1.1 [4269015.095781] enclosure megaraid_sas [
May 12 05:01:10 10.1.1.1 [4269015.095789] ehci_hcd scsi_mod [
May 12 05:01:10 10.1.1.1 [4269015.095797] thermal thermal_sys [
May 12 05:01:10 10.1.1.1 [4269015.095806] usbcore nls_base [
May 12 05:01:10 10.1.1.1 [4269015.095815] bnx2 [last unloaded: scsi_wait_scan] [
May 12 05:01:10 10.1.1.1 [4269015.095824]
May 12 05:01:10 10.1.1.1 [4269015.095826]
May 12 05:01:10 10.1.1.1 [4269015.095831] Pid: 2887, comm: Thttpd Tainted: G D 2.6.32-5-amd64 #1 PowerEdge R510
May 12 05:01:10 10.1.1.1 [4269015.095837] RIP: 0010:[
May 12 05:01:10 10.1.1.1 [4269015.095852] [
May 12 05:01:10 10.1.1.1 [4269015.095857] RSP: 0018:ffff88081bbc3b88 EFLAGS: 00010006
May 12 05:01:10 10.1.1.1 [4269015.095865] [
May 12 05:01:10 10.1.1.1 [4269015.095870] RAX: 0000000000003c00 RBX: 000000000000000f RCX: 0000000000000000
May 12 05:01:10 10.1.1.1 [4269015.095877] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000200
May 12 05:01:10 10.1.1.1 [4269015.095884] [
May 12 05:01:10 10.1.1.1 [4269015.095889] RBP: 0000000000015780 R08: 0000000000000000 R09: ffff88001dc4fc40
May 12 05:01:10 10.1.1.1 [4269015.095895] R10: 0000000000000064 R11: 0000000000000246 R12: ffff88001dc6fb30
May 12 05:01:10 10.1.1.1 [4269015.095903] [
May 12 05:01:10 10.1.1.1 [4269015.095908] R13: 000000000000000f R14: 0000000000015780 R15: ffff88001dc4fbf0
May 12 05:01:10 10.1.1.1 [4269015.095913] FS: 00007f8e9fc786e0(0000) GS:ffff88001dc60000(0000) knlGS:0000000000000000
May 12 05:01:10 10.1.1.1 [4269015.095916] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 12 05:01:10 10.1.1.1 [4269015.095919] CR2: 00007f07e2a15210 CR3: 000000081bbfb000 CR4: 00000000000006e0
May 12 05:01:10 10.1.1.1 [4269015.095922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 12 05:01:10 10.1.1.1 [4269015.095925] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 12 05:01:10 10.1.1.1 [4269015.095929] Process Thttpd (pid: 2887, threadinfo ffff88081bbc2000, task ffff8808296d1530)
May 12 05:01:10 10.1.1.1 [4269015.095931] Stack:
May 12 05:01:10 10.1.1.1 [4269015.095933] 0000000029675c00 0000000000015788 0000000000015780 0000000000000008
May 12 05:01:10 10.1.1.1 [4269015.095937] <0> 0000000000015780 0000000000015780 0000000000015780 ffff880829b7cd00
May 12 05:01:10 10.1.1.1 [4269015.095941] <0> ffff880828f00000 ffff880829b7cd00 ffffffffa000c6e0 ffffffff812501be
May 12 05:01:10 10.1.1.1 [4269015.095946] Call Trace:
May 12 05:01:10 10.1.1.1 [4269015.095955] [
May 12 05:01:10 10.1.1.1 [4269015.095961] [
May 12 05:01:10 10.1.1.1 [4269015.095968] [
May 12 05:01:10 10.1.1.1 [4269015.095973] [
May 12 05:01:10 10.1.1.1 [4269015.095978] [
May 12 05:01:10 10.1.1.1 [4269015.095982] [
May 12 05:01:10 10.1.1.1 [4269015.095987] [
May 12 05:01:10 10.1.1.1 [4269015.095993] [
May 12 05:01:10 10.1.1.1 [4269015.095999] [
May 12 05:01:10 10.1.1.1 [4269015.096004] [
May 12 05:01:10 10.1.1.1 [4269015.096009] [
May 12 05:01:10 10.1.1.1 [4269015.096013] [
May 12 05:01:10 10.1.1.1 [4269015.096016] Code: 01 00 00 00 74 13 48 8b 94 24 f0 01 00 00 c7 02 00 00 00 00 e9 91 00 00 00 41 8b 77 08 48 8b 84 24 78 01 00 00 31 d2 48 c1 e0 0a <48> f7 f6 48 8b b4 24 80 01 00 00 48 89 84 24 70 01 00 00 31 c0
May 12 05:01:10 10.1.1.1 [4269015.096042] RIP [
May 12 05:01:10 10.1.1.1 [4269015.096048] RSP
May 12 05:01:10 10.1.1.1 [4269015.096051] ---[ end trace 2ff8aecaebc3aefe ]---
查看到以下信息 查了相关资料 说是2.6.32-5内核BUG
May 12 05:01:09 10.1.1.1 [4269015.094623] Pid: 0, comm: swapper Tainted: G D 2.6.32-5-amd64 #1 PowerEdge R510
May 12 05:01:09 10.1.1.1 [4269014.583472] BUG: scheduling while atomic: swapper/0/0x10000100 May 12 05:01:10 10.1.1.1 [4269015.095604] divide error: 0000 [#3] May 12 05:01:10 10.1.1.1 [4269015.095608] Kernel panic - not syncing: Fatal exception in interrupt