一套Solaris上的9.2.0.7系统，实例意外终止，告警日志中出现以下记录:
Thu Sep 2 02:15:41 2010 Errors in file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc: ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] [] Thu Sep 2 02:15:48 2010 Errors in file /u01/app/oracle/admin/preg063/bdump/preg063_pmon_11379.trc: ORA-00474: SMON process terminated with error Thu Sep 2 02:15:48 2010 PMON: terminating instance due to error 474 Wed Sep 1 15:04:20 2010 Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_1772.trc: ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA78000] [] [] Wed Sep 1 15:06:24 2010 Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_5316.trc: ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C843DC8] [] [] Wed Sep 1 15:06:24 2010 Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_5207.trc: ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C887A70] [] [] Wed Sep 1 15:06:24 2010 Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_28532.trc: ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7C9B7FE8] [] [] Wed Sep 1 15:06:31 2010 Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_4392.trc: ORA-07445: exception encountered: core dump [0000000100624600] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CA80000] [] [] Wed Sep 1 15:06:33 2010 Errors in file /u01/app/oracle/admin/preg063/udump/preg063_ora_3748.trc: ORA-07445: exception encountered: core dump [FFFFFFFF7F400980] [SIGBUS] [Object specific hardware error]
trace日志: Dump file /u01/app/oracle/admin/preg063/bdump/preg063_smon_11391.trc Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.7.0 - Production ORACLE_HOME = /u01/app/oracle/product/920preg063 System name: SunOS 2010-09-02 02:15:41.917 *** SESSION ID:(7.1) 2010-09-02 02:15:33.090 Exception signal: 10 (SIGBUS), code: 3 (Object specific hardware error), addr: 0xffffffff7cb3bf90, PC: [0x101e05500, 0000000101E05500] *** 2010-09-02 02:15:41.936 ksedmp: internal or fatal error ORA-07445: exception encountered: core dump [0000000101E05500] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CB3BF90] [] [] Current SQL statement for this session: update sys.col_usage$ set equality_preds = equality_preds + decode(bitand(:flag,1),0,0,1), equijoin_preds = equijoin_preds + decode(bitand(:flag,2),0,0,1), nonequijoin_preds = nonequijoin_preds + decode(bitand(:flag,4),0,0,1), range_preds = range_preds + decode(bitand(:flag,8),0,0,1), like_preds = like_preds + decode(bitand(:flag,16),0,0,1), null_preds = null_preds + decode(bitand(:flag,32),0,0,1), timestamp = :time where obj# = :objn and intcol# = :coln ----- Call Stack Trace ----- calling call entry argument values in hex location type point (? means dubious value) -------------------- -------- -------------------- ---------------------------- ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ? 000000000 ? 103357D68 ? 00000003E ? FFFFFFFF7FFE3EE8 ? ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ? 103788C68 ? 10378D000 ? 000102C00 ? 000000000 ? sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ? FFFFFFFF7FFEAF80 ? 103790000 ? 10378DE30 ? 000000000 ? 103790DB0 ? kghbshrt()+96 PTR_CALL 0000000000000000 00000000A ? FFFFFFFF7FFEAF80 ? FFFFFFFF7FFEACA0 ? 000000000 ? 00000BF78 ? 0B38F0000 ? kghalo()+612 CALL kghbshrt() 103788EC8 ? 000000000 ? 10378C2E0 ? FFFFFFFF7CB30018 ? 000000458 ? 080000000 ? kghgex()+336 CALL kghalo() 000000000 ? FFFFFFFF7CB30018 ? 103788EC8 ? 000001000 ? FFFFFFFF7CB3E120 ? C0B38F000000C3D1 ? kghalf()+316 CALL kghgex() 000007C00 ? 000000440 ? 000000228 ? 103788EC8 ? 000007FFF ? 000000000 ? kdb4chk()+216 PTR_CALL 0000000000000000 103788EC8 ? 000000178 ? FFFFFFFF7CB3E180 ? FFFFFFFF7CB3E150 ? 000000000 ? 102DA35D0 ? kd4chk()+88 CALL kdb4chk() 3F5484014 ? 1007E7780 ? 000000000 ? 000000000 ? 1007E76C0 ? 000000000 ? kcbchk()+244 PTR_CALL 0000000000000000 3F5484000 ? 000000000 ? 000002000 ? 000000000 ? 1007E76C0 ? 1007E7720 ? kcoapl()+1288 CALL kcbchk() 3F5484000 ? 0001007E7 ? 1007E7000 ? 101BA4440 ? 000100400 ? 0001007E7 ? kcbapl()+96 CALL kcoapl() 000000002 ? 000000000 ? 000000001 ? 3F5484000 ? 10388DA08 ? 3F5486000 ? kcrfwr()+1716 CALL kcbapl() 10388D990 ? 3F5FA6200 ? FFFFFFFF7FFEC3B0 ? 00010378E ? 000000004 ? 000000004 ? kcbchg1()+2056 CALL kcrfwr() FFFFFFFF7FFEC638 ? 40E36C218 ? 10378F5E8 ? FFFFFFFF7FFEC3B0 ? 10378F690 ? 000000002 ? ktuchg()+1044 CALL kcbchg1() 000000000 ? FFFFFFFF7FFECE08 ? 000000000 ? 10378B530 ? 000000000 ? FFFFFFFF7FFEC730 ? ktbchg2()+132 Dump file /u01/app/oracle/admin/preg063/udump/preg063_ora_10110.trc Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.7.0 - Production ORACLE_HOME = /u01/app/oracle/product/920preg063 System name: SunOS 2010-09-01 00:29:31.655 ksedmp: internal or fatal error ORA-07445: exception encountered: core dump [0000000101E10F94] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF7CAAC000] [] [] Current SQL statement for this session: select lxOid,lxVal from lxString_c83b5454 where lxOid in (:va,:vb,:vc,:vd,:ve,:vf,:vg,:vh,:vi,:vj,:vk,:vl,:vm,:vn,:vo,:vp,:vq,:vr,:vs,:vt,:vu,:vv,:vw,:vx,:vy,:vz,:v26,:v27,:v28,:v29,:v30,:v31,:v32,:v33,:v34,:v35,:v36,:v37,:v38,:v39,:v40,:v41,:v42,:v43,:v44,:v45,:v46,:v47,:v48,:v49,:v50,:v51,:v52,:v53,:v54,:v55,:v56,:v57,:v58,:v59) and lxType=:v60 ----- Call Stack Trace ----- calling call entry argument values in hex location type point (? means dubious value) -------------------- -------- -------------------- ---------------------------- ksedmp()+328 CALL ksedst() 00000000B ? 000000000 ? 000000000 ? 103357D68 ? 00000003E ? FFFFFFFF7FFF3B98 ? ssexhd()+676 CALL ksedmp() 000103788 ? 103788000 ? 103788C68 ? 10378D000 ? 000102C00 ? 000000000 ? sigacthandler()+44 PTR_CALL 0000000000000000 000103790 ? FFFFFFFF7FFFAC30 ? 103790000 ? 10378DE30 ? 000000000 ? 103790DB0 ? kghalf()+436 PTR_CALL 0000000000000000 00000000A ? FFFFFFFF7FFFAC30 ? FFFFFFFF7FFFA950 ? 103788EC8 ? 0B38F0000 ? B38F0000000000 ? qerhjManageListElem CALL kghalf() 103788EC8 ? ent()+116 10B38F0000000001 ? FFFFFFFF7CC73418 ? FFFFFFFF7CC73418 ? FFFFFFFF7CAABFF8 ? 1034CC568 ? qerhjadf()+248 CALL qerhjManageListElem FFFFFFFF7CBA2568 ? ent() FFFFFFFF7CC733A8 ? 000000002 ? 000000049 ? FFFFFFFF7CA41E9A ? 0000000AF ? qerhjBuildHashTable CALL qerhjadf() FFFFFFFF7FFFB0F8 ? ()+4236 FFFFFFFF7C9D2DB0 ? 000000013 ? FFFFFFFF7C9D2D90 ? FFFFFFFF7CA41E9A ? 000000001 ? qerhjFetch()+772 CALL qerhjBuildHashTable 000000000 ? 1038698D8 ? () 000000040 ? FFFFFFFF7C9B16A8 ? 000000001 ? FFFFFFFF7CBA2630 ? qervwFetch()+148 PTR_CALL 0000000000000000 43C80CBB0 ? 101624E80 ? FFFFFFFF7FFFB338 ? 00000001E ? 000000008 ? 00000005F ? kpofrws()+124 PTR_CALL 0000000000000000 43C80CB58 ? 100FEF620 ? FFFFFFFF7FFFB940 ? 00000001E ? 000101400 ? 000101624 ? opifch2()+1704 CALL kpofrws() 000000004 ? 000000004 ? 100FEF620 ? FFFFFFFF7FFFB940 ? 102EC9160 ? 000000010 ? opiall0()+3900 CALL opifch2() 10101B000 ? 000000006 ?提交SR后，MOS认为可能是swap空间不足导致，建议检查系统日志:"Please check your OS logs for any errors during this time.Also please check swap space. This error is sometimes recorded when swap space becomes full"。 系统日志显示在Sep 2出现过/tmp文件系统空间耗尽，swap空间不足的问题:
Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded Aug 15 02:41:25 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded Aug 15 02:48:06 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded Sep 2 02:15:42 v08k419 tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded该内部错误通过增大swap最后解决了，that's great!