OGG 导致oracle数据库down掉 恢复后 出现 ORA-600 [13013]

ORA-00600: internal error code, arguments: [13013], [5001], [268], [8456175], [0], [8456175], [17], [], [], [], [], []

 

不停的数据库自己down掉     以下转载

-------------------------------------------------------------------------------------------------------

中午接到电话，客户的核心系统Oracle数据库实例宕机，远程过去在告警日志中看到大量的如下报错，报错很频繁：
......
Fri Jul 25 13:20:14 2014
Errors in file /u01/app/oracle/diag/rdbms/d012band/d012band/trace/d012band_smon_5964354.trc (incident=43361):
ORA-00600: internal error code, arguments: [13013], [5001], [268], [8452274], [7], [8452274], [17], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/d012band/d012band/incident/incdir_43361/d012band_smon_5964354_i43361.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non-fatal internal error happenned while SMON was doing logging scn->time mapping.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Fri Jul 25 13:20:17 2014
Dumping diagnostic data in directory=[cdmp_20140725132017], requested by (instance=1, osid=5964354 (SMON)), summary=[incident=43361].
Starting background process SMCO
......

一段时间后，最后实例被PMON进程终止，导致核心系统宕机。

客户环境：Oracle Database 11.2.0.3.0 for IBM AIX 6.1，单机，IBM HACMP主备模式保护。


参考《 New and Improved: ORA-600 [13013] "Unable to get a Stable set of Records" (文档 ID 1438920.1)》和《ORA-600 [13013] "Unable to get a Stable set of Records" (文档 ID 28185.1)》文章；该报错是由于对某个表执行DML操作，该表对应的某个索引损坏导致的，解决的办法是找出操作的表和受损的索引，重建索引即可。

下面是对ORA-600的参数说明：

6 Argument format
=================

This format relates to Oracle Server 8.0.3 and above

Arg [a] Passcount
Arg [b] Data Object number
Arg [c] Tablespace Decimal Relative DBA (RDBA) of block containing the row to be updated
Arg [d] Row Slot number
Arg [e] Decimal RDBA of block being updated (Typically same as [c])
Arg [f] Code

由此执行下面的SQL语句查询找出DML操作的表，以及表对应的索引：

SQL> SELECT OWNER,OBJECT_NAME,OBJECT_TYPE FROM DBA_OBJECTS WHERE DATA_OBJECT_ID=268;

发现问题对象是SYS用户下面的SMON_SCN_TIME表，SYS.SMON_SCN_TIME是Oracle更新非常频繁的一张表，该数据字典用于维护SCN和时间的关系。

SQL> SELECT OWNER,INDEX_NAME FROM DBA_INDEXES WHERE TABLE_NAME='SMON_SCN_TIME';

发现这个表对应有SMON_SCN_TIME_TIM_IDX和SMON_SCN_TIME_SCN_IDX两个索引。

对表进行分析，找出有问题的索引：

SQL> ANALYZE TABLE smon_scn_time VALIDATE STRUCTURE;

Table analyzed.

SQL> ANALYZE TABLE smon_scn_time VALIDATE STRUCTURE CASCADE ONLINE;

table/Index Cross Reference Failure - see trace file

能够正常的分析表，但是不能正常的分析表上的索引。

参考《OERR: ORA-1499 table/Index Cross Reference Failure - see trace file (文档 ID 1499.1)》文章，尝试找出受影响的索引：

查看trace file:
BH (0x70000078fe2df98) file#: 2 rdba: 0x0081483e (2/84030) class: 1 ba: 0x70000078ce52000
set: 174 pool: 3 bsz: 8192 bsi: 0 sflg: 2 pwc: 2272,19
dbwrid: 5 obj: 272 objn: 272 tsn: 1 afn: 2 hint: f
hash: [0x700000801ff36a0,0x700000801ff36a0] lru: [0x70000074bf81f40,0x70000074bf81ce0]
ckptq: [NULL] fileq: [NULL] objq: [0x7000007f59b4138,0x7000007f59b4138] objaq: [0x7000007f59b4128,0x7000007f59b4128]
use: [NULL] wait: [NULL] fast-cr-pins: 2
st: XCURRENT md: NULL fpin: 'kdiwh15: kdifxs' tch: 12
flags:
LRBA: [0x0.0.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [65535]
buffer tsn: 1 rdba: 0x0081483e (2/84030)
scn: 0x0000.2f4752d8 seq: 0x01 flg: 0x06 tail: 0x52d80601
frmt: 0x02 chkval: 0x0778 type: 0x06=trans data

0x0081483e要去掉0x，0x表示十六进制。

执行下面的SQL语句：

SELECT owner, segment_name, segment_type, partition_name
FROM DBA_SEGMENTS
WHERE header_file = (SELECT file#
FROM v$datafile
WHERE rfile# = dbms_utility.data_block_address_file(to_number('0080f8b2','XXXXXXXX'))
AND ts#= 1)
AND header_block = dbms_utility.data_block_address_block(to_number('0080f8b2','XXXXXXXX'));

可惜没有任何的查询结果。

继续参考《Instance Terminated With Error ORA-00474: SMON Process Terminated With Error (文档 ID 1361872.1)》文章，文章有下面一段描述：

CAUSE

ORA600 [13011] is raised due to indexes corruption. To verify the corruption run the following statements:


SQL> conn / as sysdba
SQL> ANALYZE TABLE smon_scn_time VALIDATE STRUCTURE;

Table analyzed.

-- It should come out clean giving message table analyzed.

SQL> ANALYZE TABLE smon_scn_time VALIDATE STRUCTURE CASCADE ONLINE;

-- it should fail with Ora-1499 if at least one index is corrupted

ORA-1499就是我们前面遇到的报错：

Error: ORA 1499
Text: table/Index Cross Reference Failure - see trace file
-------------------------------------------------------------------------------
Cause: An error occurred when validating an index or a table using the
ANALYZE command.
One or more entries does not point to the appropriate cross-reference.
Action: Check the trace file for more descriptive messages about the problem.
Correct these errors.

如果在分析表的时候收到ORA-1499的错误，表示至少有一个索引是损坏的。

SOLUTION

Rebuild corrupted indexes:


SQL> conn as sysdba
SQL> ALTER INDEX SMON_SCN_TIME_TIM_IDX REBUILD ONLINE;
SQL> ALTER INDEX SMON_SCN_TIME_SCN_IDX REBUILD ONLINE;

then re-run


SQL> ANALYZE TABLE smon_scn_time VALIDATE STRUCTURE CASCADE ONLINE;

Table analyzed.

Note: The last statement should not report any errors.


两个索引重建之后，ANALYZE TABLE smon_scn_time VALIDATE STRUCTURE CASCADE ONLINE语句能顺利的分析，告警日志不再有报错产生。

总结：要学会充分利用MOS，根据不同的线索搜索不同的文章来看，不要在一篇文章上吊死。

--end--