ORA-600 kclchkblk_4和2662故障---惜分飞

有客户恢复请求:由于未知原因导致aix环境的rac两台主机同时重启之后数据库无法正常启动,初步判断是由于写丢失导致故障（ORA-00742 ORA-00353）

Wed Feb 21 09:23:06 2024 ALTER DATABASE OPEN This instance was first to open Abort recovery for domain 0 Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc: ORA-01113: file 32 needs media recovery ORA-01110: data file 32: '+DATA/xff/datafile/x5sys_cs.dbf' ORA-1113 signalled during: ALTER DATABASE OPEN... Wed Feb 21 09:23:27 2024 ALTER DATABASE RECOVER  datafile '+DATA/xff/datafile/x5sys_cs.dbf'  Media Recovery Start Serial Media Recovery started WARNING! Recovering data file 32 from a fuzzy backup. It might be an online backup taken without entering the begin backup command. Recovery of Online Redo Log: Thread 2 Group 14 Seq 48490 Reading mem 0   Mem# 0: +DATA/xff/onlinelog/group_14.313.1060528521 Recovery of Online Redo Log: Thread 1 Group 7 Seq 64195 Reading mem 0   Mem# 0: +DATA/xff/onlinelog/group_7.306.1060527979 Wed Feb 21 09:24:10 2024 Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc: ORA-00742: Log read detects lost write in thread %d sequence %d block %d ORA-00334: archived log: '+DATA/xff/onlinelog/group_14.313.1060528521' Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc  (incident=336478): ORA-00353: log corruption near block 139727 change 26346459680 time 02/20/2024 20:13:50 ORA-00312: online log 14 thread 2: '+DATA/xff/onlinelog/group_14.313.1060528521'

尝试屏蔽一致性强制拉库后数据库报ORA-600 kclchkblk_4
参考:ora-600 2662和ora-600 kclchkblk_4恢复、redo异常 ORA-600 kclchkblk_4 故障恢复

Wed Feb 21 09:55:26 2024 SMON: enabling cache recovery Wed Feb 21 09:55:26 2024 Redo thread 2 internally disabled at seq 5 (CKPT) Archived Log entry 112707 added for thread 2 sequence 4 ID 0xffffffffe144183b dest 1: ARC0: Archiving disabled thread 2 sequence 5 Archived Log entry 112708 added for thread 2 sequence 5 ID 0xffffffffe144183b dest 1: Wed Feb 21 09:55:28 2024 Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc  (incident=360479): ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892] Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_360479/xff1_ora_6423264_i360479.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Thread 1 advanced to log sequence 3 (LGWR switch)   Current log# 7 seq# 3 mem# 0: +DATA/xff/onlinelog/group_7.306.1161510375 Archived Log entry 112709 added for thread 1 sequence 2 ID 0xffffffffe144183b dest 1: Wed Feb 21 09:55:31 2024 Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892] Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]

后续处理中出现和这个错误类似的ORA-600 2662错误

Wed Feb 21 15:37:35 2024 SMON: enabling cache recovery Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc  (incident=432423): ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104] Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_432423/xff1_ora_6357664_i432423.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc: ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], [] Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc: ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], [] Error 600 happened during db open, shutting down database USER (ospid: 6357664): terminating the instance due to error 600 Instance terminated by USER, pid = 6357664

通过对oracle scn进行修改,数据库open成功

SQL> recover database; Media recovery complete. SQL> SQL> SQL> SQL> oradebug setmypid Statement processed. SQL> oradebug DUMPvar SGA kcsgscn_ kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000000 00000000 00000000 00000000 SQL> oradebug poke BEFORE: [700000000019B70, 700000000019B78) = 00000000 00000000 AFTER:  [700000000019B70, 700000000019B78) = 00000006 22710D2B SQL> oradebug DUMPvar SGA kcsgscn_ kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000006 22710D2B 00000000 00000000 SQL> alter database open; Database altered.

后续检查发现obj$中的index异常（ORA-08102: index key not found, obj# 39）
类似文章:通过bbed修改obj$中dataobj$重现I_OBJ4索引报ORA-08102错误

SQL> truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX"; truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX"                        * ERROR at line 1: ORA-00604: error occurred at recursive SQL level 1 ORA-08102: index key not found, obj# 39, file 1, block 967206 (2) SQL> SQL> select object_name,object_type from dba_objects where object_id=39; OBJECT_NAME                    OBJECT_TYPE ------------------------------ ------------------- I_OBJ4                         INDEX

对于此类问题使用非常规方法把obj$字典表进行重建（需要注意undo需要为自动管理方式,temp不能为空）,参考:
bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决