难见的oracle 9i恢复---2023年----惜分飞

时过境迁,以前恢复大量oracle 8/9版本的库,现在一套oracle 9i的库都比较稀奇了.今天恢复客户一套9.2.0.6的aix环境rac库,通过分析确认主要问题:
1. 重建控制文件，resetlogs库遗漏数据文件
 

---

2. 数据库启动主要报错ORA-600 2663和ORA-600 kclchkblk_4

Tue Nov  8 09:10:05 2022 Successfully onlined Undo Tablespace 1. Dictionary check beginning Tablespace 'TEMP' #2 found in data dictionary, but not in the controlfile. Adding to controlfile. File #84 found in data dictionary but not in controlfile. Creating OFFLINE file 'MISSING00084' in the controlfile. This file can no longer be recovered so it must be dropped. Dictionary check complete Tue Nov  8 09:10:05 2022 SMON: enabling tx recovery Tue Nov  8 09:10:05 2022 Database Characterset is ZHS16GBK Tue Nov  8 09:10:05 2022 Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_536662.trc: ORA-00600: internal error code, arguments: [2663], [3301], [2638369768], [3301], [2640322622], [], [], [] Tue Nov  8 09:10:06 2022 Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_647352.trc: ORA-00600: internal error code, arguments: [kclchkblk_4], [3301], [18446744072061740072],[3301],[18446744072052954088] Tue Nov  8 09:10:06 2022 Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_536662.trc: ORA-00600: internal error code, arguments: [2663], [3301], [2638369768], [3301], [2640322622], [], [], [] Error 600 happened during db open, shutting down database USER: terminating instance due to error 600 Instance terminated by USER, pid = 536662 ORA-1092 signalled during: alter database open...

根据客户文件名称的规则,推算出来84号文件实际的文件名(因为使用的是lv[aix的hacmp管理的lv的裸设备方式]),通过dbv确认文件无坏块

DBVERIFY: Release 9.2.0.6.0 - Production on Sat May 13 16:44:09 2023 Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved. DBVERIFY - Verification starting : FILE = /dev/ra_txn_ind12.dbf DBVERIFY - Verification complete Total Pages Examined         : 256000 Total Pages Processed (Data) : 0 Total Pages Failing   (Data) : 0 Total Pages Processed (Index): 299 Total Pages Failing   (Index): 0 Total Pages Processed (Other): 13 Total Pages Processed (Seg)  : 0 Total Pages Failing   (Seg)  : 0 Total Pages Empty            : 255688 Total Pages Marked Corrupt   : 0 Total Pages Influx           : 0 Highest block SCN            : 11177081099136 (2602.1576194944)

bbed验证文件该文件是否是84号文件

$ bbed blocksize=8192 filename='/dev/ra_txn_ind12.dbf'   Password: BBED: Release 2.0.0.0.0 - Limited Production on Mon May 15 09:45:44 2023 Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved. ************* !!! For Oracle Internal Use only !!! *************** BBED> map  File: /dev/ra_txn_ind12.dbf (0)  Block: 1                                     Dba:0x00000000 ------------------------------------------------------------  Data File Header  struct kcvfh, 608 bytes                    @0        ub4 tailchk                                @8188    BBED> p kcvfh struct kcvfh, 608 bytes                     @0          struct kcvfhbfh, 20 bytes                @0             ub1 type_kcbh                         @0        0x0b       ub1 frmt_kcbh                         @1        0x02       ub1 spare1_kcbh                       @2        0x00       ub1 spare2_kcbh                       @3        0x00       ub4 rdba_kcbh                         @4        0x15000001       ub4 bas_kcbh                          @8        0x00000000       ub2 wrp_kcbh                          @12       0x0000       ub1 seq_kcbh                          @14       0x01       ub1 flg_kcbh                          @15       0x04 (KCBHFCKV)       ub2 chkval_kcbh                       @16       0x1b4a       ub2 spare3_kcbh                       @18       0x0000    struct kcvfhhdr, 76 bytes                @20            ub4 kccfhswv                          @20       0x09200000       ub4 kccfhcvn                          @24       0x08000000       ub4 kccfhdbi                          @28       0x05d15ccf       ……       ub4 kccfhcsq                          @40       0x00525a20       ub4 kccfhfsz                          @44       0x0003e800       s_blkz kccfhbsz                       @48       0x00       ub2 kccfhfno                          @52       0x0054       ub2 kccfhtyp                          @54       0x0003    ……    ub4 kcvfhrfn                             @528      0x00000054  ---确认是84号文件   ……

通过bbed修改文件相关信息,然后尝试rename文件,但是recover datafile 84报错

Mon May 15 09:49:44 2023 alter database rename file '/u01/prod/proddb/9.2.0/dbs/MISSING00084' to '/dev/ra_txn_ind12.dbf' Mon May 15 09:49:44 2023 Completed: alter database rename file '/u01/prod/proddb/9.2.0 Mon May 15 09:51:15 2023 ALTER DATABASE RECOVER  datafile 84  Media Recovery Start Mon May 15 09:51:15 2023 Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_467190.trc: ORA-07445: exception encountered: core dump [] [] [] [] [] []

通过处理之后,数据库recover 正常,但是open报ORA-600 4193错误

Mon May 15 09:57:53 2023 ALTER DATABASE RECOVER  DATABASE  Media Recovery Start Mon May 15 09:57:53 2023 Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0   Mem# 0 errs 0: /dev/rlog01a.dbf   Mem# 1 errs 0: /dev/rlog01b.dbf Media Recovery Complete Completed: ALTER DATABASE RECOVER  DATABASE  Mon May 15 09:59:24 2023 alter database open Mon May 15 09:59:24 2023 Beginning crash recovery of 1 threads Mon May 15 09:59:24 2023 Started redo scan Mon May 15 09:59:24 2023 Completed redo scan  75 redo blocks read, 0 data blocks need recovery Mon May 15 09:59:24 2023 Started recovery at  Thread 1: logseq 4, block 2, scn 3301.2638369687 Mon May 15 09:59:24 2023 Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0   Mem# 0 errs 0: /dev/rlog01a.dbf   Mem# 1 errs 0: /dev/rlog01b.dbf Mon May 15 09:59:24 2023 Completed redo application Mon May 15 09:59:24 2023 Ended recovery at  Thread 1: logseq 4, block 77, scn 3301.2638389765  0 data blocks read, 0 data blocks written, 75 redo blocks read Crash recovery completed successfully Mon May 15 09:59:25 2023 Thread 1 advanced to log sequence 5 Thread 1 opened at log sequence 5   Current log# 2 seq# 5 mem# 0: /dev/rlog02a.dbf   Current log# 2 seq# 5 mem# 1: /dev/rlog02b.dbf Successful open of redo thread 1 Mon May 15 09:59:25 2023 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Mon May 15 09:59:25 2023 SMON: enabling cache recovery Mon May 15 09:59:25 2023 ARC0: Media recovery disabled Mon May 15 09:59:25 2023 Successfully onlined Undo Tablespace 1. Dictionary check beginning Tablespace 'TEMP' #2 found in data dictionary, but not in the controlfile. Adding to controlfile. Dictionary check complete Mon May 15 09:59:25 2023 SMON: enabling tx recovery Mon May 15 09:59:25 2023 Database Characterset is ZHS16GBK Mon May 15 09:59:25 2023 Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413872.trc: ORA-00600: internal error code, arguments: [4193], [781], [6399], [], [], [], [], [] Mon May 15 09:59:25 2023 Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_844004.trc: ORA-00600: internal error code, arguments: [4193], [56042], [1895], [], [], [], [], [] Mon May 15 09:59:26 2023 Doing block recovery for fno: 12 blk: 153 Mon May 15 09:59:26 2023 Doing block recovery for fno: 12 blk: 2893 Mon May 15 09:59:26 2023 Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0   Mem# 0 errs 0: /dev/rlog02a.dbf Mon May 15 09:59:26 2023 Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0 Mon May 15 09:59:26 2023   Mem# 1 errs 0: /dev/rlog02b.dbf Mon May 15 09:59:26 2023   Mem# 0 errs 0: /dev/rlog02a.dbf   Mem# 1 errs 0: /dev/rlog02b.dbf Doing block recovery for fno: 12 blk: 3009 Mon May 15 09:59:26 2023 Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0   Mem# 0 errs 0: /dev/rlog02a.dbf   Mem# 1 errs 0: /dev/rlog02b.dbf Mon May 15 09:59:26 2023 Doing block recovery for fno: 12 blk: 89 Mon May 15 09:59:26 2023 Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0   Mem# 0 errs 0: /dev/rlog02a.dbf   Mem# 1 errs 0: /dev/rlog02b.dbf Mon May 15 09:59:26 2023 Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_844004.trc: ORA-00607: Internal error occurred while making a change to a data block ORA-00600: internal error code, arguments: [4193], [56042], [1895], [], [], [], [], [] Error 607 happened during db open, shutting down database USER: terminating instance due to error 607 Instance terminated by USER, pid = 844004 ORA-1092 signalled during: alter database open...

绕过该错误之后,数据库启动报ORA-600 2662错误

$ sqlplus "/ as sysdba" SQL*Plus: Release 9.2.0.6.0 - Production on Mon May 15 10:04:44 2023 Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved. Connected to an idle instance. SQL> startup mount pfile='/tmp/pfile' ORACLE instance started. Total System Global Area 1102023336 bytes Fixed Size                   744104 bytes Variable Size             922746880 bytes Database Buffers          167772160 bytes Redo Buffers               10760192 bytes Database mounted. SQL> alter database open; alter database open * ERROR at line 1: ORA-03113: end-of-file on communication channel

Mon May 15 10:05:03 2023 SMON: enabling cache recovery Mon May 15 10:05:03 2023 ARC0: Media recovery disabled Mon May 15 10:05:03 2023 SMON: enabling tx recovery Mon May 15 10:05:03 2023 Database Characterset is ZHS16GBK Mon May 15 10:05:03 2023 Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc: ORA-00600: internal error code, arguments: [2662], [3301], [2638409995], [3301], [2644132966], [4195678] Mon May 15 10:05:04 2023 Non-fatal internal error happenned while SMON was doing temporary segment drop. SMON encountered 1 out of maximum 100 non-fatal internal errors. Mon May 15 10:05:04 2023 Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc: ORA-00600: internal error code, arguments: [2662], [3301], [2638409998], [3301], [2644132966], [4195678] Mon May 15 10:05:04 2023 Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc: ORA-00600: internal error code, arguments: [2662], [3301], [2638409998], [3301], [2644132966], [4195678] SMON: terminating instance due to error 600 Instance terminated by SMON, pid = 413880

解决该错误之后,数据库open正常

$ sqlplus "/ as sysdba" SQL*Plus: Release 9.2.0.6.0 - Production on Mon May 15 10:10:30 2023 Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved. Connected to an idle instance. SQL> startup mount pfile='/tmp/pfile' ORACLE instance started. Total System Global Area 1102023336 bytes Fixed Size                   744104 bytes Variable Size             922746880 bytes Database Buffers          167772160 bytes Redo Buffers               10760192 bytes Database mounted. SQL> alter database open; Database altered.

逻辑方式导出数据,本次恢复任务基本完成.
以前有过的类似恢复案例(类似较多选择典型几个):
ORA-600 2663
ORA-600 2663 故障恢复
ORA-600 2662
ora-600 2662和ora-600 kclchkblk_4恢复
redo异常 ORA-600 kclchkblk_4 故障恢复
ORA-600 4193 错误说明和解决
ORA-00600 [2662]和ORA-00600 [4194]恢复