ORA-00600:[kcratr_nab_less_than_odr], 分析解决

今天，在测试库测试时，在alter database open 时发现报：
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227],[10276], [], [], [], [], [], [], []

导致原因：这可能是由于控制文件的缺失，或者在线日志文件在实例恢复时不完整
尝试以下操作均未成功：

    SQL> alter database open resetlogs;
    alter database open resetlogs
    *
    ERROR at line 1:
    ORA-01139: RESETLOGS option only valid after an incomplete database recovery
     
     
    SQL> recover database;
    ORA-00283: recovery session canceled due to errors
    ORA-00264: no recovery required
     
     
    SQL> alter database open;
    alter database open
    *
    ERROR at line 1:
    ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [3464], [12432], [12534], [], [], [], [], [], [], []

查看告警日志：

    LTER DATABASE OPEN
    Beginning crash recovery of 1 threads
    parallel recovery started with 3 processes
    Started redo scan
    Completed redo scan
    read 9919 KB redo, 1763 data blocks need recovery
    Errors in file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_7784.trc (incident=201788):
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    Incident details in: d:\app\chjayhsx\diag\rdbms\orcl\orcl\incident\incdir_201788\orcl_ora_7784_i201788.trc
    Tue Sep 15 10:08:32 2015
    Trace dumping is performing id=[cdmp_20150915100832]
    Aborting crash recovery due to error 600
    Errors in file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_7784.trc:
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    Errors in file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_7784.trc:
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    ORA-600 signalled during: ALTER DATABASE OPEN...
    Tue Sep 15 10:08:52 2015
    alter database open
    Beginning crash recovery of 1 threads
    parallel recovery started with 3 processes
    Started redo scan
    Completed redo scan
    read 9919 KB redo, 1763 data blocks need recovery
    Errors in file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_7784.trc (incident=201789):
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    Incident details in: d:\app\chjayhsx\diag\rdbms\orcl\orcl\incident\incdir_201789\orcl_ora_7784_i201789.trc
    Aborting crash recovery due to error 600
    Errors in file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_7784.trc:
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    Errors in file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_7784.trc:
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    ORA-600 signalled during: alter database open

查看相应的trace文件

    Trace file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_7784.trc
    Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Windows NT Version V6.2
    CPU : 4 - type 8664, 4 Physical Cores
    Process Affinity : 0x0x0000000000000000
    Memory (Avail/Total): Ph:1458M/3982M, Ph+PgF:4383M/8078M
    Instance name: orcl
    Redo thread mounted by this instance: 1
    Oracle process number: 23
    Windows thread id: 7784, image: ORACLE.EXE (SHAD)
    *** 2015-09-15 10:08:29.402
    *** SESSION ID:(191.3) 2015-09-15 10:08:29.402
    *** CLIENT ID:() 2015-09-15 10:08:29.402
    *** SERVICE NAME:() 2015-09-15 10:08:29.402
    *** MODULE NAME:(sqlplus.exe) 2015-09-15 10:08:29.402
    *** ACTION NAME:() 2015-09-15 10:08:29.402
    Successfully allocated 3 recovery slaves
    Using 45 overflow buffers per recovery slave
    Thread 1 checkpoint: logseq 289, block 2, scn 7805163
    *** 2015-09-15 10:08:29.590
    cache-low rba: logseq 289, block 1978
    on-disk rba: logseq 290, block 10276, scn 7808333
    start recovery at logseq 289, block 1978, scn 0
    *** 2015-09-15 10:08:30.067
    Started writing zeroblks thread 1 seq 290 blocks 8227-8234
    *** 2015-09-15 10:08:30.068
    Completed writing zeroblks thread 1 seq 290
    ==== Redo read statistics for thread 1 ====
    Total physical reads (from disk and memory): 13999Kb
    -- Redo read_disk statistics --
    Read rate (ASYNC): 9920Kb in 0.54s => 17.94 Mb/sec
    Longest record: 14Kb, moves: 0/36179 (0%)
    Change moves: 2/32 (6%), moved: 0Mb
    Longest LWN: 1024Kb, moves: 2/247 (0%), moved: 2Mb
    Last redo scn: 0x0000.00772509 (7808265)
    ----------------------------------------------
    ----- Recovery Hash Table Statistics ---------
    Hash table buckets = 32768
    Longest hash chain = 2
    Average hash chain = 1763/1726 = 1.0
    Max compares per lookup = 2
    Avg compares per lookup = 70120/71549 = 1.0
    ----------------------------------------------
    WARNING! Crash recovery of thread 1 seq 290 is
    ending at redo block 8227 but should not have ended before
    redo block 10276
    Incident 201788 created, dump file: d:\app\chjayhsx\diag\rdbms\orcl\orcl\incident\incdir_201788\orcl_ora_7784_i201788.trc
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    *** 2015-09-15 10:08:33.089
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [290], [8227], [10276], [], [], [], [], [], [], []
    *** 2015-09-15 10:08:53.024
    Successfully allocated 3 recovery slaves
    Using 45 overflow buffers per recovery slave
    Thread 1 checkpoint: logseq 289, block 2, scn 7805163
    cache-low rba: logseq 289, block 1978
    on-disk rba: logseq 290, block 10276, scn 7808333
    start recovery at logseq 289, block 1978, scn 0

可以看出ORA-00600: internal error code, arguments:[kcratr_nab_less_than_odr], [1], [290], [8227], [10276], 和TRACE里的错误提示WARNING! Crash recovery of thread 1 seq290 is ending at redo block 12432 but should not have ended before redo block 12534 ，应该是由于我强制关闭电脑，导致LGWR写联机日志文件时失败，下次重新启动数据库时，需要做实例级恢复，而又无法从联机日志文件里获取到这些redo信息，因为上次关闭时，写日志失败了。


那么ORA-00600: internal error code, arguments:[kcratr_nab_less_than_odr], [1], [290], [8227], [10276]这些参数是什么意思呢？
实例需要恢复日志序列号为290的联机日志文件，需要恢复到编号为10276的日志块，而实际上只能恢复到第8227个日志块儿，所以库就启不来了。不过，从当前日志文件信息，可以看到，当前日志组的确是290：
ps：参见eygle博客->http://www.eygle.com/archives/2010/05/kcratr_nab_less_than_odr.html


解决办法：还原或者恢复这些文件
根据MOS Alter database open fails with ORA-00600 kcratr_nab_less_than_odr (Doc ID 1296264.1)

现在来尝试下进行控制文件的重建：

    SQL> Show parameter control_files
    NAME TYPE VALUE
    ------------------------------------ ----------- ------------------------------
    control_files string D:\APP\CHJAYHSX\ORADATA\ORCL\C
    ONTROL01.CTL, D:\APP\CHJAYHSX\
    FLASH_RECOVERY_AREA\ORCL\CONTR
    OL02.CTL
    SQL> oradebug setmypid
    已处理的语句
    SQL> Alter session set tracefile_identifier='controlfilerecreate' ;
    会话已更改。
    SQL> Alter database backup controlfile to trace noresetlogs;
    数据库已更改。
    SQL> oradebug tracefile_name ;
    d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_3768_controlfilerecreate.trc
    SQL> shutdown immediate
    ORA-01109: 数据库未打开
    已经卸载数据库。
    ORACLE 例程已经关闭。
    SQL> startup nomount;
    ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
    ORACLE 例程已经启动。
    Total System Global Area 1670221824 bytes
    Fixed Size 2176328 bytes
    Variable Size 1291848376 bytes
    Database Buffers 369098752 bytes
    Redo Buffers 7098368 bytes
    SQL> CREATE CONTROLFILE REUSE DATABASE "ORCL" NORESETLOGS ARCHIVELOG
    2 MAXLOGFILES 16
    3 MAXLOGMEMBERS 3
    4 MAXDATAFILES 100
    5 MAXINSTANCES 8
    6 MAXLOGHISTORY 292
    7 LOGFILE
    8 GROUP 1 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO01.LOG' SIZE 10M BLOCKSIZE 512,
    9 GROUP 2 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO02.LOG' SIZE 10M BLOCKSIZE 512,
    10 GROUP 3 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO03.LOG' SIZE 10M BLOCKSIZE 512,
    11 GROUP 4 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO04.LOG' SIZE 10M BLOCKSIZE 512,
    12 GROUP 5 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO05.LOG' SIZE 10M BLOCKSIZE 512,
    13 GROUP 6 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO06.LOG' SIZE 10M BLOCKSIZE 512,
    14 GROUP 7 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO07.LOG' SIZE 10M BLOCKSIZE 512
    15 -- STANDBY LOGFILE
    16 DATAFILE
    17 'D:\APP\CHJAYHSX\ORADATA\ORCL\SYSTEM01.DBF',
    18 'D:\APP\CHJAYHSX\ORADATA\ORCL\SYSAUX01.DBF',
    19 'D:\APP\CHJAYHSX\ORADATA\ORCL\UNDOTBS01.DBF',
    20 'D:\APP\CHJAYHSX\ORADATA\ORCL\USERS01.DBF',
    21 'D:\APP\CHJAYHSX\ORADATA\ORCL\EXAMPLE01.DBF',
    22 'D:\APP\CHJAYHSX\ORADATA\ORCL\RMAN_TBS01.DBF',
    23 'D:\APP\CHJAYHSX\ORADATA\ORCL\MANAGE_TEST.DBF',
    24 'D:\APP\CHJAYHSX\ORADATA\ORCL\TEST01.DBF',
    25 'D:\APP\CHJAYHSX\ORADATA\ORCL\SYSTEM02.DBF'
    26 CHARACTER SET ZHS16GBK
    27 ;
    控制文件已创建。
    SQL> RECOVER DATABASE ;
    完成介质恢复。
    SQL> alter database open;
    数据库已更改。
    SQL>

PS：这个重建脚本怎么来的呢？
SQL> oradebug setmypid 
SQL> Alter session set tracefile_identifier='controlfilerecreate' ;
SQL> Alter database backup controlfile to trace noresetlogs;
SQL> oradebug tracefile_name ; --> This command will give the path and name of the trace file

执行这些后，会在trace文件目录下，生成一个新的trc文件，

    Trace file d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_3768_controlfilerecreate.trc
    Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Windows NT Version V6.2
    CPU : 4 - type 8664, 4 Physical Cores
    Process Affinity : 0x0x0000000000000000
    Memory (Avail/Total): Ph:1030M/3982M, Ph+PgF:3770M/8078M
    Instance name: orcl
    Redo thread mounted by this instance: 1
    Oracle process number: 23
    Windows thread id: 3768, image: ORACLE.EXE (SHAD)


    *** 2015-09-15 12:20:28.646
    *** SESSION ID:(191.37) 2015-09-15 12:20:28.646
    *** CLIENT ID:() 2015-09-15 12:20:28.646
    *** SERVICE NAME:() 2015-09-15 12:20:28.646
    *** MODULE NAME:(sqlplus.exe) 2015-09-15 12:20:28.646
    *** ACTION NAME:() 2015-09-15 12:20:28.646
     

    *** TRACE CONTINUED FROM FILE d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_3768.trc ***


    *** 2015-09-15 12:20:28.642
    -- The following are current System-scope REDO Log Archival related
    -- parameters and can be included in the database initialization file.
    --
    -- LOG_ARCHIVE_DEST=''
    -- LOG_ARCHIVE_DUPLEX_DEST=''
    --
    -- LOG_ARCHIVE_FORMAT=ARC%S_%R.%T
    --
    -- DB_UNIQUE_NAME="orcl"
    --
    -- LOG_ARCHIVE_CONFIG='SEND, RECEIVE, NODG_CONFIG'
    -- LOG_ARCHIVE_MAX_PROCESSES=4
    -- STANDBY_FILE_MANAGEMENT=MANUAL
    -- STANDBY_ARCHIVE_DEST=%ORACLE_HOME%\RDBMS
    -- FAL_CLIENT=''
    -- FAL_SERVER=''
    --
    -- LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST'
    -- LOG_ARCHIVE_DEST_1='MANDATORY NOREOPEN NODELAY'
    -- LOG_ARCHIVE_DEST_1='ARCH NOAFFIRM EXPEDITE NOVERIFY SYNC'
    -- LOG_ARCHIVE_DEST_1='NOREGISTER NOALTERNATE NODEPENDENCY'
    -- LOG_ARCHIVE_DEST_1='NOMAX_FAILURE NOQUOTA_SIZE NOQUOTA_USED NODB_UNIQUE_NAME'
    -- LOG_ARCHIVE_DEST_1='VALID_FOR=(PRIMARY_ROLE,ONLINE_LOGFILES)'
    -- LOG_ARCHIVE_DEST_STATE_1=ENABLE
    --
    -- The following commands will create a new control file and use it
    -- to open the database.
    -- Data used by Recovery Manager will be lost.
    -- Additional logs may be required for media recovery of offline
    -- Use this only if the current versions of all online logs are
    -- available.
    -- After mounting the created controlfile, the following SQL
    -- statement will place the database in the appropriate
    -- protection mode:
    -- ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE
    STARTUP NOMOUNT
    CREATE CONTROLFILE REUSE DATABASE "ORCL" NORESETLOGS ARCHIVELOG
        MAXLOGFILES 16
        MAXLOGMEMBERS 3
        MAXDATAFILES 100
        MAXINSTANCES 8
        MAXLOGHISTORY 292
    LOGFILE
      GROUP 1 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO01.LOG' SIZE 10M BLOCKSIZE 512,
      GROUP 2 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO02.LOG' SIZE 10M BLOCKSIZE 512,
      GROUP 3 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO03.LOG' SIZE 10M BLOCKSIZE 512,
      GROUP 4 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO04.LOG' SIZE 10M BLOCKSIZE 512,
      GROUP 5 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO05.LOG' SIZE 10M BLOCKSIZE 512,
      GROUP 6 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO06.LOG' SIZE 10M BLOCKSIZE 512,
      GROUP 7 'D:\APP\CHJAYHSX\ORADATA\ORCL\REDO07.LOG' SIZE 10M BLOCKSIZE 512
    -- STANDBY LOGFILE
    DATAFILE
      'D:\APP\CHJAYHSX\ORADATA\ORCL\SYSTEM01.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\SYSAUX01.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\UNDOTBS01.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\USERS01.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\EXAMPLE01.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\RMAN_TBS01.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\MANAGE_TEST.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\TEST01.DBF',
      'D:\APP\CHJAYHSX\ORADATA\ORCL\SYSTEM02.DBF'
    CHARACTER SET ZHS16GBK
    ;
    -- Commands to re-create incarnation table
    -- Below log names MUST be changed to existing filenames on
    -- disk. Any one log file from each branch can be used to
    -- re-create incarnation records.
    -- ALTER DATABASE REGISTER LOGFILE 'D:\APP\CHJAYHSX\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2015_09_15\O1_MF_1_1_%U_.ARC';
    -- ALTER DATABASE REGISTER LOGFILE 'D:\APP\CHJAYHSX\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2015_09_15\O1_MF_1_1_%U_.ARC';
    -- ALTER DATABASE REGISTER LOGFILE 'D:\APP\CHJAYHSX\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2015_09_15\O1_MF_1_1_%U_.ARC';
    -- Recovery is required if any of the datafiles are restored backups,
    -- or if the last shutdown was not normal or immediate.
    RECOVER DATABASE
    -- All logs need archiving and a log switch is needed.
    ALTER SYSTEM ARCHIVE LOG ALL;
    -- Database can now be opened normally.
    ALTER DATABASE OPEN;
    -- Commands to add tempfiles to temporary tablespaces.
    -- Online tempfiles have complete space information.
    -- Other tempfiles may require adjustment.
    ALTER TABLESPACE TEMP ADD TEMPFILE 'D:\APP\CHJAYHSX\ORADATA\ORCL\TEMP01.DBF' REUSE;
    -- End of tempfile additions.
    --

    *** 2015-09-15 12:20:36.610
    Processing Oradebug command 'tracefile_name '

    *** 2015-09-15 12:20:36.610
    Oradebug command 'tracefile_name ' console output:
    d:\app\chjayhsx\diag\rdbms\orcl\orcl\trace\orcl_ora_3768_controlfilerecreate.trc

这样就可以直接利用这个脚本重建控制文件了，然后打开数据库工作正常。

总结
方法一：利用在线日志还原
步骤：
SQL>Startup mount ;
SQL>Show parameter control_files
SQL> select a.member, a.group#, b.status from v$logfile a ,v$log b where a.group#=b.group# and b.status='CURRENT' ;
SQL> Shutdown immediate;
（确保有备份文件）
SQL> Startup mount ;
SQL> recover database using backup controlfile until cancel ; (得开启快闪功能)
（利用control_file的查询值恢复）
SQL> Alter database open resetlogs ;

方法二：利用RMAN
把数据库起到mount状态
$ rman target /
rman> spool log to '/tmp/rman.log';
rman> list backup ;
rman> exit
SQL> Show parameter control_files 
SQL> oradebug setmypid 
SQL> Alter session set tracefile_identifier='controlfilerecreate' ;
SQL> Alter database backup controlfile to trace noresetlogs;
SQL> oradebug tracefile_name ; --> This command will give the path and name of the trace file
--Go to this location ,Open this trace file and select the controlfile recreation script with NOResetlogs option 
SQL> Shutdown immediate;
Rename the existing controlfile to _old ---> This is Important as we need to have a backup of existing controlfile since we plan to recreate it
SQL> Startup nomount
Now run the Controlfile recreation script with NO Resetlogs option.
SQL> Alter database open ;
Rman> Catalog start with '' ;