一次oracle support的危机现场处理（之二）

天有不测风云，

IBM 的中端磁盘阵列这次又惹祸了。在微码升级的时候，

DS4800 发生故障。导致当时在这台库上的几套生产库报错了。

IBM工程师把阵列故障恢复后，拍拍屁股走人，阵列上的数据又得我们来想办法抢救了。。。。

还好有oracle support的支持我们最终化解了这次危机。下文详细介绍了整个恢复的过程。

 

 

oracle工程师到达现场后，根据错误信息，
|||||||||||||||||||||||||||||||||||||||||||||||||||||
错误信息如下所示：
Mon Apr 26 20:40:06 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/product/10.2.0/db/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =121
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 1000
  sessions                 = 1105
  sga_max_size             = 12884901888
  __shared_pool_size       = 5066719232
  __large_pool_size        = 16777216
  __java_pool_size         = 16777216
  __streams_pool_size      = 0
  sga_target               = 12884901888
  control_files            = /oradata/ccdev1/control01.ctl, /oradata/ccdev1/control02.ctl, /oradata/ccdev1/control03.ctl
  db_block_size            = 8192
  __db_cache_size          = 7767851008
  db_writer_processes      = 4
  compatible               = 10.2.0.3.0
  db_files                 = 300
  db_file_multiblock_read_count= 16
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=ccdev1XDB)
  job_queue_processes      = 10
  background_dump_dest     = /oracle/admin/ccdev1/bdump
  user_dump_dest           = /oracle/admin/ccdev1/udump
  core_dump_dest           = /oracle/admin/ccdev1/cdump
  audit_file_dest          = /oracle/admin/ccdev1/adump
  db_name                  = ccdev1
  open_cursors             = 300
  pga_aggregate_target     = 4294967296
PMON started with pid=2, OS id=12592
PSP0 started with pid=3, OS id=12594
MMAN started with pid=4, OS id=12596
DBW0 started with pid=5, OS id=12598
DBW1 started with pid=6, OS id=12600
DBW2 started with pid=7, OS id=12602
DBW3 started with pid=8, OS id=12604
LGWR started with pid=9, OS id=12606
CKPT started with pid=10, OS id=12608
SMON started with pid=11, OS id=12610
RECO started with pid=12, OS id=12612
CJQ0 started with pid=13, OS id=12614
MMON started with pid=14, OS id=12616
Mon Apr 26 20:40:07 2010
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=12618
Mon Apr 26 20:40:07 2010
starting up 1 shared server(s) ...
Mon Apr 26 20:40:07 2010
ALTER DATABASE   MOUNT
Mon Apr 26 20:40:14 2010
Setting recovery target incarnation to 2
Mon Apr 26 20:40:14 2010
Successful mount of redo thread 1, with mount id 1621305962
Mon Apr 26 20:40:14 2010
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Mon Apr 26 20:40:14 2010
ALTER DATABASE OPEN
Mon Apr 26 20:40:14 2010
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Mon Apr 26 20:40:15 2010
Started redo scan
Mon Apr 26 20:40:15 2010
Completed redo scan
 37 redo blocks read, 6 data blocks need recovery
Mon Apr 26 20:40:15 2010
Started redo application at
 Thread 1: logseq 1340, block 3
Mon Apr 26 20:40:15 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1340 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Mon Apr 26 20:40:15 2010
Completed redo application
Mon Apr 26 20:40:15 2010
Completed crash recovery at
 Thread 1: logseq 1340, block 40, scn 9549479515357
 6 data blocks read, 6 data blocks written, 37 redo blocks read
Mon Apr 26 20:40:15 2010
Thread 1 advanced to log sequence 1341 (thread open)
Thread 1 opened at log sequence 1341
  Current log# 3 seq# 1341 mem# 0: /oradata/ccdev1/redo03.log
Successful open of redo thread 1
Mon Apr 26 20:40:15 2010
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Apr 26 20:40:15 2010
SMON: enabling cache recovery
Mon Apr 26 20:40:15 2010
Successfully onlined Undo Tablespace 1.
Mon Apr 26 20:40:15 2010
SMON: enabling tx recovery
Mon Apr 26 20:40:15 2010
Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 4
Mon Apr 26 20:40:15 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2663], [2223], [1767217797], [2223], [1767218866], [], [], []
Mon Apr 26 20:40:17 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-01595: error freeing extent (10) of rollback segment (1))
ORA-00600: internal error code, arguments: [2663], [2223], [1767217797], [2223], [1767218866], [], [], []
Mon Apr 26 20:40:17 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217797], [2223], [1767218866], [205521621], [], []
Mon Apr 26 20:40:19 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217797], [2223], [1767218866], [205521621], [], []
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 2 out of maximum 100 non-fatal internal errors.
Mon Apr 26 20:40:21 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217798], [2223], [1767218866], [205521621], [], []
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 3 out of maximum 100 non-fatal internal errors.
Mon Apr 26 20:40:23 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217799], [2223], [1767218866], [205521621], [], []
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 4 out of maximum 100 non-fatal internal errors.
Mon Apr 26 20:40:34 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217803], [2223], [1767218866], [205521621], [], []
Mon Apr 26 20:40:35 2010
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 5 out of maximum 100 non-fatal internal errors.
Mon Apr 26 20:40:45 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217806], [2223], [1767218866], [205521621], [], []
Mon Apr 26 20:40:47 2010
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 6 out of maximum 100 non-fatal internal errors.
Mon Apr 26 20:40:57 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217810], [2223], [1767218866], [205521621], [], []
Mon Apr 26 20:40:58 2010
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 7 out of maximum 100 non-fatal internal errors.
Mon Apr 26 20:41:08 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12610.trc:
ORA-00600: internal error code, arguments: [2662], [2223], [1767217814], [2223], [1767218866], [205521621], [], []
Mon Apr 26 20:41:10 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_pmon_12592.trc:
ORA-00474: SMON process terminated with error
Mon Apr 26 20:41:10 2010
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 12592

||||||||||||||||||||||||||||||||||||||||||||||||||
这是一次数据库的完整的启动过程，
启动之后 数据库的报错信息如下所示
ORA-00600: internal error code, arguments: [2663], [2223], [1767217797], [2223], [1767218866], [], [], []
1767217797 是数据库的scn，
1767218866 是一个datafile的scn。
可以看到，数据库启动之后，smon将做Instance recovery操作，具体原因是数据库的database 将应用联机日志做前滚操作。
当滚到一定的时间点之后，还是没有能把数据库Open，
smon 终止并报错误
ORA-00474: SMON process terminated with error
最后pmon发现smon不存在，将把instance 终止
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 12592
   

oracle工程师分析故障原因，数据库正常运行的时候，因为磁盘阵列的故障，造成了IO丢失，这个时候，dbwr写到数据库中的
一个datafile的io顺利执行了，但是同时去跟新ctlfile的io丢失了。造成一个问题，即数据库datafile的scn(1767218866)比数据库时间(1767217797)要更新。
这样每次重启的时候，数据库的库scn将通过redo log前滚（instance recovery),可以看到因为库的scn一直是在向前追的过程中，
oracle工程师判断再重启几次之后，数据库的库时间将追上这个datafile的时间。
下面显示数据库时间（蓝色字体），datafile时间（紫色字体）

1767217797
1767217798
1767217799
1767217803
1767217806
1767217810
1767217814
1767218866

从上面可以看出，数据库的scn是一直向前追的。

果然再次启动数据库之后，2663 的600号错误小时，换成了4194的 600号错误，这个和回滚段的故障有关。
错误信息如下所示：
Mon Apr 26 20:42:05 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_12753.trc:
ORA-00600: internal error code, arguments: [4194], [21], [8], [], [], [], [], []

下一阶段的错误信息如下所示：

||||||||||||||||||||||||||||||||||||||||||||
Mon Apr 26 20:55:15 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/product/10.2.0/db/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =121
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 1000
  sessions                 = 1105
  event                    = 10015 trace name context forever, level 10
  sga_max_size             = 12884901888
  __shared_pool_size       = 5066719232
  __large_pool_size        = 16777216
  __java_pool_size         = 16777216
  __streams_pool_size      = 0
  sga_target               = 12884901888
  control_files            = /oradata/ccdev1/control01.ctl, /oradata/ccdev1/control02.ctl, /oradata/ccdev1/control03.ctl
  db_block_size            = 8192
  __db_cache_size          = 7767851008
  db_writer_processes      = 4
  compatible               = 10.2.0.3.0
  db_files                 = 300
  db_file_multiblock_read_count= 16
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=ccdev1XDB)
  job_queue_processes      = 10
  background_dump_dest     = /oracle/admin/ccdev1/bdump
  user_dump_dest           = /oracle/admin/ccdev1/udump
  core_dump_dest           = /oracle/admin/ccdev1/cdump
  audit_file_dest          = /oracle/admin/ccdev1/adump
  db_name                  = ccdev1
  open_cursors             = 300
  pga_aggregate_target     = 4294967296
PMON started with pid=2, OS id=13592
PSP0 started with pid=3, OS id=13594
MMAN started with pid=4, OS id=13596
DBW0 started with pid=5, OS id=13598
DBW1 started with pid=6, OS id=13600
DBW2 started with pid=7, OS id=13602
DBW3 started with pid=8, OS id=13604
LGWR started with pid=9, OS id=13606
CKPT started with pid=10, OS id=13608
SMON started with pid=11, OS id=13610
RECO started with pid=12, OS id=13612
CJQ0 started with pid=13, OS id=13614
MMON started with pid=14, OS id=13616
Mon Apr 26 20:55:17 2010
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=13618
Mon Apr 26 20:55:17 2010
starting up 1 shared server(s) ...
Mon Apr 26 20:55:17 2010
ALTER DATABASE   MOUNT
Mon Apr 26 20:55:24 2010
Setting recovery target incarnation to 2
Mon Apr 26 20:55:24 2010
Successful mount of redo thread 1, with mount id 1621322232
Mon Apr 26 20:55:24 2010
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Mon Apr 26 20:55:24 2010
ALTER DATABASE OPEN
Mon Apr 26 20:55:24 2010
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Mon Apr 26 20:55:24 2010
Started redo scan
Mon Apr 26 20:55:25 2010
Completed redo scan
 2 redo blocks read, 1 data blocks need recovery
Mon Apr 26 20:55:25 2010
Started redo application at
 Thread 1: logseq 1342, block 55
Mon Apr 26 20:55:25 2010
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1342 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo01.log
Mon Apr 26 20:55:25 2010
Completed redo application
Mon Apr 26 20:55:25 2010
Completed crash recovery at
 Thread 1: logseq 1342, block 57, scn 9549479558100
 1 data blocks read, 1 data blocks written, 2 redo blocks read
Mon Apr 26 20:55:25 2010
Thread 1 advanced to log sequence 1343 (thread open)
Thread 1 opened at log sequence 1343
  Current log# 2 seq# 1343 mem# 0: /oradata/ccdev1/redo02.log
Successful open of redo thread 1
Mon Apr 26 20:55:25 2010
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Apr 26 20:55:25 2010
SMON: enabling cache recovery
Mon Apr 26 20:55:25 2010
Successfully onlined Undo Tablespace 1.
Mon Apr 26 20:55:25 2010
SMON: enabling tx recovery
Mon Apr 26 20:55:25 2010
Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 4
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=22, OS id=13635
Mon Apr 26 20:55:27 2010
Errors in file /oracle/admin/ccdev1/udump/ccdev1_ora_13624.trc:
ORA-00600: internal error code, arguments: [4194], [45], [4], [], [], [], [], []
Mon Apr 26 20:55:28 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [21], [8], [], [], [], [], []
Mon Apr 26 20:55:30 2010
Doing block recovery for file 49 block 725
Block recovery from logseq 1343, block 42 to scn 9549479558193
Mon Apr 26 20:55:30 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery stopped at EOT rba 1343.43.16
Block recovery completed at rba 1343.43.16, scn 2223.1767259183
Doing block recovery for file 2 block 9
Block recovery from logseq 1343, block 42 to scn 9549479558190
Mon Apr 26 20:55:30 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.43.16, scn 2223.1767259183
Mon Apr 26 20:55:30 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-01595: error freeing extent (10) of rollback segment (1))
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [21], [8], [], [], [], [], []
Mon Apr 26 20:56:26 2010
Doing block recovery for file 50 block 5235
Doing block recovery for file 2 block 105
Block recovery from logseq 1343, block 41 to scn 9549479558189
Mon Apr 26 20:56:26 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.42.16, scn 2223.1767259182
Doing block recovery for file 50 block 5235
Block recovery from logseq 1343, block 41 to scn 9549479558215
Mon Apr 26 20:56:26 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery stopped at EOT rba 1343.45.16
Block recovery completed at rba 1343.45.16, scn 2223.1767259191
Mon Apr 26 21:10:34 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [44], [9], [], [], [], [], []
Mon Apr 26 21:10:37 2010
Doing block recovery for file 49 block 47529
Block recovery from logseq 1343, block 51 to scn 9549479564499
Mon Apr 26 21:10:37 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery stopped at EOT rba 1343.52.16
Block recovery completed at rba 1343.52.16, scn 2223.1767265490
Doing block recovery for file 2 block 153
Block recovery from logseq 1343, block 51 to scn 9549479564497
Mon Apr 26 21:10:37 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.52.16, scn 2223.1767265490
Mon Apr 26 21:10:37 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [44], [9], [], [], [], [], []
Mon Apr 26 21:10:38 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [57], [26], [], [], [], [], []
Doing block recovery for file 50 block 184225
Block recovery from logseq 1343, block 52 to scn 9549479564503
Mon Apr 26 21:10:41 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery stopped at EOT rba 1343.53.16
Block recovery completed at rba 1343.53.16, scn 2223.1767265494
Doing block recovery for file 2 block 73
Block recovery from logseq 1343, block 52 to scn 9549479564501
Mon Apr 26 21:10:41 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.53.16, scn 2223.1767265494
Mon Apr 26 21:10:41 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [57], [26], [], [], [], [], []
Mon Apr 26 21:10:42 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [59], [14], [], [], [], [], []
Doing block recovery for file 50 block 2692
Block recovery from logseq 1343, block 53 to scn 9549479564506
Mon Apr 26 21:10:45 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery stopped at EOT rba 1343.54.16
Block recovery completed at rba 1343.54.16, scn 2223.1767265497
Doing block recovery for file 2 block 89
Block recovery from logseq 1343, block 53 to scn 9549479564504
Mon Apr 26 21:10:45 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.54.16, scn 2223.1767265497
Mon Apr 26 21:10:45 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [59], [14], [], [], [], [], []
Mon Apr 26 21:10:46 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [6], [3], [], [], [], [], []
Doing block recovery for file 50 block 1405
Block recovery from logseq 1343, block 54 to scn 9549479564509
Mon Apr 26 21:10:49 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery stopped at EOT rba 1343.55.16
Block recovery completed at rba 1343.55.16, scn 2223.1767265500
Doing block recovery for file 2 block 41
Block recovery from logseq 1343, block 54 to scn 9549479564507
Mon Apr 26 21:10:49 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.55.16, scn 2223.1767265500
Mon Apr 26 21:10:49 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [6], [3], [], [], [], [], []
Mon Apr 26 21:10:50 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [44], [9], [], [], [], [], []
Doing block recovery for file 49 block 47529
Block recovery from logseq 1343, block 51 to scn 9549479564499
Mon Apr 26 21:10:53 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.52.16, scn 2223.1767265493
Doing block recovery for file 2 block 153
Block recovery from logseq 1343, block 51 to scn 9549479564513
Mon Apr 26 21:10:53 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.57.16, scn 2223.1767265506
Mon Apr 26 21:10:53 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [44], [9], [], [], [], [], []
Mon Apr 26 21:11:03 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [57], [26], [], [], [], [], []
Mon Apr 26 21:11:06 2010
Doing block recovery for file 50 block 184225
Block recovery from logseq 1343, block 52 to scn 9549479564503
Mon Apr 26 21:11:06 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.53.16, scn 2223.1767265496
Doing block recovery for file 2 block 73
Block recovery from logseq 1343, block 52 to scn 9549479564518
Mon Apr 26 21:11:06 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.58.16, scn 2223.1767265511
Mon Apr 26 21:11:06 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [57], [26], [], [], [], [], []
Mon Apr 26 21:11:16 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [21], [8], [], [], [], [], []
Mon Apr 26 21:11:19 2010
Doing block recovery for file 49 block 725
Block recovery from logseq 1343, block 58 to scn 9549479564526
Mon Apr 26 21:11:19 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery stopped at EOT rba 1343.61.16
Block recovery completed at rba 1343.61.16, scn 2223.1767265517
Doing block recovery for file 2 block 9
Block recovery from logseq 1343, block 58 to scn 9549479564524
Mon Apr 26 21:11:19 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1343 Reading mem 0
  Mem# 0: /oradata/ccdev1/redo02.log
Block recovery completed at rba 1343.61.16, scn 2223.1767265517
Mon Apr 26 21:11:19 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [21], [8], [], [], [], [], []
Mon Apr 26 21:11:29 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_smon_13610.trc:
ORA-00600: internal error code, arguments: [4194], [44], [9], [], [], [], [], []
Mon Apr 26 21:11:32 2010
Errors in file /oracle/admin/ccdev1/bdump/ccdev1_pmon_13592.trc:
ORA-00474: SMON process terminated with error
Mon Apr 26 21:11:32 2010
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 13592
 
我们查询了一下file 2和 file 50对应的 数据文件，都是Undo tablespace的datafile。
oracle 工程师解释 4194号错误是因为undo tablespace里面的坏块错误导致的数据库错误。
发生这个错误的时候，开始阶段数据库处于open状态，但是过了一会，smon因为发现了undo 的坏块将终止，
最后导致pmon 因为smon的丢失而把instance 干掉。

 

解决的方法分为两种，
查看数据库的Undo tablespace 里面是否有活动的会话。(通过10015事件）

1、如果没有，可以直接用新的Undo tablespace，然后drop掉这个老的undo tablespace
2、如果有，则需要做其他操作
查看dba_corruption_block 来找到corrupt的块

然后选择 corruption 方式来解决。

||||||||||||||||||||||||||||||||||||||||||||||||||||||

在数据库层面设置10015事件，
events = "10015 trace name context forever,level 10" scope=spfile
再重启数据库，
检查�