RAC节点频繁重启出现ORA-29702
数据库的Oracle 10204 RAC for Windows出现频繁节点重启的问题。
从告警日志看，当前节点的重启一般发生在节点刚启动或关闭时：
Thu May 03 17:22:45 2012
cluster interconnect IPC tb version:Oracle 9i Winsock2 TCP/IP IPC
IPC Vendor 0 proto 0
Version 0.0
PMON started with pid=2, OS id=1616
DIAG started with pid=3, OS id=120
PSP0 started with pid=4, OS id=6104
LMON started with pid=5, OS id=3844
LMD0 started with pid=6, OS id=6120
LMS0 started with pid=7, OS id=3548
LMS1 started with pid=8, OS id=5688
LMS2 started with pid=9, OS id=3636
LMS3 started with pid=10, OS id=3588
MMAN started with pid=11, OS id=3168
DBW0 started with pid=12, OS id=3208
DBW1 started with pid=13, OS id=5784
LGWR started with pid=14, OS id=6208
CKPT started with pid=15, OS id=3100
SMON started with pid=16, OS id=5948
RECO started with pid=17, OS id=3748
CJQ0 started with pid=18, OS id=7152
MMON started with pid=19, OS id=4552
MMNL started with pid=20, OS id=6940
Thu May 03 17:22:46 2012
lmon registered with NM - instance id 1 (internal mem no 0)
Thu May 03 17:22:46 2012
Reconfiguration started (old inc 0, new inc 8)
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Error: KGXGN aborts the instance (6)
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lmon_3844.trc:
ORA-29702: ???????????
LMON: terminating instance due to error 29702
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_pmon_1616.trc:
ORA-29702: ???????????
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_psp0_6104.trc:
ORA-29702: ???????????
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_dbw0_3208.trc:
ORA-29702: ???????????
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_mman_3168.trc:
ORA-29702: ???????????
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_dbw1_5784.trc:
ORA-29702: ???????????
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_ckpt_3100.trc:
ORA-29702: ???????????
Thu May 03 17:22:51 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lgwr_6208.trc:
ORA-29702: ???????????
Thu May 03 17:22:52 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_reco_3748.trc:
ORA-29702: ???????????
Thu May 03 17:22:52 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_smon_5948.trc:
ORA-29702: ???????????
Thu May 03 17:22:52 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lms1_5688.trc:
ORA-29702: ???????????
Thu May 03 17:22:52 2012
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_lms0_3548.trc:
ORA-29702: ???????????
Instance terminated by LMON, pid = 3844
而从CSSD日志文件中可以发现下面的信息：
[ CSSD]2012-04-29 16:26:07.953 [7112] >TRACE: clssgmReconfigThread: completed for reconfig(13), with status(1)
2012-04-30 09:07:04.718: [ OCROSD]utgdv:11:could not read reg value ocrmirrorconfig_loc os error=操作系统找不到已输入的环境选项。
2012-04-30 09:07:04.718: [ OCROSD]utgdv:11:could not read reg value ocrmirrorconfig_loc os error=操作系统找不到已输入的环境选项。
[ CSSD]2012-04-30 09:07:04.765 >USER: Copyright 2012, Oracle version 10.2.0.4.0
[ CSSD]2012-04-30 09:07:04.765 >USER: CSS daemon log for node crct-oadb, number 1, in cluster crs
[ CSSD]2012-04-30 09:07:04.765 [3780] >TRACE: clssscmain: local-only set to false
[ clsdmt]Listening to (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=61180))
[ CSSD]2012-04-30 09:07:04.781 [3780] >TRACE: clssnmReadNodeInfo: added node 1 (crct-oadb) to cluster
[ CSSD]2012-04-30 09:07:04.781 [3780] >TRACE: clssnmReadNodeInfo: added node 2 (crct-oapt) to cluster
[ CSSD]2012-04-30 09:07:04.828 [3724] >TRACE: clssnm_skgxninit: Compatible vendor clusterware not in use
[ CSSD]2012-04-30 09:07:04.828 [3724] >TRACE: clssnm_skgxnmon: skgxn init failed
[ CSSD]2012-04-30 09:07:04.843 [3780] >TRACE: clssnmNMInitialize: misscount set to (60)
[ CSSD]2012-04-30 09:07:04.843 [3780] >TRACE: clssnmNMInitialize: Network heartbeat thresholds are: impending reconfig 30000 ms, reconfig start (misscount) 60000 ms
[ CSSD]2012-04-30 09:07:04.843 [3780] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (0/\\.\votedsk1)
[ CSSD]2012-04-30 09:07:04.843 [3112] >TRACE: clssnmvDPT: spawned for disk 0 (\\.\votedsk1)
[ CSSD]2012-04-30 09:07:06.843 [3112] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (0/\\.\votedsk1)
[ CSSD]2012-04-30 09:07:06.843 [4492] >TRACE: clssnmvKillBlockThread: spawned for disk 0 (\\.\votedsk1) initial sleep interval (1000)ms
根据这些信息查询，发现属于10.2.0.4上的bug：10gR2/11gR1: Instances Abort With ORA-29702 When The Server is rebooted or shut down [ID 752399.1]。这个bug影响10.2.0.1到10.2.0.4以及11.1.0.6和11.1.0.7版本。
Oracle给出的解决方案是修改操作系统启动时调用的K96 link替换为K19 link。不过当前版本是Windows环境，显然这种解决方法并不适用。恐怕除了升级版本外，没有什么太好的其他解决方法。
将产品环境部署在Windows环境下的系统确实少见，而在Windows上部署RAC的就更是凤毛麟角了，而大多数这样部署的不只是对于Oracle不了解，连Windows和Linux的稳定性的差别都不是很清楚，出现各种问题的几率自然要大得多了。