dg备库加入集群和ORA-00304: requested INSTANCE_NUMBER is busy

我在搭建dg时，用

srvctl add database -d docare5pri -n docare5 -o /oracle/app/oracle/product/11.2.0/dbhome_1 -c RAC -p +DATA/icupri/parameter/spfileicu.ora -r physical_standby -s OPEN -t IMMEDIATE -a DATA,ARCH
srvctl add instance -d docare5pri -i docare51 -n smdb01
srvctl add instance -d docare5pri -i docare52 -n smdb02

添加数据库到ocr中，让集群管理，但是srvctl start 报错

 

srvctl启动rac数据库报错：

[oracle@smdb01 ~]$ srvctl start database -d docare5pri
PRCR-1079 : Failed to start resource ora.docare5pri.db
CRS-5017: The resource action "ora.docare5pri.db start" encountered the following error: 
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/oracle/app/11.2.0/grid/log/smdb02/agent/crsd/oraagent_oracle//oraagent_oracle.log".

CRS-2674: Start of 'ora.docare5pri.db' on 'smdb02' failed
CRS-2632: There are no more servers to try to place resource 'ora.docare5pri.db' on that would satisfy its placement policy

看到shared memeory我还以为是内存问题，看了内存又free很多，内核参数也设置正确，大页都正确加载了

 

集群状态：

ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       smdb02                                       
ora.cvu
      1        ONLINE  ONLINE       smdb02                                       
ora.docare5pri.db
      1        ONLINE  ONLINE       smdb01                   Open,Readonly       
      2        ONLINE  OFFLINE                               Instance Shutdown   
ora.icu.db
      1        OFFLINE OFFLINE                                                   
      2        OFFLINE OFFLINE                                                   
ora.icupri.db
      1        ONLINE  ONLINE       smdb01                   Open,Readonly       
      2        ONLINE  OFFLINE                               Instance Shutdown   
ora.oc4j
      1        ONLINE  ONLINE       smdb02                                       
ora.scan1.vip
      1        ONLINE  ONLINE       smdb02                                       
ora.smdb01.vip
      1        ONLINE  ONLINE       smdb01                                       
ora.smdb02.vip
      1        ONLINE  ONLINE       smdb02  

 

oraagent_oracle.log：

2020-06-11 22:26:35.380: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstConnection::connectInt: server not attached
2020-06-11 22:26:35.417: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstConnection:connectInt connected
2020-06-11 22:26:35.417: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstAgent::startup
2020-06-11 22:27:01.871: [ora.icupri.db][1215178496]{2:50963:52532} [start] ORA-00304: requested INSTANCE_NUMBER is busy

2020-06-11 22:27:01.871: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstAgent::start: 1 errcode 304
2020-06-11 22:27:01.871: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstAgent::start: ORA-304 instance busy  m_instanceType 1 m_lastOCIError 304
2020-06-11 22:27:01.872: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstAgent::start: ORA-304 instance busy  dbType RAC
2020-06-11 22:27:01.873: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstConnection::connectInt: server not attached
2020-06-11 22:27:01.916: [ora.icupri.db][1215178496]{2:50963:52532} [start] ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

2020-06-11 22:27:01.917: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstConnection::connectInt (2) Exception OCIException
2020-06-11 22:27:01.917: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstConnection:connect:excp OCIException OCI error 1034
2020-06-11 22:27:01.917: [ora.icupri.db][1215178496]{2:50963:52532} [start] InstAgent::start: 2 errcode 1034
2020-06-11 22:27:01.917: [ USRTHRD][1215178496]{2:50963:52532} InstConnection:~InstConnection: this 3000e590

 

从log日志中仔细查看发现，启动前面都没有报错，到requested INSTANCE_NUMBER is busy时报错

 

mos：

OERR: ORA-304 "requested INSTANCE_NUMBER is busy" Reference Note (Doc ID 18363.1)

   Bug 27731019 - Starting a RAC Database may Fail With Error ORA-304 (Doc ID 27731019.8)

解决办法：

Stop the clusterware and started it with 'crsctl stop crs' and 'crsctl start crs'

Bug 27731019 说是12c和19c才有这个bug，我的环境是11g！！！

 

为了彻底解决，把CRS中的数据库资源删除了

srvctl remove instance -d icupri -i icu1 
srvctl remove instance -d icupri -i icu2
srvctl remove database -d icupri

然后再root中重启集群，2个节点都需要执行

crsctl stop crs

crsctl start crs

检查启动好后加入数据库资源到CRS

srvctl add database -d docare5pri -n docare5 -o /oracle/app/oracle/product/11.2.0/dbhome_1 -c RAC -p +DATA/icupri/parameter/spfileicu.ora -r physical_standby -s OPEN -t IMMEDIATE -a DATA,ARCH
srvctl add instance -d docare5pri -i docare51 -n smdb01
srvctl add instance -d docare5pri -i docare52 -n smdb02

 

最后启动成功

srvctl start database -d icupri