RAC集群启动实例遭遇PRCR-1079、CRS-5017、ORA-01102cannot mount database in EXCLUSIVE mode

环境：Oracle 11g R2 RAC集群

在做异机恢复时，通过srvctl start database -d 命令启动数据实例，遇到了如下错误。导致只有一个节点能起来。

[grid@orcl02 grid_1]$ srvctl start database -d orcl
PRCR-1079 : Failed to start resource ora.orcl.db
CRS-5017: The resource action "ora.orcl.db start" encountered the following error: 
ORA-01102: cannot mount database in EXCLUSIVE mode
. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid_1/log/orcl02/agent/crsd/oraagent_oracle//oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'orcl02' failed
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy

[grid@orcl01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       orcl01                                       
               ONLINE  ONLINE       orcl02                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       orcl01                                       
               ONLINE  ONLINE       orcl02                                       
ora.asm
               ONLINE  ONLINE       orcl01                   Started             
               ONLINE  ONLINE       orcl02                   Started             
ora.gsd
               OFFLINE OFFLINE      orcl01                                       
               OFFLINE OFFLINE      orcl02                                       
ora.net1.network
               ONLINE  ONLINE       orcl01                                       
               ONLINE  ONLINE       orcl02                                       
ora.ons
               ONLINE  ONLINE       orcl01                                       
               ONLINE  ONLINE       orcl02                                       
ora.registry.acfs
               ONLINE  ONLINE       orcl01                                       
               ONLINE  ONLINE       orcl02                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       orcl01                                       
ora.cvu
      1        ONLINE  ONLINE       orcl02                                       
ora.oc4j
      1        ONLINE  ONLINE       orcl02                                       
ora.orcl.db
      1        ONLINE  OFFLINE                               Instance Shutdown   
      2        ONLINE  ONLINE       orcl02                   Open                
ora.orcl01.vip
      1        ONLINE  ONLINE       orcl01                                       
ora.orcl02.vip
      1        ONLINE  ONLINE       orcl02                                       
ora.scan1.vip
      1        ONLINE  ONLINE       orcl01                                       

crsd日志：

2019-05-14 00:23:48.335: [ora.orcl.db][4006668032]{2:45299:306} [start] InstAgent::stop: }
2019-05-14 00:23:48.335: [ora.orcl.db][4006668032]{2:45299:306} [start] clean  }
2019-05-14 00:23:48.335: [ora.orcl.db][4006668032]{2:45299:306} [start] InstAgent::start exception }
2019-05-14 00:23:48.336: [ora.orcl.db][4006668032]{2:45299:306} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01102: cannot mount database in EXCLUSIVE mode
. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid_1/log/orcl02/agent/crsd/oraagent_oracle//oraagent_oracle.log".

2019-05-14 00:23:48.337: [    AGFW][4004566784]{2:45299:306} Agent sending reply for: RESOURCE_START[ora.orcl.db 2 1] ID 4098:1816
2019-05-14 00:23:48.341: [ora.orcl.db][4006668032]{2:45299:306} [start] (:CLSN00107:) clsn_agent::start }
2019-05-14 00:23:48.341: [    AGFW][4006668032]{2:45299:306} Command: start for resource: ora.orcl.db 2 1 completed with status: FAIL
2019-05-14 00:23:48.342: [CLSFRAME][4025780000] TM [MultiThread] is changing desired thread # to 3. Current # is 2
2019-05-14 00:23:48.342: [CLSFRAME][4025780000] TM [MultiThread] is changing desired thread # to 4. Current # is 3
2019-05-14 00:23:48.343: [    AGFW][4004566784]{2:45299:306} Agent sending reply for: RESOURCE_START[ora.orcl.db 2 1] ID 4098:1816
2019-05-14 00:23:48.353: [ora.orcl.db][4006668032]{2:45299:306} [check] ConnectionPool::resetConnection  s_statusOfConnectionMap 00b07d28
2019-05-14 00:23:48.354: [ora.orcl.db][4006668032]{2:45299:306} [check] ConnectionPool::resetConnection sid orcl2 status  2
2019-05-14 00:23:48.354: [ora.orcl.db][4006668032]{2:45299:306} [check] Gimh::check OH /u01/app/oracle/product/11.2.0/db_1 SID orcl2
2019-05-14 00:23:48.354: [ora.orcl.db][4006668032]{2:45299:306} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 1(Instance Shutdown),7(Cluster Reconfiguration) exists
2019-05-14 00:23:48.354: [ora.orcl.db][4006668032]{2:45299:306} [check] clsnDbAgent:checkCbk 9 state shutdown shutdownBegin:1 GIMH 1
2019-05-14 00:23:48.354: [ora.orcl.db][4006668032]{2:45299:306} [check] clsnDbAgent:checkCbk clsagfw_res_status ret 2
2019-05-14 00:23:48.354: [ USRTHRD][4006668032]{2:45299:306} DbAgent::DedicatedThread::stop entry
2019-05-14 00:23:48.354: [ USRTHRD][4006668032]{2:45299:306} DbAgent::DedicatedThread::abortConnxn entry

查看挂掉的实例2日志：

lmon registered with NM - instance number 2 (internal mem no 1)
USER (ospid: 58891): terminating the instance due to error 1102
Instance terminated by USER, pid = 58891
Mon May 13 15:41:44 2019
ORA-01565: Unable to open Spfile ?/dbs/spfile@.ora.
Mon May 13 15:45:00 2019

可以看到是访问ASM上的spfile文件出现了问题。为了检查spfile。手动生成pfile。

SQL> create pfile from spfile='+data/orcl/spfileorcl.ora';

检查相关参数，*.cluster_database=FALSE，集群模式被设置成了false，导致只有集群中只能有一个实例存在。

在节点1上修改参数并重启实例1。

SQL> alter system set cluster_database=true scope=spfile sid='orcl1'; 
SQL> shutdown immediate;
SQL> startup;
SQL> show parameter cluster_database;
NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
cluster_database		     boolean	 TRUE
cluster_database_instances	     integer	 2

更改完参数后启动节点2实例。集群恢复正常