12c OCR corrupted results in CRS stack down.
1. check crsd.trc
2017-03-21 16:14:44.667838 : CRSOCR:2580299264: OCR context init failure. Error: PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]
2017-03-21 16:14:44.668018 : CRSD:2580299264: Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]
2017-03-21 16:14:44.668032 : CRSD:2580299264: [PANIC] CRSD exiting: Could not init OCR, code: 26
2017-03-21 16:14:44.668078 : CRSD:2580299264: Done.
2. ocrcheck reported same problem.
[root@testdb01 ~]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]
3. check if there's any ocr backups.
[root@testdb01 ~]# cd /u01/app/12.1.0.2/grid/cdata/cluster-test07/
[root@testdb01 cluster-test07]# ll
total 25156
-rw-r--r-- 1 oracle oinstall 1687552 Feb 22 15:49 backup_20170222_154955.ocr
-rw-r--r-- 1 oracle oinstall 1712128 Feb 22 15:49 backup_20170222_154956.ocr
-rw-r--r-- 1 oracle oinstall 1863680 Mar 8 02:06 week.ocr
-rw-r--r-- 1 oracle oinstall 1863680 Mar 15 02:07 week_.ocr
-rw-r--r-- 1 oracle oinstall 1863680 Mar 19 02:07 day.ocr
-rw-r--r-- 1 oracle oinstall 1863680 Mar 20 02:07 day_.ocr
-rw-r--r-- 1 oracle oinstall 1863680 Mar 20 06:07 backup02.ocr
-rw-r--r-- 1 oracle oinstall 1863680 Mar 20 10:07 backup01.ocr
-rw-r--r-- 1 oracle oinstall 1863680 Mar 20 14:07 backup00.ocr
-rw-r--r-- 1 root root 1859584 Mar 20 18:37 23219979.ocr
-rw-r--r-- 1 root root 1863680 Mar 20 22:37 41104710.ocr
-rw-r--r-- 1 root root 1863680 Mar 21 02:37 25823987.ocr
-rw-r--r-- 1 root root 1863680 Mar 21 06:37 42138219.ocr
-rw-r--r-- 1 root root 1863680 Mar 21 10:37 17919197.ocr
[root@testdb01 cluster-test07]# ls -lht
total 25M
-rw-r--r-- 1 root root 1.8M Mar 21 10:37 17919197.ocr
-rw-r--r-- 1 root root 1.8M Mar 21 06:37 42138219.ocr
-rw-r--r-- 1 root root 1.8M Mar 21 02:37 25823987.ocr
-rw-r--r-- 1 root root 1.8M Mar 20 22:37 41104710.ocr
-rw-r--r-- 1 root root 1.8M Mar 20 18:37 23219979.ocr
-rw-r--r-- 1 oracle oinstall 1.8M Mar 20 14:07 backup00.ocr
-rw-r--r-- 1 oracle oinstall 1.8M Mar 20 10:07 backup01.ocr
-rw-r--r-- 1 oracle oinstall 1.8M Mar 20 06:07 backup02.ocr
-rw-r--r-- 1 oracle oinstall 1.8M Mar 20 02:07 day_.ocr
-rw-r--r-- 1 oracle oinstall 1.8M Mar 19 02:07 day.ocr
-rw-r--r-- 1 oracle oinstall 1.8M Mar 15 02:07 week_.ocr
-rw-r--r-- 1 oracle oinstall 1.8M Mar 8 02:06 week.ocr
-rw-r--r-- 1 oracle oinstall 1.7M Feb 22 15:49 backup_20170222_154956.ocr
-rw-r--r-- 1 oracle oinstall 1.7M Feb 22 15:49 backup_20170222_154955.ocr
4. restore OCR
[root@testdb01 cluster-test07]# ocrconfig -restore backup00.ocr
[root@testdb01 cluster-test07]# crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'testdb01'
CRS-2676: Start of 'ora.crsd' on 'testdb01' succeeded
[root@testdb01 cluster-test07]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1652
Available space (kbytes) : 407916
ID : 1368296587
Device/File Name : +DBFS_DG
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@testdb01 cluster-test07]# crsctl check cluster -all
**************************************************************
testdb01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
test07db02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@testdb01 cluster-test07]# crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA_test07.dg
ONLINE ONLINE testdb01 STABLE
ONLINE ONLINE test07db02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE testdb01 STABLE
ONLINE ONLINE test07db02 STABLE
ora.LISTENER.lsnr
OFFLINE OFFLINE testdb01 STABLE
ONLINE ONLINE test07db02 STABLE
ora.RECO_test07.dg
ONLINE ONLINE testdb01 STABLE
ONLINE ONLINE test07db02 STABLE
ora.asm
ONLINE ONLINE testdb01 Started,STABLE
ONLINE ONLINE test07db02 Started,STABLE
ora.net1.network
ONLINE ONLINE testdb01 STABLE
ONLINE ONLINE test07db02 STABLE
ora.ons
ONLINE ONLINE testdb01 STABLE
ONLINE ONLINE test07db02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb01 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE test07db02 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE test07db02 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE test07db02 192.168.17.3 192.168
.17.4,STABLE
ora.cvu
1 ONLINE ONLINE test07db02 STABLE
ora.testdb01.vip
1 ONLINE ONLINE testdb01 STABLE
ora.test07db02.vip
1 ONLINE ONLINE test07db02 STABLE
ora.mgmtdb
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.oc4j
1 ONLINE ONLINE test07db02 STABLE
ora.scan1.vip
1 ONLINE ONLINE testdb01 STABLE
ora.scan2.vip
1 ONLINE ONLINE test07db02 STABLE
ora.scan3.vip
1 ONLINE ONLINE test07db02 STABLE
--------------------------------------------------------------------------------