Oracle Study之--Oracle 11gR2 RAC crs启动故障


系统环境：

    操作系统：RedHat EL5

    Cluster：  Oracle 11gR2 Grid

    Oracle：   Oracle 11gR2 

RAC环境中，其中一个节点crsd进程启动失败：

[grid@wyb666 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


1、查看日志
[root@wyb666 ~]# cd /u01/11.2.0/grid/log
[root@wyb666 log]# ls
crs  diag  wyb666
[root@wyb666 log]# cd wyb666/
[root@wyb666 wyb666]# ls
admin  alertwyb666.log  crsd  ctssd    evmd   gnsd   mdnsd  racg
agent  client           cssd  diskmon  gipcd  gpnpd  ohasd  srvm
[root@wyb666 wyb666]# cd crsd
[root@wyb666 crsd]# ls
crsd.log  crsdOUT.log  crsd.trc
[root@wyb666 crsd]# tail crsd.log 
2015-06-08 11:41:28.430: [  OCRASM][3046103824]proprasmcl: asmhandle is NULL
2015-06-08 11:41:28.430: [  OCRRAW][3046103824]proprinit: Could not open raw device 
2015-06-08 11:41:28.430: [  OCRASM][3046103824]proprasmcl: asmhandle is NULL
2015-06-08 11:41:28.430: [  OCRAPI][3046103824]a_init:16!: Backend init unsuccessful : [26]
2015-06-08 11:41:28.430: [  CRSOCR][3046103824] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2015-06-08 11:41:28.430: [    CRSD][3046103824][PANIC] CRSD exiting: Could not init OCR, code: 26
2015-06-08 11:41:28.430: [    CRSD][3046103824] Done.

从以上可以看出：在crs service启动时，无法访问OCR磁盘，访问ASM DISKGROUP 失败！

2、查看ASM磁盘组信息
[root@wyb666 crsd]# service oracleasm listdisks
ASM_DATA1
ASM_DATA2
ASM_RCY1
ASM_RCY2
OCR_VOTE1
OCR_VOTE2
OCR_VOTE3
[root@wyb666 crsd]# ls -l /dev/oracleasm/disks
total 0
brw-rw---- 1 grid asmadmin 8, 21 Jun  8 09:23 ASM_DATA1
brw-rw---- 1 grid asmadmin 8, 22 Jun  8 09:23 ASM_DATA2
brw-rw---- 1 grid asmadmin 8, 23 Jun  8 09:23 ASM_RCY1
brw-rw---- 1 grid asmadmin 8, 24 Jun  8 09:23 ASM_RCY2
brw-rw---- 1 grid asmadmin 8, 17 Jun  8 11:53 OCR_VOTE1
brw-rw---- 1 grid asmadmin 8, 18 Jun  8 11:53 OCR_VOTE2
brw-rw---- 1 grid asmadmin 8, 19 Jun  8 11:53 OCR_VOTE3


解决方法：

重新启动CRS服务，然后再手工启动ASM Instance：

启动ASM Instance
[root@wyb666 crsd]# su - grid

[grid@wyb666 ~]$ sqlplus '/as sysasm'
SQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 8 11:59:26 2015
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> conn /as  sysasm
Connected to an idle instance.
SQL> startup
ASM instance started
Total System Global Area  284565504 bytes
Fixed Size                  1336036 bytes
Variable Size             258063644 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled


SQL> select name,state from v$asm_diskgroup;
NAME                           STATE
------------------------------ -----------
DATA                           MOUNTED
DG1                            MOUNTED
RCY1                           MOUNTED

启动成功：
[root@wyb666 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online