10G rac 因为ocr原因导致crs不能启动的排查二例
近期遇到过两次RAC节点的主机后记 后CRS不能启动的情况。
案例1：LINUX+10.2.0.5RAC平台，OCR对应的裸设备权限在重启后不正确，因为设置裸设备权限的脚本设置有误。
案例2：主机版本为HP-UX B.11.31，使用的是的HP-UX Service Guard集群件，小机宕机重启后VG未挂载导致OCR所在磁盘无法访问。
记录如下：案例1：
LINUX+10.2.0.5RAC平台，OCR对应的裸设备权限在重启后不正确，因为设置裸设备权限的脚本设置有误。
情况如下：
[root@rac02 ~]# ps -ef|grep css
root 16820 1 0 May25 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root 16872 16818 0 May25 ? 00:01:48 /bin/sh /etc/init.d/init.cssd startcheck
root 16924 16820 0 May25 ? 00:01:38 /bin/sh /etc/init.d/init.cssd startcheck
root 17062 16823 0 May25 ? 00:01:50 /bin/sh /etc/init.d/init.cssd startcheck
root 17866 17636 0 19:32 pts/1 00:00:00 grep css
[root@rac02 ~]# tail /var/log/messages
Sep 11 19:33:04 rac02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.16924.
Sep 11 19:33:04 rac02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.17062.
Sep 11 19:33:04 rac02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.16872.
Sep 11 19:34:04 rac02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.16924.
Sep 11 19:34:04 rac02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.17062.
Sep 11 19:34:04 rac02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.16872.
[root@rac02 log]# cat /tmp/crsctl.17062
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]
[root@rac02 ~]# ls -al /dev/raw*
crw------- 1 root root 162, 0 May 25 01:46 /dev/rawctl
/dev/raw:
total 0
drwxr-xr-x 2 root root 140 May 25 01:46 .
drwxr-xr-x 14 root root 5860 May 25 01:46 ..
crw------- 1 root root 162, 10 May 25 01:46 raw10
crw------- 1 oracle oinstall 162, 3 May 25 01:46 raw3
crw------- 1 oracle oinstall 162, 4 May 25 01:46 raw4
crw------- 1 oracle oinstall 162, 5 May 25 01:46 raw5
crw------- 1 root root 162, 9 May 25 01:46 raw9
修改脚本使权限如下后正常：--注意脚本设置正确确保下次重启主机后权限仍正确 。
[root@rac02 ~]# ls -al /dev/raw*
crw------- 1 root root 162, 0 May 25 01:46 /dev/rawctl
/dev/raw:
total 0
drwxr-xr-x 2 root root 140 May 25 01:46 .
drwxr-xr-x 14 root root 5860 May 25 01:46 ..
crw-r----- 1 root oinstall 162, 10 May 25 01:46 raw10
crw-r--r-- 1 oracle oinstall 162, 3 May 25 01:46 raw3
crw-r--r-- 1 oracle oinstall 162, 4 May 25 01:46 raw4
crw-r--r-- 1 oracle oinstall 162, 5 May 25 01:46 raw5
案例2：
主机版本为HP-UX B.11.31，使用的是的HP-UX Service Guard集群件，小机宕机重启后VG未挂载导致OCR所在磁盘无法访问。
故障分析：
rac#[/etc]ps -ef|grep crs
root 2249 1 0 Nov 5 ? 0:00 /bin/sh /sbin/init.d/init.crsd run
root 29242 26214 0 16:12:54 pts/0 0:00 grep crs
rac#[/etc]ps -ef|grep init
root 1 0 0 Nov 5 ? 0:01 init
root 23 0 0 Nov 5 ? 0:00 pagetable_init_daemon
root 29368 26214 0 16:15:29 pts/0 0:00 grep init
root 2247 1 0 Nov 5 ? 0:00 /bin/sh /sbin/init.d/init.evmd run
root 2248 1 0 Nov 5 ? 0:00 /bin/sh /sbin/init.d/init.cssd fatal
root 2249 1 0 Nov 5 ? 0:00 /bin/sh /sbin/init.d/init.crsd run
root 2281 2248 0 Nov 5 ? 0:08 /bin/sh /sbin/init.d/init.cssd startcheck
root 2274 2249 0 Nov 5 ? 0:08 /bin/sh /sbin/init.d/init.cssd startcheck
root 2284 2247 0 Nov 5 ? 0:08 /bin/sh /sbin/init.d/init.cssd startcheck
rac$[/tmp]ls -lrt crsctl*
-rw-rw-rw- 1 oracle dba 155 Nov 9 15:35 crsctl.2274
-rw-rw-rw- 1 oracle dba 155 Nov 9 15:35 crsctl.2281
-rw-rw-rw- 1 oracle dba 155 Nov 9 15:35 crsctl.2284
rac$[/tmp]cat crsctl.2284
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such device or address] [6]
rac$[/tmp]cat crsctl.2281
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such device or address] [6]
rac$[/tmp]cat crsctl.2274
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such device or address] [6]
检查OCR信息
nbrbdb2$[/home/oracle]ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 130852
Used space (kbytes) : 3312
Available space (kbytes) : 127540
ID : 245644703
Device/File Name : /dev/vgora/rocr0
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
nbrbdb2$[/home/oracle]ls -al /dev/vgora/rocr0
crw-r----- 1 oracle dba 64 0x020001 Jun 14 2013 /dev/vgora/rocr0
查看节点1上的信息：
rac$[/oracle/product/10.2.0/crs_1/log/rac/cssd]ls -al /dev/vgora/rocr0
crw-r----- 1 oracle dba 64 0x020001 Sep 28 2012 /dev/vgora/rocr0
rac#[/]vgdisplay
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 4353
VGDA 2
PE Size (Mbytes) 32
Total PE 4343
Alloc PE 4073
Free PE 270
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vglog".
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vglock".
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vgora".
故障解决：
从以上信息可以看到VG未激活，导致OCR不可读写。
使用如下命令激活VG后CRS恢复正常：
#[/]vgchange -a s vgora
#[/]vgchange -a s vglog