RAC重建OCR/Voting disk遇到的一些故障

author：skate

time：2010-05-09

我的测试环境：

 

母系统：win2003
虚拟软件：vmware3.2.1
guest系统：centos4.7
oracle db：oracle10.2.1

 

以下是我在重建rac的ocr/voting disk过程中遇到的错误及解决方法，记录一下。

rac故障现象总结：

 

0. 检查crs的状态

 

直接查看进程”ps -ef |grep d.bin“

[root@rac1 oracle]# ps -ef |grep d.bin
root     15716  6979  0 11:04 pts/0    00:00:00 grep d.bin
root     28240     1  1 09:29 ?        00:01:00 /u01/crs/oracle/product/10.2.0/crs/bin/crsd.bin reboot
oracle   29059 28223  0 09:32 ?        00:00:11 /u01/crs/oracle/product/10.2.0/crs/bin/evmd.bin
oracle   29209 29181  0 09:32 ?        00:00:44 /u01/crs/oracle/product/10.2.0/crs/bin/ocssd.bin

 

 看见以上的进程，就代表crs已经正常启动了

 

 用命令查看"crsctl check crs"

 

[oracle@rac2 ~]$ crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

 

看见以上输出，crs已经正常启动。

 

像如下的情况，就代表crs没有成功启动。

 

# ps -ef | grep css

root      6929     1  0 19:56 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root      6960  6928  0 19:56 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root      6963  6929  0 19:56 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root      7064  6935  0 19:56 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

 

可以查看crs的相关日志：crsd.log，ocssd.log，evmd.log

 

 

 

 

1. crs的故障：

 

1.1 报错：Insufficient user privileges.

 

现象：
[oracle@green ~]$ crsctl stop crs
Insufficient user privileges.

 

解决：
由于root环境变量没有设置$oracle_home和crs的环境变量，所以root下提示没有这个命令

[root@rac2 ~]# crsctl check crs
-bash: crsctl: command not found

[root@rac2 ~]# su - oracle

[oracle@rac2 ~]$ crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

[oracle@rac2 ~]$ crsctl stop crs
Insufficient user privileges.

 

[oracle@rac2 ~]$ su
Password:
[root@rac2 oracle]# crsctl stop crs
Stopping resources.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rac2 oracle]#

 

 

1.2 报错：

Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

现象：
[root@rac2 oracle]# crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

 

查看下crs的进程启动情况： ps -ef |grep d.bin

 

当crs启动后（crsctl start crs），要稍等一会才能起来，如果很快就核查，就会报上面这个错误。
还有一种情况也会产生这个错误，那就是节点间时间不同步，我这次遇到的这个问题就是因为节点
间时间不同步，我用了简单的rdate保证两个节点间的同步，当然还有其他的方法，如ntpdate或建立
时间服务器。

 

也可以直接用如下文件管理：
 /etc/rc.d/init.d/init.crs
 /etc/rc.d/init.d/init.crsd
 /etc/rc.d/init.d/init.cssd
 /etc/rc.d/init.d/init.evmd

 

参考：http://www.dbspecialists.com/files/presentations/rac_quick_reference.html

######################################################################################

rac2上的asm无法启动，报如下的错误：

[oracle@rac1 ~]$ srvctl start asm -n rac2
PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [CRS-0215: Could not start resource 'ora.rac2.ASM2.asm'.]]
  [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [CRS-0215: Could not start resource 'ora.rac2.ASM2.asm'.]]

然后用crs_start单独启动，看报什么错，结果又报了一大堆错误：

[oracle@rac2 ~]$ srvctl start asm -n rac2
PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Thu May 6 15:25:59 2010
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Enter user-name: Connected to an idle instance.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL> ORA-27504: IPC error creating OSD context
rac2:ora.rac2.ASM2.asm:ORA-27300: OS system dependent operation:if_not_found failed with status: 0
rac2:ora.rac2.ASM2.asm:ORA-27301: OS failure message: Error 0
rac2:ora.rac2.ASM2.asm:ORA-27302: failure occurred at: skgxpvaddr9
rac2:ora.rac2.ASM2.asm:ORA-27303: additional information: requested interface 192.0.22.0 not found. Check output from ifconfig command
rac2:ora.rac2.ASM2.asm:SQL> Disconnected
rac2:ora.rac2.ASM2.asm:
CRS-0215: Could not start resource 'ora.rac2.ASM2.asm'.]]
  [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Thu May 6 15:25:59 2010
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Enter user-name: Connected to an idle instance.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL> ORA-27504: IPC error creating OSD context
rac2:ora.rac2.ASM2.asm:ORA-27300: OS system dependent operation:if_not_found failed with status: 0
rac2:ora.rac2.ASM2.asm:ORA-27301: OS failure message: Error 0
rac2:ora.rac2.ASM2.asm:ORA-27302: failure occurred at: skgxpvaddr9
rac2:ora.rac2.ASM2.asm:ORA-27303: additional information: requested interface 192.0.22.0 not found. Check output from ifconfig command
rac2:ora.rac2.ASM2.asm:SQL> Disconnected
rac2:ora.rac2.ASM2.asm:
CRS-0215: Could not start resource 'ora.rac2.ASM2.asm'.]]

 

一般错误“ORA-27504: IPC error creating OSD context”是因为节点间的通信的有问题

 

首先查看/etc/hosts文件

 

正确的格式应该如下：

 

[oracle@rac1 ~]$ more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1      localhost.localdomain localhost

#skate add

# Public
192.168.2.31   rac1.localdomain        rac1
192.168.2.22   rac2.localdomain        rac2
#Private
192.168.0.31   rac1-priv.localdomain   rac1-priv
192.168.0.22   rac2-priv.localdomain   rac2-priv
#Virtual
192.168.2.131   rac1-vip.localdomain    rac1-vip
192.168.2.122   rac2-vip.localdomain    rac2-vip
[oracle@rac1 ~]$

 

 

我的这个文件没有问题，在群里讨论，我和大家都比较关注下面的错误：
ORA-27303: additional information: requested interface 192.0.22.0 not found. Check output from ifconfig command

但是什么引起这个错误的呢？

先怀疑网卡设置，可能是ip设置有问题，或者MUT有问题。不过经过检查我的网卡设置都是正常的

 

rac1节点网络：

[root@rac1 tmp]# ip a
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:2a:81:d3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.31/24 brd 192.168.2.255 scope global eth0
    inet 192.168.2.131/24 brd 192.168.2.255 scope global secondary eth0:1
    inet6 fe80::20c:29ff:fe2a:81d3/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:2a:81:dd brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.31/24 brd 192.168.0.255 scope global eth1
    inet6 fe80::20c:29ff:fe2a:81dd/64 scope link
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

rac2节点网络：

[root@rac2 ~]# ip a
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:81:22:38 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.22/24 brd 192.168.2.255 scope global eth0
    inet 192.168.2.122/24 brd 192.168.2.255 scope global secondary eth0:1
    inet6 fe80::20c:29ff:fe81:2238/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:81:22:42 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.22/24 brd 192.168.0.255 scope global eth1
    inet6 fe80::20c:29ff:fe81:2242/64 scope link
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

 

 

我又google了半天，找到一个帖子，说是尝试如下修改，可以解决

 

1、关闭 Oracle 实例－instance。
2、cd $ORACLE_HOME/rdbms/lib
3、make -f ins_rdbms.mk rac_off
4、make -f ins_rdbms.mk ioracle

 

我按其操作后，没起作用，反而出来如下的错误：

 

[oracle@rac2 lib]$ srvctl start asm -n rac2
PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Thu May 6 16:34:23 2010
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Enter user-name: Connected to an idle instance.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL> ORA-00439: feature not enabled: Real Application Clusters
rac2:ora.rac2.ASM2.asm:SQL> Disconnected
rac2:ora.rac2.ASM2.asm:
CRS-0215: Could not start resource 'ora.rac2.ASM2.asm'.]]
  [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "rac2", [rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Thu May 6 16:34:23 2010
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:Enter user-name: Connected to an idle instance.
rac2:ora.rac2.ASM2.asm:
rac2:ora.rac2.ASM2.asm:SQL> ORA-00439: feature not enabled: Real Application Clusters
rac2:ora.rac2.ASM2.asm:SQL> Disconnected
rac2:ora.rac2.ASM2.asm:
CRS-0215: Could not start resource 'ora.rac2.ASM2.asm'.]]

 

从错误码“ORA-00439: feature not enabled: Real Application Clusters”可以看出已经禁用了集群功能，于是我有反向执行

 

1、关闭 Oracle 实例－instance。（这步我没操作，因为我的实例就没起来，呵呵）
2、cd $ORACLE_HOME/rdbms/lib
3、make -f ins_rdbms.mk rac_on
4、make -f ins_rdbms.mk ioracle

 

执行后，又恢复到以前的额错误了，查看相应的alertlog都没有错误，不过在asm2的alertlog中最后两行有错误

 

[root@rac2 ~]# tail -50 /u01/app/oracle/admin/+ASM/bdump/alert_+ASM2.log |more
USER: terminating instance due to error 27504
Instance terminated by USER, pid = 27244
Fri May  7 03:26:47 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2
.0 _1 s/arch
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  large_pool_size          = 12582912
  instance_type            = asm
  cluster_interconnects    = 192,168.0.22
  cluster_database         = TRUE
  instance_number          = 2
  remote_login_passwordfile= EXCLUSIVE
  background_dump_dest     = /u01/app/oracle/admin/+ASM/bdump
  user_dump_dest           = /u01/app/oracle/admin/+ASM/udump
  core_dump_dest           = /u01/app/oracle/admin/+ASM ump
  asm_diskgroups           = DATA
USER: terminating instance due to error 27504
Instance terminated by USER, pid = 29732
Fri May  7 03:28:40 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2
.0 _1 s/arch
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  large_pool_size          = 12582912
  instance_type            = asm
  cluster_interconnects    = 192,168.0.22
  cluster_database         = TRUE
  instance_number          = 2
  remote_login_passwordfile= EXCLUSIVE
  background_dump_dest     = /u01/app/oracle/admin/+ASM/bdump
  user_dump_dest           = /u01/app/oracle/admin/+ASM/udump
  core_dump_dest           = /u01/app/oracle/admin/+ASM ump
  asm_diskgroups           = DATA
USER: terminating instance due to error 27504
Instance terminated by USER, pid = 31201

这个信息也不能定位错误在哪，在这过程中，我用sqlplus在rac1中可以成功启动数据库。最后一个群里的朋友说看看
asm2的参数文件内容

我经过检查发现我把+ASM2.cluster_interconnects='192.168.0.22' 写成+ASM2.cluster_interconnects='192,168.0.22'

把逗点写成了逗号，马上改正过了，然后在启动asm2实例，就可以启动了。

 

现在在回头想，报错：ORA-27303: additional information: requested interface 192.0.22.0 not found. Check output from ifconfig command
就可以理解了，因为节点间通信有问题，才会报这个错误.

 

 

#####################################################################################

 

启动数据库报错

 

[oracle@rac2 ~]$ srvctl start database -d rac
PRKP-1001 : Error starting instance rac1 on node rac1
CRS-0215: Could not start resource 'ora.rac.rac1.inst'.
PRKP-1001 : Error starting instance rac2 on node rac2
CRS-0215: Could not start resource 'ora.rac.rac2.inst'.

 

虽然用srvctl无法启动数据库，但是可以用sqlplus分别在两个节点正常启动数据库

抱这个错误，网上有说按如下方法可以解决：

 

as root:
crsctl stop crs
rm -f /var/tmp/.oracle/*
crsctl start crs

 

等一会，crs正常启动后，就可以正常启动数据库了

 

但对我的环境，问题依旧。这是我突然想到数据库名和实例名的大小写的问题，

我刚才注册到ocr里的都是小写的，怀疑可能是这个原因。于是删除原来小写的
，从新注册大写的

 

这是原来注册的小写的：

[oracle@rac2 ~]$ srvctl add database -d rac -o /u01/app/oracle/product/10.2.0/db_1
[oracle@rac2 ~]$ srvctl add instance -d rac -i rac1 -n rac1
[oracle@rac2 ~]$ srvctl add instance -d rac -i rac2 -n rac2
[oracle@rac2 ~]$ srvctl modify instance -d rac -i rac1 -s +ASM1
[oracle@rac2 ~]$ srvctl modify instance -d rac -i rac2 -s +ASM2

 

把小写的删除

 

[oracle@rac2 ~]$ srvctl remove instance -d rac -i rac1
Remove instance rac1 from the database rac? (y/[n]) y
[oracle@rac2 ~]$ srvctl remove instance -d rac -i rac2
Remove instance rac2 from the database rac? (y/[n]) y
[oracle@rac2 ~]$ srvctl remove database -d rac
Remove the database rac? (y/[n]) y
[oracle@rac2 ~]$

 

把database和instance注册成大写的

 

[oracle@rac2 ~]$ srvctl add database -d RAC -o $ORACLE_HOME
[oracle@rac2 ~]$ srvctl add instance -d RAC -i RAC1 -n rac1
[oracle@rac2 ~]$ srvctl add instance -d RAC -i RAC2 -n rac2
[oracle@rac2 ~]$ srvctl modify  instance -d RAC -i RAC1 -s +ASM1
[oracle@rac2 ~]$ srvctl modify  instance -d RAC -i RAC2 -s +ASM2

 

然后在启动数据库，居然启动了啊。

 

[oracle@rac2 ~]$ srvctl start database -d rac

 

 

######################################################################

在ocr中删除instance和database的报错：

 

[oracle@rac2 ~]$ srvctl remove instance -d rac -i rac1
Remove instance rac1 from the database rac? (y/[n]) y
PRKP-1023 : The instance {0} is still running.rac
[oracle@rac2 ~]$ srvctl remove instance -d rac -i rac1
Remove instance rac1 from the database rac? (y/[n]) y
PRKP-1023 : The instance {0} is still running.rac
[oracle@rac2 ~]$ srvctl remove instance -d rac
PRKO-2001 : Invalid command line syntax
[oracle@rac2 ~]$ srvctl remove database -d rac
Remove the database rac? (y/[n]) y
PRKP-1022 : The database rac is still running.

 

解决方式：用crs_stop -all停掉所有的服务，然后用crs_stat -t -v 检查各服务的状态
如果有服务的state是UNKNO的，那就只能一个一个的停掉了。

 

################################################################

 

onsctl启动的问题:

 

[oracle@rac1 ~]$ onsctl ping
Number of onsconfiguration retrieved, numcfg = 0
ons is not running ...

 

解决：
[oracle@rac1 ~]$ onsctl start
Number of onsconfiguration retrieved, numcfg = 0
Number of onsconfiguration retrieved, numcfg = 0
onsctl: ons started

[oracle@rac1 ~]$ onsctl ping
Number of onsconfiguration retrieved, numcfg = 0
ons is running ...

##########################################################

 

错误现象oifcfg getif 没有返回值

 

[root@rac2 public]# /u01/crs/oracle/product/10.2.0/crs/bin/oifcfg iflist
eth0  192.168.2.0
eth1  192.168.0.0
[root@rac2 public]# /u01/crs/oracle/product/10.2.0/crs/bin/oifcfg getif
[root@rac2 public]# /u01/crs/oracle/product/10.2.0/crs/bin/oifcfg getif -global
[root@rac2 public]# /u01/crs/oracle/product/10.2.0/crs/bin/oifcfg getif -global rac1
[root@rac2 public]# /u01/crs/oracle/product/10.2.0/crs/bin/oifcfg getif -global rac2

 

手工把网络信息注册到ocr中

 

[oracle@rac2 public]# oifcfg setif -global eth0/192.168.2.0:public
[oracle@rac2 public]# oifcfg setif -global eth1/192.168.0.0:cluster_interconnect

然后就可以查了啊

 

[oracle@rac1 ~]$ oifcfg getif
eth0  192.168.2.0  global  public
eth1  192.168.0.0  global  cluster_interconnect

 

#########################################################################

 

 

以上的错误是我在重建ocr/voting disk所遇到的一些错误。

 

 

----------end---------