虚拟机下11g r2 RAC安装root时报：Timed out waiting for the CRS stack to start.

      虚拟机Linux环境下，安装11G r2的RAC，使用的是grid用户，共两个节点，第二个节点 root时遇到Timed out waiting for the CRS stack to start.
全部完全卸载、磁盘dd操作后重新安装多次，问题依旧。如下：

[root@dbserver2 ~]# /apps/oracle/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /apps/oracle/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-06-04 03:04:12: Parsing the host name
2012-06-04 03:04:12: Checking for super user privileges
2012-06-04 03:04:12: User has super user privileges
Using configuration parameter file: /apps/oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on centos-release-4-8

 

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node dbserver1, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'dbserver2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'dbserver2' succeeded
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'dbserver2'
CRS-2676: Start of 'ora.mdnsd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'dbserver2'
CRS-2676: Start of 'ora.gipcd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dbserver2'
CRS-2676: Start of 'ora.gpnpd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbserver2'
CRS-2676: Start of 'ora.cssdmonitor' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dbserver2'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbserver2'
CRS-2676: Start of 'ora.diskmon' on 'dbserver2' succeeded
CRS-2676: Start of 'ora.cssd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'dbserver2'
CRS-2676: Start of 'ora.ctssd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dbserver2'
CRS-2676: Start of 'ora.asm' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'dbserver2'
CRS-2676: Start of 'ora.crsd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'dbserver2'
CRS-2676: Start of 'ora.evmd' on 'dbserver2' succeeded
Timed out waiting for the CRS stack to start.

重新执行root还是不行，如下：

[root@dbserver2 ~]# cd /apps/oracle/11.2.0/grid/crs/install/
[root@dbserver2 install]# /apps/oracle/11.2.0/grid/crs/install/rootcrs.pl -verbose -deconfig -force
……
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node

[root@dbserver2 ~]# /apps/oracle/11.2.0/grid/root.sh
……
CRS-2676: Start of 'ora.evmd' on 'dbserver2' succeeded
Timed out waiting for the CRS stack to start.

查看日志：

[root@dbserver2 dbserver2]# pwd
/apps/oracle/11.2.0/grid/log/dbserver2

[root@dbserver2 dbserver2]# tail -f /apps/oracle/11.2.0/grid/log/dbserver2/alertdbserver2.log

2012-06-01 20:03:44.976
[crsd(11217)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.
2012-06-01 20:03:45.315
[ohasd(10460)]CRS-2765:Resource 'ora.crsd' has failed on server 'dbserver2'.
2012-06-01 20:03:47.083
[crsd(11235)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.
2012-06-01 20:03:47.631
[ohasd(10460)]CRS-2765:Resource 'ora.crsd' has failed on server 'dbserver2'.
2012-06-01 20:03:49.501
[crsd(11250)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.
2012-06-01 20:03:49.986
[ohasd(10460)]CRS-2765:Resource 'ora.crsd' has failed on server 'dbserver2'.
2012-06-01 20:03:49.988
[ohasd(10460)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

继续查看： /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log.

[root@dbserver2 crsd]# tail -200 /apps/oracle/11.2.0/grid/log/dbserver2/crsd/crsd.log

gipcretSuccess (0)
2012-06-01 20:16:12.859: [  OCRASM][3086657216]proprasmo: Error in open/create file in dg [crs]
[  OCRASM][3086657216]SLOS : SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CRS.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CRS.255.4294967295
ORA-15001: diskgroup "CRS"
2012-06-01 20:03:49.499: [  OCRASM][3086726848]proprasmo: kgfoCheckMount returned [6]
2012-06-01 20:03:49.499: [  OCRASM][3086726848]proprasmo: The ASM disk group crs is not found or not mounted
2012-06-01 20:03:49.501: [  OCRRAW][3086726848]proprioo: Failed to open [+crs]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2012-06-01 20:03:49.502: [  OCRRAW][3086726848]proprioo: No OCR/OLR devices are usable
2012-06-01 20:03:49.502: [  OCRASM][3086726848]proprasmcl: asmhandle is NULL
2012-06-01 20:03:49.502: [  OCRRAW][3086726848]proprinit: Could not open raw device
2012-06-01 20:03:49.502: [  OCRASM][3086726848]proprasmcl: asmhandle is NULL
2012-06-01 20:03:49.505: [  OCRAPI][3086726848]a_init:16!: Backend init unsuccessful : [26]
2012-06-01 20:03:49.508: [  CRSOCR][3086726848] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error[SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CRS.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CRS.255.4294967295
ORA-15001: diskgroup "CRS"
] [8]
2012-06-01 20:03:49.508: [    CRSD][3086726848][PANIC] CRSD exiting: Could not init OCR, code: 26
2012-06-01 20:03:49.509: [    CRSD][3086726848] Done.

检查如下：

[grid@dbserver2 ~]$ asmcmd
ASMCMD> ls
没有任何内容 。

检查磁盘：
[root@dbserver2 dbserver2]# fdisk -l
……
Disk /dev/sdh: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1               1         261     2096451   83  Linux

Disk /dev/sdi: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdi1               1         261     2096451   83  Linux

Disk /dev/sdj: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdj1               1         261     2096451   83  Linux

[root@dbserver2 dbserver2]# /etc/init.d/oracleasm listdisks

[root@dbserver2 dbserver2]# /etc/init.d/oracleasm listdisks
CRS1
CRS2
CRS3
[root@dbserver2 dbserver2]#

[root@dbserver2 dbserver2]# cd /dev/oracleasm/disks/
[root@dbserver2 disks]# ll
total 0
brw-rw----  1 grid asmadmin 8, 113 Jun  2 10:44 CRS1
brw-rw----  1 grid asmadmin 8, 129 Jun  2 10:44 CRS2
brw-rw----  1 grid asmadmin 8, 145 Jun  2 10:44 CRS3
[root@dbserver2 disks]#

下面的检查：
[root@dbserver2 oracle]# cd /apps/oracle/11.2.0/grid/bin/
[root@dbserver2 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6e49b9f535744ffcbf42cf5c1315be5c (/dev/oracleasm/disks/CRS1) [CRS]
 2. ONLINE   8319759558834fc8bf89f783a8177b5b (/dev/oracleasm/disks/CRS2) [CRS]
 3. ONLINE   94e522f261334f18bfc59d576f112016 (/dev/oracleasm/disks/CRS3) [CRS]
Located 3 voting disk(s).
[root@dbserver2 bin]#

重新删除后再安装一次，问题依旧。

dd if=/dev/zero of=/dev/sdh1 bs=1024 count=300
rm -rf /apps/*
chown -R grid:oinstall /apps/
rm -rf /tmp/*
rm -rf /etc/ora*

又未成功

继续检查：
[root@dbserver2 bin]# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=6, opn=kgfo, dep=0, loc=kgfoCkMt03
diskgroup CRS not mounted ()
] [6]
[root@dbserver2 bin]#
在另一个节点 ：
[root@dbserver1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2280
         Available space (kbytes) :     259840
         ID                       : 1856240294
         Device/File Name         :       +CRS
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded
SQL> select name from V$asm_diskgroup
  2  ;

NAME
------------------------------
CRS

SQL> alter diskgroup CRS mount;
alter diskgroup CRS mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "CRS" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "CRS"
SQL> select group_number,disk_number,mode_status,name from V$asm_disk;

GROUP_NUMBER DISK_NUMBER MODE_ST NAME
------------ ----------- ------- ------------------------------
           0           0 ONLINE
           0           2 ONLINE
           0           1 ONLINE

 

重新安装ASMLIB

[root@dbserver1 bin]# rpm -qa|grep oracleasm
oracleasm-support-2.1.7-1.el4
oracleasmlib-2.0.4-1.el4
oracleasm-2.6.9-89.EL-2.0.5-1.el4
oracleasm-2.6.9-89.ELsmp-2.0.5-1.el4

[root@dbserver1 bin]# rpm -e oracleasm-2.6.9-89.EL-2.0.5-1.el4
[root@dbserver1 bin]# rpm -e oracleasmlib-2.0.4-1.el4
[root@dbserver1 bin]# rpm -e oracleasm-2.6.9-89.ELsmp-2.0.5-1.el4
[root@dbserver1 bin]# rpm -e oracleasm-support-2.1.7-1.el4
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
[root@dbserver1 soft]# rpm -ivh oracleasm*

 

SQL> show parameter asm

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string
asm_diskstring                       string
asm_power_limit                      integer     1
asm_preferred_read_failure_groups    string

 

SQL> select group_number,disk_number,mount_status,header_status,path from V$asm_disk;

GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU PATH
------------ ----------- ------- ------------ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
           0           0 CLOSED  MEMBER       /dev/oracleasm/disks/CRS1
           0           2 CLOSED  MEMBER       /dev/oracleasm/disks/CRS3
           0           1 CLOSED  MEMBER       /dev/oracleasm/disks/CRS2

SQL>

最后偿试的解决办法

在虚拟机的配置中加入了：
scsi1:0.deviceType = "disk"
scsi1:1.deviceType = "disk"
scsi1:2.deviceType = "disk"
scsi1:3.deviceType = "disk"
scsi1:4.deviceType = "disk"
scsi1:5.deviceType = "disk"
scsi1:6.deviceType = "disk"
scsi1:7.deviceType = "disk"
scsi1:8.deviceType = "disk"
scsi1:9.deviceType = "disk"
scsi1:10.deviceType = "disk"
scsi1:11.deviceType = "disk"
再次重试

dd if=/dev/zero of=/dev/sdh1 bs=1024 count=300
dd if=/dev/zero of=/dev/sdi1 bs=1024 count=300
dd if=/dev/zero of=/dev/sdj1 bs=1024 count=300
rm -rf /apps/*
chown -R grid:oinstall /apps/
rm -rf /tmp/*
rm -rf /etc/ora*
rm -f /usr/local/bin/dbhome
rm -f /usr/local/bin/oraenv
rm -f /usr/local/bin/coraenv
chown -R grid:oinstall /soft/

重新安装，居然成功，如下所示：

oracleasm scandisks
oracleasm listdisks
[root@dbserver1 ~]# oracleasm createdisk vol1 /dev/sdh1
[root@dbserver1 ~]# oracleasm createdisk vol2 /dev/sdi1
[root@dbserver1 ~]# oracleasm createdisk vol3 /dev/sdj1
[root@dbserver1 ~]# oracleasm listdisks
……
[root@dbserver2 ~]# /apps/oracle/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /apps/oracle/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2012-06-05 15:52:57: Parsing the host name
2012-06-05 15:52:57: Checking for super user privileges
2012-06-05 15:52:57: User has super user privileges
Using configuration parameter file: /apps/oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on centos-release-4-8

 

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node dbserver1, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'dbserver2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'dbserver2' succeeded
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'dbserver2'
CRS-2676: Start of 'ora.mdnsd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'dbserver2'
CRS-2676: Start of 'ora.gipcd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dbserver2'
CRS-2676: Start of 'ora.gpnpd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbserver2'
CRS-2676: Start of 'ora.cssdmonitor' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dbserver2'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbserver2'
CRS-2676: Start of 'ora.diskmon' on 'dbserver2' succeeded
CRS-2676: Start of 'ora.cssd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'dbserver2'
CRS-2676: Start of 'ora.ctssd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dbserver2'
CRS-2676: Start of 'ora.asm' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'dbserver2'
CRS-2676: Start of 'ora.crsd' on 'dbserver2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'dbserver2'
CRS-2676: Start of 'ora.evmd' on 'dbserver2' succeeded
/apps/oracle/11.2.0/grid/bin/srvctl start vip -i dbserver2 ... failed
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... failed
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3602 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /apps/oraInventory
'UpdateNodeList' was successful.

看来问题的原因还就是虚拟机的设置问题。不过此次安装两个节点都有另外的问题，
/apps/oracle/11.2.0/grid/bin/srvctl start nodeapps -n dbserver1 ... failed
Configure Oracle Grid Infrastructure for a Cluster ... failed
这个问题再通过另的办法解决。