【RAC】打完补丁后，发现只有一台rac可以启动，另一台无法启动
安装11Gr2单机asm后，打完11.2.0.3.7的psu后，发现启动不起来数据库，alert日志内容如下：
Errors in file /u01/app/oracle/diag/rdbms/bdspoc/bdspoc/trace/bdspoc_rbal_11187.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u01/app/oracle/diag/rdbms/bdspoc/bdspoc/trace/bdspoc_rbal_11187.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup SDATA was mounted
NOTE: dependency between database bdspoc and diskgroup resource ora.SDATA.dg is established
Errors in file /u01/app/oracle/diag/rdbms/bdspoc/bdspoc/trace/bdspoc_ora_11104.trc (incident=20001):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/bdspoc/bdspoc/incident/incdir_20001/bdspoc_ora_11104_i20001.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/bdspoc/bdspoc/trace/bdspoc_ora_11104.trc (incident=20002):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/bdspoc/bdspoc/incident/incdir_20002/bdspoc_ora_11104_i20002.trcDumping diagnostic data in directory=[cdmp_20130925101448], requested by (instance=1, osid=11104), summary=[incident=20001].通过查找后，发现11.2安装Grid Infrastructure均需要安装执行如下命令
$ su - grid
$ cd
$ ./setasmgidwrap o=<11.2 RDBMS Home>/bin/oracle
[grid@bdspoc_st bin]$ ./setasmgidwrap o=/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
出现如上错误原因主要有两点，我的问题属于case1.
CAUSE
Case #1 ] Group permission of "oracle" executable from RDBMS home should have the same group information for ASM devices according to note 1084186.1.
$ ls -l $GRID_HOME/bin/oracle
-rwsr-s--x 1 grid oinstall 228954465 Jul 1 13:37 /oh1/grid/product/11.2.0/bin/oracle
$ ls -l $RDBMS_HOME/bin/oracle
-rwsr-s--x 1 oracle asmadmin 228954465 Jul 1 13:37 /oh1/oracle/product/11.2.0/bin/oracle
$ ls -l $ASM_DEVICE/*
brw-rw---- 1 grid asmadmin 8, 33 Feb 15 08:11 /dev/oracleasm/disks/ASMD1
brw-rw---- 1 grid asmadmin 8, 49 Feb 15 08:11 /dev/oracleasm/disks/ASMD2
brw-rw---- 1 grid asmadmin 8, 17 May 4 22:30 /dev/oracleasm/disks/CRSD1
But in this case, "oracle" executable from RDBMS shows different group information which is different from group information for ASM devices.
ORA-600[kfioTranslateIO03] and [17090] occurrs due to the permission issue.
$ ls -l $RDBMS_HOME/bin/oracle
-rwsr-s--x 1 oracle oinstall 228954465 Jul 1 13:37 /oh1/oracle/product/11.2.0/bin/oracle
---- it should be "asmadmin" or at least should be the same group of all ASM devices.
Case #2 ] The following RPMs were installed with kernel 2.6.18-308.16.1.el5.
# rpm -qa | grep oracleasm
oracleasm-support-2.1.7-1.el5
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-308.11.1.el5-2.0.5-1.el5 <<<---- This is wrong version of ASMlib