Oracle 11G RAC，2号节点执行root.sh报错，ORA-15018，multipath多路径相关

安装Oracle 11G RAC （11.2.0.4），grid安装完毕在2号节点执行root.sh报错，如下：

[root@jcsjdb2 ~]# /u01/app/oraInventory/orainstRoot.sh 
Changing permissions of /u01/app/oraInventory. 
Adding read,write permissions for group. 
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall. 
The execution of the script is complete. 
[root@jcsjdb2 ~]# /u01/app/11.2.0/grid_1/root.sh 
Performing root user operation for Oracle 11g

The following environment variables are set as: 
ORACLE_OWNER= grid 
ORACLE_HOME= /u01/app/11.2.0/grid_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
Copying dbhome to /usr/local/bin … 
Copying oraenv to /usr/local/bin … 
Copying coraenv to /usr/local/bin …

Creating /etc/oratab file… 
Entries will be added to the /etc/oratab file as needed by 
Database Configuration Assistant when a database is created 
Finished running generic part of root script. 
Now product-specific root actions will be performed. 
Using configuration parameter file: /u01/app/11.2.0/grid_1/crs/install/crsconfig_params 
Creating trace directory 
User ignored Prerequisites during installation 
Installing Trace File Analyzer 
OLR initialization - successful 
Adding Clusterware entries to upstart 
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘jcsjdb2’ 
CRS-2676: Start of ‘ora.mdnsd’ on ‘jcsjdb2’ succeeded 
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘jcsjdb2’ 
CRS-2676: Start of ‘ora.gpnpd’ on ‘jcsjdb2’ succeeded 
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘jcsjdb2’ 
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘jcsjdb2’ 
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘jcsjdb2’ succeeded 
CRS-2676: Start of ‘ora.gipcd’ on ‘jcsjdb2’ succeeded 
CRS-2672: Attempting to start ‘ora.cssd’ on ‘jcsjdb2’ 
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘jcsjdb2’ 
CRS-2676: Start of ‘ora.diskmon’ on ‘jcsjdb2’ succeeded 
CRS-2676: Start of ‘ora.cssd’ on ‘jcsjdb2’ succeeded

Disk Group OCRVOTE creation failed with the following message: 
ORA-15018: diskgroup cannot be created 
ORA-15072: command requires at least 2 regular failure groups, discovered only 0 
ORA-15080: synchronous I/O operation to a disk failed

Configuration of ASM … failed 
see asmca logs at /u01/app/grid/cfgtoollogs/asmca for details 
Did not succssfully configure and start ASM at /u01/app/11.2.0/grid_1/crs/install/crsconfig_lib.pm line 6912. 
/u01/app/11.2.0/grid_1/perl/bin/perl -I/u01/app/11.2.0/grid_1/perl/lib -I/u01/app/11.2.0/grid_1/crs/install /u01/app/11.2.0/grid_1/crs/install/rootcrs.pl execution failed

在MOS上查询到是配置了多路径软件(nultipath)的原因。

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.4 [Release 11.2]
Linux x86-64

SYMPTOMS

  root.sh on second node failing with ASM errors, 

Disk Group DG_OCR_VOTING_HRB creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 1 regular failure groups, discovered only 0
ORA-15080: synchronous I/O operation to a disk failed

Configuration of ASM ... failed
see asmca logs at /u01/app/oracle/cfgtoollogs/asmca for details
Did not successfully configure and start ASM at /u01/grid/oracle/crs/product/11.2.0.3/crs/install/crsconfig_lib.pm line 6763.

  

CAUSE

  [oracle@xxx011 ~]$ ls -l /dev/oracleasm/disks
total 0
brw-rw---- 1 oracle dba 253, 7 Apr 30 05:46 DISK_OCR_01
brw-rw---- 1 oracle dba 253, 8 Apr 30 05:47 DISK_VOTING_01

<<< on node 1 the disks are multipath disks  -- major number is 253


[root@xxx012 asmca]# ls -l /dev/oracleasm/disks
total 0
brw-rw---- 1 oracle dba 65,  48 Apr 30 06:30 DISK_OCR_01
brw-rw---- 1 oracle dba  8, 240 Apr 30 06:30 DISK_VOTING_01

<<< on node 2 the disks are not multipath disks

  

SOLUTION

 1) Make sure following parameters in the  /etc/sysconfig/oracleasm file are set for multipathing 

eg:

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="mpath dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

When using multipathing, you want to force oracleasm to scan the multipathed devices first.


2) # service oracleasm restart
or
# /etc/init.d/oracleasm restart

3)  check /dev/oracleasm/disks/*:

ls -ltr /dev/oracleasm/disks/*   ---->   check major number =253

 

Both nodes need to see the multipath disks.

eg:

[root@xxx011 ~]# ls -ltr /dev/oracleasm/disks/*
brw-rw---- 1 oracle dba 253, 7 Apr 30 17:11 /dev/oracleasm/disks/DISK_OCR_01
brw-rw---- 1 oracle dba 253, 8 Apr 30 17:11 /dev/oracleasm/disks/DISK_VOTING_01


[root@xxx012 ~]# ls -ltr /dev/oracleasm/disks/*
brw-rw---- 1 oracle dba 253, 7 Apr 30 17:10 /dev/oracleasm/disks/DISK_OCR_01
brw-rw---- 1 oracle dba 253, 8 Apr 30 17:10 /dev/oracleasm/disks/DISK_VOTING_01

 

Then reran the root.sh on node2 - which completed successfully

 

根据文档描述，在节点上验证，二号节点的oracleasm并没有正确的绑定多路径软件生成的mpath或dm设备，而是读取到了sd设备。

一号节点：

[grid@jcsjdb1 bin]$ ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 253, 7 Jan  4 20:59 DATA01
brw-rw---- 1 grid asmadmin 253, 3 Jan  4 20:59 DATA01_02
brw-rw---- 1 grid asmadmin 253, 4 Jan  4 20:59 OCR_VOTE01
brw-rw---- 1 grid asmadmin 253, 6 Jan  4 20:59 OCR_VOTE02
brw-rw---- 1 grid asmadmin 253, 5 Jan  4 20:59 OCR_VOTE03

二号节点：

[root@jcsjdb2 ~]# ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 8, 128 Jan  4 21:37 DATA01
brw-rw---- 1 grid asmadmin 8,  48 Jan  4 21:37 DATA01_02
brw-rw---- 1 grid asmadmin 8,  32 Jan  4 21:37 OCR_VOTE01
brw-rw---- 1 grid asmadmin 8,  64 Jan  4 21:37 OCR_VOTE02
brw-rw---- 1 grid asmadmin 8,  96 Jan  4 21:37 OCR_VOTE03

二号节点存储信息：

[root@jcsjdb2 ~]# ll /dev/dm*
brw-rw---- 1 root disk  253, 0 Jan  4 21:36 /dev/dm-0
brw-rw---- 1 root disk  253, 1 Jan  4 21:36 /dev/dm-1
brw-rw---- 1 root disk  253, 2 Jan  4 21:36 /dev/dm-2
brw-rw---- 1 root disk  253, 3 Jan  4 21:37 /dev/dm-3
brw-rw---- 1 root disk  253, 4 Jan  4 21:37 /dev/dm-4
brw-rw---- 1 root disk  253, 5 Jan  4 21:37 /dev/dm-5
brw-rw---- 1 root disk  253, 6 Jan  4 21:37 /dev/dm-6
brw-rw---- 1 root disk  253, 7 Jan  4 21:37 /dev/dm-7
crw-rw---- 1 root audio  14, 9 Jan  4 21:36 /dev/dmmidi
[root@jcsjdb2 ~]# ll /dev/mapper/mpath*
lrwxrwxrwx 1 root root 7 Jan  4 21:37 /dev/mapper/mpatha -> ../dm-5
lrwxrwxrwx 1 root root 7 Jan  4 21:37 /dev/mapper/mpathb -> ../dm-3
lrwxrwxrwx 1 root root 7 Jan  4 21:37 /dev/mapper/mpathc -> ../dm-6
lrwxrwxrwx 1 root root 7 Jan  4 21:37 /dev/mapper/mpathd -> ../dm-7
lrwxrwxrwx 1 root root 7 Jan  4 21:37 /dev/mapper/mpathe -> ../dm-4
[root@jcsjdb2 ~]# ll /dev/sd*
brw-rw---- 1 root disk 8,   0 Jan  4 21:36 /dev/sda
brw-rw---- 1 root disk 8,   1 Jan  4 21:36 /dev/sda1
brw-rw---- 1 root disk 8,   2 Jan  4 21:36 /dev/sda2
brw-rw---- 1 root disk 8,  16 Jan  4 21:36 /dev/sdb
brw-rw---- 1 root disk 8,  32 Jan  4 21:37 /dev/sdc
brw-rw---- 1 root disk 8,  48 Jan  4 21:37 /dev/sdd
brw-rw---- 1 root disk 8,  64 Jan  4 21:37 /dev/sde
brw-rw---- 1 root disk 8,  80 Jan  4 21:37 /dev/sdf
brw-rw---- 1 root disk 8,  96 Jan  4 21:37 /dev/sdg
brw-rw---- 1 root disk 8, 112 Jan  4 21:37 /dev/sdh
brw-rw---- 1 root disk 8, 128 Jan  4 21:37 /dev/sdi
brw-rw---- 1 root disk 8, 144 Jan  4 21:37 /dev/sdj
brw-rw---- 1 root disk 8, 160 Jan  4 21:37 /dev/sdk
brw-rw---- 1 root disk 8, 176 Jan  4 21:37 /dev/sdl

按照文档中的解决方法，修改/etc/sysconfig/oracleasm文件，指定扫描mpath或dm设备，排除sd的设备，并重启oracleasm进程，问题解决，如果不设置该配置文件，服务器重启后，crs进程也无法正常启动，将会报asm磁盘挂载的相关错误。