安装Oracle 11G RAC (11.2.0.4),grid安装完毕在2号节点执行root.sh报错,如下:
[root@jcsjdb2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@jcsjdb2 ~]# /u01/app/11.2.0/grid_1/root.sh
Performing root user operation for Oracle 11gThe following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid_1Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin …
Copying oraenv to /usr/local/bin …
Copying coraenv to /usr/local/bin …Creating /etc/oratab file…
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid_1/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘jcsjdb2’
CRS-2676: Start of ‘ora.mdnsd’ on ‘jcsjdb2’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘jcsjdb2’
CRS-2676: Start of ‘ora.gpnpd’ on ‘jcsjdb2’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘jcsjdb2’
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘jcsjdb2’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘jcsjdb2’ succeeded
CRS-2676: Start of ‘ora.gipcd’ on ‘jcsjdb2’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘jcsjdb2’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘jcsjdb2’
CRS-2676: Start of ‘ora.diskmon’ on ‘jcsjdb2’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘jcsjdb2’ succeededDisk Group OCRVOTE creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 0
ORA-15080: synchronous I/O operation to a disk failedConfiguration of ASM … failed
see asmca logs at /u01/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /u01/app/11.2.0/grid_1/crs/install/crsconfig_lib.pm line 6912.
/u01/app/11.2.0/grid_1/perl/bin/perl -I/u01/app/11.2.0/grid_1/perl/lib -I/u01/app/11.2.0/grid_1/crs/install /u01/app/11.2.0/grid_1/crs/install/rootcrs.pl execution failed
在MOS上查询到是配置了多路径软件(nultipath)的原因。
1) Make sure following parameters in the /etc/sysconfig/oracleasm file are set for multipathing
eg:
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="mpath dm"
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"
When using multipathing, you want to force oracleasm to scan the multipathed devices first.
2) # service oracleasm restart
or
# /etc/init.d/oracleasm restart
3) check /dev/oracleasm/disks/*:
ls -ltr /dev/oracleasm/disks/* ----> check major number =253
Both nodes need to see the multipath disks.
eg:
[root@xxx011 ~]# ls -ltr /dev/oracleasm/disks/*
brw-rw---- 1 oracle dba 253, 7 Apr 30 17:11 /dev/oracleasm/disks/DISK_OCR_01
brw-rw---- 1 oracle dba 253, 8 Apr 30 17:11 /dev/oracleasm/disks/DISK_VOTING_01
[root@xxx012 ~]# ls -ltr /dev/oracleasm/disks/*
brw-rw---- 1 oracle dba 253, 7 Apr 30 17:10 /dev/oracleasm/disks/DISK_OCR_01
brw-rw---- 1 oracle dba 253, 8 Apr 30 17:10 /dev/oracleasm/disks/DISK_VOTING_01
Then reran the root.sh on node2 - which completed successfully
根据文档描述,在节点上验证,二号节点的oracleasm并没有正确的绑定多路径软件生成的mpath或dm设备,而是读取到了sd设备。
一号节点:
[grid@jcsjdb1 bin]$ ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 253, 7 Jan 4 20:59 DATA01
brw-rw---- 1 grid asmadmin 253, 3 Jan 4 20:59 DATA01_02
brw-rw---- 1 grid asmadmin 253, 4 Jan 4 20:59 OCR_VOTE01
brw-rw---- 1 grid asmadmin 253, 6 Jan 4 20:59 OCR_VOTE02
brw-rw---- 1 grid asmadmin 253, 5 Jan 4 20:59 OCR_VOTE03
二号节点:
[root@jcsjdb2 ~]# ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 8, 128 Jan 4 21:37 DATA01
brw-rw---- 1 grid asmadmin 8, 48 Jan 4 21:37 DATA01_02
brw-rw---- 1 grid asmadmin 8, 32 Jan 4 21:37 OCR_VOTE01
brw-rw---- 1 grid asmadmin 8, 64 Jan 4 21:37 OCR_VOTE02
brw-rw---- 1 grid asmadmin 8, 96 Jan 4 21:37 OCR_VOTE03
二号节点存储信息:
[root@jcsjdb2 ~]# ll /dev/dm*
brw-rw---- 1 root disk 253, 0 Jan 4 21:36 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Jan 4 21:36 /dev/dm-1
brw-rw---- 1 root disk 253, 2 Jan 4 21:36 /dev/dm-2
brw-rw---- 1 root disk 253, 3 Jan 4 21:37 /dev/dm-3
brw-rw---- 1 root disk 253, 4 Jan 4 21:37 /dev/dm-4
brw-rw---- 1 root disk 253, 5 Jan 4 21:37 /dev/dm-5
brw-rw---- 1 root disk 253, 6 Jan 4 21:37 /dev/dm-6
brw-rw---- 1 root disk 253, 7 Jan 4 21:37 /dev/dm-7
crw-rw---- 1 root audio 14, 9 Jan 4 21:36 /dev/dmmidi
[root@jcsjdb2 ~]# ll /dev/mapper/mpath*
lrwxrwxrwx 1 root root 7 Jan 4 21:37 /dev/mapper/mpatha -> ../dm-5
lrwxrwxrwx 1 root root 7 Jan 4 21:37 /dev/mapper/mpathb -> ../dm-3
lrwxrwxrwx 1 root root 7 Jan 4 21:37 /dev/mapper/mpathc -> ../dm-6
lrwxrwxrwx 1 root root 7 Jan 4 21:37 /dev/mapper/mpathd -> ../dm-7
lrwxrwxrwx 1 root root 7 Jan 4 21:37 /dev/mapper/mpathe -> ../dm-4
[root@jcsjdb2 ~]# ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Jan 4 21:36 /dev/sda
brw-rw---- 1 root disk 8, 1 Jan 4 21:36 /dev/sda1
brw-rw---- 1 root disk 8, 2 Jan 4 21:36 /dev/sda2
brw-rw---- 1 root disk 8, 16 Jan 4 21:36 /dev/sdb
brw-rw---- 1 root disk 8, 32 Jan 4 21:37 /dev/sdc
brw-rw---- 1 root disk 8, 48 Jan 4 21:37 /dev/sdd
brw-rw---- 1 root disk 8, 64 Jan 4 21:37 /dev/sde
brw-rw---- 1 root disk 8, 80 Jan 4 21:37 /dev/sdf
brw-rw---- 1 root disk 8, 96 Jan 4 21:37 /dev/sdg
brw-rw---- 1 root disk 8, 112 Jan 4 21:37 /dev/sdh
brw-rw---- 1 root disk 8, 128 Jan 4 21:37 /dev/sdi
brw-rw---- 1 root disk 8, 144 Jan 4 21:37 /dev/sdj
brw-rw---- 1 root disk 8, 160 Jan 4 21:37 /dev/sdk
brw-rw---- 1 root disk 8, 176 Jan 4 21:37 /dev/sdl
按照文档中的解决方法,修改/etc/sysconfig/oracleasm文件,指定扫描mpath或dm设备,排除sd的设备,并重启oracleasm进程,问题解决,如果不设置该配置文件,服务器重启后,crs进程也无法正常启动,将会报asm磁盘挂载的相关错误。